[PATCH bpf-next 0/3] Autogenerating API documentation

[PATCH bpf-next 0/3] Autogenerating API documentation

[grantseltzer]

This series of patches is meant to start the initiative to document libbpf.
It includes .rst files which are text documentation describing building, 
API naming convention, as well as an index to generated API documentation.

The generated API documentation is enabled by Doxygen, which actually 
parses the code for documentation comment strings and generates XML.
A tool called Sphinx then reads this XML with the help of the breathe
plugin, as well as the above mentioned .rst files and generates beautiful
HTML output.

The goal of this is for readthedocs.io to be able to pick up that generated
documentation which will be made possible with the help of readthedoc's 
github integration and libbpf's official github mirror. Minor setup 
is required in that mirror once this patch series is merged.

grantseltzer (3):
  bpf: Add sphinx documentation build files
  bpf: Add doxygen configuration file
  bpf: Add rst docs for libbpf

 tools/lib/bpf/docs/api.rst                    |  60 ++++
 tools/lib/bpf/docs/build.rst                  |  39 +++
 tools/lib/bpf/docs/conf.py                    |  38 +++
 tools/lib/bpf/docs/index.rst                  |  21 ++
 .../naming_convention.rst}                    |  18 +-
 tools/lib/bpf/docs/sphinx/Makefile            |   9 +
 tools/lib/bpf/docs/sphinx/doxygen/Doxyfile    | 320 ++++++++++++++++++
 tools/lib/bpf/docs/sphinx/requirements.txt    |   1 +
 8 files changed, 499 insertions(+), 7 deletions(-)
 create mode 100644 tools/lib/bpf/docs/api.rst
 create mode 100644 tools/lib/bpf/docs/build.rst
 create mode 100644 tools/lib/bpf/docs/conf.py
 create mode 100644 tools/lib/bpf/docs/index.rst
 rename tools/lib/bpf/{README.rst => docs/naming_convention.rst} (97%)
 create mode 100644 tools/lib/bpf/docs/sphinx/Makefile
 create mode 100644 tools/lib/bpf/docs/sphinx/doxygen/Doxyfile
 create mode 100644 tools/lib/bpf/docs/sphinx/requirements.txt

-- 
2.29.2

[PATCH bpf-next 1/3] bpf: Add sphinx documentation build files

[grantseltzer]

This adds the ability for sphinx, a tool for creating html documentation
to read in .rst documentation files. It also enables it to use the breathe
plugin to read in xml files that are generated by doxygen, a code
documentation tool.

Sphinx pulls in rst or xml documentation and generates pretty html
which also conveniently can be hosted by readthedocs.org.

The requirements.txt file tells sphinx that it requires the 'breathe'
plugin to be installed. breathe is a plugin that acts as the translation
layer between doxygen and sphinx.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 tools/lib/bpf/docs/conf.py                 | 38 ++++++++++++++++++++++
 tools/lib/bpf/docs/index.rst               | 15 +++++++++
 tools/lib/bpf/docs/sphinx/Makefile         |  9 +++++
 tools/lib/bpf/docs/sphinx/requirements.txt |  1 +
 4 files changed, 63 insertions(+)
 create mode 100644 tools/lib/bpf/docs/conf.py
 create mode 100644 tools/lib/bpf/docs/index.rst
 create mode 100644 tools/lib/bpf/docs/sphinx/Makefile
 create mode 100644 tools/lib/bpf/docs/sphinx/requirements.txt

diff --git a/tools/lib/bpf/docs/conf.py b/tools/lib/bpf/docs/conf.py
new file mode 100644
index 000000000..a67dff0dd
--- /dev/null
+++ b/tools/lib/bpf/docs/conf.py
@@ -0,0 +1,38 @@
+#!/usr/bin/env python3
+# SPDX-License-Identifier: GPL-2.0
+# Configuration file for the Sphinx documentation builder.
+#
+# This file only contains a selection of the most common options. For a full
+# list see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+import os
+import subprocess
+
+extensions = [
+    'sphinx.ext.autodoc',
+    'sphinx.ext.doctest',
+    'sphinx.ext.mathjax',
+    'sphinx.ext.viewcode',
+    'sphinx.ext.imgmath',
+    'sphinx.ext.todo',
+    'breathe',
+]
+
+# List of patterns, relative to source directory, that match files and
+# directories to ignore when looking for source files.
+# This pattern also affects html_static_path and html_extra_path.
+exclude_patterns = []
+
+read_the_docs_build = os.environ.get('READTHEDOCS', None) == 'True'
+
+if read_the_docs_build:
+    subprocess.call('make clean', shell=True)
+    subprocess.call('cd sphinx/doxygen ; doxygen', shell=True)
+
+html_theme = 'sphinx_rtd_theme'
+
+breathe_projects = { "libbpf": "./sphinx/doxygen/build/xml/" }
+breathe_default_project = "libbpf"
+breathe_show_define_initializer = True
+breathe_show_enumvalue_initializer = True
diff --git a/tools/lib/bpf/docs/index.rst b/tools/lib/bpf/docs/index.rst
new file mode 100644
index 000000000..31a6ecfab
--- /dev/null
+++ b/tools/lib/bpf/docs/index.rst
@@ -0,0 +1,15 @@
+libbpf documentation
+=======================================
+
+This is documentation for libbpf, a userspace library for loading and
+interacting with bpf programs.
+
+All general BPF questions, including kernel functionality, libbpf APIs and
+their application, should be sent to bpf@vger.kernel.org mailing list.
+You can subscribe to it `here <http://vger.kernel.org/vger-lists.html#bpf>`_
+and search its archive `here <https://lore.kernel.org/bpf/>`_.
+Please search the archive before asking new questions. It very well might
+be that this was already addressed or answered before.
+
+.. toctree::
+   :caption: Documentation:
diff --git a/tools/lib/bpf/docs/sphinx/Makefile b/tools/lib/bpf/docs/sphinx/Makefile
new file mode 100644
index 000000000..69305958f
--- /dev/null
+++ b/tools/lib/bpf/docs/sphinx/Makefile
@@ -0,0 +1,9 @@
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = ..
+BUILDDIR      = build
+
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)"
+
+%:
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)"
diff --git a/tools/lib/bpf/docs/sphinx/requirements.txt b/tools/lib/bpf/docs/sphinx/requirements.txt
new file mode 100644
index 000000000..188f51e62
--- /dev/null
+++ b/tools/lib/bpf/docs/sphinx/requirements.txt
@@ -0,0 +1 @@
+breathe
\ No newline at end of file
-- 
2.29.2

[PATCH bpf-next 2/3] bpf: Add doxygen configuration file

[grantseltzer]

This adds Doxyfile, the configuration file for Doxygen.
It can be evoked by navigating to it's containing directory
and running `doxygen`. Doxyfile is the default filename.

Evoking sphinx from the tools/lib/bpf/sphinx directory will
also evoke doxygen. This integrates doxygen XML output into
sphinx HTML output.

The majority of these configuraiton settings are defaults,
the main important settings are INPUT, OUTPUT_DIRECTORY,
GENERATE_XML, EXCLUDE_SYMBOLS, and OPTIMIZE_OUTPUT_FOR_C.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 tools/lib/bpf/docs/sphinx/doxygen/Doxyfile | 320 +++++++++++++++++++++
 1 file changed, 320 insertions(+)
 create mode 100644 tools/lib/bpf/docs/sphinx/doxygen/Doxyfile

diff --git a/tools/lib/bpf/docs/sphinx/doxygen/Doxyfile b/tools/lib/bpf/docs/sphinx/doxygen/Doxyfile
new file mode 100644
index 000000000..905eace7b
--- /dev/null
+++ b/tools/lib/bpf/docs/sphinx/doxygen/Doxyfile
@@ -0,0 +1,320 @@
+DOXYFILE_ENCODING      = UTF-8
+PROJECT_NAME           = "libbpf"
+PROJECT_NUMBER         =
+PROJECT_BRIEF          =
+PROJECT_LOGO           =
+OUTPUT_DIRECTORY       = ./build
+CREATE_SUBDIRS         = NO
+ALLOW_UNICODE_NAMES    = NO
+OUTPUT_LANGUAGE        = English
+OUTPUT_TEXT_DIRECTION  = None
+BRIEF_MEMBER_DESC      = YES
+REPEAT_BRIEF           = YES
+ALWAYS_DETAILED_SEC    = NO
+INLINE_INHERITED_MEMB  = NO
+FULL_PATH_NAMES        = YES
+STRIP_FROM_PATH        =
+STRIP_FROM_INC_PATH    =
+SHORT_NAMES            = NO
+JAVADOC_AUTOBRIEF      = NO
+JAVADOC_BANNER         = NO
+QT_AUTOBRIEF           = NO
+MULTILINE_CPP_IS_BRIEF = NO
+PYTHON_DOCSTRING       = YES
+INHERIT_DOCS           = YES
+SEPARATE_MEMBER_PAGES  = NO
+TAB_SIZE               = 4
+ALIASES                =
+OPTIMIZE_OUTPUT_FOR_C  = YES
+OPTIMIZE_OUTPUT_JAVA   = NO
+OPTIMIZE_FOR_FORTRAN   = NO
+OPTIMIZE_OUTPUT_VHDL   = NO
+OPTIMIZE_OUTPUT_SLICE  = NO
+EXTENSION_MAPPING      =
+MARKDOWN_SUPPORT       = YES
+TOC_INCLUDE_HEADINGS   = 5
+AUTOLINK_SUPPORT       = YES
+BUILTIN_STL_SUPPORT    = NO
+CPP_CLI_SUPPORT        = NO
+SIP_SUPPORT            = NO
+IDL_PROPERTY_SUPPORT   = YES
+DISTRIBUTE_GROUP_DOC   = NO
+GROUP_NESTED_COMPOUNDS = NO
+SUBGROUPING            = YES
+INLINE_GROUPED_CLASSES = NO
+INLINE_SIMPLE_STRUCTS  = NO
+TYPEDEF_HIDES_STRUCT   = NO
+LOOKUP_CACHE_SIZE      = 0
+NUM_PROC_THREADS       = 1
+EXTRACT_ALL            = NO
+EXTRACT_PRIVATE        = NO
+EXTRACT_PRIV_VIRTUAL   = NO
+EXTRACT_PACKAGE        = NO
+EXTRACT_STATIC         = NO
+EXTRACT_LOCAL_CLASSES  = YES
+EXTRACT_LOCAL_METHODS  = NO
+EXTRACT_ANON_NSPACES   = NO
+RESOLVE_UNNAMED_PARAMS = YES
+HIDE_UNDOC_MEMBERS     = NO
+HIDE_UNDOC_CLASSES     = NO
+HIDE_FRIEND_COMPOUNDS  = NO
+HIDE_IN_BODY_DOCS      = NO
+INTERNAL_DOCS          = NO
+CASE_SENSE_NAMES       = YES
+HIDE_SCOPE_NAMES       = NO
+HIDE_COMPOUND_REFERENCE= NO
+SHOW_INCLUDE_FILES     = YES
+SHOW_GROUPED_MEMB_INC  = NO
+FORCE_LOCAL_INCLUDES   = NO
+INLINE_INFO            = YES
+SORT_MEMBER_DOCS       = YES
+SORT_BRIEF_DOCS        = NO
+SORT_MEMBERS_CTORS_1ST = NO
+SORT_GROUP_NAMES       = NO
+SORT_BY_SCOPE_NAME     = NO
+STRICT_PROTO_MATCHING  = NO
+GENERATE_TODOLIST      = YES
+GENERATE_TESTLIST      = YES
+GENERATE_BUGLIST       = YES
+GENERATE_DEPRECATEDLIST= YES
+ENABLED_SECTIONS       =
+MAX_INITIALIZER_LINES  = 30
+SHOW_USED_FILES        = YES
+SHOW_FILES             = YES
+SHOW_NAMESPACES        = YES
+FILE_VERSION_FILTER    =
+LAYOUT_FILE            =
+CITE_BIB_FILES         =
+QUIET                  = NO
+WARNINGS               = YES
+WARN_IF_UNDOCUMENTED   = YES
+WARN_IF_DOC_ERROR      = YES
+WARN_NO_PARAMDOC       = NO
+WARN_AS_ERROR          = NO
+WARN_FORMAT            = "$file:$line: $text"
+WARN_LOGFILE           =
+INPUT                  = ../../..
+INPUT_ENCODING         = UTF-8
+FILE_PATTERNS          = *.c \
+                         *.cc \
+                         *.cxx \
+                         *.cpp \
+                         *.c++ \
+                         *.java \
+                         *.ii \
+                         *.ixx \
+                         *.ipp \
+                         *.i++ \
+                         *.inl \
+                         *.idl \
+                         *.ddl \
+                         *.odl \
+                         *.h \
+                         *.hh \
+                         *.hxx \
+                         *.hpp \
+                         *.h++ \
+                         *.cs \
+                         *.d \
+                         *.php \
+                         *.php4 \
+                         *.php5 \
+                         *.phtml \
+                         *.inc \
+                         *.m \
+                         *.markdown \
+                         *.md \
+                         *.mm \
+                         *.dox \
+                         *.py \
+                         *.pyw \
+                         *.f90 \
+                         *.f95 \
+                         *.f03 \
+                         *.f08 \
+                         *.f18 \
+                         *.f \
+                         *.for \
+                         *.vhd \
+                         *.vhdl \
+                         *.ucf \
+                         *.qsf \
+                         *.ice
+RECURSIVE              = NO
+EXCLUDE                =
+EXCLUDE_SYMLINKS       = NO
+EXCLUDE_PATTERNS       =
+EXCLUDE_SYMBOLS        = ___*
+EXAMPLE_PATH           =
+EXAMPLE_PATTERNS       = *
+EXAMPLE_RECURSIVE      = NO
+IMAGE_PATH             =
+INPUT_FILTER           =
+FILTER_PATTERNS        =
+FILTER_SOURCE_FILES    = NO
+FILTER_SOURCE_PATTERNS =
+USE_MDFILE_AS_MAINPAGE = YES
+SOURCE_BROWSER         = NO
+INLINE_SOURCES         = NO
+STRIP_CODE_COMMENTS    = YES
+REFERENCED_BY_RELATION = NO
+REFERENCES_RELATION    = NO
+REFERENCES_LINK_SOURCE = YES
+SOURCE_TOOLTIPS        = YES
+USE_HTAGS              = NO
+VERBATIM_HEADERS       = YES
+ALPHABETICAL_INDEX     = YES
+IGNORE_PREFIX          =
+GENERATE_HTML          = NO
+HTML_OUTPUT            = html
+HTML_FILE_EXTENSION    = .html
+HTML_HEADER            =
+HTML_FOOTER            =
+HTML_STYLESHEET        =
+HTML_EXTRA_STYLESHEET  =
+HTML_EXTRA_FILES       =
+HTML_COLORSTYLE_HUE    = 220
+HTML_COLORSTYLE_SAT    = 100
+HTML_COLORSTYLE_GAMMA  = 80
+HTML_TIMESTAMP         = NO
+HTML_DYNAMIC_MENUS     = YES
+HTML_DYNAMIC_SECTIONS  = NO
+HTML_INDEX_NUM_ENTRIES = 100
+GENERATE_DOCSET        = NO
+DOCSET_FEEDNAME        = "Doxygen generated docs"
+DOCSET_BUNDLE_ID       = org.doxygen.Project
+DOCSET_PUBLISHER_ID    = org.doxygen.Publisher
+DOCSET_PUBLISHER_NAME  = Publisher
+GENERATE_HTMLHELP      = NO
+CHM_FILE               =
+HHC_LOCATION           =
+GENERATE_CHI           = NO
+CHM_INDEX_ENCODING     =
+BINARY_TOC             = NO
+TOC_EXPAND             = NO
+GENERATE_QHP           = NO
+QCH_FILE               =
+QHP_NAMESPACE          = org.doxygen.Project
+QHP_VIRTUAL_FOLDER     = doc
+QHP_CUST_FILTER_NAME   =
+QHP_CUST_FILTER_ATTRS  =
+QHP_SECT_FILTER_ATTRS  =
+QHG_LOCATION           =
+GENERATE_ECLIPSEHELP   = NO
+ECLIPSE_DOC_ID         = org.doxygen.Project
+DISABLE_INDEX          = NO
+GENERATE_TREEVIEW      = NO
+ENUM_VALUES_PER_LINE   = 4
+TREEVIEW_WIDTH         = 250
+EXT_LINKS_IN_WINDOW    = NO
+HTML_FORMULA_FORMAT    = png
+FORMULA_FONTSIZE       = 10
+FORMULA_TRANSPARENT    = YES
+FORMULA_MACROFILE      =
+USE_MATHJAX            = NO
+MATHJAX_FORMAT         = HTML-CSS
+MATHJAX_RELPATH        = https://cdn.jsdelivr.net/npm/mathjax@2
+MATHJAX_EXTENSIONS     =
+MATHJAX_CODEFILE       =
+SEARCHENGINE           = YES
+SERVER_BASED_SEARCH    = NO
+EXTERNAL_SEARCH        = NO
+SEARCHENGINE_URL       =
+SEARCHDATA_FILE        = searchdata.xml
+EXTERNAL_SEARCH_ID     =
+EXTRA_SEARCH_MAPPINGS  =
+GENERATE_LATEX         = NO
+LATEX_OUTPUT           = latex
+LATEX_CMD_NAME         =
+MAKEINDEX_CMD_NAME     = makeindex
+LATEX_MAKEINDEX_CMD    = makeindex
+COMPACT_LATEX          = NO
+PAPER_TYPE             = a4
+EXTRA_PACKAGES         =
+LATEX_HEADER           =
+LATEX_FOOTER           =
+LATEX_EXTRA_STYLESHEET =
+LATEX_EXTRA_FILES      =
+PDF_HYPERLINKS         = YES
+USE_PDFLATEX           = YES
+LATEX_BATCHMODE        = NO
+LATEX_HIDE_INDICES     = NO
+LATEX_SOURCE_CODE      = NO
+LATEX_BIB_STYLE        = plain
+LATEX_TIMESTAMP        = NO
+LATEX_EMOJI_DIRECTORY  =
+GENERATE_RTF           = NO
+RTF_OUTPUT             = rtf
+COMPACT_RTF            = NO
+RTF_HYPERLINKS         = NO
+RTF_STYLESHEET_FILE    =
+RTF_EXTENSIONS_FILE    =
+RTF_SOURCE_CODE        = NO
+GENERATE_MAN           = NO
+MAN_OUTPUT             = man
+MAN_EXTENSION          = .3
+MAN_SUBDIR             =
+MAN_LINKS              = NO
+GENERATE_XML           = YES
+XML_OUTPUT             = xml
+XML_PROGRAMLISTING     = YES
+XML_NS_MEMB_FILE_SCOPE = NO
+GENERATE_DOCBOOK       = NO
+DOCBOOK_OUTPUT         = docbook
+DOCBOOK_PROGRAMLISTING = NO
+GENERATE_AUTOGEN_DEF   = NO
+GENERATE_PERLMOD       = NO
+PERLMOD_LATEX          = NO
+PERLMOD_PRETTY         = YES
+PERLMOD_MAKEVAR_PREFIX =
+ENABLE_PREPROCESSING   = YES
+MACRO_EXPANSION        = YES
+EXPAND_ONLY_PREDEF     = NO
+SEARCH_INCLUDES        = YES
+INCLUDE_PATH           =
+INCLUDE_FILE_PATTERNS  =
+PREDEFINED             =
+EXPAND_AS_DEFINED      =
+SKIP_FUNCTION_MACROS   = YES
+TAGFILES               =
+GENERATE_TAGFILE       =
+ALLEXTERNALS           = NO
+EXTERNAL_GROUPS        = YES
+EXTERNAL_PAGES         = YES
+CLASS_DIAGRAMS         = YES
+DIA_PATH               =
+HIDE_UNDOC_RELATIONS   = YES
+HAVE_DOT               = NO
+DOT_NUM_THREADS        = 0
+DOT_FONTNAME           = Helvetica
+DOT_FONTSIZE           = 10
+DOT_FONTPATH           =
+CLASS_GRAPH            = YES
+COLLABORATION_GRAPH    = YES
+GROUP_GRAPHS           = YES
+UML_LOOK               = NO
+UML_LIMIT_NUM_FIELDS   = 10
+DOT_UML_DETAILS        = NO
+DOT_WRAP_THRESHOLD     = 17
+TEMPLATE_RELATIONS     = NO
+INCLUDE_GRAPH          = YES
+INCLUDED_BY_GRAPH      = YES
+CALL_GRAPH             = NO
+CALLER_GRAPH           = NO
+GRAPHICAL_HIERARCHY    = YES
+DIRECTORY_GRAPH        = YES
+DOT_IMAGE_FORMAT       = png
+INTERACTIVE_SVG        = NO
+DOT_PATH               =
+DOTFILE_DIRS           =
+MSCFILE_DIRS           =
+DIAFILE_DIRS           =
+PLANTUML_JAR_PATH      =
+PLANTUML_CFG_FILE      =
+PLANTUML_INCLUDE_PATH  =
+DOT_GRAPH_MAX_NODES    = 50
+MAX_DOT_GRAPH_DEPTH    = 0
+DOT_TRANSPARENT        = NO
+DOT_MULTI_TARGETS      = NO
+GENERATE_LEGEND        = YES
+DOT_CLEANUP            = YES
-- 
2.29.2

[PATCH bpf-next 3/3] bpf: Add rst docs for libbpf

[grantseltzer]

This adds rst files containg documentation files relevant for
libbpf development. naming_convention.rst is pulled from the
previous README.rst file. api.rst is an index page that links
to the api documentation generationg from doxygen+breathe.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 tools/lib/bpf/docs/api.rst                    | 60 +++++++++++++++++++
 tools/lib/bpf/docs/build.rst                  | 39 ++++++++++++
 tools/lib/bpf/docs/index.rst                  |  6 ++
 .../naming_convention.rst}                    | 18 +++---
 4 files changed, 116 insertions(+), 7 deletions(-)
 create mode 100644 tools/lib/bpf/docs/api.rst
 create mode 100644 tools/lib/bpf/docs/build.rst
 rename tools/lib/bpf/{README.rst => docs/naming_convention.rst} (97%)

diff --git a/tools/lib/bpf/docs/api.rst b/tools/lib/bpf/docs/api.rst
new file mode 100644
index 000000000..36bac417b
--- /dev/null
+++ b/tools/lib/bpf/docs/api.rst
@@ -0,0 +1,60 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+.. _api:
+
+.. contents:: Table of Contents
+   :local:
+   :depth: 1
+
+LIBBPF API
+==================
+
+
+libbpf.h
+------------------
+
+.. doxygenfile:: libbpf.h
+   :project: libbpf
+
+bpf_core_read.h
+------------------
+
+.. doxygenfile:: bpf_core_read.h
+   :project: libbpf
+
+btf.h
+------------------
+
+.. doxygenfile:: btf.h
+   :project: libbpf
+
+bpf_endian.h
+------------------
+
+.. doxygenfile:: bpf_endian.h
+   :project: libbpf
+
+libbpf_common.h
+------------------
+
+.. doxygenfile:: libbpf_common.h
+   :project: libbpf
+
+hashmap.h
+------------------
+
+.. doxygenfile:: hashmap.h
+   :project: libbpf
+
+
+bpf_helpers.h
+------------------
+
+.. doxygenfile:: bpf_helpers.h
+   :project: libbpf
+
+bpf_helper_defs.h
+------------------
+
+.. doxygenfile:: hashmap.h
+   :project: libbpf
diff --git a/tools/lib/bpf/docs/build.rst b/tools/lib/bpf/docs/build.rst
new file mode 100644
index 000000000..749f96dd2
--- /dev/null
+++ b/tools/lib/bpf/docs/build.rst
@@ -0,0 +1,39 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+.. _build:
+
+Building libbpf
+=======================================
+
+libelf is an internal dependency of libbpf and thus it is required to link
+against and must be installed on the system for applications to work.
+pkg-config is used by default to find libelf, and the program called
+can be overridden with PKG_CONFIG.
+
+If using pkg-config at build time is not desired, it can be disabled by
+setting NO_PKG_CONFIG=1 when calling make.
+
+To build both static libbpf.a and shared libbpf.so:
+
+.. code-block:: bash
+
+    $ cd src
+    $ make
+
+To build only static libbpf.a library in directory build/ and install them
+together with libbpf headers in a staging directory root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ mkdir build root
+    $ BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install
+
+To build both static libbpf.a and shared libbpf.so against a custom libelf
+dependency installed in /build/root/ and install them together with libbpf
+headers in a build directory /build/root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ PKG_CONFIG_PATH=/build/root/lib64/pkgconfig DESTDIR=/build/root make
\ No newline at end of file
diff --git a/tools/lib/bpf/docs/index.rst b/tools/lib/bpf/docs/index.rst
index 31a6ecfab..76bb93580 100644
--- a/tools/lib/bpf/docs/index.rst
+++ b/tools/lib/bpf/docs/index.rst
@@ -1,3 +1,5 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
 libbpf documentation
 =======================================
 
@@ -13,3 +15,7 @@ be that this was already addressed or answered before.
 
 .. toctree::
    :caption: Documentation:
+
+   api
+   naming_convention
+   build
\ No newline at end of file
diff --git a/tools/lib/bpf/README.rst b/tools/lib/bpf/docs/naming_convention.rst
similarity index 97%
rename from tools/lib/bpf/README.rst
rename to tools/lib/bpf/docs/naming_convention.rst
index 8928f7787..6b9ae9701 100644
--- a/tools/lib/bpf/README.rst
+++ b/tools/lib/bpf/docs/naming_convention.rst
@@ -1,6 +1,8 @@
 .. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 
-libbpf API naming convention
+.. _naming_convention:
+
+API naming convention
 ============================
 
 libbpf API provides access to a few logically separated groups of
@@ -76,7 +78,7 @@ Please take a look at Documentation/networking/af_xdp.rst in the Linux
 kernel source tree on how to use XDP sockets and for some common
 mistakes in case you do not get any traffic up to user space.
 
-libbpf ABI
+ABI
 ==========
 
 libbpf can be both linked statically or used as DSO. To avoid possible
@@ -116,7 +118,8 @@ This bump in ABI version is at most once per kernel development cycle.
 
 For example, if current state of ``libbpf.map`` is:
 
-.. code-block::
+.. code-block:: c
+
         LIBBPF_0.0.1 {
         	global:
                         bpf_func_a;
@@ -128,7 +131,8 @@ For example, if current state of ``libbpf.map`` is:
 , and a new symbol ``bpf_func_c`` is being introduced, then
 ``libbpf.map`` should be changed like this:
 
-.. code-block::
+.. code-block:: c
+
         LIBBPF_0.0.1 {
         	global:
                         bpf_func_a;
@@ -148,7 +152,7 @@ Format of version script and ways to handle ABI changes, including
 incompatible ones, described in details in [1].
 
 Stand-alone build
-=================
+-------------------
 
 Under https://github.com/libbpf/libbpf there is a (semi-)automated
 mirror of the mainline's version of libbpf for a stand-alone build.
@@ -157,12 +161,12 @@ However, all changes to libbpf's code base must be upstreamed through
 the mainline kernel tree.
 
 License
-=======
+-------------------
 
 libbpf is dual-licensed under LGPL 2.1 and BSD 2-Clause.
 
 Links
-=====
+-------------------
 
 [1] https://www.akkadia.org/drepper/dsohowto.pdf
     (Chapter 3. Maintaining APIs and ABIs).
-- 
2.29.2

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Jonathan Corbet]

grantseltzer <grantseltzer@gmail.com> writes:

> This series of patches is meant to start the initiative to document libbpf.
> It includes .rst files which are text documentation describing building, 
> API naming convention, as well as an index to generated API documentation.

So I'm totally in favor of documenting libbpf...

> The generated API documentation is enabled by Doxygen, which actually 
> parses the code for documentation comment strings and generates XML.
> A tool called Sphinx then reads this XML with the help of the breathe
> plugin, as well as the above mentioned .rst files and generates beautiful
> HTML output.
>
> The goal of this is for readthedocs.io to be able to pick up that generated
> documentation which will be made possible with the help of readthedoc's 
> github integration and libbpf's official github mirror. Minor setup 
> is required in that mirror once this patch series is merged.

...but I do have to wonder why you are doing something outside of the
kernel's documentation system, which just happens to be based on a tool
called Sphinx.  It would be Really Nice to have this documentation part
of our doc tree; it would also be good to not bring in yet another tool
for building kernel docs.

Do you really need to do your own thing here?

Thanks,

jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Thu, Apr 29, 2021 at 6:57 PM Jonathan Corbet <corbet@lwn.net> wrote:
>
> grantseltzer <grantseltzer@gmail.com> writes:
>
> > This series of patches is meant to start the initiative to document libbpf.
> > It includes .rst files which are text documentation describing building,
> > API naming convention, as well as an index to generated API documentation.
>
> So I'm totally in favor of documenting libbpf...
>
> > The generated API documentation is enabled by Doxygen, which actually
> > parses the code for documentation comment strings and generates XML.
> > A tool called Sphinx then reads this XML with the help of the breathe
> > plugin, as well as the above mentioned .rst files and generates beautiful
> > HTML output.
> >
> > The goal of this is for readthedocs.io to be able to pick up that generated
> > documentation which will be made possible with the help of readthedoc's
> > github integration and libbpf's official github mirror. Minor setup
> > is required in that mirror once this patch series is merged.
>
> ...but I do have to wonder why you are doing something outside of the
> kernel's documentation system, which just happens to be based on a tool
> called Sphinx.  It would be Really Nice to have this documentation part
> of our doc tree; it would also be good to not bring in yet another tool
> for building kernel docs.
>
> Do you really need to do your own thing here?

Hm, yes I do agree that it'd be nice to use existing tooling but I
just have a couple concerns for this but please point me in the right
direction because i'm sure i'm missing something. I was told to ask on
the linux-doc mailing list because you'd have valuable input anway.
This is based on reading
https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments

1. We'd want the ability to pull documentation from the code itself to
make it so documentation never falls out of date with code. Based on
the docs on kernel.org/doc it seems that we'd have to be explicit with
specifying which functions/types are included in an .rst file and
submit a patch to update the documentation everytime the libbpf api
changes. Perhaps if this isn't a thing already I can figure out how to
contribute it.

2. Would it be possible (or necessary) to separate libbpf
documentation from the kernel readthedocs page since libbpf isn't part
of the kernel?

Thanks so much,
Grant

> Thanks,
>
> jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Jonathan Corbet]

Grant Seltzer Richman <grantseltzer@gmail.com> writes:

> Hm, yes I do agree that it'd be nice to use existing tooling but I
> just have a couple concerns for this but please point me in the right
> direction because i'm sure i'm missing something. I was told to ask on
> the linux-doc mailing list because you'd have valuable input anway.
> This is based on reading
> https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
>
> 1. We'd want the ability to pull documentation from the code itself to
> make it so documentation never falls out of date with code. Based on
> the docs on kernel.org/doc it seems that we'd have to be explicit with
> specifying which functions/types are included in an .rst file and
> submit a patch to update the documentation everytime the libbpf api
> changes. Perhaps if this isn't a thing already I can figure out how to
> contribute it.

No, you can tell it to pull out docs for all of the functions in a given
file.  You only need to name things if you want to narrow things down.

> 2. Would it be possible (or necessary) to separate libbpf
> documentation from the kernel readthedocs page since libbpf isn't part
> of the kernel?

It could certainly be built as a separate "book", as are many of the
kernel books now.  I could see it as something that gets pulled into the
user-space API book, but there could also perhaps be an argument made
for creating a new "libraries" book instead.

Thanks,

jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
>
> Grant Seltzer Richman <grantseltzer@gmail.com> writes:
>
> > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > just have a couple concerns for this but please point me in the right
> > direction because i'm sure i'm missing something. I was told to ask on
> > the linux-doc mailing list because you'd have valuable input anway.
> > This is based on reading
> > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> >
> > 1. We'd want the ability to pull documentation from the code itself to
> > make it so documentation never falls out of date with code. Based on
> > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > specifying which functions/types are included in an .rst file and
> > submit a patch to update the documentation everytime the libbpf api
> > changes. Perhaps if this isn't a thing already I can figure out how to
> > contribute it.
>
> No, you can tell it to pull out docs for all of the functions in a given
> file.  You only need to name things if you want to narrow things down.

Alright, I will figure out how to do this and adjust the patch
accordingly. My biggest overall goal is making it as easy as possible
to contribute documentation. I think even adding just one doc string
above an API function is a great opportunity for new contributors to
familiarize themselves with the mailing list/patch process.

>
> > 2. Would it be possible (or necessary) to separate libbpf
> > documentation from the kernel readthedocs page since libbpf isn't part
> > of the kernel?
>
> It could certainly be built as a separate "book", as are many of the
> kernel books now.  I could see it as something that gets pulled into the
> user-space API book, but there could also perhaps be an argument made
> for creating a new "libraries" book instead.

Yea if I can figure this out for the libbpf API it'd be great to
replicate it for any API!

>
> Thanks,
>
> jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> >
> > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> >
> > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > just have a couple concerns for this but please point me in the right
> > > direction because i'm sure i'm missing something. I was told to ask on
> > > the linux-doc mailing list because you'd have valuable input anway.
> > > This is based on reading
> > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > >
> > > 1. We'd want the ability to pull documentation from the code itself to
> > > make it so documentation never falls out of date with code. Based on
> > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > specifying which functions/types are included in an .rst file and
> > > submit a patch to update the documentation everytime the libbpf api
> > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > contribute it.
> >
> > No, you can tell it to pull out docs for all of the functions in a given
> > file.  You only need to name things if you want to narrow things down.
>
> Alright, I will figure out how to do this and adjust the patch
> accordingly. My biggest overall goal is making it as easy as possible
> to contribute documentation. I think even adding just one doc string
> above an API function is a great opportunity for new contributors to
> familiarize themselves with the mailing list/patch process.
>
> >
> > > 2. Would it be possible (or necessary) to separate libbpf
> > > documentation from the kernel readthedocs page since libbpf isn't part
> > > of the kernel?
> >
> > It could certainly be built as a separate "book", as are many of the
> > kernel books now.  I could see it as something that gets pulled into the
> > user-space API book, but there could also perhaps be an argument made
> > for creating a new "libraries" book instead.
>
> Yea if I can figure this out for the libbpf API it'd be great to
> replicate it for any API!

It would be great if it was possible to have this libbpf
auto-generated documentation as part of the kernel documentation, but
also be able to generate and export it into our Github mirror to be
pulled by readthedocs.io. If that can be done, it would be the best of
both kernel and external worlds. We have a sync script that already
auto-generates and checks in BPF helpers header, so we have a
precedent of checking in auto-generated stuff into Github. So it's
mostly about figuring out the mechanics of doc generation.

>
> >
> > Thanks,
> >
> > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > >
> > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > >
> > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > just have a couple concerns for this but please point me in the right
> > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > This is based on reading
> > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > >
> > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > make it so documentation never falls out of date with code. Based on
> > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > specifying which functions/types are included in an .rst file and
> > > > submit a patch to update the documentation everytime the libbpf api
> > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > contribute it.
> > >
> > > No, you can tell it to pull out docs for all of the functions in a given
> > > file.  You only need to name things if you want to narrow things down.
> >
> > Alright, I will figure out how to do this and adjust the patch
> > accordingly. My biggest overall goal is making it as easy as possible
> > to contribute documentation. I think even adding just one doc string
> > above an API function is a great opportunity for new contributors to
> > familiarize themselves with the mailing list/patch process.
> >
> > >
> > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > of the kernel?
> > >
> > > It could certainly be built as a separate "book", as are many of the
> > > kernel books now.  I could see it as something that gets pulled into the
> > > user-space API book, but there could also perhaps be an argument made
> > > for creating a new "libraries" book instead.
> >
> > Yea if I can figure this out for the libbpf API it'd be great to
> > replicate it for any API!
>
> It would be great if it was possible to have this libbpf
> auto-generated documentation as part of the kernel documentation, but
> also be able to generate and export it into our Github mirror to be
> pulled by readthedocs.io. If that can be done, it would be the best of
> both kernel and external worlds. We have a sync script that already
> auto-generates and checks in BPF helpers header, so we have a
> precedent of checking in auto-generated stuff into Github. So it's
> mostly about figuring out the mechanics of doc generation.

Agreed, the mirror will have to bring in the documentation
subdirectory as well so the output could be seperate.

Just want to update in this thread that i've been really preoccupied
with other obligations and will get back to this next week.

>
> >
> > >
> > > Thanks,
> > >
> > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Mon, May 10, 2021 at 7:59 AM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> > <grantseltzer@gmail.com> wrote:
> > >
> > > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > > >
> > > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > > >
> > > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > > just have a couple concerns for this but please point me in the right
> > > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > > This is based on reading
> > > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > > >
> > > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > > make it so documentation never falls out of date with code. Based on
> > > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > > specifying which functions/types are included in an .rst file and
> > > > > submit a patch to update the documentation everytime the libbpf api
> > > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > > contribute it.
> > > >
> > > > No, you can tell it to pull out docs for all of the functions in a given
> > > > file.  You only need to name things if you want to narrow things down.
> > >
> > > Alright, I will figure out how to do this and adjust the patch
> > > accordingly. My biggest overall goal is making it as easy as possible
> > > to contribute documentation. I think even adding just one doc string
> > > above an API function is a great opportunity for new contributors to
> > > familiarize themselves with the mailing list/patch process.
> > >
> > > >
> > > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > > of the kernel?
> > > >
> > > > It could certainly be built as a separate "book", as are many of the
> > > > kernel books now.  I could see it as something that gets pulled into the
> > > > user-space API book, but there could also perhaps be an argument made
> > > > for creating a new "libraries" book instead.
> > >
> > > Yea if I can figure this out for the libbpf API it'd be great to
> > > replicate it for any API!
> >
> > It would be great if it was possible to have this libbpf
> > auto-generated documentation as part of the kernel documentation, but
> > also be able to generate and export it into our Github mirror to be
> > pulled by readthedocs.io. If that can be done, it would be the best of
> > both kernel and external worlds. We have a sync script that already
> > auto-generates and checks in BPF helpers header, so we have a
> > precedent of checking in auto-generated stuff into Github. So it's
> > mostly about figuring out the mechanics of doc generation.
>
> Agreed, the mirror will have to bring in the documentation
> subdirectory as well so the output could be seperate.
>
> Just want to update in this thread that i've been really preoccupied
> with other obligations and will get back to this next week.

No worries. Thanks for the update!

>
> >
> > >
> > > >
> > > > Thanks,
> > > >
> > > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Mon, May 10, 2021 at 1:48 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 10, 2021 at 7:59 AM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> > > <grantseltzer@gmail.com> wrote:
> > > >
> > > > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > > > >
> > > > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > > > >
> > > > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > > > just have a couple concerns for this but please point me in the right
> > > > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > > > This is based on reading
> > > > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > > > >
> > > > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > > > make it so documentation never falls out of date with code. Based on
> > > > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > > > specifying which functions/types are included in an .rst file and
> > > > > > submit a patch to update the documentation everytime the libbpf api
> > > > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > > > contribute it.
> > > > >
> > > > > No, you can tell it to pull out docs for all of the functions in a given
> > > > > file.  You only need to name things if you want to narrow things down.
> > > >
> > > > Alright, I will figure out how to do this and adjust the patch
> > > > accordingly. My biggest overall goal is making it as easy as possible
> > > > to contribute documentation. I think even adding just one doc string
> > > > above an API function is a great opportunity for new contributors to
> > > > familiarize themselves with the mailing list/patch process.
> > > >
> > > > >
> > > > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > > > of the kernel?
> > > > >
> > > > > It could certainly be built as a separate "book", as are many of the
> > > > > kernel books now.  I could see it as something that gets pulled into the
> > > > > user-space API book, but there could also perhaps be an argument made
> > > > > for creating a new "libraries" book instead.
> > > >
> > > > Yea if I can figure this out for the libbpf API it'd be great to
> > > > replicate it for any API!
> > >
> > > It would be great if it was possible to have this libbpf
> > > auto-generated documentation as part of the kernel documentation, but
> > > also be able to generate and export it into our Github mirror to be
> > > pulled by readthedocs.io. If that can be done, it would be the best of
> > > both kernel and external worlds. We have a sync script that already
> > > auto-generates and checks in BPF helpers header, so we have a
> > > precedent of checking in auto-generated stuff into Github. So it's
> > > mostly about figuring out the mechanics of doc generation.
> >
> > Agreed, the mirror will have to bring in the documentation
> > subdirectory as well so the output could be seperate.
> >
> > Just want to update in this thread that i've been really preoccupied
> > with other obligations and will get back to this next week.
>
> No worries. Thanks for the update!

Finally catching up on this, thanks for all of your patience!

I've discovered that it's actually very easy, even trivial, to add API
documentation for libbpf using the existing kernel sphinx
documentation system. Adding a couple files with directives under
`Documentaiton/bpf` is enough to pull in any comment-documented
functions/structs in libbpf code. I'm not sure who owns the CI/CD
infrastructure that recompiles the documentation and hosts on
kernel.org/doc but I've been building them locally with `make
htmldocs` with no problem. That would require a single patch and we
can start adding comment documentation to libbpf.  I can submit a
patch for that if you'd like to test it yourself. In this system the
html output is not checked into git though.

Andrii - what do you think of having libbpf API documentation hosted
on the kernel.org readthedocs? It would be nice to have it seperate
from the rest of kernel documentation for simplicity, though it is
nice to use/contribute-to existing infrastructure. If you'd like to
have it seperate we can have the libbpf mirror run `make htmldocs`,
take the output, and host our own readthedocs site.

I'd love to have this all set up and have the full API documented by
the time you cut the libbpf 1.0 release!

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Tue, May 25, 2021 at 8:22 PM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Mon, May 10, 2021 at 1:48 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Mon, May 10, 2021 at 7:59 AM Grant Seltzer Richman
> > <grantseltzer@gmail.com> wrote:
> > >
> > > On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > > >
> > > > On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> > > > <grantseltzer@gmail.com> wrote:
> > > > >
> > > > > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > > > > >
> > > > > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > > > > >
> > > > > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > > > > just have a couple concerns for this but please point me in the right
> > > > > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > > > > This is based on reading
> > > > > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > > > > >
> > > > > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > > > > make it so documentation never falls out of date with code. Based on
> > > > > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > > > > specifying which functions/types are included in an .rst file and
> > > > > > > submit a patch to update the documentation everytime the libbpf api
> > > > > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > > > > contribute it.
> > > > > >
> > > > > > No, you can tell it to pull out docs for all of the functions in a given
> > > > > > file.  You only need to name things if you want to narrow things down.
> > > > >
> > > > > Alright, I will figure out how to do this and adjust the patch
> > > > > accordingly. My biggest overall goal is making it as easy as possible
> > > > > to contribute documentation. I think even adding just one doc string
> > > > > above an API function is a great opportunity for new contributors to
> > > > > familiarize themselves with the mailing list/patch process.
> > > > >
> > > > > >
> > > > > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > > > > of the kernel?
> > > > > >
> > > > > > It could certainly be built as a separate "book", as are many of the
> > > > > > kernel books now.  I could see it as something that gets pulled into the
> > > > > > user-space API book, but there could also perhaps be an argument made
> > > > > > for creating a new "libraries" book instead.
> > > > >
> > > > > Yea if I can figure this out for the libbpf API it'd be great to
> > > > > replicate it for any API!
> > > >
> > > > It would be great if it was possible to have this libbpf
> > > > auto-generated documentation as part of the kernel documentation, but
> > > > also be able to generate and export it into our Github mirror to be
> > > > pulled by readthedocs.io. If that can be done, it would be the best of
> > > > both kernel and external worlds. We have a sync script that already
> > > > auto-generates and checks in BPF helpers header, so we have a
> > > > precedent of checking in auto-generated stuff into Github. So it's
> > > > mostly about figuring out the mechanics of doc generation.
> > >
> > > Agreed, the mirror will have to bring in the documentation
> > > subdirectory as well so the output could be seperate.
> > >
> > > Just want to update in this thread that i've been really preoccupied
> > > with other obligations and will get back to this next week.
> >
> > No worries. Thanks for the update!
>
> Finally catching up on this, thanks for all of your patience!
>
> I've discovered that it's actually very easy, even trivial, to add API
> documentation for libbpf using the existing kernel sphinx
> documentation system. Adding a couple files with directives under
> `Documentaiton/bpf` is enough to pull in any comment-documented
> functions/structs in libbpf code. I'm not sure who owns the CI/CD

Hopefully Jonathan will know.

> infrastructure that recompiles the documentation and hosts on
> kernel.org/doc but I've been building them locally with `make
> htmldocs` with no problem. That would require a single patch and we
> can start adding comment documentation to libbpf.  I can submit a
> patch for that if you'd like to test it yourself. In this system the
> html output is not checked into git though.
>
> Andrii - what do you think of having libbpf API documentation hosted
> on the kernel.org readthedocs? It would be nice to have it seperate
> from the rest of kernel documentation for simplicity, though it is
> nice to use/contribute-to existing infrastructure. If you'd like to
> have it seperate we can have the libbpf mirror run `make htmldocs`,
> take the output, and host our own readthedocs site.

Yeah, sure, let's start with libbpf docs as part of the kernel docs,
it doesn't hurt. But libbpf versioning is separate from kernel
versioning and most libbpf users "consume" libbpf from Github repo, so
we might need to do `make htmldocs` trick while syncing. But let's do
one step at a time.

>
> I'd love to have this all set up and have the full API documented by
> the time you cut the libbpf 1.0 release!

Yep, I agree.

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Wed, May 26, 2021 at 4:46 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Tue, May 25, 2021 at 8:22 PM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Mon, May 10, 2021 at 1:48 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Mon, May 10, 2021 at 7:59 AM Grant Seltzer Richman
> > > <grantseltzer@gmail.com> wrote:
> > > >
> > > > On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
> > > > <andrii.nakryiko@gmail.com> wrote:
> > > > >
> > > > > On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> > > > > <grantseltzer@gmail.com> wrote:
> > > > > >
> > > > > > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > > > > > >
> > > > > > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > > > > > >
> > > > > > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > > > > > just have a couple concerns for this but please point me in the right
> > > > > > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > > > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > > > > > This is based on reading
> > > > > > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > > > > > >
> > > > > > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > > > > > make it so documentation never falls out of date with code. Based on
> > > > > > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > > > > > specifying which functions/types are included in an .rst file and
> > > > > > > > submit a patch to update the documentation everytime the libbpf api
> > > > > > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > > > > > contribute it.
> > > > > > >
> > > > > > > No, you can tell it to pull out docs for all of the functions in a given
> > > > > > > file.  You only need to name things if you want to narrow things down.
> > > > > >
> > > > > > Alright, I will figure out how to do this and adjust the patch
> > > > > > accordingly. My biggest overall goal is making it as easy as possible
> > > > > > to contribute documentation. I think even adding just one doc string
> > > > > > above an API function is a great opportunity for new contributors to
> > > > > > familiarize themselves with the mailing list/patch process.
> > > > > >
> > > > > > >
> > > > > > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > > > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > > > > > of the kernel?
> > > > > > >
> > > > > > > It could certainly be built as a separate "book", as are many of the
> > > > > > > kernel books now.  I could see it as something that gets pulled into the
> > > > > > > user-space API book, but there could also perhaps be an argument made
> > > > > > > for creating a new "libraries" book instead.
> > > > > >
> > > > > > Yea if I can figure this out for the libbpf API it'd be great to
> > > > > > replicate it for any API!
> > > > >
> > > > > It would be great if it was possible to have this libbpf
> > > > > auto-generated documentation as part of the kernel documentation, but
> > > > > also be able to generate and export it into our Github mirror to be
> > > > > pulled by readthedocs.io. If that can be done, it would be the best of
> > > > > both kernel and external worlds. We have a sync script that already
> > > > > auto-generates and checks in BPF helpers header, so we have a
> > > > > precedent of checking in auto-generated stuff into Github. So it's
> > > > > mostly about figuring out the mechanics of doc generation.
> > > >
> > > > Agreed, the mirror will have to bring in the documentation
> > > > subdirectory as well so the output could be seperate.
> > > >
> > > > Just want to update in this thread that i've been really preoccupied
> > > > with other obligations and will get back to this next week.
> > >
> > > No worries. Thanks for the update!
> >
> > Finally catching up on this, thanks for all of your patience!
> >
> > I've discovered that it's actually very easy, even trivial, to add API
> > documentation for libbpf using the existing kernel sphinx
> > documentation system. Adding a couple files with directives under
> > `Documentaiton/bpf` is enough to pull in any comment-documented
> > functions/structs in libbpf code. I'm not sure who owns the CI/CD
>
> Hopefully Jonathan will know.
>
> > infrastructure that recompiles the documentation and hosts on
> > kernel.org/doc but I've been building them locally with `make
> > htmldocs` with no problem. That would require a single patch and we
> > can start adding comment documentation to libbpf.  I can submit a
> > patch for that if you'd like to test it yourself. In this system the
> > html output is not checked into git though.
> >
> > Andrii - what do you think of having libbpf API documentation hosted
> > on the kernel.org readthedocs? It would be nice to have it seperate
> > from the rest of kernel documentation for simplicity, though it is
> > nice to use/contribute-to existing infrastructure. If you'd like to
> > have it seperate we can have the libbpf mirror run `make htmldocs`,
> > take the output, and host our own readthedocs site.
>
> Yeah, sure, let's start with libbpf docs as part of the kernel docs,
> it doesn't hurt. But libbpf versioning is separate from kernel
> versioning and most libbpf users "consume" libbpf from Github repo, so
> we might need to do `make htmldocs` trick while syncing. But let's do
> one step at a time.

The more I think about this, I don't feel it makes sense to put the
libbpf docs in the main kernel ones if it isn't what works best for
libbpf. Sure we can stitch something together so we can version and
publish output from kernel docs but that'd cause too much confusion.

I'd rather go with this patch series, having a sphinx build script in
tools/lib/bpf. This requires no changes to how we sync the libbpf
mirror and we get the ability to pin documentation for each release
for free.

Jonathan - am I missing something about how we can version libbpf
separately from the actual kernel docs?

> >
> > I'd love to have this all set up and have the full API documented by
> > the time you cut the libbpf 1.0 release!
>
> Yep, I agree.

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Jonathan Corbet]

Grant Seltzer Richman <grantseltzer@gmail.com> writes:

> Jonathan - am I missing something about how we can version libbpf
> separately from the actual kernel docs?

Sorry, I missed this before.  I guess I still don't understand the
question, though; could you explain what the problem is here?  Apologies
for being so slow...

Thanks,

jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Tue, Jun 1, 2021 at 3:01 PM Jonathan Corbet <corbet@lwn.net> wrote:
>
> Grant Seltzer Richman <grantseltzer@gmail.com> writes:
>
> > Jonathan - am I missing something about how we can version libbpf
> > separately from the actual kernel docs?
>
> Sorry, I missed this before.  I guess I still don't understand the
> question, though; could you explain what the problem is here?  Apologies
> for being so slow...

No worries, thanks for your response and feedback! I made it confusing
by not including v2 in my second patch series (this is v1)

Andrii cuts releases of libbpf using the github mirror at
github.com/libbpf/libbpf. There's more context in the README there,
but most of the major distributions package libbpf from this mirror.
Since developers that use libbpf in their applications include libbpf
based on these github releases instead of versions of Linux (i.e. I
use libbpf 0.4, not libbpf from linux 5.14), it's important to have
the API documentation be labeled by the github release versions. Is
there any mechanism in the kernel docs that would allow us to do that?
Would it make more sense for the libbpf community to maintain their
own documentation system/website for this purpose?

>
> Thanks,
>
> jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Jonathan Corbet]

Grant Seltzer Richman <grantseltzer@gmail.com> writes:

> Andrii cuts releases of libbpf using the github mirror at
> github.com/libbpf/libbpf. There's more context in the README there,
> but most of the major distributions package libbpf from this mirror.
> Since developers that use libbpf in their applications include libbpf
> based on these github releases instead of versions of Linux (i.e. I
> use libbpf 0.4, not libbpf from linux 5.14), it's important to have
> the API documentation be labeled by the github release versions. Is
> there any mechanism in the kernel docs that would allow us to do that?
> Would it make more sense for the libbpf community to maintain their
> own documentation system/website for this purpose?

It depends on how you want that labeling to look, I guess.  One simple
thing might be to put a DOC: block into the libbpf code that holds the
version number, then use a kernel-doc directive to pull it in in the
appropriate place.  Alternatives might include adding a bit of magic to
Documentation/conf.py to fetch a "#define VERSION# out of the source
somewhere and stash the information away.

If you're wanting to replace the version code that appears at the top of
the left column in the HTML output, though, it's going to be a bit
harder.  I don't doubt we can do it, but it may require messing around
with template files and such.

Thanks,

jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Tue, Jun 1, 2021 at 7:19 PM Jonathan Corbet <corbet@lwn.net> wrote:
>
> Grant Seltzer Richman <grantseltzer@gmail.com> writes:
>
> > Andrii cuts releases of libbpf using the github mirror at
> > github.com/libbpf/libbpf. There's more context in the README there,
> > but most of the major distributions package libbpf from this mirror.
> > Since developers that use libbpf in their applications include libbpf
> > based on these github releases instead of versions of Linux (i.e. I
> > use libbpf 0.4, not libbpf from linux 5.14), it's important to have
> > the API documentation be labeled by the github release versions. Is
> > there any mechanism in the kernel docs that would allow us to do that?
> > Would it make more sense for the libbpf community to maintain their
> > own documentation system/website for this purpose?
>
> It depends on how you want that labeling to look, I guess.  One simple
> thing might be to put a DOC: block into the libbpf code that holds the
> version number, then use a kernel-doc directive to pull it in in the
> appropriate place.  Alternatives might include adding a bit of magic to
> Documentation/conf.py to fetch a "#define VERSION# out of the source
> somewhere and stash the information away.

Gotcha, I will investigate these approaches. Thanks!
>
> If you're wanting to replace the version code that appears at the top of
> the left column in the HTML output, though, it's going to be a bit
> harder.  I don't doubt we can do it, but it may require messing around
> with template files and such.
>
> Thanks,
>
> jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Tue, Jun 1, 2021 at 9:06 PM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Tue, Jun 1, 2021 at 7:19 PM Jonathan Corbet <corbet@lwn.net> wrote:
> >
> > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> >
> > > Andrii cuts releases of libbpf using the github mirror at
> > > github.com/libbpf/libbpf. There's more context in the README there,
> > > but most of the major distributions package libbpf from this mirror.
> > > Since developers that use libbpf in their applications include libbpf
> > > based on these github releases instead of versions of Linux (i.e. I
> > > use libbpf 0.4, not libbpf from linux 5.14), it's important to have
> > > the API documentation be labeled by the github release versions. Is
> > > there any mechanism in the kernel docs that would allow us to do that?
> > > Would it make more sense for the libbpf community to maintain their
> > > own documentation system/website for this purpose?
> >
> > It depends on how you want that labeling to look, I guess.  One simple
> > thing might be to put a DOC: block into the libbpf code that holds the
> > version number, then use a kernel-doc directive to pull it in in the
> > appropriate place.  Alternatives might include adding a bit of magic to
> > Documentation/conf.py to fetch a "#define VERSION# out of the source
> > somewhere and stash the information away.
>
> Gotcha, I will investigate these approaches. Thanks!

After investigating/attempting these approaches, my opinion is that it
would be better to have a separate libbpf documentation system (sphinx
configuration files). This way we can maintain separate versions of
the documentation for each release/version without having duplicate
pages, and without having to heavily change the kernel docs files to
fit libbpf specific needs.

If you check out libbpf.readthedocs.io you can see what that would
look like. I made a test release (v21.21.21) to show how easy this is.
That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.

I'm fine with having this new sphinx configuration in the kernel tree,
I'm also fine with having it on the github mirror. Both make sense to
me. Either way the comment docs have to be submitted through the
mailing list.

One last idea I have is to have the non-api docs (for example, the
document describing naming convention in libbpf) in the kernel tree,
and sync it in the github mirror.

Please feel free to ask questions, I've been thinking a lot about
this! Once we decide on which way to go I can have this up and running
almost immediately.
> >
> > If you're wanting to replace the version code that appears at the top of
> > the left column in the HTML output, though, it's going to be a bit
> > harder.  I don't doubt we can do it, but it may require messing around
> > with template files and such.
> >
> > Thanks,
> >
> > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Fri, Jun 4, 2021 at 2:19 PM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Tue, Jun 1, 2021 at 9:06 PM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Tue, Jun 1, 2021 at 7:19 PM Jonathan Corbet <corbet@lwn.net> wrote:
> > >
> > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > >
> > > > Andrii cuts releases of libbpf using the github mirror at
> > > > github.com/libbpf/libbpf. There's more context in the README there,
> > > > but most of the major distributions package libbpf from this mirror.
> > > > Since developers that use libbpf in their applications include libbpf
> > > > based on these github releases instead of versions of Linux (i.e. I
> > > > use libbpf 0.4, not libbpf from linux 5.14), it's important to have
> > > > the API documentation be labeled by the github release versions. Is
> > > > there any mechanism in the kernel docs that would allow us to do that?
> > > > Would it make more sense for the libbpf community to maintain their
> > > > own documentation system/website for this purpose?
> > >
> > > It depends on how you want that labeling to look, I guess.  One simple
> > > thing might be to put a DOC: block into the libbpf code that holds the
> > > version number, then use a kernel-doc directive to pull it in in the
> > > appropriate place.  Alternatives might include adding a bit of magic to
> > > Documentation/conf.py to fetch a "#define VERSION# out of the source
> > > somewhere and stash the information away.
> >
> > Gotcha, I will investigate these approaches. Thanks!
>
> After investigating/attempting these approaches, my opinion is that it
> would be better to have a separate libbpf documentation system (sphinx
> configuration files). This way we can maintain separate versions of
> the documentation for each release/version without having duplicate
> pages, and without having to heavily change the kernel docs files to
> fit libbpf specific needs.

So I assume you looked at the DOC: block that Jonathan suggested
above? Can you walk us a bit on how that would look like and why you
think it's not going to work?

>
> If you check out libbpf.readthedocs.io you can see what that would
> look like. I made a test release (v21.21.21) to show how easy this is.
> That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.

It looks pretty nice. Where does v21.21.21 come from, though? It's
also weird that docs are under src/docs, not just under docs/, but I
assume that's a quick hack to demonstrate this?

>
> I'm fine with having this new sphinx configuration in the kernel tree,
> I'm also fine with having it on the github mirror. Both make sense to
> me. Either way the comment docs have to be submitted through the
> mailing list.
>
> One last idea I have is to have the non-api docs (for example, the
> document describing naming convention in libbpf) in the kernel tree,
> and sync it in the github mirror.
>
> Please feel free to ask questions, I've been thinking a lot about
> this! Once we decide on which way to go I can have this up and running
> almost immediately.

To be entirely honest, I'm already a bit lost on all the
possibilities. It would be great if you can summarize what's possible
and how it would look like.

As a general guidance, I think we should try to keep all the
documentation in one place (which means kernel sources, because that's
where API documentation will have to leave). As for config files,
unless they will "stick out" too much, I'd keep them close to the docs
themselves. If not, putting them in Github mirror is fine by me as
well.

Pretty much the only important aspect, from my point of view, is that
docs are versioned according to the libbpf version, not kernel
version. Otherwise huge confusion will ensue for all the users of
libbpf, most of not all of which are using Github mirror.

Does this make sense and is doable?

> > >
> > > If you're wanting to replace the version code that appears at the top of
> > > the left column in the HTML output, though, it's going to be a bit
> > > harder.  I don't doubt we can do it, but it may require messing around
> > > with template files and such.
> > >
> > > Thanks,
> > >
> > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Mon, Jun 7, 2021 at 6:45 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:

> So I assume you looked at the DOC: block that Jonathan suggested
> above? Can you walk us a bit on how that would look like and why you
> think it's not going to work?

If I understand correctly, the using the  DOC: comment block and
extracting it would require you to manually update a comment in code
that labels the libbpf version number and then cut a release. The
problems are:

1) Having to do that manually
2) The docs on kernel.org would not show previous libbpf releases
(just the ones that were released with different kernel versions)

> > If you check out libbpf.readthedocs.io you can see what that would
> > look like. I made a test release (v21.21.21) to show how easy this is.
> > That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.
>
> It looks pretty nice. Where does v21.21.21 come from, though?

libbpf.readthedocs.org currently tracks my libbpf fork at
github.com/grantseltzer/libbpf, I cut the 21.21.21 release there.

What this demonstrates is that readthedocs pulls in any release or
branch I enable in the admin panel and builds a seperate site for each
one. You can see the release/branches by toggling them in the bottom
left.

> also weird that docs are under src/docs, not just under docs/, but I
> assume that's a quick hack to demonstrate this?

I can move that no problem. All the config files have relative paths
that point at each other.

> To be entirely honest, I'm already a bit lost on all the
> possibilities. It would be great if you can summarize what's possible
> and how it would look like.

That's my fault, I'm not making this easy to review (2 different patch
series, plus github PR). Here is a summary of the 3 different
approaches I've presented. At the bottom I have my recommendation.

1) Integrate libbpf docs into the kernel documentation system:

- Add libbpf documentation files (e.g. 'naming-convention',
'building-libbpf') under the Linux 'Documentation' directory.
- Include a file which has sphinx directives to pull in API
documentation from the code under tools/lib/bpf.
- This documentation would appear on kernel.org/doc alongside all
other kernel documentation

Pro: Make use of existing kernel documentation infrastructure
Con: In order to have the libbpf documentation be versioned based on
github releases it would require manually updating comments in code.
To get it how we really want it (i.e. select a libbpf release and see
the documentation specific for it), it would likely require
rearranging/duplicating code in libbpf under subdirectories.
Contributing simple documentation has to go through the mailing list
which is daunting for new contributors.

2) Have all libbpf documentation in the kernel repo under
tools/lib/bpf/docs, including libbpf's own sphinx configuration files

- The github mirror would pull in these documentation files and sphinx
configuration files.
- readthedocs would be configured to build off of the github libbpf mirror repo.
- The documentation would be accessible at libbpf.readthedocs.org.
- Users would be able to navigate documentation of different releases
of libbpf easily. Maintainers would not need to do any manual work to
make that happen.

Pro: Documentation all in one place, in the kernel repo, integrates
easily with the github mirror, and we easily get to version
documentation by libbpf releases.
Con: We don't make use of existing kernel documentation
infrastructure. Contributing simple documentation has to go through
the mailing list which is daunting for new contributors.

3) Host all the documentation on the libbpf github mirror

- Documents (e.g. 'naming-convention', 'building-libbpf') would be
available at libbpf.readthedocs.org
- Sphinx configuration files would live in the libbpf github mirror
- Code comment documentation would still go through the mailing list

Pro: Documentation is explorable by different versions. Some
documentation can be contributed via github which is easier than the
mailing list.
Con: Documentation would be scattered across different places.

My recommendation:

I would go with option 2, it feels like the best compromise. However,
I'd like to contribute changes to the kernel documentation such that
it can support versioning particular documentation files based on
specified git hashes. That way we can have libbpf docs on kernel.org
but still get to version docs based on libbpf releases, not kernel
ones.

I'm happy to explore other options, but if one of these sounds like
something that we can agree on for now, I can submit a fresh patch for
it.


>
> As a general guidance, I think we should try to keep all the
> documentation in one place (which means kernel sources, because that's
> where API documentation will have to leave). As for config files,
> unless they will "stick out" too much, I'd keep them close to the docs
> themselves. If not, putting them in Github mirror is fine by me as
> well.
>
> Pretty much the only important aspect, from my point of view, is that
> docs are versioned according to the libbpf version, not kernel
> version. Otherwise huge confusion will ensue for all the users of
> libbpf, most of not all of which are using Github mirror.
>
> Does this make sense and is doable?
>
> > > >
> > > > If you're wanting to replace the version code that appears at the top of
> > > > the left column in the HTML output, though, it's going to be a bit
> > > > harder.  I don't doubt we can do it, but it may require messing around
> > > > with template files and such.
> > > >
> > > > Thanks,
> > > >
> > > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Wed, Jun 9, 2021 at 10:04 AM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Mon, Jun 7, 2021 at 6:45 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
>
> > So I assume you looked at the DOC: block that Jonathan suggested
> > above? Can you walk us a bit on how that would look like and why you
> > think it's not going to work?
>
> If I understand correctly, the using the  DOC: comment block and
> extracting it would require you to manually update a comment in code
> that labels the libbpf version number and then cut a release. The
> problems are:
>
> 1) Having to do that manually
> 2) The docs on kernel.org would not show previous libbpf releases
> (just the ones that were released with different kernel versions)
>
> > > If you check out libbpf.readthedocs.io you can see what that would
> > > look like. I made a test release (v21.21.21) to show how easy this is.
> > > That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.
> >
> > It looks pretty nice. Where does v21.21.21 come from, though?
>
> libbpf.readthedocs.org currently tracks my libbpf fork at
> github.com/grantseltzer/libbpf, I cut the 21.21.21 release there.
>
> What this demonstrates is that readthedocs pulls in any release or
> branch I enable in the admin panel and builds a seperate site for each
> one. You can see the release/branches by toggling them in the bottom
> left.
>
> > also weird that docs are under src/docs, not just under docs/, but I
> > assume that's a quick hack to demonstrate this?
>
> I can move that no problem. All the config files have relative paths
> that point at each other.
>
> > To be entirely honest, I'm already a bit lost on all the
> > possibilities. It would be great if you can summarize what's possible
> > and how it would look like.
>
> That's my fault, I'm not making this easy to review (2 different patch
> series, plus github PR). Here is a summary of the 3 different
> approaches I've presented. At the bottom I have my recommendation.
>
> 1) Integrate libbpf docs into the kernel documentation system:
>
> - Add libbpf documentation files (e.g. 'naming-convention',
> 'building-libbpf') under the Linux 'Documentation' directory.
> - Include a file which has sphinx directives to pull in API
> documentation from the code under tools/lib/bpf.
> - This documentation would appear on kernel.org/doc alongside all
> other kernel documentation
>
> Pro: Make use of existing kernel documentation infrastructure
> Con: In order to have the libbpf documentation be versioned based on
> github releases it would require manually updating comments in code.
> To get it how we really want it (i.e. select a libbpf release and see
> the documentation specific for it), it would likely require
> rearranging/duplicating code in libbpf under subdirectories.
> Contributing simple documentation has to go through the mailing list
> which is daunting for new contributors.
>
> 2) Have all libbpf documentation in the kernel repo under
> tools/lib/bpf/docs, including libbpf's own sphinx configuration files
>
> - The github mirror would pull in these documentation files and sphinx
> configuration files.
> - readthedocs would be configured to build off of the github libbpf mirror repo.
> - The documentation would be accessible at libbpf.readthedocs.org.
> - Users would be able to navigate documentation of different releases
> of libbpf easily. Maintainers would not need to do any manual work to
> make that happen.
>
> Pro: Documentation all in one place, in the kernel repo, integrates
> easily with the github mirror, and we easily get to version
> documentation by libbpf releases.
> Con: We don't make use of existing kernel documentation
> infrastructure. Contributing simple documentation has to go through
> the mailing list which is daunting for new contributors.
>
> 3) Host all the documentation on the libbpf github mirror
>
> - Documents (e.g. 'naming-convention', 'building-libbpf') would be
> available at libbpf.readthedocs.org
> - Sphinx configuration files would live in the libbpf github mirror
> - Code comment documentation would still go through the mailing list
>
> Pro: Documentation is explorable by different versions. Some
> documentation can be contributed via github which is easier than the
> mailing list.
> Con: Documentation would be scattered across different places.
>
> My recommendation:
>
> I would go with option 2, it feels like the best compromise. However,
> I'd like to contribute changes to the kernel documentation such that
> it can support versioning particular documentation files based on
> specified git hashes. That way we can have libbpf docs on kernel.org
> but still get to version docs based on libbpf releases, not kernel
> ones.
>
> I'm happy to explore other options, but if one of these sounds like
> something that we can agree on for now, I can submit a fresh patch for
> it.

Honestly, I think option #3 would be best for actually getting more
docs contributions, because it will reduce the friction of getting
started. Figuring out the kernel mailing list submission process is
not a trivial problem.

But I'm sure some people will have strong feelings about keeping docs
in the kernel source tree. Assuming we'd get a lot of push back for
option #3, I think the next best thing would be a hybrid of #1 and #2.
Host docs in kernel docs under Documentation/bpf/libbpf, so it will
get into kernel sources automatically. But also copy those docs into
Github mirror during sync under /docs and have readthedocs pull and
version it properly. That has a chance of creating a bit of confusion
for where exactly libbpf docs are (it would be libbpf.readthedocs.org,
versioned properly), but given docs will have the same source of truth
(modulo minor versioning differences, potentially), I think this might
be ok. WDYT? Are there any problems with that?

Hosting docs under tools/libbpf/docs in kernel repo is the worst of
both worlds, IMO. Hard to contribute, but also not integrated with the
rest of kernel docs.

>
>
> >
> > As a general guidance, I think we should try to keep all the
> > documentation in one place (which means kernel sources, because that's
> > where API documentation will have to leave). As for config files,
> > unless they will "stick out" too much, I'd keep them close to the docs
> > themselves. If not, putting them in Github mirror is fine by me as
> > well.
> >
> > Pretty much the only important aspect, from my point of view, is that
> > docs are versioned according to the libbpf version, not kernel
> > version. Otherwise huge confusion will ensue for all the users of
> > libbpf, most of not all of which are using Github mirror.
> >
> > Does this make sense and is doable?
> >
> > > > >
> > > > > If you're wanting to replace the version code that appears at the top of
> > > > > the left column in the HTML output, though, it's going to be a bit
> > > > > harder.  I don't doubt we can do it, but it may require messing around
> > > > > with template files and such.
> > > > >
> > > > > Thanks,
> > > > >
> > > > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman]

On Thu, Jun 10, 2021 at 1:14 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Wed, Jun 9, 2021 at 10:04 AM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Mon, Jun 7, 2021 at 6:45 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> >
> > > So I assume you looked at the DOC: block that Jonathan suggested
> > > above? Can you walk us a bit on how that would look like and why you
> > > think it's not going to work?
> >
> > If I understand correctly, the using the  DOC: comment block and
> > extracting it would require you to manually update a comment in code
> > that labels the libbpf version number and then cut a release. The
> > problems are:
> >
> > 1) Having to do that manually
> > 2) The docs on kernel.org would not show previous libbpf releases
> > (just the ones that were released with different kernel versions)
> >
> > > > If you check out libbpf.readthedocs.io you can see what that would
> > > > look like. I made a test release (v21.21.21) to show how easy this is.
> > > > That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.
> > >
> > > It looks pretty nice. Where does v21.21.21 come from, though?
> >
> > libbpf.readthedocs.org currently tracks my libbpf fork at
> > github.com/grantseltzer/libbpf, I cut the 21.21.21 release there.
> >
> > What this demonstrates is that readthedocs pulls in any release or
> > branch I enable in the admin panel and builds a seperate site for each
> > one. You can see the release/branches by toggling them in the bottom
> > left.
> >
> > > also weird that docs are under src/docs, not just under docs/, but I
> > > assume that's a quick hack to demonstrate this?
> >
> > I can move that no problem. All the config files have relative paths
> > that point at each other.
> >
> > > To be entirely honest, I'm already a bit lost on all the
> > > possibilities. It would be great if you can summarize what's possible
> > > and how it would look like.
> >
> > That's my fault, I'm not making this easy to review (2 different patch
> > series, plus github PR). Here is a summary of the 3 different
> > approaches I've presented. At the bottom I have my recommendation.
> >
> > 1) Integrate libbpf docs into the kernel documentation system:
> >
> > - Add libbpf documentation files (e.g. 'naming-convention',
> > 'building-libbpf') under the Linux 'Documentation' directory.
> > - Include a file which has sphinx directives to pull in API
> > documentation from the code under tools/lib/bpf.
> > - This documentation would appear on kernel.org/doc alongside all
> > other kernel documentation
> >
> > Pro: Make use of existing kernel documentation infrastructure
> > Con: In order to have the libbpf documentation be versioned based on
> > github releases it would require manually updating comments in code.
> > To get it how we really want it (i.e. select a libbpf release and see
> > the documentation specific for it), it would likely require
> > rearranging/duplicating code in libbpf under subdirectories.
> > Contributing simple documentation has to go through the mailing list
> > which is daunting for new contributors.
> >
> > 2) Have all libbpf documentation in the kernel repo under
> > tools/lib/bpf/docs, including libbpf's own sphinx configuration files
> >
> > - The github mirror would pull in these documentation files and sphinx
> > configuration files.
> > - readthedocs would be configured to build off of the github libbpf mirror repo.
> > - The documentation would be accessible at libbpf.readthedocs.org.
> > - Users would be able to navigate documentation of different releases
> > of libbpf easily. Maintainers would not need to do any manual work to
> > make that happen.
> >
> > Pro: Documentation all in one place, in the kernel repo, integrates
> > easily with the github mirror, and we easily get to version
> > documentation by libbpf releases.
> > Con: We don't make use of existing kernel documentation
> > infrastructure. Contributing simple documentation has to go through
> > the mailing list which is daunting for new contributors.
> >
> > 3) Host all the documentation on the libbpf github mirror
> >
> > - Documents (e.g. 'naming-convention', 'building-libbpf') would be
> > available at libbpf.readthedocs.org
> > - Sphinx configuration files would live in the libbpf github mirror
> > - Code comment documentation would still go through the mailing list
> >
> > Pro: Documentation is explorable by different versions. Some
> > documentation can be contributed via github which is easier than the
> > mailing list.
> > Con: Documentation would be scattered across different places.
> >
> > My recommendation:
> >
> > I would go with option 2, it feels like the best compromise. However,
> > I'd like to contribute changes to the kernel documentation such that
> > it can support versioning particular documentation files based on
> > specified git hashes. That way we can have libbpf docs on kernel.org
> > but still get to version docs based on libbpf releases, not kernel
> > ones.
> >
> > I'm happy to explore other options, but if one of these sounds like
> > something that we can agree on for now, I can submit a fresh patch for
> > it.
>
> Honestly, I think option #3 would be best for actually getting more
> docs contributions, because it will reduce the friction of getting
> started. Figuring out the kernel mailing list submission process is
> not a trivial problem.

I can agree to that. This is also the easiest option to change,
completely undo, or migrate off of. Since it's on github only we don't
have to worry about all the different mailing lists. I think that
makes it a really good starting point regardless.

My PR on the github repo is up to date and ready for review. Once
merged I can seamlessly move libbpf.readthedocs.org to pull from that
repo instead of my fork.

Also I'd like to add you and whoever else as an admin, if you can sign
up for readthedocs just let me know your username and i'll add you.
And feel free to add any other libbpf maintainers that you think it'd
be appropriate.

>
> But I'm sure some people will have strong feelings about keeping docs
> in the kernel source tree. Assuming we'd get a lot of push back for
> option #3, I think the next best thing would be a hybrid of #1 and #2.
> Host docs in kernel docs under Documentation/bpf/libbpf, so it will
> get into kernel sources automatically. But also copy those docs into
> Github mirror during sync under /docs and have readthedocs pull and
> version it properly. That has a chance of creating a bit of confusion
> for where exactly libbpf docs are (it would be libbpf.readthedocs.org,
> versioned properly), but given docs will have the same source of truth
> (modulo minor versioning differences, potentially), I think this might
> be ok. WDYT? Are there any problems with that?

The only issue I foresee is having the 'latest' docs on kernel.org/doc
and having versioned docs on libbpf.readthedocs.org. That'd confuse
me.

>
> Hosting docs under tools/libbpf/docs in kernel repo is the worst of
> both worlds, IMO. Hard to contribute, but also not integrated with the
> rest of kernel docs.

Yea, that's also a good point!
>
> >
> >
> > >
> > > As a general guidance, I think we should try to keep all the
> > > documentation in one place (which means kernel sources, because that's
> > > where API documentation will have to leave). As for config files,
> > > unless they will "stick out" too much, I'd keep them close to the docs
> > > themselves. If not, putting them in Github mirror is fine by me as
> > > well.
> > >
> > > Pretty much the only important aspect, from my point of view, is that
> > > docs are versioned according to the libbpf version, not kernel
> > > version. Otherwise huge confusion will ensue for all the users of
> > > libbpf, most of not all of which are using Github mirror.
> > >
> > > Does this make sense and is doable?
> > >
> > > > > >
> > > > > > If you're wanting to replace the version code that appears at the top of
> > > > > > the left column in the HTML output, though, it's going to be a bit
> > > > > > harder.  I don't doubt we can do it, but it may require messing around
> > > > > > with template files and such.
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > jon

Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Andrii Nakryiko]

On Fri, Jun 11, 2021 at 1:01 PM Grant Seltzer Richman
<grantseltzer@gmail.com> wrote:
>
> On Thu, Jun 10, 2021 at 1:14 PM Andrii Nakryiko
> <andrii.nakryiko@gmail.com> wrote:
> >
> > On Wed, Jun 9, 2021 at 10:04 AM Grant Seltzer Richman
> > <grantseltzer@gmail.com> wrote:
> > >
> > > On Mon, Jun 7, 2021 at 6:45 PM Andrii Nakryiko
> > > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > > So I assume you looked at the DOC: block that Jonathan suggested
> > > > above? Can you walk us a bit on how that would look like and why you
> > > > think it's not going to work?
> > >
> > > If I understand correctly, the using the  DOC: comment block and
> > > extracting it would require you to manually update a comment in code
> > > that labels the libbpf version number and then cut a release. The
> > > problems are:
> > >
> > > 1) Having to do that manually
> > > 2) The docs on kernel.org would not show previous libbpf releases
> > > (just the ones that were released with different kernel versions)
> > >
> > > > > If you check out libbpf.readthedocs.io you can see what that would
> > > > > look like. I made a test release (v21.21.21) to show how easy this is.
> > > > > That is being pulled from my PR at github.com/libbpf/libbpf/pull/260.
> > > >
> > > > It looks pretty nice. Where does v21.21.21 come from, though?
> > >
> > > libbpf.readthedocs.org currently tracks my libbpf fork at
> > > github.com/grantseltzer/libbpf, I cut the 21.21.21 release there.
> > >
> > > What this demonstrates is that readthedocs pulls in any release or
> > > branch I enable in the admin panel and builds a seperate site for each
> > > one. You can see the release/branches by toggling them in the bottom
> > > left.
> > >
> > > > also weird that docs are under src/docs, not just under docs/, but I
> > > > assume that's a quick hack to demonstrate this?
> > >
> > > I can move that no problem. All the config files have relative paths
> > > that point at each other.
> > >
> > > > To be entirely honest, I'm already a bit lost on all the
> > > > possibilities. It would be great if you can summarize what's possible
> > > > and how it would look like.
> > >
> > > That's my fault, I'm not making this easy to review (2 different patch
> > > series, plus github PR). Here is a summary of the 3 different
> > > approaches I've presented. At the bottom I have my recommendation.
> > >
> > > 1) Integrate libbpf docs into the kernel documentation system:
> > >
> > > - Add libbpf documentation files (e.g. 'naming-convention',
> > > 'building-libbpf') under the Linux 'Documentation' directory.
> > > - Include a file which has sphinx directives to pull in API
> > > documentation from the code under tools/lib/bpf.
> > > - This documentation would appear on kernel.org/doc alongside all
> > > other kernel documentation
> > >
> > > Pro: Make use of existing kernel documentation infrastructure
> > > Con: In order to have the libbpf documentation be versioned based on
> > > github releases it would require manually updating comments in code.
> > > To get it how we really want it (i.e. select a libbpf release and see
> > > the documentation specific for it), it would likely require
> > > rearranging/duplicating code in libbpf under subdirectories.
> > > Contributing simple documentation has to go through the mailing list
> > > which is daunting for new contributors.
> > >
> > > 2) Have all libbpf documentation in the kernel repo under
> > > tools/lib/bpf/docs, including libbpf's own sphinx configuration files
> > >
> > > - The github mirror would pull in these documentation files and sphinx
> > > configuration files.
> > > - readthedocs would be configured to build off of the github libbpf mirror repo.
> > > - The documentation would be accessible at libbpf.readthedocs.org.
> > > - Users would be able to navigate documentation of different releases
> > > of libbpf easily. Maintainers would not need to do any manual work to
> > > make that happen.
> > >
> > > Pro: Documentation all in one place, in the kernel repo, integrates
> > > easily with the github mirror, and we easily get to version
> > > documentation by libbpf releases.
> > > Con: We don't make use of existing kernel documentation
> > > infrastructure. Contributing simple documentation has to go through
> > > the mailing list which is daunting for new contributors.
> > >
> > > 3) Host all the documentation on the libbpf github mirror
> > >
> > > - Documents (e.g. 'naming-convention', 'building-libbpf') would be
> > > available at libbpf.readthedocs.org
> > > - Sphinx configuration files would live in the libbpf github mirror
> > > - Code comment documentation would still go through the mailing list
> > >
> > > Pro: Documentation is explorable by different versions. Some
> > > documentation can be contributed via github which is easier than the
> > > mailing list.
> > > Con: Documentation would be scattered across different places.
> > >
> > > My recommendation:
> > >
> > > I would go with option 2, it feels like the best compromise. However,
> > > I'd like to contribute changes to the kernel documentation such that
> > > it can support versioning particular documentation files based on
> > > specified git hashes. That way we can have libbpf docs on kernel.org
> > > but still get to version docs based on libbpf releases, not kernel
> > > ones.
> > >
> > > I'm happy to explore other options, but if one of these sounds like
> > > something that we can agree on for now, I can submit a fresh patch for
> > > it.
> >
> > Honestly, I think option #3 would be best for actually getting more
> > docs contributions, because it will reduce the friction of getting
> > started. Figuring out the kernel mailing list submission process is
> > not a trivial problem.
>
> I can agree to that. This is also the easiest option to change,
> completely undo, or migrate off of. Since it's on github only we don't
> have to worry about all the different mailing lists. I think that
> makes it a really good starting point regardless.

Let's just go with option #1, let's put the docs in kernel repo under
Documentation/bpf/libbpf and copy/sync them in the sync script. That
will keep most libbpf stuff (except Github-specific Travis CI configs)
in one repo (kernel), reviewed by the bigger set of folks (not just me
on Github). Let's see how confusing having two libbpf documentation
sites will be (we can always add a sentence to README.md on Github to
point to the "canonical" source of documentation at
libbpf.readthedocs.io).

>
> My PR on the github repo is up to date and ready for review. Once
> merged I can seamlessly move libbpf.readthedocs.org to pull from that
> repo instead of my fork.
>
> Also I'd like to add you and whoever else as an admin, if you can sign
> up for readthedocs just let me know your username and i'll add you.
> And feel free to add any other libbpf maintainers that you think it'd
> be appropriate.

I signed up, my user is anakryiko, thanks.

>
> >
> > But I'm sure some people will have strong feelings about keeping docs
> > in the kernel source tree. Assuming we'd get a lot of push back for
> > option #3, I think the next best thing would be a hybrid of #1 and #2.
> > Host docs in kernel docs under Documentation/bpf/libbpf, so it will
> > get into kernel sources automatically. But also copy those docs into
> > Github mirror during sync under /docs and have readthedocs pull and
> > version it properly. That has a chance of creating a bit of confusion
> > for where exactly libbpf docs are (it would be libbpf.readthedocs.org,
> > versioned properly), but given docs will have the same source of truth
> > (modulo minor versioning differences, potentially), I think this might
> > be ok. WDYT? Are there any problems with that?
>
> The only issue I foresee is having the 'latest' docs on kernel.org/doc
> and having versioned docs on libbpf.readthedocs.org. That'd confuse
> me.

Yep, I share the feeling, but let's start with being good citizens in
the kernel. If that turns out to be too confusing and painful, we'll
reassess.

>
> >
> > Hosting docs under tools/libbpf/docs in kernel repo is the worst of
> > both worlds, IMO. Hard to contribute, but also not integrated with the
> > rest of kernel docs.
>
> Yea, that's also a good point!
> >
> > >
> > >
> > > >
> > > > As a general guidance, I think we should try to keep all the
> > > > documentation in one place (which means kernel sources, because that's
> > > > where API documentation will have to leave). As for config files,
> > > > unless they will "stick out" too much, I'd keep them close to the docs
> > > > themselves. If not, putting them in Github mirror is fine by me as
> > > > well.
> > > >
> > > > Pretty much the only important aspect, from my point of view, is that
> > > > docs are versioned according to the libbpf version, not kernel
> > > > version. Otherwise huge confusion will ensue for all the users of
> > > > libbpf, most of not all of which are using Github mirror.
> > > >
> > > > Does this make sense and is doable?
> > > >
> > > > > > >
> > > > > > > If you're wanting to replace the version code that appears at the top of
> > > > > > > the left column in the HTML output, though, it's going to be a bit
> > > > > > > harder.  I don't doubt we can do it, but it may require messing around
> > > > > > > with template files and such.
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > jon