[PATCH bpf-next 0/2] Autogenerating libbpf API documentation

[PATCH bpf-next 0/2] Autogenerating libbpf API documentation

[grantseltzer]

This patch series is meant to start the initiative to document libbpf.
It includes .rst files which are text documentation describing building, 
API naming convention, as well as an index to generated API documentation.

In this approach the generated API documentation is enabled by the kernels
existing kernel documentation system which uses sphinx. The resulting docs
would then be synced to kernel.org/doc

You can test this by running `make htmldocs` and serving the html in 
Documentation/output. Since libbpf does not yet have comments in kernel
doc format, see kernel.org/doc/html/latest/doc-guide/kernel-doc.html for
an example so you can test this.

The advantage of this approach is to use the existing sphinx
infrastructure that the kernel has, and have libbpf docs in
the same place as everything else.

The perhaps large disadvantage of this approach is that libbpf versions
independently from the kernel. If it's possible to version libbpf
separately without having duplicates that would be the ideal scenario.

grantseltzer (2):
  Add documentation for libbpf including API autogen
  Remove duplicate README doc from libbpf

 Documentation/bpf/index.rst                   | 13 +++++++
 Documentation/bpf/libbpf.rst                  | 14 +++++++
 Documentation/bpf/libbpf_api.rst              | 18 +++++++++
 Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++
 .../bpf/libbpf_naming_convention.rst          | 20 +++++-----
 5 files changed, 93 insertions(+), 9 deletions(-)
 create mode 100644 Documentation/bpf/libbpf.rst
 create mode 100644 Documentation/bpf/libbpf_api.rst
 create mode 100644 Documentation/bpf/libbpf_build.rst
 rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (96%)

-- 
2.29.2

[PATCH bpf-next 1/2] Add documentation for libbpf including API autogen

[grantseltzer]

This adds rst files containing documentation for libbpf. This includes
the addition of libbpf_api.rst which pulls comment documentation from
header files in libbpf under tools/lib/bpf/. The comment docs would be
of the standard kernel doc format.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 Documentation/bpf/index.rst                   |  13 ++
 Documentation/bpf/libbpf.rst                  |  14 ++
 Documentation/bpf/libbpf_api.rst              |  18 ++
 Documentation/bpf/libbpf_build.rst            |  37 ++++
 .../bpf/libbpf_naming_convention.rst          | 170 ++++++++++++++++++
 5 files changed, 252 insertions(+)
 create mode 100644 Documentation/bpf/libbpf.rst
 create mode 100644 Documentation/bpf/libbpf_api.rst
 create mode 100644 Documentation/bpf/libbpf_build.rst
 create mode 100644 Documentation/bpf/libbpf_naming_convention.rst

diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index a702f67dd..44f646735 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst
@@ -12,6 +12,19 @@ BPF instruction-set.
 The Cilium project also maintains a `BPF and XDP Reference Guide`_
 that goes into great technical depth about the BPF Architecture.
 
+libbpf
+======
+
+Libbpf is a userspace library for loading and interacting with bpf programs.
+
+.. toctree::
+   :maxdepth: 1
+
+   libbpf
+   libbpf_api
+   libbpf_build
+   libbpf_naming_convention
+
 BPF Type Format (BTF)
 =====================
 
diff --git a/Documentation/bpf/libbpf.rst b/Documentation/bpf/libbpf.rst
new file mode 100644
index 000000000..515e5f341
--- /dev/null
+++ b/Documentation/bpf/libbpf.rst
@@ -0,0 +1,14 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+libbpf
+======
+
+This is documentation for libbpf, a userspace library for loading and
+interacting with bpf programs.
+
+All general BPF questions, including kernel functionality, libbpf APIs and
+their application, should be sent to bpf@vger.kernel.org mailing list.
+You can subscribe to it `here <http://vger.kernel.org/vger-lists.html#bpf>`_
+and search its archive `here <https://lore.kernel.org/bpf/>`_.
+Please search the archive before asking new questions. It very well might
+be that this was already addressed or answered before.
diff --git a/Documentation/bpf/libbpf_api.rst b/Documentation/bpf/libbpf_api.rst
new file mode 100644
index 000000000..fd6b25e03
--- /dev/null
+++ b/Documentation/bpf/libbpf_api.rst
@@ -0,0 +1,18 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+API
+===
+
+This documentation is autogenerated from header files in libbpf, tools/lib/bpf
+
+.. kernel-doc:: tools/lib/bpf/libbpf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/bpf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/btf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/xsk.h
+   :internal:
diff --git a/Documentation/bpf/libbpf_build.rst b/Documentation/bpf/libbpf_build.rst
new file mode 100644
index 000000000..b8240eaaa
--- /dev/null
+++ b/Documentation/bpf/libbpf_build.rst
@@ -0,0 +1,37 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Building libbpf
+===============
+
+libelf is an internal dependency of libbpf and thus it is required to link
+against and must be installed on the system for applications to work.
+pkg-config is used by default to find libelf, and the program called
+can be overridden with PKG_CONFIG.
+
+If using pkg-config at build time is not desired, it can be disabled by
+setting NO_PKG_CONFIG=1 when calling make.
+
+To build both static libbpf.a and shared libbpf.so:
+
+.. code-block:: bash
+
+    $ cd src
+    $ make
+
+To build only static libbpf.a library in directory build/ and install them
+together with libbpf headers in a staging directory root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ mkdir build root
+    $ BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install
+
+To build both static libbpf.a and shared libbpf.so against a custom libelf
+dependency installed in /build/root/ and install them together with libbpf
+headers in a build directory /build/root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ PKG_CONFIG_PATH=/build/root/lib64/pkgconfig DESTDIR=/build/root make
\ No newline at end of file
diff --git a/Documentation/bpf/libbpf_naming_convention.rst b/Documentation/bpf/libbpf_naming_convention.rst
new file mode 100644
index 000000000..048be7933
--- /dev/null
+++ b/Documentation/bpf/libbpf_naming_convention.rst
@@ -0,0 +1,170 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+API naming convention
+=====================
+
+libbpf API provides access to a few logically separated groups of
+functions and types. Every group has its own naming convention
+described here. It's recommended to follow these conventions whenever a
+new function or type is added to keep libbpf API clean and consistent.
+
+All types and functions provided by libbpf API should have one of the
+following prefixes: ``bpf_``, ``btf_``, ``libbpf_``, ``xsk_``,
+``perf_buffer_``.
+
+System call wrappers
+--------------------
+
+System call wrappers are simple wrappers for commands supported by
+sys_bpf system call. These wrappers should go to ``bpf.h`` header file
+and map one-on-one to corresponding commands.
+
+For example ``bpf_map_lookup_elem`` wraps ``BPF_MAP_LOOKUP_ELEM``
+command of sys_bpf, ``bpf_prog_attach`` wraps ``BPF_PROG_ATTACH``, etc.
+
+Objects
+-------
+
+Another class of types and functions provided by libbpf API is "objects"
+and functions to work with them. Objects are high-level abstractions
+such as BPF program or BPF map. They're represented by corresponding
+structures such as ``struct bpf_object``, ``struct bpf_program``,
+``struct bpf_map``, etc.
+
+Structures are forward declared and access to their fields should be
+provided via corresponding getters and setters rather than directly.
+
+These objects are associated with corresponding parts of ELF object that
+contains compiled BPF programs.
+
+For example ``struct bpf_object`` represents ELF object itself created
+from an ELF file or from a buffer, ``struct bpf_program`` represents a
+program in ELF object and ``struct bpf_map`` is a map.
+
+Functions that work with an object have names built from object name,
+double underscore and part that describes function purpose.
+
+For example ``bpf_object__open`` consists of the name of corresponding
+object, ``bpf_object``, double underscore and ``open`` that defines the
+purpose of the function to open ELF file and create ``bpf_object`` from
+it.
+
+Another example: ``bpf_program__load`` is named for corresponding
+object, ``bpf_program``, that is separated from other part of the name
+by double underscore.
+
+All objects and corresponding functions other than BTF related should go
+to ``libbpf.h``. BTF types and functions should go to ``btf.h``.
+
+Auxiliary functions
+-------------------
+
+Auxiliary functions and types that don't fit well in any of categories
+described above should have ``libbpf_`` prefix, e.g.
+``libbpf_get_error`` or ``libbpf_prog_type_by_name``.
+
+AF_XDP functions
+-------------------
+
+AF_XDP functions should have an ``xsk_`` prefix, e.g.
+``xsk_umem__get_data`` or ``xsk_umem__create``. The interface consists
+of both low-level ring access functions and high-level configuration
+functions. These can be mixed and matched. Note that these functions
+are not reentrant for performance reasons.
+
+Please take a look at Documentation/networking/af_xdp.rst in the Linux
+kernel source tree on how to use XDP sockets and for some common
+mistakes in case you do not get any traffic up to user space.
+
+ABI
+==========
+
+libbpf can be both linked statically or used as DSO. To avoid possible
+conflicts with other libraries an application is linked with, all
+non-static libbpf symbols should have one of the prefixes mentioned in
+API documentation above. See API naming convention to choose the right
+name for a new symbol.
+
+Symbol visibility
+-----------------
+
+libbpf follow the model when all global symbols have visibility "hidden"
+by default and to make a symbol visible it has to be explicitly
+attributed with ``LIBBPF_API`` macro. For example:
+
+.. code-block:: c
+
+        LIBBPF_API int bpf_prog_get_fd_by_id(__u32 id);
+
+This prevents from accidentally exporting a symbol, that is not supposed
+to be a part of ABI what, in turn, improves both libbpf developer- and
+user-experiences.
+
+ABI versionning
+---------------
+
+To make future ABI extensions possible libbpf ABI is versioned.
+Versioning is implemented by ``libbpf.map`` version script that is
+passed to linker.
+
+Version name is ``LIBBPF_`` prefix + three-component numeric version,
+starting from ``0.0.1``.
+
+Every time ABI is being changed, e.g. because a new symbol is added or
+semantic of existing symbol is changed, ABI version should be bumped.
+This bump in ABI version is at most once per kernel development cycle.
+
+For example, if current state of ``libbpf.map`` is:
+
+.. code-block:: c
+
+        LIBBPF_0.0.1 {
+        	global:
+                        bpf_func_a;
+                        bpf_func_b;
+        	local:
+        		\*;
+        };
+
+, and a new symbol ``bpf_func_c`` is being introduced, then
+``libbpf.map`` should be changed like this:
+
+.. code-block:: c
+
+        LIBBPF_0.0.1 {
+        	global:
+                        bpf_func_a;
+                        bpf_func_b;
+        	local:
+        		\*;
+        };
+        LIBBPF_0.0.2 {
+                global:
+                        bpf_func_c;
+        } LIBBPF_0.0.1;
+
+, where new version ``LIBBPF_0.0.2`` depends on the previous
+``LIBBPF_0.0.1``.
+
+Format of version script and ways to handle ABI changes, including
+incompatible ones, described in details in [1].
+
+Stand-alone build
+-------------------
+
+Under https://github.com/libbpf/libbpf there is a (semi-)automated
+mirror of the mainline's version of libbpf for a stand-alone build.
+
+However, all changes to libbpf's code base must be upstreamed through
+the mainline kernel tree.
+
+License
+-------------------
+
+libbpf is dual-licensed under LGPL 2.1 and BSD 2-Clause.
+
+Links
+-------------------
+
+[1] https://www.akkadia.org/drepper/dsohowto.pdf
+    (Chapter 3. Maintaining APIs and ABIs).
-- 
2.29.2

Re: [PATCH bpf-next 1/2] Add documentation for libbpf including API autogen

[Matthew Wilcox]

On Mon, May 31, 2021 at 07:55:52PM +0000, grantseltzer wrote:
> +++ b/Documentation/bpf/libbpf.rst
> @@ -0,0 +1,14 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +libbpf
> +======
> +
> +This is documentation for libbpf, a userspace library for loading and
> +interacting with bpf programs.
> +
> +All general BPF questions, including kernel functionality, libbpf APIs and
> +their application, should be sent to bpf@vger.kernel.org mailing list.
> +You can subscribe to it `here <http://vger.kernel.org/vger-lists.html#bpf>`_
> +and search its archive `here <https://lore.kernel.org/bpf/>`_.

https://www.w3.org/QA/Tips/noClickHere

I suggest:

You can `subscribe <http://vger.kernel.org/vger-lists.html#bpf>`_ to the
mailing list or search its `archives <https://lore.kernel.org/bpf/>`_.

Re: [PATCH bpf-next 1/2] Add documentation for libbpf including API autogen

[Andrii Nakryiko]

On Mon, May 31, 2021 at 12:56 PM grantseltzer <grantseltzer@gmail.com> wrote:
>
> This adds rst files containing documentation for libbpf. This includes
> the addition of libbpf_api.rst which pulls comment documentation from
> header files in libbpf under tools/lib/bpf/. The comment docs would be
> of the standard kernel doc format.
>
> Signed-off-by: grantseltzer <grantseltzer@gmail.com>
> ---

Looks good, thanks! See few comments below. Let's figure out what to
do with libbpf docs versioning and land it through bpf-next tree.

>  Documentation/bpf/index.rst                   |  13 ++
>  Documentation/bpf/libbpf.rst                  |  14 ++
>  Documentation/bpf/libbpf_api.rst              |  18 ++
>  Documentation/bpf/libbpf_build.rst            |  37 ++++
>  .../bpf/libbpf_naming_convention.rst          | 170 ++++++++++++++++++
>  5 files changed, 252 insertions(+)
>  create mode 100644 Documentation/bpf/libbpf.rst
>  create mode 100644 Documentation/bpf/libbpf_api.rst
>  create mode 100644 Documentation/bpf/libbpf_build.rst
>  create mode 100644 Documentation/bpf/libbpf_naming_convention.rst
>

[...]

> +API
> +===
> +
> +This documentation is autogenerated from header files in libbpf, tools/lib/bpf
> +
> +.. kernel-doc:: tools/lib/bpf/libbpf.h
> +   :internal:
> +
> +.. kernel-doc:: tools/lib/bpf/bpf.h
> +   :internal:
> +
> +.. kernel-doc:: tools/lib/bpf/btf.h
> +   :internal:
> +
> +.. kernel-doc:: tools/lib/bpf/xsk.h
> +   :internal:

Libbpf API has a BPF side as well (bpf_helpers.h which pulls in
auto-generated bpf_helper_defs.h with all BPF helper definitions,
bpf_tracing.h, bpf_core_read.h, bpf_endian.h), we should probably
expose them as well?

> diff --git a/Documentation/bpf/libbpf_build.rst b/Documentation/bpf/libbpf_build.rst
> new file mode 100644
> index 000000000..b8240eaaa
> --- /dev/null
> +++ b/Documentation/bpf/libbpf_build.rst
> @@ -0,0 +1,37 @@
> +.. SPDX-License-Identifier: GPL-2.0
> +
> +Building libbpf
> +===============
> +
> +libelf is an internal dependency of libbpf and thus it is required to link

zlib is another dependency, can you please mention it as well?

> +against and must be installed on the system for applications to work.
> +pkg-config is used by default to find libelf, and the program called
> +can be overridden with PKG_CONFIG.
> +

[...]

> +API naming convention
> +=====================
> +
> +libbpf API provides access to a few logically separated groups of
> +functions and types. Every group has its own naming convention
> +described here. It's recommended to follow these conventions whenever a
> +new function or type is added to keep libbpf API clean and consistent.
> +
> +All types and functions provided by libbpf API should have one of the
> +following prefixes: ``bpf_``, ``btf_``, ``libbpf_``, ``xsk_``,
> +``perf_buffer_``.

ring_buffer_ and btf_dump_ are two others that we use. But I don't
know how important it is to have an exhaustive list here.

> +
> +System call wrappers
> +--------------------
> +
> +System call wrappers are simple wrappers for commands supported by
> +sys_bpf system call. These wrappers should go to ``bpf.h`` header file
> +and map one-on-one to corresponding commands.

typo: one-to-one?

> +
> +For example ``bpf_map_lookup_elem`` wraps ``BPF_MAP_LOOKUP_ELEM``
> +command of sys_bpf, ``bpf_prog_attach`` wraps ``BPF_PROG_ATTACH``, etc.
> +
> +Objects
> +-------
> +
> +Another class of types and functions provided by libbpf API is "objects"
> +and functions to work with them. Objects are high-level abstractions
> +such as BPF program or BPF map. They're represented by corresponding
> +structures such as ``struct bpf_object``, ``struct bpf_program``,
> +``struct bpf_map``, etc.
> +
> +Structures are forward declared and access to their fields should be
> +provided via corresponding getters and setters rather than directly.
> +
> +These objects are associated with corresponding parts of ELF object that
> +contains compiled BPF programs.
> +
> +For example ``struct bpf_object`` represents ELF object itself created
> +from an ELF file or from a buffer, ``struct bpf_program`` represents a
> +program in ELF object and ``struct bpf_map`` is a map.
> +
> +Functions that work with an object have names built from object name,
> +double underscore and part that describes function purpose.
> +
> +For example ``bpf_object__open`` consists of the name of corresponding
> +object, ``bpf_object``, double underscore and ``open`` that defines the
> +purpose of the function to open ELF file and create ``bpf_object`` from
> +it.
> +
> +Another example: ``bpf_program__load`` is named for corresponding
> +object, ``bpf_program``, that is separated from other part of the name
> +by double underscore.

let's drop this example, bpf_program__load is a bad example and is
going to be deprecated.  We can use btf__parse() as an example here.

> +
> +All objects and corresponding functions other than BTF related should go
> +to ``libbpf.h``. BTF types and functions should go to ``btf.h``.
> +
> +Auxiliary functions
> +-------------------
> +
> +Auxiliary functions and types that don't fit well in any of categories
> +described above should have ``libbpf_`` prefix, e.g.
> +``libbpf_get_error`` or ``libbpf_prog_type_by_name``.
> +
> +AF_XDP functions
> +-------------------
> +
> +AF_XDP functions should have an ``xsk_`` prefix, e.g.
> +``xsk_umem__get_data`` or ``xsk_umem__create``. The interface consists
> +of both low-level ring access functions and high-level configuration
> +functions. These can be mixed and matched. Note that these functions
> +are not reentrant for performance reasons.
> +
> +Please take a look at Documentation/networking/af_xdp.rst in the Linux
> +kernel source tree on how to use XDP sockets and for some common
> +mistakes in case you do not get any traffic up to user space.

I'd probably drop this section, given we move xsk.{c,h} into libxdp.

> +
> +ABI
> +==========
> +
> +libbpf can be both linked statically or used as DSO. To avoid possible
> +conflicts with other libraries an application is linked with, all
> +non-static libbpf symbols should have one of the prefixes mentioned in
> +API documentation above. See API naming convention to choose the right
> +name for a new symbol.
> +

[...]

Re: [PATCH bpf-next 1/2] Add documentation for libbpf including API autogen

[Grant Seltzer Richman]

On Wed, Jun 2, 2021 at 4:36 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 31, 2021 at 12:56 PM grantseltzer <grantseltzer@gmail.com> wrote:
> >
> > This adds rst files containing documentation for libbpf. This includes
> > the addition of libbpf_api.rst which pulls comment documentation from
> > header files in libbpf under tools/lib/bpf/. The comment docs would be
> > of the standard kernel doc format.
> >
> > Signed-off-by: grantseltzer <grantseltzer@gmail.com>
> > ---
>
> Looks good, thanks! See few comments below. Let's figure out what to
> do with libbpf docs versioning and land it through bpf-next tree.
>
> >  Documentation/bpf/index.rst                   |  13 ++
> >  Documentation/bpf/libbpf.rst                  |  14 ++
> >  Documentation/bpf/libbpf_api.rst              |  18 ++
> >  Documentation/bpf/libbpf_build.rst            |  37 ++++
> >  .../bpf/libbpf_naming_convention.rst          | 170 ++++++++++++++++++
> >  5 files changed, 252 insertions(+)
> >  create mode 100644 Documentation/bpf/libbpf.rst
> >  create mode 100644 Documentation/bpf/libbpf_api.rst
> >  create mode 100644 Documentation/bpf/libbpf_build.rst
> >  create mode 100644 Documentation/bpf/libbpf_naming_convention.rst
> >
>
> [...]
>
> > +API
> > +===
> > +
> > +This documentation is autogenerated from header files in libbpf, tools/lib/bpf
> > +
> > +.. kernel-doc:: tools/lib/bpf/libbpf.h
> > +   :internal:
> > +
> > +.. kernel-doc:: tools/lib/bpf/bpf.h
> > +   :internal:
> > +
> > +.. kernel-doc:: tools/lib/bpf/btf.h
> > +   :internal:
> > +
> > +.. kernel-doc:: tools/lib/bpf/xsk.h
> > +   :internal:
>
> Libbpf API has a BPF side as well (bpf_helpers.h which pulls in
> auto-generated bpf_helper_defs.h with all BPF helper definitions,
> bpf_tracing.h, bpf_core_read.h, bpf_endian.h), we should probably
> expose them as well?
>
> > diff --git a/Documentation/bpf/libbpf_build.rst b/Documentation/bpf/libbpf_build.rst
> > new file mode 100644
> > index 000000000..b8240eaaa
> > --- /dev/null
> > +++ b/Documentation/bpf/libbpf_build.rst
> > @@ -0,0 +1,37 @@
> > +.. SPDX-License-Identifier: GPL-2.0
> > +
> > +Building libbpf
> > +===============
> > +
> > +libelf is an internal dependency of libbpf and thus it is required to link
>
> zlib is another dependency, can you please mention it as well?
>
> > +against and must be installed on the system for applications to work.
> > +pkg-config is used by default to find libelf, and the program called
> > +can be overridden with PKG_CONFIG.
> > +
>
> [...]
>
> > +API naming convention
> > +=====================
> > +
> > +libbpf API provides access to a few logically separated groups of
> > +functions and types. Every group has its own naming convention
> > +described here. It's recommended to follow these conventions whenever a
> > +new function or type is added to keep libbpf API clean and consistent.
> > +
> > +All types and functions provided by libbpf API should have one of the
> > +following prefixes: ``bpf_``, ``btf_``, ``libbpf_``, ``xsk_``,
> > +``perf_buffer_``.
>
> ring_buffer_ and btf_dump_ are two others that we use. But I don't
> know how important it is to have an exhaustive list here.
>
> > +
> > +System call wrappers
> > +--------------------
> > +
> > +System call wrappers are simple wrappers for commands supported by
> > +sys_bpf system call. These wrappers should go to ``bpf.h`` header file
> > +and map one-on-one to corresponding commands.
>
> typo: one-to-one?
>
> > +
> > +For example ``bpf_map_lookup_elem`` wraps ``BPF_MAP_LOOKUP_ELEM``
> > +command of sys_bpf, ``bpf_prog_attach`` wraps ``BPF_PROG_ATTACH``, etc.
> > +
> > +Objects
> > +-------
> > +
> > +Another class of types and functions provided by libbpf API is "objects"
> > +and functions to work with them. Objects are high-level abstractions
> > +such as BPF program or BPF map. They're represented by corresponding
> > +structures such as ``struct bpf_object``, ``struct bpf_program``,
> > +``struct bpf_map``, etc.
> > +
> > +Structures are forward declared and access to their fields should be
> > +provided via corresponding getters and setters rather than directly.
> > +
> > +These objects are associated with corresponding parts of ELF object that
> > +contains compiled BPF programs.
> > +
> > +For example ``struct bpf_object`` represents ELF object itself created
> > +from an ELF file or from a buffer, ``struct bpf_program`` represents a
> > +program in ELF object and ``struct bpf_map`` is a map.
> > +
> > +Functions that work with an object have names built from object name,
> > +double underscore and part that describes function purpose.
> > +
> > +For example ``bpf_object__open`` consists of the name of corresponding
> > +object, ``bpf_object``, double underscore and ``open`` that defines the
> > +purpose of the function to open ELF file and create ``bpf_object`` from
> > +it.
> > +
> > +Another example: ``bpf_program__load`` is named for corresponding
> > +object, ``bpf_program``, that is separated from other part of the name
> > +by double underscore.
>
> let's drop this example, bpf_program__load is a bad example and is
> going to be deprecated.  We can use btf__parse() as an example here.
>
> > +
> > +All objects and corresponding functions other than BTF related should go
> > +to ``libbpf.h``. BTF types and functions should go to ``btf.h``.
> > +
> > +Auxiliary functions
> > +-------------------
> > +
> > +Auxiliary functions and types that don't fit well in any of categories
> > +described above should have ``libbpf_`` prefix, e.g.
> > +``libbpf_get_error`` or ``libbpf_prog_type_by_name``.
> > +
> > +AF_XDP functions
> > +-------------------
> > +
> > +AF_XDP functions should have an ``xsk_`` prefix, e.g.
> > +``xsk_umem__get_data`` or ``xsk_umem__create``. The interface consists
> > +of both low-level ring access functions and high-level configuration
> > +functions. These can be mixed and matched. Note that these functions
> > +are not reentrant for performance reasons.
> > +
> > +Please take a look at Documentation/networking/af_xdp.rst in the Linux
> > +kernel source tree on how to use XDP sockets and for some common
> > +mistakes in case you do not get any traffic up to user space.
>
> I'd probably drop this section, given we move xsk.{c,h} into libxdp.
>
> > +
> > +ABI
> > +==========
> > +
> > +libbpf can be both linked statically or used as DSO. To avoid possible
> > +conflicts with other libraries an application is linked with, all
> > +non-static libbpf symbols should have one of the prefixes mentioned in
> > +API documentation above. See API naming convention to choose the right
> > +name for a new symbol.
> > +
>
> [...]

Thanks for the feedback on this, I've fixed them on a local copy and
will incorporate them into my final patch/PR depending on what we go
with. See my most recent response on '[PATCH bpf-next 0/3]
Autogenerating API documentation'

[PATCH bpf-next 2/2] Remove duplicate README doc from libbpf

[grantseltzer]

This removes the README.rst file from libbpf code which is moved into
Documentation/bpf/libbpf in the previous commit

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 tools/lib/bpf/README.rst | 168 ---------------------------------------
 1 file changed, 168 deletions(-)
 delete mode 100644 tools/lib/bpf/README.rst

diff --git a/tools/lib/bpf/README.rst b/tools/lib/bpf/README.rst
deleted file mode 100644
index 8928f7787..000000000
--- a/tools/lib/bpf/README.rst
+++ /dev/null
@@ -1,168 +0,0 @@
-.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
-
-libbpf API naming convention
-============================
-
-libbpf API provides access to a few logically separated groups of
-functions and types. Every group has its own naming convention
-described here. It's recommended to follow these conventions whenever a
-new function or type is added to keep libbpf API clean and consistent.
-
-All types and functions provided by libbpf API should have one of the
-following prefixes: ``bpf_``, ``btf_``, ``libbpf_``, ``xsk_``,
-``perf_buffer_``.
-
-System call wrappers
---------------------
-
-System call wrappers are simple wrappers for commands supported by
-sys_bpf system call. These wrappers should go to ``bpf.h`` header file
-and map one-on-one to corresponding commands.
-
-For example ``bpf_map_lookup_elem`` wraps ``BPF_MAP_LOOKUP_ELEM``
-command of sys_bpf, ``bpf_prog_attach`` wraps ``BPF_PROG_ATTACH``, etc.
-
-Objects
--------
-
-Another class of types and functions provided by libbpf API is "objects"
-and functions to work with them. Objects are high-level abstractions
-such as BPF program or BPF map. They're represented by corresponding
-structures such as ``struct bpf_object``, ``struct bpf_program``,
-``struct bpf_map``, etc.
-
-Structures are forward declared and access to their fields should be
-provided via corresponding getters and setters rather than directly.
-
-These objects are associated with corresponding parts of ELF object that
-contains compiled BPF programs.
-
-For example ``struct bpf_object`` represents ELF object itself created
-from an ELF file or from a buffer, ``struct bpf_program`` represents a
-program in ELF object and ``struct bpf_map`` is a map.
-
-Functions that work with an object have names built from object name,
-double underscore and part that describes function purpose.
-
-For example ``bpf_object__open`` consists of the name of corresponding
-object, ``bpf_object``, double underscore and ``open`` that defines the
-purpose of the function to open ELF file and create ``bpf_object`` from
-it.
-
-Another example: ``bpf_program__load`` is named for corresponding
-object, ``bpf_program``, that is separated from other part of the name
-by double underscore.
-
-All objects and corresponding functions other than BTF related should go
-to ``libbpf.h``. BTF types and functions should go to ``btf.h``.
-
-Auxiliary functions
--------------------
-
-Auxiliary functions and types that don't fit well in any of categories
-described above should have ``libbpf_`` prefix, e.g.
-``libbpf_get_error`` or ``libbpf_prog_type_by_name``.
-
-AF_XDP functions
--------------------
-
-AF_XDP functions should have an ``xsk_`` prefix, e.g.
-``xsk_umem__get_data`` or ``xsk_umem__create``. The interface consists
-of both low-level ring access functions and high-level configuration
-functions. These can be mixed and matched. Note that these functions
-are not reentrant for performance reasons.
-
-Please take a look at Documentation/networking/af_xdp.rst in the Linux
-kernel source tree on how to use XDP sockets and for some common
-mistakes in case you do not get any traffic up to user space.
-
-libbpf ABI
-==========
-
-libbpf can be both linked statically or used as DSO. To avoid possible
-conflicts with other libraries an application is linked with, all
-non-static libbpf symbols should have one of the prefixes mentioned in
-API documentation above. See API naming convention to choose the right
-name for a new symbol.
-
-Symbol visibility
------------------
-
-libbpf follow the model when all global symbols have visibility "hidden"
-by default and to make a symbol visible it has to be explicitly
-attributed with ``LIBBPF_API`` macro. For example:
-
-.. code-block:: c
-
-        LIBBPF_API int bpf_prog_get_fd_by_id(__u32 id);
-
-This prevents from accidentally exporting a symbol, that is not supposed
-to be a part of ABI what, in turn, improves both libbpf developer- and
-user-experiences.
-
-ABI versionning
----------------
-
-To make future ABI extensions possible libbpf ABI is versioned.
-Versioning is implemented by ``libbpf.map`` version script that is
-passed to linker.
-
-Version name is ``LIBBPF_`` prefix + three-component numeric version,
-starting from ``0.0.1``.
-
-Every time ABI is being changed, e.g. because a new symbol is added or
-semantic of existing symbol is changed, ABI version should be bumped.
-This bump in ABI version is at most once per kernel development cycle.
-
-For example, if current state of ``libbpf.map`` is:
-
-.. code-block::
-        LIBBPF_0.0.1 {
-        	global:
-                        bpf_func_a;
-                        bpf_func_b;
-        	local:
-        		\*;
-        };
-
-, and a new symbol ``bpf_func_c`` is being introduced, then
-``libbpf.map`` should be changed like this:
-
-.. code-block::
-        LIBBPF_0.0.1 {
-        	global:
-                        bpf_func_a;
-                        bpf_func_b;
-        	local:
-        		\*;
-        };
-        LIBBPF_0.0.2 {
-                global:
-                        bpf_func_c;
-        } LIBBPF_0.0.1;
-
-, where new version ``LIBBPF_0.0.2`` depends on the previous
-``LIBBPF_0.0.1``.
-
-Format of version script and ways to handle ABI changes, including
-incompatible ones, described in details in [1].
-
-Stand-alone build
-=================
-
-Under https://github.com/libbpf/libbpf there is a (semi-)automated
-mirror of the mainline's version of libbpf for a stand-alone build.
-
-However, all changes to libbpf's code base must be upstreamed through
-the mainline kernel tree.
-
-License
-=======
-
-libbpf is dual-licensed under LGPL 2.1 and BSD 2-Clause.
-
-Links
-=====
-
-[1] https://www.akkadia.org/drepper/dsohowto.pdf
-    (Chapter 3. Maintaining APIs and ABIs).
-- 
2.29.2

Re: [PATCH bpf-next 2/2] Remove duplicate README doc from libbpf

[Andrii Nakryiko]

On Mon, May 31, 2021 at 12:56 PM grantseltzer <grantseltzer@gmail.com> wrote:
>
> This removes the README.rst file from libbpf code which is moved into
> Documentation/bpf/libbpf in the previous commit
>
> Signed-off-by: grantseltzer <grantseltzer@gmail.com>
> ---

Let's keep this removal together with the previous patch, that way git
should be able to detect the file move and minimize the diff.

>  tools/lib/bpf/README.rst | 168 ---------------------------------------
>  1 file changed, 168 deletions(-)
>  delete mode 100644 tools/lib/bpf/README.rst
>

[...]

Re: [PATCH bpf-next 0/2] Autogenerating libbpf API documentation

[Jonathan Corbet]

grantseltzer <grantseltzer@gmail.com> writes:

> This patch series is meant to start the initiative to document libbpf.
> It includes .rst files which are text documentation describing building, 
> API naming convention, as well as an index to generated API documentation.
>
> In this approach the generated API documentation is enabled by the kernels
> existing kernel documentation system which uses sphinx. The resulting docs
> would then be synced to kernel.org/doc
>
> You can test this by running `make htmldocs` and serving the html in 
> Documentation/output. Since libbpf does not yet have comments in kernel
> doc format, see kernel.org/doc/html/latest/doc-guide/kernel-doc.html for
> an example so you can test this.
>
> The advantage of this approach is to use the existing sphinx
> infrastructure that the kernel has, and have libbpf docs in
> the same place as everything else.
>
> The perhaps large disadvantage of this approach is that libbpf versions
> independently from the kernel. If it's possible to version libbpf
> separately without having duplicates that would be the ideal scenario.

I'm happy to see things going this direction; it looks like a good start
to me.

Let me know if you'd like this to go through the docs tree, or feel free
to add:

  Acked-by: Jonathan Corbet <corbet@lwn.net>

if you want to route it via some other path.

Thanks,

jon

Re: [PATCH bpf-next 0/2] Autogenerating libbpf API documentation

[Andrii Nakryiko]

On Tue, Jun 1, 2021 at 11:58 AM Jonathan Corbet <corbet@lwn.net> wrote:
>
> grantseltzer <grantseltzer@gmail.com> writes:
>
> > This patch series is meant to start the initiative to document libbpf.
> > It includes .rst files which are text documentation describing building,
> > API naming convention, as well as an index to generated API documentation.
> >
> > In this approach the generated API documentation is enabled by the kernels
> > existing kernel documentation system which uses sphinx. The resulting docs
> > would then be synced to kernel.org/doc
> >
> > You can test this by running `make htmldocs` and serving the html in
> > Documentation/output. Since libbpf does not yet have comments in kernel
> > doc format, see kernel.org/doc/html/latest/doc-guide/kernel-doc.html for
> > an example so you can test this.
> >
> > The advantage of this approach is to use the existing sphinx
> > infrastructure that the kernel has, and have libbpf docs in
> > the same place as everything else.
> >
> > The perhaps large disadvantage of this approach is that libbpf versions
> > independently from the kernel. If it's possible to version libbpf
> > separately without having duplicates that would be the ideal scenario.
>
> I'm happy to see things going this direction; it looks like a good start
> to me.
>
> Let me know if you'd like this to go through the docs tree, or feel free
> to add:
>
>   Acked-by: Jonathan Corbet <corbet@lwn.net>

Thanks, Jonathan! I prefer to take this through bpf-next, which will
make it easier to keep it in sync on Github (if we do that, of
course).

>
> if you want to route it via some other path.
>
> Thanks,
>
> jon