[PATCH bpf-next v3 0/1] Autogenerating libbpf API documentation

[PATCH bpf-next v3 0/1] Autogenerating libbpf API documentation

[grantseltzer]

This patch series is meant to start the initiative to document libbpf.
It includes .rst files which are text documentation describing building,
API naming convention, as well as an index to generated API documentation.

In this approach the generated API documentation is enabled by the kernels
existing kernel documentation system which uses sphinx. The resulting docs
would then be synced to kernel.org/doc

You can test this by running `make htmldocs` and serving the html in
Documentation/output. Since libbpf does not yet have comments in kernel
doc format, see kernel.org/doc/html/latest/doc-guide/kernel-doc.html for
an example so you can test this.

The advantage of this approach is to use the existing sphinx
infrastructure that the kernel has, and have libbpf docs in
the same place as everything else.

The current plan is to have the libbpf mirror sync the generated docs 
and version them based on the libbpf releases which are cut on github.

grantseltzer (1):
  Add documentation for libbpf including API autogen

 Documentation/bpf/index.rst                   | 13 +++++++
 Documentation/bpf/libbpf.rst                  | 14 +++++++
 Documentation/bpf/libbpf_api.rst              | 27 ++++++++++++++
 Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++
 .../bpf/libbpf_naming_convention.rst          | 32 +++++++---------
 5 files changed, 104 insertions(+), 19 deletions(-)
 create mode 100644 Documentation/bpf/libbpf.rst
 create mode 100644 Documentation/bpf/libbpf_api.rst
 create mode 100644 Documentation/bpf/libbpf_build.rst
 rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (89%)

-- 
2.31.1

[PATCH bpf-next v3 1/1] Add documentation for libbpf including API autogen

[grantseltzer]

This adds rst files containing documentation for libbpf. This includes
the addition of libbpf_api.rst which pulls comment documentation from
header files in libbpf under tools/lib/bpf/. The comment docs would be
of the standard kernel doc format.

Signed-off-by: grantseltzer <grantseltzer@gmail.com>
---
 Documentation/bpf/index.rst                   | 13 +++++++
 Documentation/bpf/libbpf.rst                  | 14 +++++++
 Documentation/bpf/libbpf_api.rst              | 27 ++++++++++++++
 Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++
 .../bpf/libbpf_naming_convention.rst          | 32 +++++++---------
 5 files changed, 104 insertions(+), 19 deletions(-)
 create mode 100644 Documentation/bpf/libbpf.rst
 create mode 100644 Documentation/bpf/libbpf_api.rst
 create mode 100644 Documentation/bpf/libbpf_build.rst
 rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (89%)

diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
index a702f67dd..44f646735 100644
--- a/Documentation/bpf/index.rst
+++ b/Documentation/bpf/index.rst
@@ -12,6 +12,19 @@ BPF instruction-set.
 The Cilium project also maintains a `BPF and XDP Reference Guide`_
 that goes into great technical depth about the BPF Architecture.
 
+libbpf
+======
+
+Libbpf is a userspace library for loading and interacting with bpf programs.
+
+.. toctree::
+   :maxdepth: 1
+
+   libbpf
+   libbpf_api
+   libbpf_build
+   libbpf_naming_convention
+
 BPF Type Format (BTF)
 =====================
 
diff --git a/Documentation/bpf/libbpf.rst b/Documentation/bpf/libbpf.rst
new file mode 100644
index 000000000..2e62cadee
--- /dev/null
+++ b/Documentation/bpf/libbpf.rst
@@ -0,0 +1,14 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+libbpf
+======
+
+This is documentation for libbpf, a userspace library for loading and
+interacting with bpf programs.
+
+All general BPF questions, including kernel functionality, libbpf APIs and
+their application, should be sent to bpf@vger.kernel.org mailing list.
+You can `subscribe <http://vger.kernel.org/vger-lists.html#bpf>`_ to the
+mailing list search its `archive <https://lore.kernel.org/bpf/>`_.
+Please search the archive before asking new questions. It very well might
+be that this was already addressed or answered before.
diff --git a/Documentation/bpf/libbpf_api.rst b/Documentation/bpf/libbpf_api.rst
new file mode 100644
index 000000000..b54bd7470
--- /dev/null
+++ b/Documentation/bpf/libbpf_api.rst
@@ -0,0 +1,27 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+API
+===
+
+This documentation is autogenerated from header files in libbpf, tools/lib/bpf
+
+.. kernel-doc:: tools/lib/bpf/libbpf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/bpf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/btf.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/xsk.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/bpf_tracing.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/bpf_core_read.h
+   :internal:
+
+.. kernel-doc:: tools/lib/bpf/bpf_endian.h
+   :internal:
\ No newline at end of file
diff --git a/Documentation/bpf/libbpf_build.rst b/Documentation/bpf/libbpf_build.rst
new file mode 100644
index 000000000..584be90f1
--- /dev/null
+++ b/Documentation/bpf/libbpf_build.rst
@@ -0,0 +1,37 @@
+.. SPDX-License-Identifier: GPL-2.0
+
+Building libbpf
+===============
+
+libelf and zlib are internal dependencies of libbpf and thus are required to link
+against and must be installed on the system for applications to work.
+pkg-config is used by default to find libelf, and the program called
+can be overridden with PKG_CONFIG.
+
+If using pkg-config at build time is not desired, it can be disabled by
+setting NO_PKG_CONFIG=1 when calling make.
+
+To build both static libbpf.a and shared libbpf.so:
+
+.. code-block:: bash
+
+    $ cd src
+    $ make
+
+To build only static libbpf.a library in directory build/ and install them
+together with libbpf headers in a staging directory root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ mkdir build root
+    $ BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=root make install
+
+To build both static libbpf.a and shared libbpf.so against a custom libelf
+dependency installed in /build/root/ and install them together with libbpf
+headers in a build directory /build/root/:
+
+.. code-block:: bash
+
+    $ cd src
+    $ PKG_CONFIG_PATH=/build/root/lib64/pkgconfig DESTDIR=/build/root make
\ No newline at end of file
diff --git a/tools/lib/bpf/README.rst b/Documentation/bpf/libbpf_naming_convention.rst
similarity index 89%
rename from tools/lib/bpf/README.rst
rename to Documentation/bpf/libbpf_naming_convention.rst
index 8928f7787..b6dc5c592 100644
--- a/tools/lib/bpf/README.rst
+++ b/Documentation/bpf/libbpf_naming_convention.rst
@@ -1,7 +1,7 @@
-.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+.. SPDX-License-Identifier: GPL-2.0
 
-libbpf API naming convention
-============================
+API naming convention
+=====================
 
 libbpf API provides access to a few logically separated groups of
 functions and types. Every group has its own naming convention
@@ -10,14 +10,14 @@ new function or type is added to keep libbpf API clean and consistent.
 
 All types and functions provided by libbpf API should have one of the
 following prefixes: ``bpf_``, ``btf_``, ``libbpf_``, ``xsk_``,
-``perf_buffer_``.
+``btf_dump_``, ``ring_buffer_``, ``perf_buffer_``.
 
 System call wrappers
 --------------------
 
 System call wrappers are simple wrappers for commands supported by
 sys_bpf system call. These wrappers should go to ``bpf.h`` header file
-and map one-on-one to corresponding commands.
+and map one to one to corresponding commands.
 
 For example ``bpf_map_lookup_elem`` wraps ``BPF_MAP_LOOKUP_ELEM``
 command of sys_bpf, ``bpf_prog_attach`` wraps ``BPF_PROG_ATTACH``, etc.
@@ -49,10 +49,6 @@ object, ``bpf_object``, double underscore and ``open`` that defines the
 purpose of the function to open ELF file and create ``bpf_object`` from
 it.
 
-Another example: ``bpf_program__load`` is named for corresponding
-object, ``bpf_program``, that is separated from other part of the name
-by double underscore.
-
 All objects and corresponding functions other than BTF related should go
 to ``libbpf.h``. BTF types and functions should go to ``btf.h``.
 
@@ -72,11 +68,7 @@ of both low-level ring access functions and high-level configuration
 functions. These can be mixed and matched. Note that these functions
 are not reentrant for performance reasons.
 
-Please take a look at Documentation/networking/af_xdp.rst in the Linux
-kernel source tree on how to use XDP sockets and for some common
-mistakes in case you do not get any traffic up to user space.
-
-libbpf ABI
+ABI
 ==========
 
 libbpf can be both linked statically or used as DSO. To avoid possible
@@ -116,7 +108,8 @@ This bump in ABI version is at most once per kernel development cycle.
 
 For example, if current state of ``libbpf.map`` is:
 
-.. code-block::
+.. code-block:: c
+
         LIBBPF_0.0.1 {
         	global:
                         bpf_func_a;
@@ -128,7 +121,8 @@ For example, if current state of ``libbpf.map`` is:
 , and a new symbol ``bpf_func_c`` is being introduced, then
 ``libbpf.map`` should be changed like this:
 
-.. code-block::
+.. code-block:: c
+
         LIBBPF_0.0.1 {
         	global:
                         bpf_func_a;
@@ -148,7 +142,7 @@ Format of version script and ways to handle ABI changes, including
 incompatible ones, described in details in [1].
 
 Stand-alone build
-=================
+-------------------
 
 Under https://github.com/libbpf/libbpf there is a (semi-)automated
 mirror of the mainline's version of libbpf for a stand-alone build.
@@ -157,12 +151,12 @@ However, all changes to libbpf's code base must be upstreamed through
 the mainline kernel tree.
 
 License
-=======
+-------------------
 
 libbpf is dual-licensed under LGPL 2.1 and BSD 2-Clause.
 
 Links
-=====
+-------------------
 
 [1] https://www.akkadia.org/drepper/dsohowto.pdf
     (Chapter 3. Maintaining APIs and ABIs).
-- 
2.31.1

Re: [PATCH bpf-next v3 1/1] Add documentation for libbpf including API autogen

[Andrii Nakryiko]

On Thu, Jun 17, 2021 at 11:20 AM grantseltzer <grantseltzer@gmail.com> wrote:
>
> This adds rst files containing documentation for libbpf. This includes
> the addition of libbpf_api.rst which pulls comment documentation from
> header files in libbpf under tools/lib/bpf/. The comment docs would be
> of the standard kernel doc format.
>
> Signed-off-by: grantseltzer <grantseltzer@gmail.com>
> ---
>  Documentation/bpf/index.rst                   | 13 +++++++
>  Documentation/bpf/libbpf.rst                  | 14 +++++++
>  Documentation/bpf/libbpf_api.rst              | 27 ++++++++++++++
>  Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++

Didn't we agree to have docs under Documentation/bpf/libbpf? That
should make it clear that each is libbpf-specific and probably would
make copying/syncing easier. Plus it will be a libbpf sub-section in
the docs, no?

>  .../bpf/libbpf_naming_convention.rst          | 32 +++++++---------
>  5 files changed, 104 insertions(+), 19 deletions(-)
>  create mode 100644 Documentation/bpf/libbpf.rst
>  create mode 100644 Documentation/bpf/libbpf_api.rst
>  create mode 100644 Documentation/bpf/libbpf_build.rst
>  rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (89%)
>
> diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> index a702f67dd..44f646735 100644
> --- a/Documentation/bpf/index.rst
> +++ b/Documentation/bpf/index.rst
> @@ -12,6 +12,19 @@ BPF instruction-set.
>  The Cilium project also maintains a `BPF and XDP Reference Guide`_
>  that goes into great technical depth about the BPF Architecture.
>
> +libbpf
> +======
> +
> +Libbpf is a userspace library for loading and interacting with bpf programs.
> +
> +.. toctree::
> +   :maxdepth: 1
> +
> +   libbpf
> +   libbpf_api
> +   libbpf_build
> +   libbpf_naming_convention
> +
>  BPF Type Format (BTF)
>  =====================
>
> diff --git a/Documentation/bpf/libbpf.rst b/Documentation/bpf/libbpf.rst
> new file mode 100644
> index 000000000..2e62cadee
> --- /dev/null
> +++ b/Documentation/bpf/libbpf.rst
> @@ -0,0 +1,14 @@
> +.. SPDX-License-Identifier: GPL-2.0

Should we use dual-license LGPL-2.1 OR BSD-2-Clause like the rest of libbpf?

> +
> +libbpf
> +======
> +
> +This is documentation for libbpf, a userspace library for loading and
> +interacting with bpf programs.
> +

[...]

> +    $ cd src
> +    $ PKG_CONFIG_PATH=/build/root/lib64/pkgconfig DESTDIR=/build/root make
> \ No newline at end of file
> diff --git a/tools/lib/bpf/README.rst b/Documentation/bpf/libbpf_naming_convention.rst
> similarity index 89%
> rename from tools/lib/bpf/README.rst
> rename to Documentation/bpf/libbpf_naming_convention.rst
> index 8928f7787..b6dc5c592 100644
> --- a/tools/lib/bpf/README.rst
> +++ b/Documentation/bpf/libbpf_naming_convention.rst
> @@ -1,7 +1,7 @@
> -.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> +.. SPDX-License-Identifier: GPL-2.0

I don't think we can just easily re-license without asking original
contributor. But see above, I think we should stick to the
dual-license to stay consistent with libbpf sources?


[...]

Re: [PATCH bpf-next v3 0/1] Autogenerating libbpf API documentation

[Andrii Nakryiko]

On Thu, Jun 17, 2021 at 11:20 AM grantseltzer <grantseltzer@gmail.com> wrote:
>
> This patch series is meant to start the initiative to document libbpf.
> It includes .rst files which are text documentation describing building,
> API naming convention, as well as an index to generated API documentation.
>
> In this approach the generated API documentation is enabled by the kernels
> existing kernel documentation system which uses sphinx. The resulting docs
> would then be synced to kernel.org/doc
>
> You can test this by running `make htmldocs` and serving the html in
> Documentation/output. Since libbpf does not yet have comments in kernel
> doc format, see kernel.org/doc/html/latest/doc-guide/kernel-doc.html for
> an example so you can test this.
>
> The advantage of this approach is to use the existing sphinx
> infrastructure that the kernel has, and have libbpf docs in
> the same place as everything else.
>
> The current plan is to have the libbpf mirror sync the generated docs
> and version them based on the libbpf releases which are cut on github.
>
> grantseltzer (1):
>   Add documentation for libbpf including API autogen
>

You don't need the cover letter if you are submitting a single patch.
Just put all of the context in cover letter into the patch
description.

>  Documentation/bpf/index.rst                   | 13 +++++++
>  Documentation/bpf/libbpf.rst                  | 14 +++++++
>  Documentation/bpf/libbpf_api.rst              | 27 ++++++++++++++
>  Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++
>  .../bpf/libbpf_naming_convention.rst          | 32 +++++++---------
>  5 files changed, 104 insertions(+), 19 deletions(-)
>  create mode 100644 Documentation/bpf/libbpf.rst
>  create mode 100644 Documentation/bpf/libbpf_api.rst
>  create mode 100644 Documentation/bpf/libbpf_build.rst
>  rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (89%)
>
> --
> 2.31.1

Re: [PATCH bpf-next v3 1/1] Add documentation for libbpf including API autogen

[Grant Seltzer Richman]

On Thu, Jun 17, 2021 at 4:36 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Thu, Jun 17, 2021 at 11:20 AM grantseltzer <grantseltzer@gmail.com> wrote:
> >
> > This adds rst files containing documentation for libbpf. This includes
> > the addition of libbpf_api.rst which pulls comment documentation from
> > header files in libbpf under tools/lib/bpf/. The comment docs would be
> > of the standard kernel doc format.
> >
> > Signed-off-by: grantseltzer <grantseltzer@gmail.com>
> > ---
> >  Documentation/bpf/index.rst                   | 13 +++++++
> >  Documentation/bpf/libbpf.rst                  | 14 +++++++
> >  Documentation/bpf/libbpf_api.rst              | 27 ++++++++++++++
> >  Documentation/bpf/libbpf_build.rst            | 37 +++++++++++++++++++
>
> Didn't we agree to have docs under Documentation/bpf/libbpf? That
> should make it clear that each is libbpf-specific and probably would
> make copying/syncing easier. Plus it will be a libbpf sub-section in
> the docs, no?

Ah sure, that works.

>
> >  .../bpf/libbpf_naming_convention.rst          | 32 +++++++---------
> >  5 files changed, 104 insertions(+), 19 deletions(-)
> >  create mode 100644 Documentation/bpf/libbpf.rst
> >  create mode 100644 Documentation/bpf/libbpf_api.rst
> >  create mode 100644 Documentation/bpf/libbpf_build.rst
> >  rename tools/lib/bpf/README.rst => Documentation/bpf/libbpf_naming_convention.rst (89%)
> >
> > diff --git a/Documentation/bpf/index.rst b/Documentation/bpf/index.rst
> > index a702f67dd..44f646735 100644
> > --- a/Documentation/bpf/index.rst
> > +++ b/Documentation/bpf/index.rst
> > @@ -12,6 +12,19 @@ BPF instruction-set.
> >  The Cilium project also maintains a `BPF and XDP Reference Guide`_
> >  that goes into great technical depth about the BPF Architecture.
> >
> > +libbpf
> > +======
> > +
> > +Libbpf is a userspace library for loading and interacting with bpf programs.
> > +
> > +.. toctree::
> > +   :maxdepth: 1
> > +
> > +   libbpf
> > +   libbpf_api
> > +   libbpf_build
> > +   libbpf_naming_convention
> > +
> >  BPF Type Format (BTF)
> >  =====================
> >
> > diff --git a/Documentation/bpf/libbpf.rst b/Documentation/bpf/libbpf.rst
> > new file mode 100644
> > index 000000000..2e62cadee
> > --- /dev/null
> > +++ b/Documentation/bpf/libbpf.rst
> > @@ -0,0 +1,14 @@
> > +.. SPDX-License-Identifier: GPL-2.0
>
> Should we use dual-license LGPL-2.1 OR BSD-2-Clause like the rest of libbpf?
>
> > +
> > +libbpf
> > +======
> > +
> > +This is documentation for libbpf, a userspace library for loading and
> > +interacting with bpf programs.
> > +
>
> [...]
>
> > +    $ cd src
> > +    $ PKG_CONFIG_PATH=/build/root/lib64/pkgconfig DESTDIR=/build/root make
> > \ No newline at end of file
> > diff --git a/tools/lib/bpf/README.rst b/Documentation/bpf/libbpf_naming_convention.rst
> > similarity index 89%
> > rename from tools/lib/bpf/README.rst
> > rename to Documentation/bpf/libbpf_naming_convention.rst
> > index 8928f7787..b6dc5c592 100644
> > --- a/tools/lib/bpf/README.rst
> > +++ b/Documentation/bpf/libbpf_naming_convention.rst
> > @@ -1,7 +1,7 @@
> > -.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> > +.. SPDX-License-Identifier: GPL-2.0
>
> I don't think we can just easily re-license without asking original
> contributor. But see above, I think we should stick to the
> dual-license to stay consistent with libbpf sources?

This change was not at all intentional. I'll change it back to the
dual license.
>
>
> [...]