Re: [PATCH bpf-next 0/3] Autogenerating API documentation

[Grant Seltzer Richman <grantseltzer@gmail.com>]

On Mon, May 10, 2021 at 1:48 PM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Mon, May 10, 2021 at 7:59 AM Grant Seltzer Richman
> <grantseltzer@gmail.com> wrote:
> >
> > On Fri, Apr 30, 2021 at 1:31 PM Andrii Nakryiko
> > <andrii.nakryiko@gmail.com> wrote:
> > >
> > > On Fri, Apr 30, 2021 at 7:27 AM Grant Seltzer Richman
> > > <grantseltzer@gmail.com> wrote:
> > > >
> > > > On Fri, Apr 30, 2021 at 10:22 AM Jonathan Corbet <corbet@lwn.net> wrote:
> > > > >
> > > > > Grant Seltzer Richman <grantseltzer@gmail.com> writes:
> > > > >
> > > > > > Hm, yes I do agree that it'd be nice to use existing tooling but I
> > > > > > just have a couple concerns for this but please point me in the right
> > > > > > direction because i'm sure i'm missing something. I was told to ask on
> > > > > > the linux-doc mailing list because you'd have valuable input anway.
> > > > > > This is based on reading
> > > > > > https://www.kernel.org/doc/html/v4.9/kernel-documentation.html#including-kernel-doc-comments
> > > > > >
> > > > > > 1. We'd want the ability to pull documentation from the code itself to
> > > > > > make it so documentation never falls out of date with code. Based on
> > > > > > the docs on kernel.org/doc it seems that we'd have to be explicit with
> > > > > > specifying which functions/types are included in an .rst file and
> > > > > > submit a patch to update the documentation everytime the libbpf api
> > > > > > changes. Perhaps if this isn't a thing already I can figure out how to
> > > > > > contribute it.
> > > > >
> > > > > No, you can tell it to pull out docs for all of the functions in a given
> > > > > file.  You only need to name things if you want to narrow things down.
> > > >
> > > > Alright, I will figure out how to do this and adjust the patch
> > > > accordingly. My biggest overall goal is making it as easy as possible
> > > > to contribute documentation. I think even adding just one doc string
> > > > above an API function is a great opportunity for new contributors to
> > > > familiarize themselves with the mailing list/patch process.
> > > >
> > > > >
> > > > > > 2. Would it be possible (or necessary) to separate libbpf
> > > > > > documentation from the kernel readthedocs page since libbpf isn't part
> > > > > > of the kernel?
> > > > >
> > > > > It could certainly be built as a separate "book", as are many of the
> > > > > kernel books now.  I could see it as something that gets pulled into the
> > > > > user-space API book, but there could also perhaps be an argument made
> > > > > for creating a new "libraries" book instead.
> > > >
> > > > Yea if I can figure this out for the libbpf API it'd be great to
> > > > replicate it for any API!
> > >
> > > It would be great if it was possible to have this libbpf
> > > auto-generated documentation as part of the kernel documentation, but
> > > also be able to generate and export it into our Github mirror to be
> > > pulled by readthedocs.io. If that can be done, it would be the best of
> > > both kernel and external worlds. We have a sync script that already
> > > auto-generates and checks in BPF helpers header, so we have a
> > > precedent of checking in auto-generated stuff into Github. So it's
> > > mostly about figuring out the mechanics of doc generation.
> >
> > Agreed, the mirror will have to bring in the documentation
> > subdirectory as well so the output could be seperate.
> >
> > Just want to update in this thread that i've been really preoccupied
> > with other obligations and will get back to this next week.
>
> No worries. Thanks for the update!

Finally catching up on this, thanks for all of your patience!

I've discovered that it's actually very easy, even trivial, to add API
documentation for libbpf using the existing kernel sphinx
documentation system. Adding a couple files with directives under
`Documentaiton/bpf` is enough to pull in any comment-documented
functions/structs in libbpf code. I'm not sure who owns the CI/CD
infrastructure that recompiles the documentation and hosts on
kernel.org/doc but I've been building them locally with `make
htmldocs` with no problem. That would require a single patch and we
can start adding comment documentation to libbpf.  I can submit a
patch for that if you'd like to test it yourself. In this system the
html output is not checked into git though.

Andrii - what do you think of having libbpf API documentation hosted
on the kernel.org readthedocs? It would be nice to have it seperate
from the rest of kernel documentation for simplicity, though it is
nice to use/contribute-to existing infrastructure. If you'd like to
have it seperate we can have the libbpf mirror run `make htmldocs`,
take the output, and host our own readthedocs site.

I'd love to have this all set up and have the full API documented by
the time you cut the libbpf 1.0 release!