[PATCH bpf-next v2 0/2] Add table of BPF program types to docs

[PATCH bpf-next v2 0/2] Add table of BPF program types to docs

[Donald Hunter]

Extend the libbpf documentation with a table of program types, attach
points and ELF section names. The table uses data from program_types.csv
which is generated from tools/lib/bpf/libbpf.c during the documentation
build.

Patch 1 adds subdir support to Documentation/Makefile and changes
userspace-api/media to use this instead of being a special case.

Patch 2 adds the the program_types documentation with a new makefile in
the libbpf doc directory to generate program_types.csv

I plan to look at adding info about format of section "extras" as a
follow-on.

v1 -> v2:
Automate the generation of program_types.csv as suggested by
Andrii Nakryiko.


Donald Hunter (2):
  Add subdir support to Documentation makefile
  Add table of BPF program types to libbpf docs

 Documentation/Makefile                     | 16 ++++++++--
 Documentation/bpf/libbpf/Makefile          | 36 ++++++++++++++++++++++
 Documentation/bpf/libbpf/index.rst         |  3 ++
 Documentation/bpf/libbpf/program_types.rst | 18 +++++++++++
 Documentation/bpf/programs.rst             |  3 ++
 Documentation/userspace-api/media/Makefile |  2 ++
 6 files changed, 76 insertions(+), 2 deletions(-)
 create mode 100644 Documentation/bpf/libbpf/Makefile
 create mode 100644 Documentation/bpf/libbpf/program_types.rst

-- 
2.35.1

[PATCH bpf-next v2 1/2] Add subdir support to Documentation makefile

[Donald Hunter]

Run make in list of subdirs to build generated sources and migrate
userspace-api/media to use this instead of being a special case.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/Makefile                     | 15 +++++++++++++--
 Documentation/userspace-api/media/Makefile |  2 ++
 2 files changed, 15 insertions(+), 2 deletions(-)

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 64d44c1ecad3..8a63ef2dcd1c 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -65,6 +65,12 @@ I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 # commands; the 'cmd' from scripts/Kbuild.include is not *loopable*
 loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
 
+BUILD_SUBDIRS = \
+	Documentation/userspace-api/media
+
+quiet_cmd_build_subdir = SUBDIR  $2
+      cmd_build_subdir = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=$2 $3
+
 # $2 sphinx builder e.g. "html"
 # $3 name of the build subfolder / e.g. "userspace-api/media", used as:
 #    * dest folder relative to $(BUILDDIR) and
@@ -74,7 +80,7 @@ loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
 #    e.g. "userspace-api/media" for the linux-tv book-set at ./Documentation/userspace-api/media
 
 quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
-      cmd_sphinx = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media $2 && \
+      cmd_sphinx =  \
 	PYTHONDONTWRITEBYTECODE=1 \
 	BUILDDIR=$(abspath $(BUILDDIR)) SPHINX_CONF=$(abspath $(srctree)/$(src)/$5/$(SPHINX_CONF)) \
 	$(PYTHON3) $(srctree)/scripts/jobserver-exec \
@@ -93,6 +99,7 @@ quiet_cmd_sphinx = SPHINX  $@ --> file://$(abspath $(BUILDDIR)/$3/$4)
 
 htmldocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),html))
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,html,$(var),,$(var)))
 
 linkcheckdocs:
@@ -100,6 +107,7 @@ linkcheckdocs:
 
 latexdocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),latex))
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,latex,$(var),latex,$(var)))
 
 ifeq ($(HAVE_PDFLATEX),0)
@@ -112,6 +120,7 @@ else # HAVE_PDFLATEX
 
 pdfdocs: latexdocs
 	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),latex))
 	$(foreach var,$(SPHINXDIRS), \
 	   $(MAKE) PDFLATEX="$(PDFLATEX)" LATEXOPTS="$(LATEXOPTS)" -C $(BUILDDIR)/$(var)/latex || exit; \
 	   mkdir -p $(BUILDDIR)/$(var)/pdf; \
@@ -122,10 +131,12 @@ endif # HAVE_PDFLATEX
 
 epubdocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),epub))
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,epub,$(var),epub,$(var)))
 
 xmldocs:
 	@$(srctree)/scripts/sphinx-pre-install --version-check
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),xml))
 	@+$(foreach var,$(SPHINXDIRS),$(call loop_cmd,sphinx,xml,$(var),xml,$(var)))
 
 endif # HAVE_SPHINX
@@ -138,7 +149,7 @@ refcheckdocs:
 
 cleandocs:
 	$(Q)rm -rf $(BUILDDIR)
-	$(Q)$(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=Documentation/userspace-api/media clean
+	@+$(foreach var,$(BUILD_SUBDIRS),$(call loop_cmd,build_subdir,$(var),clean))
 
 dochelp:
 	@echo  ' Linux kernel internal documentation in different formats from ReST:'
diff --git a/Documentation/userspace-api/media/Makefile b/Documentation/userspace-api/media/Makefile
index 00922aa7efde..783c6f880b72 100644
--- a/Documentation/userspace-api/media/Makefile
+++ b/Documentation/userspace-api/media/Makefile
@@ -50,6 +50,8 @@ $(BUILDDIR)/lirc.h.rst: ${UAPI}/lirc.h ${PARSER} $(SRC_DIR)/lirc.h.rst.exception
 .PHONY: all html epub xml latex
 
 all: $(IMGDOT) $(BUILDDIR) ${TARGETS}
+	@:
+
 html: all
 epub: all
 xml: all
-- 
2.35.1

[PATCH bpf-next v2 2/2] Add table of BPF program types to libbpf docs

[Donald Hunter]

Extend the libbpf documentation with a table of program types,
attach points and ELF section names. This table uses data from
program_types.csv which is generated from tools/lib/bpf/libbpf.c
during the documentation build.

Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
---
 Documentation/Makefile                     |  3 +-
 Documentation/bpf/libbpf/Makefile          | 36 ++++++++++++++++++++++
 Documentation/bpf/libbpf/index.rst         |  3 ++
 Documentation/bpf/libbpf/program_types.rst | 18 +++++++++++
 Documentation/bpf/programs.rst             |  3 ++
 5 files changed, 62 insertions(+), 1 deletion(-)
 create mode 100644 Documentation/bpf/libbpf/Makefile
 create mode 100644 Documentation/bpf/libbpf/program_types.rst

diff --git a/Documentation/Makefile b/Documentation/Makefile
index 8a63ef2dcd1c..f007314770e1 100644
--- a/Documentation/Makefile
+++ b/Documentation/Makefile
@@ -66,7 +66,8 @@ I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
 loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
 
 BUILD_SUBDIRS = \
-	Documentation/userspace-api/media
+	Documentation/userspace-api/media \
+	Documentation/bpf/libbpf
 
 quiet_cmd_build_subdir = SUBDIR  $2
       cmd_build_subdir = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=$2 $3
diff --git a/Documentation/bpf/libbpf/Makefile b/Documentation/bpf/libbpf/Makefile
new file mode 100644
index 000000000000..c0c2811c4dd6
--- /dev/null
+++ b/Documentation/bpf/libbpf/Makefile
@@ -0,0 +1,36 @@
+# SPDX-License-Identifier: GPL-2.0
+#
+# Rules to convert BPF program types in tools/lib/bpf/libbpf.c
+# into a .csv file
+
+FILES = program_types.csv
+
+TARGETS := $(addprefix $(BUILDDIR)/, $(FILES))
+
+$(BUILDDIR)/program_types.csv:	$(srctree)/tools/lib/bpf/libbpf.c
+	$(Q)awk -F'[",[:space:]]+' \
+	'BEGIN { print "Program Type,Attach Type,ELF Section Name,Sleepable" } \
+	/SEC_DEF\(\"/ && !/SEC_DEPRECATED/ { \
+	type = "``BPF_PROG_TYPE_" $$4 "``"; \
+	attach = index($$5, "0") ? "" : "``" $$5 "``"; \
+	section = "``" $$3 "``"; \
+	sleepable = index($$0, "SEC_SLEEPABLE") ? "Yes" : ""; \
+	print type "," attach "," section "," sleepable }' \
+	$< > $@
+
+.PHONY: all html epub xml latex linkcheck clean
+
+all: $(BUILDDIR) ${TARGETS}
+	@:
+
+html: all
+epub: all
+xml: all
+latex: all
+linkcheck:
+
+clean:
+	-$(Q)rm -f ${TARGETS} 2>/dev/null
+
+$(BUILDDIR):
+	$(Q)mkdir -p $@
diff --git a/Documentation/bpf/libbpf/index.rst b/Documentation/bpf/libbpf/index.rst
index 3722537d1384..2c04a9b3aa1f 100644
--- a/Documentation/bpf/libbpf/index.rst
+++ b/Documentation/bpf/libbpf/index.rst
@@ -1,5 +1,7 @@
 .. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 
+.. _libbpf:
+
 libbpf
 ======
 
@@ -9,6 +11,7 @@ libbpf
    API Documentation <https://libbpf.readthedocs.io/en/latest/api.html>
    libbpf_naming_convention
    libbpf_build
+   program_types
 
 This is documentation for libbpf, a userspace library for loading and
 interacting with bpf programs.
diff --git a/Documentation/bpf/libbpf/program_types.rst b/Documentation/bpf/libbpf/program_types.rst
new file mode 100644
index 000000000000..dc65ede09eef
--- /dev/null
+++ b/Documentation/bpf/libbpf/program_types.rst
@@ -0,0 +1,18 @@
+.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+
+.. _program_types_and_elf:
+
+Program Types  and ELF Sections
+===============================
+
+The table below lists the program types, their attach types where relevant and the ELF section
+names supported by libbpf for them. The ELF section names follow these rules:
+
+- ``type`` is an exact match, e.g. ``SEC("socket")``
+- ``type+`` means it can be either exact ``SEC("type")`` or well-formed ``SEC("type/extras")``
+  with a ‘``/``’ separator, e.g. ``SEC("tracepoint/syscalls/sys_enter_open")``
+
+.. csv-table:: Program Types and Their ELF Section Names
+   :file: ../../output/program_types.csv
+   :widths: 40 30 20 10
+   :header-rows: 1
diff --git a/Documentation/bpf/programs.rst b/Documentation/bpf/programs.rst
index 620eb667ac7a..c99000ab6d9b 100644
--- a/Documentation/bpf/programs.rst
+++ b/Documentation/bpf/programs.rst
@@ -7,3 +7,6 @@ Program Types
    :glob:
 
    prog_*
+
+For a list of all program types, see :ref:`program_types_and_elf` in
+the :ref:`libbpf` documentation.
-- 
2.35.1

Re: [PATCH bpf-next v2 2/2] Add table of BPF program types to libbpf docs

[Andrii Nakryiko]

On Wed, Aug 24, 2022 at 3:30 PM Donald Hunter <donald.hunter@gmail.com> wrote:
>
> Extend the libbpf documentation with a table of program types,
> attach points and ELF section names. This table uses data from
> program_types.csv which is generated from tools/lib/bpf/libbpf.c
> during the documentation build.
>
> Signed-off-by: Donald Hunter <donald.hunter@gmail.com>
> ---
>  Documentation/Makefile                     |  3 +-
>  Documentation/bpf/libbpf/Makefile          | 36 ++++++++++++++++++++++
>  Documentation/bpf/libbpf/index.rst         |  3 ++
>  Documentation/bpf/libbpf/program_types.rst | 18 +++++++++++
>  Documentation/bpf/programs.rst             |  3 ++
>  5 files changed, 62 insertions(+), 1 deletion(-)
>  create mode 100644 Documentation/bpf/libbpf/Makefile
>  create mode 100644 Documentation/bpf/libbpf/program_types.rst
>
> diff --git a/Documentation/Makefile b/Documentation/Makefile
> index 8a63ef2dcd1c..f007314770e1 100644
> --- a/Documentation/Makefile
> +++ b/Documentation/Makefile
> @@ -66,7 +66,8 @@ I18NSPHINXOPTS  = $(PAPEROPT_$(PAPER)) $(SPHINXOPTS) .
>  loop_cmd = $(echo-cmd) $(cmd_$(1)) || exit;
>
>  BUILD_SUBDIRS = \
> -       Documentation/userspace-api/media
> +       Documentation/userspace-api/media \
> +       Documentation/bpf/libbpf
>
>  quiet_cmd_build_subdir = SUBDIR  $2
>        cmd_build_subdir = $(MAKE) BUILDDIR=$(abspath $(BUILDDIR)) $(build)=$2 $3
> diff --git a/Documentation/bpf/libbpf/Makefile b/Documentation/bpf/libbpf/Makefile
> new file mode 100644
> index 000000000000..c0c2811c4dd6
> --- /dev/null
> +++ b/Documentation/bpf/libbpf/Makefile
> @@ -0,0 +1,36 @@
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# Rules to convert BPF program types in tools/lib/bpf/libbpf.c
> +# into a .csv file
> +
> +FILES = program_types.csv
> +
> +TARGETS := $(addprefix $(BUILDDIR)/, $(FILES))
> +
> +$(BUILDDIR)/program_types.csv: $(srctree)/tools/lib/bpf/libbpf.c
> +       $(Q)awk -F'[",[:space:]]+' \
> +       'BEGIN { print "Program Type,Attach Type,ELF Section Name,Sleepable" } \
> +       /SEC_DEF\(\"/ && !/SEC_DEPRECATED/ { \
> +       type = "``BPF_PROG_TYPE_" $$4 "``"; \
> +       attach = index($$5, "0") ? "" : "``" $$5 "``"; \
> +       section = "``" $$3 "``"; \
> +       sleepable = index($$0, "SEC_SLEEPABLE") ? "Yes" : ""; \
> +       print type "," attach "," section "," sleepable }' \
> +       $< > $@
> +
> +.PHONY: all html epub xml latex linkcheck clean
> +
> +all: $(BUILDDIR) ${TARGETS}
> +       @:
> +
> +html: all
> +epub: all
> +xml: all
> +latex: all
> +linkcheck:
> +
> +clean:
> +       -$(Q)rm -f ${TARGETS} 2>/dev/null
> +
> +$(BUILDDIR):
> +       $(Q)mkdir -p $@
> diff --git a/Documentation/bpf/libbpf/index.rst b/Documentation/bpf/libbpf/index.rst
> index 3722537d1384..2c04a9b3aa1f 100644
> --- a/Documentation/bpf/libbpf/index.rst
> +++ b/Documentation/bpf/libbpf/index.rst
> @@ -1,5 +1,7 @@
>  .. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>
> +.. _libbpf:
> +
>  libbpf
>  ======
>
> @@ -9,6 +11,7 @@ libbpf
>     API Documentation <https://libbpf.readthedocs.io/en/latest/api.html>

I'd put program_types here, it's more relevant and important than
libbpf naming conventions

>     libbpf_naming_convention
>     libbpf_build
> +   program_types
>
>  This is documentation for libbpf, a userspace library for loading and
>  interacting with bpf programs.
> diff --git a/Documentation/bpf/libbpf/program_types.rst b/Documentation/bpf/libbpf/program_types.rst
> new file mode 100644
> index 000000000000..dc65ede09eef
> --- /dev/null
> +++ b/Documentation/bpf/libbpf/program_types.rst
> @@ -0,0 +1,18 @@
> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> +
> +.. _program_types_and_elf:
> +
> +Program Types  and ELF Sections

nit: two spaces?

> +===============================
> +
> +The table below lists the program types, their attach types where relevant and the ELF section
> +names supported by libbpf for them. The ELF section names follow these rules:
> +
> +- ``type`` is an exact match, e.g. ``SEC("socket")``
> +- ``type+`` means it can be either exact ``SEC("type")`` or well-formed ``SEC("type/extras")``
> +  with a ‘``/``’ separator, e.g. ``SEC("tracepoint/syscalls/sys_enter_open")``

'/' is always going to be a type and "extras" separator, but extra
section format is not formalized. We have cases where it's all '/'s
(like tracepoint you mentioned), but newer and more complicated format
uses ':' as separator, e.g.
SEC("usdt/<path-to-binary>:<usdt_provide>:<usdt_name>") (let's mention
the latter as well to not create false impression of only ever having
'/' as separator)

> +
> +.. csv-table:: Program Types and Their ELF Section Names
> +   :file: ../../output/program_types.csv
> +   :widths: 40 30 20 10
> +   :header-rows: 1

it would be helpful to include a short snippet from generated CSV file
to give a general idea of the output

> diff --git a/Documentation/bpf/programs.rst b/Documentation/bpf/programs.rst
> index 620eb667ac7a..c99000ab6d9b 100644
> --- a/Documentation/bpf/programs.rst
> +++ b/Documentation/bpf/programs.rst
> @@ -7,3 +7,6 @@ Program Types
>     :glob:
>
>     prog_*
> +
> +For a list of all program types, see :ref:`program_types_and_elf` in
> +the :ref:`libbpf` documentation.
> --
> 2.35.1
>

Re: [PATCH bpf-next v2 2/2] Add table of BPF program types to libbpf docs

[Donald Hunter]

Andrii Nakryiko <andrii.nakryiko@gmail.com> writes:
>
>>  libbpf
>>  ======
>>
>> @@ -9,6 +11,7 @@ libbpf
>>     API Documentation <https://libbpf.readthedocs.io/en/latest/api.html>
>
> I'd put program_types here, it's more relevant and important than
> libbpf naming conventions

Good suggestion, thanks.

>>     libbpf_naming_convention
>>     libbpf_build
>> +   program_types
>>
>>  This is documentation for libbpf, a userspace library for loading and
>>  interacting with bpf programs.
>> diff --git a/Documentation/bpf/libbpf/program_types.rst b/Documentation/bpf/libbpf/program_types.rst
>> new file mode 100644
>> index 000000000000..dc65ede09eef
>> --- /dev/null
>> +++ b/Documentation/bpf/libbpf/program_types.rst
>> @@ -0,0 +1,18 @@
>> +.. SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>> +
>> +.. _program_types_and_elf:
>> +
>> +Program Types  and ELF Sections
>
> nit: two spaces?

Ack.

>> +===============================
>> +
>> +The table below lists the program types, their attach types where relevant and the ELF section
>> +names supported by libbpf for them. The ELF section names follow these rules:
>> +
>> +- ``type`` is an exact match, e.g. ``SEC("socket")``
>> +- ``type+`` means it can be either exact ``SEC("type")`` or well-formed ``SEC("type/extras")``
>> +  with a ‘``/``’ separator, e.g. ``SEC("tracepoint/syscalls/sys_enter_open")``
>
> '/' is always going to be a type and "extras" separator, but extra
> section format is not formalized. We have cases where it's all '/'s
> (like tracepoint you mentioned), but newer and more complicated format
> uses ':' as separator, e.g.
> SEC("usdt/<path-to-binary>:<usdt_provide>:<usdt_name>") (let's mention
> the latter as well to not create false impression of only ever having
> '/' as separator)

I will add a more detailed description extras format and include the
USDT example.

>> +
>> +.. csv-table:: Program Types and Their ELF Section Names
>> +   :file: ../../output/program_types.csv
>> +   :widths: 40 30 20 10
>> +   :header-rows: 1
>
> it would be helpful to include a short snippet from generated CSV file
> to give a general idea of the output

Ack.