[PATCH bpf-next v3 0/3] tools: bpftool: add an option for debug output from libbpf and verifier

[PATCH bpf-next v3 0/3] tools: bpftool: add an option for debug output from libbpf and verifier

[Quentin Monnet]

Hi,
This series adds an option to bpftool to make it print additional
information via libbpf and the kernel verifier when attempting to load
programs.

A new API function is added to libbpf in order to pass the log_level from
bpftool with the bpf_object__* part of the API.

Options for a finer control over the log levels to use for libbpf and the
verifier could be added in the future, if desired.

v3:
- Fix and clarify commit logs.

v2:
- Do not add distinct options for libbpf and verifier logs, just keep the
  one that sets all log levels to their maximum. Rename the option.
- Do not offer a way to pick desired log levels. The choice is "use the
  option to print all logs" or "stick with the defaults".
- Do not export BPF_LOG_* flags to user header.
- Update all man pages (most bpftool operations use libbpf and may print
  libbpf logs). Verifier logs are only used when attempting to load
  programs for now, so bpftool-prog.rst and bpftool.rst remain the only
  pages updated in that regard.

Previous discussion available at:
https://lore.kernel.org/bpf/20190523105426.3938-1-quentin.monnet@netronome.com/
https://lore.kernel.org/bpf/20190429095227.9745-1-quentin.monnet@netronome.com/

Quentin Monnet (3):
  tools: bpftool: add -d option to get debug output from libbpf
  libbpf: add bpf_object__load_xattr() API function to pass log_level
  tools: bpftool: make -d option print debug output from verifier

 .../bpf/bpftool/Documentation/bpftool-btf.rst |  4 +++
 .../bpftool/Documentation/bpftool-cgroup.rst  |  4 +++
 .../bpftool/Documentation/bpftool-feature.rst |  4 +++
 .../bpf/bpftool/Documentation/bpftool-map.rst |  4 +++
 .../bpf/bpftool/Documentation/bpftool-net.rst |  4 +++
 .../bpftool/Documentation/bpftool-perf.rst    |  4 +++
 .../bpftool/Documentation/bpftool-prog.rst    |  5 ++++
 tools/bpf/bpftool/Documentation/bpftool.rst   |  4 +++
 tools/bpf/bpftool/bash-completion/bpftool     |  2 +-
 tools/bpf/bpftool/main.c                      | 16 ++++++++++-
 tools/bpf/bpftool/main.h                      |  1 +
 tools/bpf/bpftool/prog.c                      | 27 ++++++++++++-------
 tools/lib/bpf/Makefile                        |  2 +-
 tools/lib/bpf/libbpf.c                        | 20 +++++++++++---
 tools/lib/bpf/libbpf.h                        |  6 +++++
 tools/lib/bpf/libbpf.map                      |  5 ++++
 16 files changed, 96 insertions(+), 16 deletions(-)

-- 
2.17.1

[PATCH bpf-next v3 1/3] tools: bpftool: add -d option to get debug output from libbpf

[Quentin Monnet]

libbpf has three levels of priority for output messages: warn, info,
debug. By default, debug output is not printed to the console.

Add a new "--debug" (short name: "-d") option to bpftool to print libbpf
logs for all three levels.

Internally, we simply use the function provided by libbpf to replace the
default printing function by one that prints logs regardless of their
level.

v2:
- Remove the possibility to select the log-levels to use (v1 offered a
  combination of "warn", "info" and "debug").
- Rename option and offer a short name: -d|--debug.
- Add option description to all bpftool manual pages (instead of
  bpftool-prog.rst only), as all commands use libbpf.

Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
 tools/bpf/bpftool/Documentation/bpftool-btf.rst    |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool-cgroup.rst |  4 ++++
 .../bpf/bpftool/Documentation/bpftool-feature.rst  |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool-map.rst    |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool-net.rst    |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool-perf.rst   |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool-prog.rst   |  4 ++++
 tools/bpf/bpftool/Documentation/bpftool.rst        |  3 +++
 tools/bpf/bpftool/bash-completion/bpftool          |  2 +-
 tools/bpf/bpftool/main.c                           | 14 +++++++++++++-
 10 files changed, 45 insertions(+), 2 deletions(-)

diff --git a/tools/bpf/bpftool/Documentation/bpftool-btf.rst b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
index 2dbc1413fabd..00668df1bf7a 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-btf.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-btf.rst
@@ -67,6 +67,10 @@ OPTIONS
 	-p, --pretty
 		  Generate human-readable JSON output. Implies **-j**.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 **# bpftool btf dump id 1226**
diff --git a/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst b/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst
index ac26876389c2..36807735e2a5 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-cgroup.rst
@@ -113,6 +113,10 @@ OPTIONS
 	-f, --bpffs
 		  Show file names of pinned programs.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 |
diff --git a/tools/bpf/bpftool/Documentation/bpftool-feature.rst b/tools/bpf/bpftool/Documentation/bpftool-feature.rst
index 14180e887082..4d08f35034a2 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-feature.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-feature.rst
@@ -73,6 +73,10 @@ OPTIONS
 	-p, --pretty
 		  Generate human-readable JSON output. Implies **-j**.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 SEE ALSO
 ========
 	**bpf**\ (2),
diff --git a/tools/bpf/bpftool/Documentation/bpftool-map.rst b/tools/bpf/bpftool/Documentation/bpftool-map.rst
index 13ef27b39f20..490b4501cb6e 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-map.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-map.rst
@@ -152,6 +152,10 @@ OPTIONS
 		  Do not automatically attempt to mount any virtual file system
 		  (such as tracefs or BPF virtual file system) when necessary.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 **# bpftool map show**
diff --git a/tools/bpf/bpftool/Documentation/bpftool-net.rst b/tools/bpf/bpftool/Documentation/bpftool-net.rst
index 934580850f42..d8e5237a2085 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-net.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-net.rst
@@ -65,6 +65,10 @@ OPTIONS
 	-p, --pretty
 		  Generate human-readable JSON output. Implies **-j**.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 
diff --git a/tools/bpf/bpftool/Documentation/bpftool-perf.rst b/tools/bpf/bpftool/Documentation/bpftool-perf.rst
index 0c7576523a21..e252bd0bc434 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-perf.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-perf.rst
@@ -53,6 +53,10 @@ OPTIONS
 	-p, --pretty
 		  Generate human-readable JSON output. Implies **-j**.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 
diff --git a/tools/bpf/bpftool/Documentation/bpftool-prog.rst b/tools/bpf/bpftool/Documentation/bpftool-prog.rst
index e8118544d118..9a92614569e6 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-prog.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-prog.rst
@@ -174,6 +174,10 @@ OPTIONS
 		  Do not automatically attempt to mount any virtual file system
 		  (such as tracefs or BPF virtual file system) when necessary.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
+
 EXAMPLES
 ========
 **# bpftool prog show**
diff --git a/tools/bpf/bpftool/Documentation/bpftool.rst b/tools/bpf/bpftool/Documentation/bpftool.rst
index 3e562d7fd56f..43dba0717953 100644
--- a/tools/bpf/bpftool/Documentation/bpftool.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool.rst
@@ -66,6 +66,9 @@ OPTIONS
 		  Do not automatically attempt to mount any virtual file system
 		  (such as tracefs or BPF virtual file system) when necessary.
 
+	-d, --debug
+		  Print all logs available from libbpf, including debug-level
+		  information.
 
 SEE ALSO
 ========
diff --git a/tools/bpf/bpftool/bash-completion/bpftool b/tools/bpf/bpftool/bash-completion/bpftool
index 50e402a5a9c8..3a476e25d046 100644
--- a/tools/bpf/bpftool/bash-completion/bpftool
+++ b/tools/bpf/bpftool/bash-completion/bpftool
@@ -181,7 +181,7 @@ _bpftool()
 
     # Deal with options
     if [[ ${words[cword]} == -* ]]; then
-        local c='--version --json --pretty --bpffs --mapcompat'
+        local c='--version --json --pretty --bpffs --mapcompat --debug'
         COMPREPLY=( $( compgen -W "$c" -- "$cur" ) )
         return 0
     fi
diff --git a/tools/bpf/bpftool/main.c b/tools/bpf/bpftool/main.c
index 1ac1fc520e6a..d74293938a05 100644
--- a/tools/bpf/bpftool/main.c
+++ b/tools/bpf/bpftool/main.c
@@ -10,6 +10,7 @@
 #include <string.h>
 
 #include <bpf.h>
+#include <libbpf.h>
 
 #include "main.h"
 
@@ -77,6 +78,13 @@ static int do_version(int argc, char **argv)
 	return 0;
 }
 
+static int __printf(2, 0)
+print_all_levels(__maybe_unused enum libbpf_print_level level,
+		 const char *format, va_list args)
+{
+	return vfprintf(stderr, format, args);
+}
+
 int cmd_select(const struct cmd *cmds, int argc, char **argv,
 	       int (*help)(int argc, char **argv))
 {
@@ -317,6 +325,7 @@ int main(int argc, char **argv)
 		{ "bpffs",	no_argument,	NULL,	'f' },
 		{ "mapcompat",	no_argument,	NULL,	'm' },
 		{ "nomount",	no_argument,	NULL,	'n' },
+		{ "debug",	no_argument,	NULL,	'd' },
 		{ 0 }
 	};
 	int opt, ret;
@@ -332,7 +341,7 @@ int main(int argc, char **argv)
 	hash_init(map_table.table);
 
 	opterr = 0;
-	while ((opt = getopt_long(argc, argv, "Vhpjfmn",
+	while ((opt = getopt_long(argc, argv, "Vhpjfmnd",
 				  options, NULL)) >= 0) {
 		switch (opt) {
 		case 'V':
@@ -362,6 +371,9 @@ int main(int argc, char **argv)
 		case 'n':
 			block_mount = true;
 			break;
+		case 'd':
+			libbpf_set_print(print_all_levels);
+			break;
 		default:
 			p_err("unrecognized option '%s'", argv[optind - 1]);
 			if (json_output)
-- 
2.17.1

[PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Quentin Monnet]

libbpf was recently made aware of the log_level attribute for programs,
used to specify the level of information expected to be dumped by the
verifier. Function bpf_prog_load_xattr() got support for this log_level
parameter.

But some applications using libbpf rely on another function to load
programs, bpf_object__load(), which does accept any parameter for log
level. Create an API function based on bpf_object__load(), but accepting
an "attr" object as a parameter. Then add a log_level field to that
object, so that applications calling the new bpf_object__load_xattr()
can pick the desired log level.

v3:
- Rewrite commit log.

v2:
- We are in a new cycle, bump libbpf extraversion number.

Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
 tools/lib/bpf/Makefile   |  2 +-
 tools/lib/bpf/libbpf.c   | 20 +++++++++++++++++---
 tools/lib/bpf/libbpf.h   |  6 ++++++
 tools/lib/bpf/libbpf.map |  5 +++++
 4 files changed, 29 insertions(+), 4 deletions(-)

diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile
index a2aceadf68db..9312066a1ae3 100644
--- a/tools/lib/bpf/Makefile
+++ b/tools/lib/bpf/Makefile
@@ -3,7 +3,7 @@
 
 BPF_VERSION = 0
 BPF_PATCHLEVEL = 0
-BPF_EXTRAVERSION = 3
+BPF_EXTRAVERSION = 4
 
 MAKEFLAGS += --no-print-directory
 
diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
index 197b574406b3..1c6fb7a3201e 100644
--- a/tools/lib/bpf/libbpf.c
+++ b/tools/lib/bpf/libbpf.c
@@ -2222,7 +2222,7 @@ static bool bpf_program__is_function_storage(struct bpf_program *prog,
 }
 
 static int
-bpf_object__load_progs(struct bpf_object *obj)
+bpf_object__load_progs(struct bpf_object *obj, int log_level)
 {
 	size_t i;
 	int err;
@@ -2230,6 +2230,7 @@ bpf_object__load_progs(struct bpf_object *obj)
 	for (i = 0; i < obj->nr_programs; i++) {
 		if (bpf_program__is_function_storage(&obj->programs[i], obj))
 			continue;
+		obj->programs[i].log_level = log_level;
 		err = bpf_program__load(&obj->programs[i],
 					obj->license,
 					obj->kern_version);
@@ -2381,10 +2382,14 @@ int bpf_object__unload(struct bpf_object *obj)
 	return 0;
 }
 
-int bpf_object__load(struct bpf_object *obj)
+int bpf_object__load_xattr(struct bpf_object_load_attr *attr)
 {
+	struct bpf_object *obj;
 	int err;
 
+	if (!attr)
+		return -EINVAL;
+	obj = attr->obj;
 	if (!obj)
 		return -EINVAL;
 
@@ -2397,7 +2402,7 @@ int bpf_object__load(struct bpf_object *obj)
 
 	CHECK_ERR(bpf_object__create_maps(obj), err, out);
 	CHECK_ERR(bpf_object__relocate(obj), err, out);
-	CHECK_ERR(bpf_object__load_progs(obj), err, out);
+	CHECK_ERR(bpf_object__load_progs(obj, attr->log_level), err, out);
 
 	return 0;
 out:
@@ -2406,6 +2411,15 @@ int bpf_object__load(struct bpf_object *obj)
 	return err;
 }
 
+int bpf_object__load(struct bpf_object *obj)
+{
+	struct bpf_object_load_attr attr = {
+		.obj = obj,
+	};
+
+	return bpf_object__load_xattr(&attr);
+}
+
 static int check_path(const char *path)
 {
 	char *cp, errmsg[STRERR_BUFSIZE];
diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
index c5ff00515ce7..e1c748db44f6 100644
--- a/tools/lib/bpf/libbpf.h
+++ b/tools/lib/bpf/libbpf.h
@@ -89,8 +89,14 @@ LIBBPF_API int bpf_object__unpin_programs(struct bpf_object *obj,
 LIBBPF_API int bpf_object__pin(struct bpf_object *object, const char *path);
 LIBBPF_API void bpf_object__close(struct bpf_object *object);
 
+struct bpf_object_load_attr {
+	struct bpf_object *obj;
+	int log_level;
+};
+
 /* Load/unload object into/from kernel */
 LIBBPF_API int bpf_object__load(struct bpf_object *obj);
+LIBBPF_API int bpf_object__load_xattr(struct bpf_object_load_attr *attr);
 LIBBPF_API int bpf_object__unload(struct bpf_object *obj);
 LIBBPF_API const char *bpf_object__name(struct bpf_object *obj);
 LIBBPF_API unsigned int bpf_object__kversion(struct bpf_object *obj);
diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
index 673001787cba..6ce61fa0baf3 100644
--- a/tools/lib/bpf/libbpf.map
+++ b/tools/lib/bpf/libbpf.map
@@ -164,3 +164,8 @@ LIBBPF_0.0.3 {
 		bpf_map_freeze;
 		btf__finalize_data;
 } LIBBPF_0.0.2;
+
+LIBBPF_0.0.4 {
+	global:
+		bpf_object__load_xattr;
+} LIBBPF_0.0.3;
-- 
2.17.1

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Jesper Dangaard Brouer]

On Fri, 24 May 2019 11:36:47 +0100
Quentin Monnet <quentin.monnet@netronome.com> wrote:

> libbpf was recently made aware of the log_level attribute for programs,
> used to specify the level of information expected to be dumped by the
> verifier. Function bpf_prog_load_xattr() got support for this log_level
> parameter.
> 
> But some applications using libbpf rely on another function to load
> programs, bpf_object__load(), which does accept any parameter for log
> level. Create an API function based on bpf_object__load(), but accepting
> an "attr" object as a parameter. Then add a log_level field to that
> object, so that applications calling the new bpf_object__load_xattr()
> can pick the desired log level.

Does this allow us to extend struct bpf_object_load_attr later?

> v3:
> - Rewrite commit log.
> 
> v2:
> - We are in a new cycle, bump libbpf extraversion number.
> 
> Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
> ---
>  tools/lib/bpf/Makefile   |  2 +-
>  tools/lib/bpf/libbpf.c   | 20 +++++++++++++++++---
>  tools/lib/bpf/libbpf.h   |  6 ++++++
>  tools/lib/bpf/libbpf.map |  5 +++++
>  4 files changed, 29 insertions(+), 4 deletions(-)
> 
> diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile
> index a2aceadf68db..9312066a1ae3 100644
> --- a/tools/lib/bpf/Makefile
> +++ b/tools/lib/bpf/Makefile
> @@ -3,7 +3,7 @@
>  
>  BPF_VERSION = 0
>  BPF_PATCHLEVEL = 0
> -BPF_EXTRAVERSION = 3
> +BPF_EXTRAVERSION = 4
>  
>  MAKEFLAGS += --no-print-directory
>  
> diff --git a/tools/lib/bpf/libbpf.c b/tools/lib/bpf/libbpf.c
> index 197b574406b3..1c6fb7a3201e 100644
> --- a/tools/lib/bpf/libbpf.c
> +++ b/tools/lib/bpf/libbpf.c
> @@ -2222,7 +2222,7 @@ static bool bpf_program__is_function_storage(struct bpf_program *prog,
>  }
>  
>  static int
> -bpf_object__load_progs(struct bpf_object *obj)
> +bpf_object__load_progs(struct bpf_object *obj, int log_level)
>  {
>  	size_t i;
>  	int err;
> @@ -2230,6 +2230,7 @@ bpf_object__load_progs(struct bpf_object *obj)
>  	for (i = 0; i < obj->nr_programs; i++) {
>  		if (bpf_program__is_function_storage(&obj->programs[i], obj))
>  			continue;
> +		obj->programs[i].log_level = log_level;
>  		err = bpf_program__load(&obj->programs[i],
>  					obj->license,
>  					obj->kern_version);
> @@ -2381,10 +2382,14 @@ int bpf_object__unload(struct bpf_object *obj)
>  	return 0;
>  }
>  
> -int bpf_object__load(struct bpf_object *obj)
> +int bpf_object__load_xattr(struct bpf_object_load_attr *attr)
>  {
> +	struct bpf_object *obj;
>  	int err;
>  
> +	if (!attr)
> +		return -EINVAL;
> +	obj = attr->obj;
>  	if (!obj)
>  		return -EINVAL;
>  
> @@ -2397,7 +2402,7 @@ int bpf_object__load(struct bpf_object *obj)
>  
>  	CHECK_ERR(bpf_object__create_maps(obj), err, out);
>  	CHECK_ERR(bpf_object__relocate(obj), err, out);
> -	CHECK_ERR(bpf_object__load_progs(obj), err, out);
> +	CHECK_ERR(bpf_object__load_progs(obj, attr->log_level), err, out);
>  
>  	return 0;
>  out:
> @@ -2406,6 +2411,15 @@ int bpf_object__load(struct bpf_object *obj)
>  	return err;
>  }
>  
> +int bpf_object__load(struct bpf_object *obj)
> +{
> +	struct bpf_object_load_attr attr = {
> +		.obj = obj,
> +	};
> +
> +	return bpf_object__load_xattr(&attr);
> +}
> +
>  static int check_path(const char *path)
>  {
>  	char *cp, errmsg[STRERR_BUFSIZE];
> diff --git a/tools/lib/bpf/libbpf.h b/tools/lib/bpf/libbpf.h
> index c5ff00515ce7..e1c748db44f6 100644
> --- a/tools/lib/bpf/libbpf.h
> +++ b/tools/lib/bpf/libbpf.h
> @@ -89,8 +89,14 @@ LIBBPF_API int bpf_object__unpin_programs(struct bpf_object *obj,
>  LIBBPF_API int bpf_object__pin(struct bpf_object *object, const char *path);
>  LIBBPF_API void bpf_object__close(struct bpf_object *object);
>  
> +struct bpf_object_load_attr {
> +	struct bpf_object *obj;
> +	int log_level;
> +};

Can this be extended later?

>  /* Load/unload object into/from kernel */
>  LIBBPF_API int bpf_object__load(struct bpf_object *obj);
> +LIBBPF_API int bpf_object__load_xattr(struct bpf_object_load_attr *attr);
>  LIBBPF_API int bpf_object__unload(struct bpf_object *obj);
>  LIBBPF_API const char *bpf_object__name(struct bpf_object *obj);
>  LIBBPF_API unsigned int bpf_object__kversion(struct bpf_object *obj);
> diff --git a/tools/lib/bpf/libbpf.map b/tools/lib/bpf/libbpf.map
> index 673001787cba..6ce61fa0baf3 100644
> --- a/tools/lib/bpf/libbpf.map
> +++ b/tools/lib/bpf/libbpf.map
> @@ -164,3 +164,8 @@ LIBBPF_0.0.3 {
>  		bpf_map_freeze;
>  		btf__finalize_data;
>  } LIBBPF_0.0.2;
> +
> +LIBBPF_0.0.4 {
> +	global:
> +		bpf_object__load_xattr;
> +} LIBBPF_0.0.3;



-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Quentin Monnet]

2019-05-24 13:22 UTC+0200 ~ Jesper Dangaard Brouer <brouer@redhat.com>
> On Fri, 24 May 2019 11:36:47 +0100
> Quentin Monnet <quentin.monnet@netronome.com> wrote:
> 
>> libbpf was recently made aware of the log_level attribute for programs,
>> used to specify the level of information expected to be dumped by the
>> verifier. Function bpf_prog_load_xattr() got support for this log_level
>> parameter.
>>
>> But some applications using libbpf rely on another function to load
>> programs, bpf_object__load(), which does accept any parameter for log
>> level. Create an API function based on bpf_object__load(), but accepting
>> an "attr" object as a parameter. Then add a log_level field to that
>> object, so that applications calling the new bpf_object__load_xattr()
>> can pick the desired log level.
> 
> Does this allow us to extend struct bpf_object_load_attr later?

I see no reason why it could not. Having the _xattr() version of the
function is precisely a way to have something extensible in the future,
without having to create additional API functions each time we want to
pass a new parameter. And e.g. struct bpf_prog_load_attr (used with
bpf_prog_load_xattr()) has already been extended in the past. So, yeah,
we can add to it in the future. Do you have something in mind?

Quentin

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Jesper Dangaard Brouer]

On Fri, 24 May 2019 12:51:14 +0100
Quentin Monnet <quentin.monnet@netronome.com> wrote:

> 2019-05-24 13:22 UTC+0200 ~ Jesper Dangaard Brouer <brouer@redhat.com>
> > On Fri, 24 May 2019 11:36:47 +0100
> > Quentin Monnet <quentin.monnet@netronome.com> wrote:
> >   
> >> libbpf was recently made aware of the log_level attribute for programs,
> >> used to specify the level of information expected to be dumped by the
> >> verifier. Function bpf_prog_load_xattr() got support for this log_level
> >> parameter.
> >>
> >> But some applications using libbpf rely on another function to load
> >> programs, bpf_object__load(), which does accept any parameter for log
> >> level. Create an API function based on bpf_object__load(), but accepting
> >> an "attr" object as a parameter. Then add a log_level field to that
> >> object, so that applications calling the new bpf_object__load_xattr()
> >> can pick the desired log level.  
> > 
> > Does this allow us to extend struct bpf_object_load_attr later?  
> 
> I see no reason why it could not. Having the _xattr() version of the
> function is precisely a way to have something extensible in the future,
> without having to create additional API functions each time we want to
> pass a new parameter. And e.g. struct bpf_prog_load_attr (used with
> bpf_prog_load_xattr()) has already been extended in the past. So, yeah,
> we can add to it in the future.

Great.  I just don't know/understand how user-space handle this. If a
binary is compiled with libbpf as dynamic loadable lib, then it e.g. saw
libbpf.so.2 when it was compiled, then can't it choose to use libbpf.so.3
then? (e.g. when libbpf.so.2 is not on the system). (I would actually
like to learn/understand this, so links are welcome).

> Do you have something in mind?

I was playing with extending bpf_prog_load_attr, but instead I created a
bpf_prog_load_attr_maps instead and a new function
bpf_prog_load_xattr_maps(), e.g. see:

https://github.com/xdp-project/xdp-tutorial/blob/master/common/common_libbpf.h
https://github.com/xdp-project/xdp-tutorial/blob/master/common/common_libbpf.c

I guess, I could just extend bpf_prog_load_attr instead, right?

-- 
Best regards,
  Jesper Dangaard Brouer
  MSc.CS, Principal Kernel Engineer at Red Hat
  LinkedIn: http://www.linkedin.com/in/brouer

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Quentin Monnet]

2019-05-24 14:49 UTC+0200 ~ Jesper Dangaard Brouer <brouer@redhat.com>
> On Fri, 24 May 2019 12:51:14 +0100
> Quentin Monnet <quentin.monnet@netronome.com> wrote:
> 
>> 2019-05-24 13:22 UTC+0200 ~ Jesper Dangaard Brouer <brouer@redhat.com>
>>> On Fri, 24 May 2019 11:36:47 +0100
>>> Quentin Monnet <quentin.monnet@netronome.com> wrote:
>>>   
>>>> libbpf was recently made aware of the log_level attribute for programs,
>>>> used to specify the level of information expected to be dumped by the
>>>> verifier. Function bpf_prog_load_xattr() got support for this log_level
>>>> parameter.
>>>>
>>>> But some applications using libbpf rely on another function to load
>>>> programs, bpf_object__load(), which does accept any parameter for log
>>>> level. Create an API function based on bpf_object__load(), but accepting
>>>> an "attr" object as a parameter. Then add a log_level field to that
>>>> object, so that applications calling the new bpf_object__load_xattr()
>>>> can pick the desired log level.  
>>>
>>> Does this allow us to extend struct bpf_object_load_attr later?  
>>
>> I see no reason why it could not. Having the _xattr() version of the
>> function is precisely a way to have something extensible in the future,
>> without having to create additional API functions each time we want to
>> pass a new parameter. And e.g. struct bpf_prog_load_attr (used with
>> bpf_prog_load_xattr()) has already been extended in the past. So, yeah,
>> we can add to it in the future.
> 
> Great.  I just don't know/understand how user-space handle this. If a
> binary is compiled with libbpf as dynamic loadable lib, then it e.g. saw
> libbpf.so.2 when it was compiled, then can't it choose to use libbpf.so.3
> then? (e.g. when libbpf.so.2 is not on the system). (I would actually
> like to learn/understand this, so links are welcome).

Well I'm no library expert, so don't take my word for it. As far as I
understand, the soname of the library is selected at link time. So if
your app is linked again libbpf.so.2, you will need version 2.* of the
library to be installed on your system, because increasing the version
number usually implies ABI breakage. You can usually check which version
of the libraries is needed with ldd ("ldd bpftool", except that you
won't see libbpf because it's statically linked for bpftool).

This being said, for now the version number for libbpf has not been
incremented and is still at 0, we only had the extraversion increasing.
Since it's not part of the soname ("-Wl,-soname,libbpf.so.$(VERSION)" in
libbpf Makefile), it is not taken into account when searching for the
lib on the system. What I mean is that if the program is linked against
libbpf.so.0, it could pick libbpf.so.0.0.2 or libbpf.so.0.0.3
indifferently depending on what it finds on the system (I assume it
takes the newest?). There should not be any ABI breakage between the
two, so programs compiled against an older patchlevel or extraversion of
the library should still be able to use a newer one.

There is some documentation on libraries here (I should take some time
to finish reading it myself!):

http://tldp.org/HOWTO/Program-Library-HOWTO/

There are also interesting elements in the documentation that was cited
when Andrey introduced the LIBPPF_API macros in libbpf:

https://www.akkadia.org/drepper/dsohowto.pdf

> 
>> Do you have something in mind?
> 
> I was playing with extending bpf_prog_load_attr, but instead I created a
> bpf_prog_load_attr_maps instead and a new function
> bpf_prog_load_xattr_maps(), e.g. see:
> 
> https://github.com/xdp-project/xdp-tutorial/blob/master/common/common_libbpf.h
> https://github.com/xdp-project/xdp-tutorial/blob/master/common/common_libbpf.c
> 
> I guess, I could just extend bpf_prog_load_attr instead, right?
> 

I believe so.

Best,
Quentin

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Alexei Starovoitov]

On Fri, May 24, 2019 at 3:36 AM Quentin Monnet
<quentin.monnet@netronome.com> wrote:
>
> libbpf was recently made aware of the log_level attribute for programs,
> used to specify the level of information expected to be dumped by the
> verifier. Function bpf_prog_load_xattr() got support for this log_level
> parameter.
>
> But some applications using libbpf rely on another function to load
> programs, bpf_object__load(), which does accept any parameter for log
> level. Create an API function based on bpf_object__load(), but accepting
> an "attr" object as a parameter. Then add a log_level field to that
> object, so that applications calling the new bpf_object__load_xattr()
> can pick the desired log level.
>
> v3:
> - Rewrite commit log.
>
> v2:
> - We are in a new cycle, bump libbpf extraversion number.
>
> Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
> ---
>  tools/lib/bpf/Makefile   |  2 +-
>  tools/lib/bpf/libbpf.c   | 20 +++++++++++++++++---
>  tools/lib/bpf/libbpf.h   |  6 ++++++
>  tools/lib/bpf/libbpf.map |  5 +++++
>  4 files changed, 29 insertions(+), 4 deletions(-)

This commit broke ./test_progs -s
prog_tests/bpf_verif_scale.c no longer passes log_level.
Could you please take a look?

Re: [PATCH bpf-next v3 2/3] libbpf: add bpf_object__load_xattr() API function to pass log_level

[Quentin Monnet]

2019-05-28 17:35 UTC-0700 ~ Alexei Starovoitov
<alexei.starovoitov@gmail.com>
> On Fri, May 24, 2019 at 3:36 AM Quentin Monnet
> <quentin.monnet@netronome.com> wrote:
>>
>> libbpf was recently made aware of the log_level attribute for programs,
>> used to specify the level of information expected to be dumped by the
>> verifier. Function bpf_prog_load_xattr() got support for this log_level
>> parameter.
>>
>> But some applications using libbpf rely on another function to load
>> programs, bpf_object__load(), which does accept any parameter for log
>> level. Create an API function based on bpf_object__load(), but accepting
>> an "attr" object as a parameter. Then add a log_level field to that
>> object, so that applications calling the new bpf_object__load_xattr()
>> can pick the desired log level.
>>
>> v3:
>> - Rewrite commit log.
>>
>> v2:
>> - We are in a new cycle, bump libbpf extraversion number.
>>
>> Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
>> Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
>> ---
>>  tools/lib/bpf/Makefile   |  2 +-
>>  tools/lib/bpf/libbpf.c   | 20 +++++++++++++++++---
>>  tools/lib/bpf/libbpf.h   |  6 ++++++
>>  tools/lib/bpf/libbpf.map |  5 +++++
>>  4 files changed, 29 insertions(+), 4 deletions(-)
> 
> This commit broke ./test_progs -s
> prog_tests/bpf_verif_scale.c no longer passes log_level.
> Could you please take a look?
> 

Indeed, I forgot that bpf_load_prog_xattr() would eventually call
bpf_object__load_progs() as well, where the log_level is now overwritten.

Fix incoming, sorry about that.

Quentin

[PATCH bpf-next v3 3/3] tools: bpftool: make -d option print debug output from verifier

[Quentin Monnet]

The "-d" option is used to require all logs available for bpftool. So
far it meant telling libbpf to print even debug-level information. But
there is another source of info that can be made more verbose: when we
attemt to load programs with bpftool, we can pass a log_level parameter
to the verifier in order to control the amount of information that is
printed to the console.

Reuse the "-d" option to print all information the verifier can tell. At
this time, this means logs related to BPF_LOG_LEVEL1, BPF_LOG_LEVEL2 and
BPF_LOG_STATS. As mentioned in the discussion on the first version of
this set, these macros are internal to the kernel
(include/linux/bpf_verifier.h) and are not meant to be part of the
stable user API, therefore we simply use the related constants to print
whatever we can at this time, without trying to tell users what is
log_level1 or what is statistics.

Verifier logs are only used when loading programs for now (In the
future: for loading BTF objects with bpftool? Although libbpf does not
currently offer to print verifier info at debug level if no error
occurred when loading BTF objects), so bpftool.rst and bpftool-prog.rst
are the only man pages to get the update.

v3:
- Add details on log level and BTF loading at the end of commit log.

v2:
- Remove the possibility to select the log levels to use (v1 offered a
  combination of "log_level1", "log_level2" and "stats").
- The macros from kernel header bpf_verifier.h are not used (and
  therefore not moved to UAPI header).
- In v1 this was a distinct option, but is now merged in the only "-d"
  switch to activate libbpf and verifier debug-level logs all at the
  same time.

Signed-off-by: Quentin Monnet <quentin.monnet@netronome.com>
Reviewed-by: Jakub Kicinski <jakub.kicinski@netronome.com>
---
 .../bpftool/Documentation/bpftool-prog.rst    |  5 ++--
 tools/bpf/bpftool/Documentation/bpftool.rst   |  5 ++--
 tools/bpf/bpftool/main.c                      |  2 ++
 tools/bpf/bpftool/main.h                      |  1 +
 tools/bpf/bpftool/prog.c                      | 27 ++++++++++++-------
 5 files changed, 26 insertions(+), 14 deletions(-)

diff --git a/tools/bpf/bpftool/Documentation/bpftool-prog.rst b/tools/bpf/bpftool/Documentation/bpftool-prog.rst
index 9a92614569e6..228a5c863cc7 100644
--- a/tools/bpf/bpftool/Documentation/bpftool-prog.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool-prog.rst
@@ -175,8 +175,9 @@ OPTIONS
 		  (such as tracefs or BPF virtual file system) when necessary.
 
 	-d, --debug
-		  Print all logs available from libbpf, including debug-level
-		  information.
+		  Print all logs available, even debug-level information. This
+		  includes logs from libbpf as well as from the verifier, when
+		  attempting to load programs.
 
 EXAMPLES
 ========
diff --git a/tools/bpf/bpftool/Documentation/bpftool.rst b/tools/bpf/bpftool/Documentation/bpftool.rst
index 43dba0717953..6a9c52ef84a9 100644
--- a/tools/bpf/bpftool/Documentation/bpftool.rst
+++ b/tools/bpf/bpftool/Documentation/bpftool.rst
@@ -67,8 +67,9 @@ OPTIONS
 		  (such as tracefs or BPF virtual file system) when necessary.
 
 	-d, --debug
-		  Print all logs available from libbpf, including debug-level
-		  information.
+		  Print all logs available, even debug-level information. This
+		  includes logs from libbpf as well as from the verifier, when
+		  attempting to load programs.
 
 SEE ALSO
 ========
diff --git a/tools/bpf/bpftool/main.c b/tools/bpf/bpftool/main.c
index d74293938a05..4879f6395c7e 100644
--- a/tools/bpf/bpftool/main.c
+++ b/tools/bpf/bpftool/main.c
@@ -26,6 +26,7 @@ bool pretty_output;
 bool json_output;
 bool show_pinned;
 bool block_mount;
+bool verifier_logs;
 int bpf_flags;
 struct pinned_obj_table prog_table;
 struct pinned_obj_table map_table;
@@ -373,6 +374,7 @@ int main(int argc, char **argv)
 			break;
 		case 'd':
 			libbpf_set_print(print_all_levels);
+			verifier_logs = true;
 			break;
 		default:
 			p_err("unrecognized option '%s'", argv[optind - 1]);
diff --git a/tools/bpf/bpftool/main.h b/tools/bpf/bpftool/main.h
index 3d63feb7f852..28a2a5857e14 100644
--- a/tools/bpf/bpftool/main.h
+++ b/tools/bpf/bpftool/main.h
@@ -91,6 +91,7 @@ extern json_writer_t *json_wtr;
 extern bool json_output;
 extern bool show_pinned;
 extern bool block_mount;
+extern bool verifier_logs;
 extern int bpf_flags;
 extern struct pinned_obj_table prog_table;
 extern struct pinned_obj_table map_table;
diff --git a/tools/bpf/bpftool/prog.c b/tools/bpf/bpftool/prog.c
index 26336bad0442..1f209c80d906 100644
--- a/tools/bpf/bpftool/prog.c
+++ b/tools/bpf/bpftool/prog.c
@@ -750,10 +750,11 @@ static int do_detach(int argc, char **argv)
 
 static int load_with_options(int argc, char **argv, bool first_prog_only)
 {
-	enum bpf_attach_type expected_attach_type;
-	struct bpf_object_open_attr attr = {
-		.prog_type	= BPF_PROG_TYPE_UNSPEC,
+	struct bpf_object_load_attr load_attr = { 0 };
+	struct bpf_object_open_attr open_attr = {
+		.prog_type = BPF_PROG_TYPE_UNSPEC,
 	};
+	enum bpf_attach_type expected_attach_type;
 	struct map_replace *map_replace = NULL;
 	struct bpf_program *prog = NULL, *pos;
 	unsigned int old_map_fds = 0;
@@ -767,7 +768,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
 
 	if (!REQ_ARGS(2))
 		return -1;
-	attr.file = GET_ARG();
+	open_attr.file = GET_ARG();
 	pinfile = GET_ARG();
 
 	while (argc) {
@@ -776,7 +777,7 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
 
 			NEXT_ARG();
 
-			if (attr.prog_type != BPF_PROG_TYPE_UNSPEC) {
+			if (open_attr.prog_type != BPF_PROG_TYPE_UNSPEC) {
 				p_err("program type already specified");
 				goto err_free_reuse_maps;
 			}
@@ -793,7 +794,8 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
 			strcat(type, *argv);
 			strcat(type, "/");
 
-			err = libbpf_prog_type_by_name(type, &attr.prog_type,
+			err = libbpf_prog_type_by_name(type,
+						       &open_attr.prog_type,
 						       &expected_attach_type);
 			free(type);
 			if (err < 0)
@@ -881,16 +883,16 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
 
 	set_max_rlimit();
 
-	obj = __bpf_object__open_xattr(&attr, bpf_flags);
+	obj = __bpf_object__open_xattr(&open_attr, bpf_flags);
 	if (IS_ERR_OR_NULL(obj)) {
 		p_err("failed to open object file");
 		goto err_free_reuse_maps;
 	}
 
 	bpf_object__for_each_program(pos, obj) {
-		enum bpf_prog_type prog_type = attr.prog_type;
+		enum bpf_prog_type prog_type = open_attr.prog_type;
 
-		if (attr.prog_type == BPF_PROG_TYPE_UNSPEC) {
+		if (open_attr.prog_type == BPF_PROG_TYPE_UNSPEC) {
 			const char *sec_name = bpf_program__title(pos, false);
 
 			err = libbpf_prog_type_by_name(sec_name, &prog_type,
@@ -960,7 +962,12 @@ static int load_with_options(int argc, char **argv, bool first_prog_only)
 		goto err_close_obj;
 	}
 
-	err = bpf_object__load(obj);
+	load_attr.obj = obj;
+	if (verifier_logs)
+		/* log_level1 + log_level2 + stats, but not stable UAPI */
+		load_attr.log_level = 1 + 2 + 4;
+
+	err = bpf_object__load_xattr(&load_attr);
 	if (err) {
 		p_err("failed to load object file");
 		goto err_close_obj;
-- 
2.17.1

Re: [PATCH bpf-next v3 0/3] tools: bpftool: add an option for debug output from libbpf and verifier

[Yonghong Song]

On 5/24/19 3:36 AM, Quentin Monnet wrote:
> Hi,
> This series adds an option to bpftool to make it print additional
> information via libbpf and the kernel verifier when attempting to load
> programs.
> 
> A new API function is added to libbpf in order to pass the log_level from
> bpftool with the bpf_object__* part of the API.
> 
> Options for a finer control over the log levels to use for libbpf and the
> verifier could be added in the future, if desired.
> 
> v3:
> - Fix and clarify commit logs.
> 
> v2:
> - Do not add distinct options for libbpf and verifier logs, just keep the
>    one that sets all log levels to their maximum. Rename the option.
> - Do not offer a way to pick desired log levels. The choice is "use the
>    option to print all logs" or "stick with the defaults".
> - Do not export BPF_LOG_* flags to user header.
> - Update all man pages (most bpftool operations use libbpf and may print
>    libbpf logs). Verifier logs are only used when attempting to load
>    programs for now, so bpftool-prog.rst and bpftool.rst remain the only
>    pages updated in that regard.
> 
> Previous discussion available at:
> https://lore.kernel.org/bpf/20190523105426.3938-1-quentin.monnet@netronome.com/
> https://lore.kernel.org/bpf/20190429095227.9745-1-quentin.monnet@netronome.com/

The series looks good to me.
Acked-by: Yonghong Song <yhs@fb.com>

> 
> Quentin Monnet (3):
>    tools: bpftool: add -d option to get debug output from libbpf
>    libbpf: add bpf_object__load_xattr() API function to pass log_level
>    tools: bpftool: make -d option print debug output from verifier
> 
>   .../bpf/bpftool/Documentation/bpftool-btf.rst |  4 +++
>   .../bpftool/Documentation/bpftool-cgroup.rst  |  4 +++
>   .../bpftool/Documentation/bpftool-feature.rst |  4 +++
>   .../bpf/bpftool/Documentation/bpftool-map.rst |  4 +++
>   .../bpf/bpftool/Documentation/bpftool-net.rst |  4 +++
>   .../bpftool/Documentation/bpftool-perf.rst    |  4 +++
>   .../bpftool/Documentation/bpftool-prog.rst    |  5 ++++
>   tools/bpf/bpftool/Documentation/bpftool.rst   |  4 +++
>   tools/bpf/bpftool/bash-completion/bpftool     |  2 +-
>   tools/bpf/bpftool/main.c                      | 16 ++++++++++-
>   tools/bpf/bpftool/main.h                      |  1 +
>   tools/bpf/bpftool/prog.c                      | 27 ++++++++++++-------
>   tools/lib/bpf/Makefile                        |  2 +-
>   tools/lib/bpf/libbpf.c                        | 20 +++++++++++---
>   tools/lib/bpf/libbpf.h                        |  6 +++++
>   tools/lib/bpf/libbpf.map                      |  5 ++++
>   16 files changed, 96 insertions(+), 16 deletions(-)
> 

Re: [PATCH bpf-next v3 0/3] tools: bpftool: add an option for debug output from libbpf and verifier

[Daniel Borkmann]

On 05/24/2019 12:36 PM, Quentin Monnet wrote:
> Hi,
> This series adds an option to bpftool to make it print additional
> information via libbpf and the kernel verifier when attempting to load
> programs.
> 
> A new API function is added to libbpf in order to pass the log_level from
> bpftool with the bpf_object__* part of the API.
> 
> Options for a finer control over the log levels to use for libbpf and the
> verifier could be added in the future, if desired.
> 
> v3:
> - Fix and clarify commit logs.
> 
> v2:
> - Do not add distinct options for libbpf and verifier logs, just keep the
>   one that sets all log levels to their maximum. Rename the option.
> - Do not offer a way to pick desired log levels. The choice is "use the
>   option to print all logs" or "stick with the defaults".
> - Do not export BPF_LOG_* flags to user header.
> - Update all man pages (most bpftool operations use libbpf and may print
>   libbpf logs). Verifier logs are only used when attempting to load
>   programs for now, so bpftool-prog.rst and bpftool.rst remain the only
>   pages updated in that regard.
> 
> Previous discussion available at:
> https://lore.kernel.org/bpf/20190523105426.3938-1-quentin.monnet@netronome.com/
> https://lore.kernel.org/bpf/20190429095227.9745-1-quentin.monnet@netronome.com/
> 
> Quentin Monnet (3):
>   tools: bpftool: add -d option to get debug output from libbpf
>   libbpf: add bpf_object__load_xattr() API function to pass log_level
>   tools: bpftool: make -d option print debug output from verifier
> 
>  .../bpf/bpftool/Documentation/bpftool-btf.rst |  4 +++
>  .../bpftool/Documentation/bpftool-cgroup.rst  |  4 +++
>  .../bpftool/Documentation/bpftool-feature.rst |  4 +++
>  .../bpf/bpftool/Documentation/bpftool-map.rst |  4 +++
>  .../bpf/bpftool/Documentation/bpftool-net.rst |  4 +++
>  .../bpftool/Documentation/bpftool-perf.rst    |  4 +++
>  .../bpftool/Documentation/bpftool-prog.rst    |  5 ++++
>  tools/bpf/bpftool/Documentation/bpftool.rst   |  4 +++
>  tools/bpf/bpftool/bash-completion/bpftool     |  2 +-
>  tools/bpf/bpftool/main.c                      | 16 ++++++++++-
>  tools/bpf/bpftool/main.h                      |  1 +
>  tools/bpf/bpftool/prog.c                      | 27 ++++++++++++-------
>  tools/lib/bpf/Makefile                        |  2 +-
>  tools/lib/bpf/libbpf.c                        | 20 +++++++++++---
>  tools/lib/bpf/libbpf.h                        |  6 +++++
>  tools/lib/bpf/libbpf.map                      |  5 ++++
>  16 files changed, 96 insertions(+), 16 deletions(-)
> 

Applied, thanks!