[PATCH bpf-next 0/3] Maintain selftest configuration in-tree

[PATCH bpf-next 0/3] Maintain selftest configuration in-tree

[Daniel Müller]

BPF selftests mandate certain kernel configuration options to be present in
order to pass. Currently the "reference" config files containing these options
are hosted in a separate repository [0]. From there they are picked up by the
BPF continuous integration system as well as the in-tree vmtest.sh helper
script, which allows for running tests in a VM-based setup locally.

But it gets worse, as "BPF CI" is really two CI systems: one for libbpf
(mentioned above) and one for the bpf-next kernel repository (or more precisely:
family of repositories, as bpf-rc is using the system). As such, we have an
additional -- and slightly divergent -- copy of these configurations.

This patch set proposes the merging of said configurations into this repository.
Doing so provides several benefits:
1) the vmtest.sh script is now self-contained, no longer requiring to pull
   configurations over the network
2) we can have a single copy of these configurations, eliminating the
   maintenance burden of keeping two versions in-sync
3) the kernel tree is the place where most development happens, so it is the
   most natural to adjust configurations as changes are proposed there, as
   opposed to out-of-tree, where they would always remain an afterthought

The patch set is structed in such a way that we first integrate the first set of
configurations [0], then we add the difference to the second one [1], and lastly
we adjust the vmtest.sh script to pick up the local configuration instead of
reaching out to GitHub.

[0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
[1] https://github.com/kernel-patches/vmtest/tree/831ee8eb72ddb7e03babb8f7e050d52a451237aa/travis-ci/vmtest/configs

Daniel Müller (3):
  selftests/bpf: Copy over libbpf configs
  selftests/bpf: Integrate vmtest configs
  selftests/bpf: Adjust vmtest.sh to use local kernel configuration

 .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
 .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
 .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
 .../bpf/configs/config-latest.x86_64          | 3073 +++++++++++++++++
 .../bpf/configs/denylist/DENYLIST-5.5.0       |  117 +
 .../bpf/configs/denylist/DENYLIST-latest      |   11 +
 .../configs/denylist/DENYLIST-latest.s390x    |   68 +
 tools/testing/selftests/bpf/vmtest.sh         |   28 +-
 8 files changed, 6055 insertions(+), 16 deletions(-)
 create mode 100644 tools/testing/selftests/bpf/configs/allowlist/ALLOWLIST-4.9.0
 create mode 100644 tools/testing/selftests/bpf/configs/allowlist/ALLOWLIST-5.5.0
 create mode 100644 tools/testing/selftests/bpf/configs/config-latest.s390x
 create mode 100644 tools/testing/selftests/bpf/configs/config-latest.x86_64
 create mode 100644 tools/testing/selftests/bpf/configs/denylist/DENYLIST-5.5.0
 create mode 100644 tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
 create mode 100644 tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x

-- 
2.30.2

[PATCH bpf-next 2/3] selftests/bpf: Integrate vmtest configs

[Daniel Müller]

This change integrates the configuration from the vmtest repository [0],
where it is currently used for testing kernel patches into the existing
configuration pulled in with an earlier patch. The result is a super set
of the configs from the two repositories.

[0]: https://github.com/kernel-patches/vmtest/tree/831ee8eb72ddb7e03babb8f7e050d52a451237aa/travis-ci/vmtest/configs

Signed-off-by: Daniel Müller <deso@posteo.net>
---
 tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest | 5 +++++
 .../selftests/bpf/configs/denylist/DENYLIST-latest.s390x     | 1 +
 2 files changed, 6 insertions(+)

diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
index 939de574..ddf8a0c5 100644
--- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
+++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
@@ -4,3 +4,8 @@ stacktrace_build_id_nmi
 stacktrace_build_id
 task_fd_query_rawtp
 varlen
+btf_dump/btf_dump: syntax
+kprobe_multi_test/bench_attach
+core_reloc/enum64val
+core_reloc/size___diff_sz
+core_reloc/type_based___diff_sz
diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
index e33cab..36574b0 100644
--- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
+++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
@@ -63,5 +63,6 @@ bpf_cookie                               # failed to open_and_load program: -524
 xdp_do_redirect                          # prog_run_max_size unexpected error: -22 (errno 22)
 send_signal                              # intermittently fails to receive signal
 select_reuseport                         # intermittently fails on new s390x setup
+tc_redirect/tc_redirect_dtime            # very flaky
 xdp_synproxy                             # JIT does not support calling kernel function                                (kfunc)
 unpriv_bpf_disabled                      # fentry
-- 
2.30.2

[PATCH bpf-next 3/3] selftests/bpf: Adjust vmtest.sh to use local kernel configuration

[Daniel Müller]

So far the vmtest.sh script, which can be used as a convenient way to
run bpf selftests, has obtained the kernel config safe to use for
testing from the libbpf/libbpf GitHub repository [0].
Given that we now have included this configuration into this very
repository, we can just consume it from here as well, eliminating the
necessity of remote accesses.
With this change we adjust the logic in the script to use the
configuration from below tools/testing/selftests/bpf/configs/ instead of
pulling it over the network.

[0]: https://github.com/libbpf/libbpf

Signed-off-by: Daniel Müller <deso@posteo.net>
---
 tools/testing/selftests/bpf/vmtest.sh | 28 ++++++++++++---------------
 1 file changed, 12 insertions(+), 16 deletions(-)

diff --git a/tools/testing/selftests/bpf/vmtest.sh b/tools/testing/selftests/bpf/vmtest.sh
index e0bb04a..fdab48 100755
--- a/tools/testing/selftests/bpf/vmtest.sh
+++ b/tools/testing/selftests/bpf/vmtest.sh
@@ -30,8 +30,7 @@ DEFAULT_COMMAND="./test_progs"
 MOUNT_DIR="mnt"
 ROOTFS_IMAGE="root.img"
 OUTPUT_DIR="$HOME/.bpf_selftests"
-KCONFIG_URL="https://raw.githubusercontent.com/libbpf/libbpf/master/travis-ci/vmtest/configs/config-latest.${ARCH}"
-KCONFIG_API_URL="https://api.github.com/repos/libbpf/libbpf/contents/travis-ci/vmtest/configs/config-latest.${ARCH}"
+KCONFIG_REL_PATH="tools/testing/selftests/bpf/configs/config-latest.${ARCH}"
 INDEX_URL="https://raw.githubusercontent.com/libbpf/ci/master/INDEX"
 NUM_COMPILE_JOBS="$(nproc)"
 LOG_FILE_BASE="$(date +"bpf_selftests.%Y-%m-%d_%H-%M-%S")"
@@ -271,20 +270,17 @@ is_rel_path()
 
 update_kconfig()
 {
-	local kconfig_file="$1"
-	local update_command="curl -sLf ${KCONFIG_URL} -o ${kconfig_file}"
-	# Github does not return the "last-modified" header when retrieving the
-	# raw contents of the file. Use the API call to get the last-modified
-	# time of the kernel config and only update the config if it has been
-	# updated after the previously cached config was created. This avoids
-	# unnecessarily compiling the kernel and selftests.
+	local kernel_checkout="$1"
+	local kconfig_file="$2"
+	local kconfig_src="${kernel_checkout}/${KCONFIG_REL_PATH}"
+	local update_command="cp ${kconfig_src} ${kconfig_file}"
 	if [[ -f "${kconfig_file}" ]]; then
-		local last_modified_date="$(curl -sL -D - "${KCONFIG_API_URL}" -o /dev/null | \
-			grep "last-modified" | awk -F ': ' '{print $2}')"
-		local remote_modified_timestamp="$(date -d "${last_modified_date}" +"%s")"
-		local local_creation_timestamp="$(stat -c %Y "${kconfig_file}")"
-
-		if [[ "${remote_modified_timestamp}" -gt "${local_creation_timestamp}" ]]; then
+		local src_modified="$(stat -c %Y "${kconfig_src}")"
+		local local_modified="$(stat -c %Y "${kconfig_file}")"
+		# Only update the config if it has been updated after the
+		# previously cached config was created. This avoids
+		# unnecessarily compiling the kernel and selftests.
+		if [[ "${src_modified}" -gt "${local_modified}" ]]; then
 			${update_command}
 		fi
 	else
@@ -372,7 +368,7 @@ main()
 
 	mkdir -p "${OUTPUT_DIR}"
 	mkdir -p "${mount_dir}"
-	update_kconfig "${kconfig_file}"
+	update_kconfig "${kernel_checkout}" "${kconfig_file}"
 
 	recompile_kernel "${kernel_checkout}" "${make_command}"
 
-- 
2.30.2

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Alexei Starovoitov]

On Tue, Jul 12, 2022 at 2:21 PM Daniel Müller <deso@posteo.net> wrote:
>
> This change integrates the libbpf maintained configurations and
> black/white lists [0] into the repository, co-located with the BPF
> selftests themselves. The only differences from the source is that we
> replaced the terms blacklist & whitelist with denylist and allowlist,
> respectively.
>
> [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
>
> Signed-off-by: Daniel Müller <deso@posteo.net>
> ---
>  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
>  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
>  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
>  .../bpf/configs/config-latest.x86_64          | 3073 +++++++++++++++++

Instead of checking in the full config please trim it to
relevant dependencies like existing selftests/bpf/config.
Otherwise every update/addition would trigger massive patches.

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Daniel Müller]

On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> On Tue, Jul 12, 2022 at 2:21 PM Daniel Müller <deso@posteo.net> wrote:
> >
> > This change integrates the libbpf maintained configurations and
> > black/white lists [0] into the repository, co-located with the BPF
> > selftests themselves. The only differences from the source is that we
> > replaced the terms blacklist & whitelist with denylist and allowlist,
> > respectively.
> >
> > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> >
> > Signed-off-by: Daniel Müller <deso@posteo.net>
> > ---
> >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> >  .../bpf/configs/config-latest.x86_64          | 3073 +++++++++++++++++
> 
> Instead of checking in the full config please trim it to
> relevant dependencies like existing selftests/bpf/config.
> Otherwise every update/addition would trigger massive patches.

Thanks for taking a look. Sure. Do we have some kind of tooling for that or are
there any suggestions on the best approach to minimize?

Thanks,
Daniel

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[sdf]

On 07/12, Daniel M�ller wrote:
> On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > >
> > > This change integrates the libbpf maintained configurations and
> > > black/white lists [0] into the repository, co-located with the BPF
> > > selftests themselves. The only differences from the source is that we
> > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > respectively.
> > >
> > > [0]  
> https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > >
> > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > ---
> > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > >  .../bpf/configs/config-latest.x86_64          | 3073  
> +++++++++++++++++
> >
> > Instead of checking in the full config please trim it to
> > relevant dependencies like existing selftests/bpf/config.
> > Otherwise every update/addition would trigger massive patches.

> Thanks for taking a look. Sure. Do we have some kind of tooling for that  
> or are
> there any suggestions on the best approach to minimize?

I would be interested to know as well if somebody knows some tricks on
how to deal with kconfig. I've spent some time yesterday manually
crafting various minimal bpf configs (for build tests), running make
olddefconfig and then verifying that all my options are still present in
the final config file.

It seems like kconfig tool can resolve some of the dependencies,
but there is a lot of if/endif that can break in non-obvious ways.
For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
won't get you CONFIG_TRACING=y in the final .config

So the only thing, for me, that helped, was to manually go through
the kconfig files trying to see what the dependencies are.
I've tried scripts/kconfig/merge_config.sh, but it doesn't
seem to bring anything new to the table..

So here is what I ended up with, I don't think it will help you that
much, but at least can highlight the moving parts (I was thinking that
maybe we can eventually put them in the CI as well to make sure all weird
configurations are build-tested?):

.config+all:CONFIG_MODULES=y
.config+all:CONFIG_HAVE_EBPF_JIT=y
.config+all:CONFIG_BPF=y
.config+all:CONFIG_BPF_SYSCALL=y
.config+all:CONFIG_BPF_JIT=y
.config+all:CONFIG_BPF_JIT_ALWAYS_ON=y
.config+all:CONFIG_BPF_JIT_DEFAULT_ON=y
.config+all:CONFIG_CGROUPS=y
.config+all:CONFIG_CGROUP_BPF=y
.config+all:CONFIG_SECURITY=y
.config+all:CONFIG_KPROBES=y
.config+all:CONFIG_TRACING=y
.config+all:CONFIG_FTRACE=y
.config+all:CONFIG_BPF_KPROBE_OVERRIDE=y
.config+all:CONFIG_BPF_EVENTS=y
.config+all:CONFIG_BPF_LSM=y
.config+all:CONFIG_NET=y
.config+all:CONFIG_INET=y
.config+all:CONFIG_NET_SCHED=y
.config+all:CONFIG_NET_CLS_ACT=y
.config+all:CONFIG_BPF_STREAM_PARSER=y
.config+all:CONFIG_NET_ACT_BPF=y
.config+all:CONFIG_NET_CLS_BPF=y
.config+all:CONFIG_TEST_BPF=y

.config-all:CONFIG_BPFILTER=n
.config-all:CONFIG_BPF=n
.config-all:CONFIG_BPF_JIT=n
.config-all:CONFIG_BPF_SYSCALL=n
.config-all:CONFIG_HAVE_EBPF_JIT=n
.config-all:CONFIG_NET_ACT_BPF=n
.config-all:CONFIG_NET_CLS_BPF=n
.config-all:CONFIG_TEST_BPF=n

.config-net:CONFIG_MODULES=y
.config-net:CONFIG_HAVE_EBPF_JIT=y
.config-net:CONFIG_BPF=y
.config-net:CONFIG_BPF_SYSCALL=y
.config-net:CONFIG_BPF_JIT=y
.config-net:CONFIG_BPF_JIT_ALWAYS_ON=y
.config-net:CONFIG_BPF_JIT_DEFAULT_ON=y
.config-net:CONFIG_CGROUPS=y
.config-net:CONFIG_CGROUP_BPF=y
.config-net:CONFIG_SECURITY=y
.config-net:CONFIG_KPROBES=y
.config-net:CONFIG_TRACING=y
.config-net:CONFIG_FTRACE=y
.config-net:CONFIG_BPF_KPROBE_OVERRIDE=y
.config-net:CONFIG_BPF_EVENTS=y
.config-net:CONFIG_BPF_LSM=y
.config-net:CONFIG_NET=n
.config-net:CONFIG_INET=n
.config-net:CONFIG_NET_SCHED=n
.config-net:CONFIG_NET_CLS_ACT=n
.config-net:CONFIG_BPF_STREAM_PARSER=n
.config-net:CONFIG_NET_ACT_BPF=n
.config-net:CONFIG_NET_CLS_BPF=n
.config-net:CONFIG_TEST_BPF=n

.config-cg-lsm:CONFIG_MODULES=y
.config-cg-lsm:CONFIG_HAVE_EBPF_JIT=y
.config-cg-lsm:CONFIG_BPF=y
.config-cg-lsm:CONFIG_BPF_SYSCALL=y
.config-cg-lsm:CONFIG_BPF_JIT=y
.config-cg-lsm:CONFIG_BPF_JIT_ALWAYS_ON=y
.config-cg-lsm:CONFIG_BPF_JIT_DEFAULT_ON=y
.config-cg-lsm:CONFIG_SECURITY=y
.config-cg-lsm:CONFIG_KPROBES=y
.config-cg-lsm:CONFIG_TRACING=n
.config-cg-lsm:CONFIG_FTRACE=n
.config-cg-lsm:CONFIG_BPF_EVENTS=n
.config-cg-lsm:CONFIG_BPF_LSM=n
.config-cg-lsm:CONFIG_CGROUP_BPF=n

.config+classic:CONFIG_MODULES=y
.config+classic:CONFIG_HAVE_EBPF_JIT=y
.config+classic:CONFIG_BPF=y
.config+classic:CONFIG_BPF_SYSCALL=n
.config+classic:CONFIG_BPF_JIT=y
.config+classic:CONFIG_BPF_JIT_ALWAYS_ON=n
.config+classic:CONFIG_BPF_JIT_DEFAULT_ON=n
.config+classic:CONFIG_SECURITY=n
.config+classic:CONFIG_KPROBES=n
.config+classic:CONFIG_TRACING=n
.config+classic:CONFIG_FTRACE=n
.config+classic:CONFIG_BPF_EVENTS=n
.config+classic:CONFIG_BPF_LSM=n
.config+classic:CONFIG_CGROUP_BPF=n
.config+classic:CONFIG_NET=y
.config+classic:CONFIG_INET=y

.config+syscall:CONFIG_MODULES=y
.config+syscall:CONFIG_HAVE_EBPF_JIT=y
.config+syscall:CONFIG_BPF=y
.config+syscall:CONFIG_BPF_SYSCALL=y
.config+syscall:CONFIG_BPF_JIT=n
.config+syscall:CONFIG_BPF_JIT_ALWAYS_ON=n
.config+syscall:CONFIG_BPF_JIT_DEFAULT_ON=n
.config+syscall:CONFIG_SECURITY=n
.config+syscall:CONFIG_KPROBES=n
.config+syscall:CONFIG_TRACING=n
.config+syscall:CONFIG_FTRACE=n
.config+syscall:CONFIG_BPF_EVENTS=n
.config+syscall:CONFIG_BPF_LSM=n
.config+syscall:CONFIG_CGROUP_BPF=n

.config-cg+lsm:CONFIG_MODULES=y
.config-cg+lsm:CONFIG_HAVE_EBPF_JIT=y
.config-cg+lsm:CONFIG_BPF=y
.config-cg+lsm:CONFIG_BPF_SYSCALL=y
.config-cg+lsm:CONFIG_BPF_JIT=y
.config-cg+lsm:CONFIG_BPF_JIT_ALWAYS_ON=y
.config-cg+lsm:CONFIG_BPF_JIT_DEFAULT_ON=y
.config-cg+lsm:CONFIG_SECURITY=y
.config-cg+lsm:CONFIG_KPROBES=y
.config-cg+lsm:CONFIG_TRACING=y
.config-cg+lsm:CONFIG_FTRACE=y
.config-cg+lsm:CONFIG_BPF_EVENTS=y
.config-cg+lsm:CONFIG_BPF_LSM=y
.config-cg+lsm:CONFIG_CGROUP_BPF=n

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Daniel Müller]

On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> On 07/12, Daniel M�ller wrote:
> > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > > >
> > > > This change integrates the libbpf maintained configurations and
> > > > black/white lists [0] into the repository, co-located with the BPF
> > > > selftests themselves. The only differences from the source is that we
> > > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > > respectively.
> > > >
> > > > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > >
> > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > ---
> > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > +++++++++++++++++
> > >
> > > Instead of checking in the full config please trim it to
> > > relevant dependencies like existing selftests/bpf/config.
> > > Otherwise every update/addition would trigger massive patches.
> 
> > Thanks for taking a look. Sure. Do we have some kind of tooling for that
> > or are
> > there any suggestions on the best approach to minimize?
> 
> I would be interested to know as well if somebody knows some tricks on
> how to deal with kconfig. I've spent some time yesterday manually
> crafting various minimal bpf configs (for build tests), running make
> olddefconfig and then verifying that all my options are still present in
> the final config file.
> 
> It seems like kconfig tool can resolve some of the dependencies,
> but there is a lot of if/endif that can break in non-obvious ways.
> For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
> won't get you CONFIG_TRACING=y in the final .config
> 
> So the only thing, for me, that helped, was to manually go through
> the kconfig files trying to see what the dependencies are.
> I've tried scripts/kconfig/merge_config.sh, but it doesn't
> seem to bring anything new to the table..
> 
> So here is what I ended up with, I don't think it will help you that
> much, but at least can highlight the moving parts (I was thinking that
> maybe we can eventually put them in the CI as well to make sure all weird
> configurations are build-tested?):

[...]

I *think* that make savedefconfig [0] is the way to go, at least for my use
case. That cuts down the config file to <350 lines. However, it does change some
configurations from 'm' to 'y', which I can't say I quite understand or would
have expected (but perhaps minimal implies no modules or similar; I haven't
investigated).
I am still verifying that the result is working as expected, though.

Thanks,
Daniel

[0] https://lwn.net/Articles/397363/

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Andrii Nakryiko]

On Tue, Jul 12, 2022 at 4:01 PM Daniel Müller <deso@posteo.net> wrote:
>
> On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> > On 07/12, Daniel M�ller wrote:
> > > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > > > >
> > > > > This change integrates the libbpf maintained configurations and
> > > > > black/white lists [0] into the repository, co-located with the BPF
> > > > > selftests themselves. The only differences from the source is that we
> > > > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > > > respectively.
> > > > >
> > > > > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > > >
> > > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > > ---
> > > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > > +++++++++++++++++
> > > >
> > > > Instead of checking in the full config please trim it to
> > > > relevant dependencies like existing selftests/bpf/config.
> > > > Otherwise every update/addition would trigger massive patches.
> >
> > > Thanks for taking a look. Sure. Do we have some kind of tooling for that
> > > or are
> > > there any suggestions on the best approach to minimize?
> >
> > I would be interested to know as well if somebody knows some tricks on
> > how to deal with kconfig. I've spent some time yesterday manually
> > crafting various minimal bpf configs (for build tests), running make
> > olddefconfig and then verifying that all my options are still present in
> > the final config file.
> >
> > It seems like kconfig tool can resolve some of the dependencies,
> > but there is a lot of if/endif that can break in non-obvious ways.
> > For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
> > won't get you CONFIG_TRACING=y in the final .config
> >
> > So the only thing, for me, that helped, was to manually go through
> > the kconfig files trying to see what the dependencies are.
> > I've tried scripts/kconfig/merge_config.sh, but it doesn't
> > seem to bring anything new to the table..
> >
> > So here is what I ended up with, I don't think it will help you that
> > much, but at least can highlight the moving parts (I was thinking that
> > maybe we can eventually put them in the CI as well to make sure all weird
> > configurations are build-tested?):
>
> [...]
>
> I *think* that make savedefconfig [0] is the way to go, at least for my use
> case. That cuts down the config file to <350 lines. However, it does change some
> configurations from 'm' to 'y', which I can't say I quite understand or would
> have expected (but perhaps minimal implies no modules or similar; I haven't
> investigated).
> I am still verifying that the result is working as expected, though.

I think ideally we'd do defconfig first, then append whatever is in
selftests/bpf/config, do olddefconfig to fill in all the unspecified
options, and then use the result as the config. Yes, that requires
that selftests/bpf/config has some of the dependent values specified,
which is an annoying mostly one-time thing, but I think it's
beneficial to all the new BPF users, because it *really* shows what
needs to be added to kernel config to make everything work. We can
also split it into BPF-specific and non-BPF (dependencies) configs, if
that is cleaner.

Also, I don't think we should move 4.9.0 and 5.5.0 lists here, let's
keep them in libbpf CI, they are very specific there. Here we should
only maintain the latest per-arch configs and allow/deny lists only.

>
> Thanks,
> Daniel
>
> [0] https://lwn.net/Articles/397363/

Re: [PATCH bpf-next 2/3] selftests/bpf: Integrate vmtest configs

[Andrii Nakryiko]

On Tue, Jul 12, 2022 at 2:21 PM Daniel Müller <deso@posteo.net> wrote:
>
> This change integrates the configuration from the vmtest repository [0],
> where it is currently used for testing kernel patches into the existing
> configuration pulled in with an earlier patch. The result is a super set
> of the configs from the two repositories.
>
> [0]: https://github.com/kernel-patches/vmtest/tree/831ee8eb72ddb7e03babb8f7e050d52a451237aa/travis-ci/vmtest/configs
>
> Signed-off-by: Daniel Müller <deso@posteo.net>
> ---
>  tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest | 5 +++++
>  .../selftests/bpf/configs/denylist/DENYLIST-latest.s390x     | 1 +
>  2 files changed, 6 insertions(+)
>
> diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> index 939de574..ddf8a0c5 100644
> --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> @@ -4,3 +4,8 @@ stacktrace_build_id_nmi
>  stacktrace_build_id
>  task_fd_query_rawtp
>  varlen
> +btf_dump/btf_dump: syntax
> +kprobe_multi_test/bench_attach
> +core_reloc/enum64val
> +core_reloc/size___diff_sz
> +core_reloc/type_based___diff_sz

I don't think any of these are necessary anymore. Some of them were
due to nightly Clang was stale.

> diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> index e33cab..36574b0 100644
> --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> @@ -63,5 +63,6 @@ bpf_cookie                               # failed to open_and_load program: -524
>  xdp_do_redirect                          # prog_run_max_size unexpected error: -22 (errno 22)
>  send_signal                              # intermittently fails to receive signal
>  select_reuseport                         # intermittently fails on new s390x setup
> +tc_redirect/tc_redirect_dtime            # very flaky

same for this, yes it's flaky, but this shouldn't be in this list (I'd
rather people actually fix the flakiness, of course). These configs
should be "known not working" test cases (e.g., like BPF
trampoline-based for s390x, that feature is just not implemented). But
flaky tests should go here, they should be ideally fixed and not be
blessed officially to be ignored.

>  xdp_synproxy                             # JIT does not support calling kernel function                                (kfunc)
>  unpriv_bpf_disabled                      # fentry
> --
> 2.30.2
>

Re: [PATCH bpf-next 2/3] selftests/bpf: Integrate vmtest configs

[Daniel Müller]

On Wed, Jul 13, 2022 at 10:07:02PM -0700, Andrii Nakryiko wrote:
> On Tue, Jul 12, 2022 at 2:21 PM Daniel Müller <deso@posteo.net> wrote:
> >
> > This change integrates the configuration from the vmtest repository [0],
> > where it is currently used for testing kernel patches into the existing
> > configuration pulled in with an earlier patch. The result is a super set
> > of the configs from the two repositories.
> >
> > [0]: https://github.com/kernel-patches/vmtest/tree/831ee8eb72ddb7e03babb8f7e050d52a451237aa/travis-ci/vmtest/configs
> >
> > Signed-off-by: Daniel Müller <deso@posteo.net>
> > ---
> >  tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest | 5 +++++
> >  .../selftests/bpf/configs/denylist/DENYLIST-latest.s390x     | 1 +
> >  2 files changed, 6 insertions(+)
> >
> > diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > index 939de574..ddf8a0c5 100644
> > --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > @@ -4,3 +4,8 @@ stacktrace_build_id_nmi
> >  stacktrace_build_id
> >  task_fd_query_rawtp
> >  varlen
> > +btf_dump/btf_dump: syntax
> > +kprobe_multi_test/bench_attach
> > +core_reloc/enum64val
> > +core_reloc/size___diff_sz
> > +core_reloc/type_based___diff_sz
> 
> I don't think any of these are necessary anymore. Some of them were
> due to nightly Clang was stale.
> 
> > diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > index e33cab..36574b0 100644
> > --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > @@ -63,5 +63,6 @@ bpf_cookie                               # failed to open_and_load program: -524
> >  xdp_do_redirect                          # prog_run_max_size unexpected error: -22 (errno 22)
> >  send_signal                              # intermittently fails to receive signal
> >  select_reuseport                         # intermittently fails on new s390x setup
> > +tc_redirect/tc_redirect_dtime            # very flaky
> 
> same for this, yes it's flaky, but this shouldn't be in this list (I'd
> rather people actually fix the flakiness, of course). These configs
> should be "known not working" test cases (e.g., like BPF
> trampoline-based for s390x, that feature is just not implemented). But
> flaky tests should go here, they should be ideally fixed and not be
> blessed officially to be ignored.

I can remove this change from the set. But really from my perspective
the entire patch set's concern is not with cleaning up any of the lists
-- it is about merging and integrating existing configuration from two
others repositories into this one, while preserving what has been done
and why in a way that can be followed when looking back at repository
histories.
My observation has been that at least on x86_64, none of the denied
tests caused actual failures when run. And yet, that is best cleaned up
subsequently if it were for me.

Thanks,
Daniel

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Daniel Müller]

On Wed, Jul 13, 2022 at 09:48:32PM -0700, Andrii Nakryiko wrote:
> On Tue, Jul 12, 2022 at 4:01 PM Daniel Müller <deso@posteo.net> wrote:
> >
> > On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> > > On 07/12, Daniel M�ller wrote:
> > > > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > > > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > > > > >
> > > > > > This change integrates the libbpf maintained configurations and
> > > > > > black/white lists [0] into the repository, co-located with the BPF
> > > > > > selftests themselves. The only differences from the source is that we
> > > > > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > > > > respectively.
> > > > > >
> > > > > > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > > > >
> > > > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > > > ---
> > > > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > > > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > > > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > > > +++++++++++++++++
> > > > >
> > > > > Instead of checking in the full config please trim it to
> > > > > relevant dependencies like existing selftests/bpf/config.
> > > > > Otherwise every update/addition would trigger massive patches.
> > >
> > > > Thanks for taking a look. Sure. Do we have some kind of tooling for that
> > > > or are
> > > > there any suggestions on the best approach to minimize?
> > >
> > > I would be interested to know as well if somebody knows some tricks on
> > > how to deal with kconfig. I've spent some time yesterday manually
> > > crafting various minimal bpf configs (for build tests), running make
> > > olddefconfig and then verifying that all my options are still present in
> > > the final config file.
> > >
> > > It seems like kconfig tool can resolve some of the dependencies,
> > > but there is a lot of if/endif that can break in non-obvious ways.
> > > For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
> > > won't get you CONFIG_TRACING=y in the final .config
> > >
> > > So the only thing, for me, that helped, was to manually go through
> > > the kconfig files trying to see what the dependencies are.
> > > I've tried scripts/kconfig/merge_config.sh, but it doesn't
> > > seem to bring anything new to the table..
> > >
> > > So here is what I ended up with, I don't think it will help you that
> > > much, but at least can highlight the moving parts (I was thinking that
> > > maybe we can eventually put them in the CI as well to make sure all weird
> > > configurations are build-tested?):
> >
> > [...]
> >
> > I *think* that make savedefconfig [0] is the way to go, at least for my use
> > case. That cuts down the config file to <350 lines. However, it does change some
> > configurations from 'm' to 'y', which I can't say I quite understand or would
> > have expected (but perhaps minimal implies no modules or similar; I haven't
> > investigated).
> > I am still verifying that the result is working as expected, though.
> 
> I think ideally we'd do defconfig first, then append whatever is in
> selftests/bpf/config, do olddefconfig to fill in all the unspecified
> options, and then use the result as the config. Yes, that requires
> that selftests/bpf/config has some of the dependent values specified,
> which is an annoying mostly one-time thing, but I think it's
> beneficial to all the new BPF users, because it *really* shows what
> needs to be added to kernel config to make everything work. We can
> also split it into BPF-specific and non-BPF (dependencies) configs, if
> that is cleaner.

Agreed, we should do that eventually. But let's not put everything into
this patch set, which was never intended to rework everything we have,
okay? It contains a few steps towards where we want to head.

If we start diverging massively now, while also moving configurations
between multiple repositories, we end up with a mess of a history that
will be hard to follow.

So unless there exist very strong arguments forcing us to do that here
and now (such as us regressing on one front, which I don't see here),
I'd rather we go about it at a later point after other check boxes have
been ticked. What do you think?

> Also, I don't think we should move 4.9.0 and 5.5.0 lists here, let's
> keep them in libbpf CI, they are very specific there. Here we should
> only maintain the latest per-arch configs and allow/deny lists only.

Sounds good, will remove them.

Thanks,
Daniel

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Daniel Müller]

On Tue, Jul 12, 2022 at 11:01:38PM +0000, Daniel Müller wrote:
> On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> > On 07/12, Daniel M�ller wrote:
> > > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > > > >
> > > > > This change integrates the libbpf maintained configurations and
> > > > > black/white lists [0] into the repository, co-located with the BPF
> > > > > selftests themselves. The only differences from the source is that we
> > > > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > > > respectively.
> > > > >
> > > > > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > > >
> > > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > > ---
> > > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > > +++++++++++++++++
> > > >
> > > > Instead of checking in the full config please trim it to
> > > > relevant dependencies like existing selftests/bpf/config.
> > > > Otherwise every update/addition would trigger massive patches.
> > 
> > > Thanks for taking a look. Sure. Do we have some kind of tooling for that
> > > or are
> > > there any suggestions on the best approach to minimize?
> > 
> > I would be interested to know as well if somebody knows some tricks on
> > how to deal with kconfig. I've spent some time yesterday manually
> > crafting various minimal bpf configs (for build tests), running make
> > olddefconfig and then verifying that all my options are still present in
> > the final config file.
> > 
> > It seems like kconfig tool can resolve some of the dependencies,
> > but there is a lot of if/endif that can break in non-obvious ways.
> > For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
> > won't get you CONFIG_TRACING=y in the final .config
> > 
> > So the only thing, for me, that helped, was to manually go through
> > the kconfig files trying to see what the dependencies are.
> > I've tried scripts/kconfig/merge_config.sh, but it doesn't
> > seem to bring anything new to the table..
> > 
> > So here is what I ended up with, I don't think it will help you that
> > much, but at least can highlight the moving parts (I was thinking that
> > maybe we can eventually put them in the CI as well to make sure all weird
> > configurations are build-tested?):
> 
> [...]
> 
> I *think* that make savedefconfig [0] is the way to go, at least for my use
> case. That cuts down the config file to <350 lines. However, it does change some
> configurations from 'm' to 'y', which I can't say I quite understand or would
> have expected (but perhaps minimal implies no modules or similar; I haven't
> investigated).
> I am still verifying that the result is working as expected, though.

Just to add to that...it turns out that while savedefconfig works, it can
produce faulty configs by silently disabling functionality when used with an
architecture that is not the native one (i.e., make ARCH=xxx savedefconfig).

The reason being that some options may directly or indirectly dependent on the
compiler supporting certain features. But if one "just" wants to minimize a
configuration for a different architecture, a fully blown cross-compilation
toolchain may not be present. So the feature availability check fails and,
consequently, options may be disabled.

That so happened in my case where we want to have a config for s390. I noticed
that BPF_JIT (which was set in the to-be-minimized config), for example, always
ended up being disabled. The reason turned out to be that it transitively
depends on MARCH_Z196 which "depends on $(cc-option,-march=z196)", which, well,
my compiler does not know about.

I am unsure if there is a better way than to manually fiddle around to get those
options back or whether I was using it wrong. I also have not come across any
warnings about such behavior, but perhaps I missed it; or it's "too obvious".

Daniel

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Andrii Nakryiko]

On Thu, Jul 14, 2022 at 7:36 AM Daniel Müller <deso@posteo.net> wrote:
>
> On Wed, Jul 13, 2022 at 09:48:32PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jul 12, 2022 at 4:01 PM Daniel Müller <deso@posteo.net> wrote:
> > >
> > > On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> > > > On 07/12, Daniel M�ller wrote:
> > > > > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei Starovoitov wrote:
> > > > > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller <deso@posteo.net> wrote:
> > > > > > >
> > > > > > > This change integrates the libbpf maintained configurations and
> > > > > > > black/white lists [0] into the repository, co-located with the BPF
> > > > > > > selftests themselves. The only differences from the source is that we
> > > > > > > replaced the terms blacklist & whitelist with denylist and allowlist,
> > > > > > > respectively.
> > > > > > >
> > > > > > > [0] https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > > > > >
> > > > > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > > > > ---
> > > > > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > > > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > > > > >  .../selftests/bpf/configs/config-latest.s390x | 2711 +++++++++++++++
> > > > > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > > > > +++++++++++++++++
> > > > > >
> > > > > > Instead of checking in the full config please trim it to
> > > > > > relevant dependencies like existing selftests/bpf/config.
> > > > > > Otherwise every update/addition would trigger massive patches.
> > > >
> > > > > Thanks for taking a look. Sure. Do we have some kind of tooling for that
> > > > > or are
> > > > > there any suggestions on the best approach to minimize?
> > > >
> > > > I would be interested to know as well if somebody knows some tricks on
> > > > how to deal with kconfig. I've spent some time yesterday manually
> > > > crafting various minimal bpf configs (for build tests), running make
> > > > olddefconfig and then verifying that all my options are still present in
> > > > the final config file.
> > > >
> > > > It seems like kconfig tool can resolve some of the dependencies,
> > > > but there is a lot of if/endif that can break in non-obvious ways.
> > > > For example, putting CONFIG_TRACING=y and doing 'make olddefconfig'
> > > > won't get you CONFIG_TRACING=y in the final .config
> > > >
> > > > So the only thing, for me, that helped, was to manually go through
> > > > the kconfig files trying to see what the dependencies are.
> > > > I've tried scripts/kconfig/merge_config.sh, but it doesn't
> > > > seem to bring anything new to the table..
> > > >
> > > > So here is what I ended up with, I don't think it will help you that
> > > > much, but at least can highlight the moving parts (I was thinking that
> > > > maybe we can eventually put them in the CI as well to make sure all weird
> > > > configurations are build-tested?):
> > >
> > > [...]
> > >
> > > I *think* that make savedefconfig [0] is the way to go, at least for my use
> > > case. That cuts down the config file to <350 lines. However, it does change some
> > > configurations from 'm' to 'y', which I can't say I quite understand or would
> > > have expected (but perhaps minimal implies no modules or similar; I haven't
> > > investigated).
> > > I am still verifying that the result is working as expected, though.
> >
> > I think ideally we'd do defconfig first, then append whatever is in
> > selftests/bpf/config, do olddefconfig to fill in all the unspecified
> > options, and then use the result as the config. Yes, that requires
> > that selftests/bpf/config has some of the dependent values specified,
> > which is an annoying mostly one-time thing, but I think it's
> > beneficial to all the new BPF users, because it *really* shows what
> > needs to be added to kernel config to make everything work. We can
> > also split it into BPF-specific and non-BPF (dependencies) configs, if
> > that is cleaner.
>
> Agreed, we should do that eventually. But let's not put everything into
> this patch set, which was never intended to rework everything we have,
> okay? It contains a few steps towards where we want to head.
>
> If we start diverging massively now, while also moving configurations
> between multiple repositories, we end up with a mess of a history that
> will be hard to follow.
>
> So unless there exist very strong arguments forcing us to do that here
> and now (such as us regressing on one front, which I don't see here),
> I'd rather we go about it at a later point after other check boxes have
> been ticked. What do you think?
>

You mean the part about checking in huge Kconfigs for x86 and s390x? I
don't think we should do that as a first step. Yes it's an annoying
(but also very important) part to figure out the minimal set of added
configs on top of default config, but I think we should do that from
the beginning instead of polluting Git history with massive configs.
It will also keep selftests/bpf/config "honest" instead of putting it
on new users to figure out other missed or dependent configs by
themselves.

With s390x config, at least, I hope that Ilya can ease the pain,
especially that he was the one who came up with that config in the
first place (cc'ed Ilya).


> > Also, I don't think we should move 4.9.0 and 5.5.0 lists here, let's
> > keep them in libbpf CI, they are very specific there. Here we should
> > only maintain the latest per-arch configs and allow/deny lists only.
>
> Sounds good, will remove them.
>
> Thanks,
> Daniel

Re: [PATCH bpf-next 2/3] selftests/bpf: Integrate vmtest configs

[Andrii Nakryiko]

On Thu, Jul 14, 2022 at 7:04 AM Daniel Müller <deso@posteo.net> wrote:
>
> On Wed, Jul 13, 2022 at 10:07:02PM -0700, Andrii Nakryiko wrote:
> > On Tue, Jul 12, 2022 at 2:21 PM Daniel Müller <deso@posteo.net> wrote:
> > >
> > > This change integrates the configuration from the vmtest repository [0],
> > > where it is currently used for testing kernel patches into the existing
> > > configuration pulled in with an earlier patch. The result is a super set
> > > of the configs from the two repositories.
> > >
> > > [0]: https://github.com/kernel-patches/vmtest/tree/831ee8eb72ddb7e03babb8f7e050d52a451237aa/travis-ci/vmtest/configs
> > >
> > > Signed-off-by: Daniel Müller <deso@posteo.net>
> > > ---
> > >  tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest | 5 +++++
> > >  .../selftests/bpf/configs/denylist/DENYLIST-latest.s390x     | 1 +
> > >  2 files changed, 6 insertions(+)
> > >
> > > diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > > index 939de574..ddf8a0c5 100644
> > > --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > > +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest
> > > @@ -4,3 +4,8 @@ stacktrace_build_id_nmi
> > >  stacktrace_build_id
> > >  task_fd_query_rawtp
> > >  varlen
> > > +btf_dump/btf_dump: syntax
> > > +kprobe_multi_test/bench_attach
> > > +core_reloc/enum64val
> > > +core_reloc/size___diff_sz
> > > +core_reloc/type_based___diff_sz
> >
> > I don't think any of these are necessary anymore. Some of them were
> > due to nightly Clang was stale.
> >
> > > diff --git a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > > index e33cab..36574b0 100644
> > > --- a/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > > +++ b/tools/testing/selftests/bpf/configs/denylist/DENYLIST-latest.s390x
> > > @@ -63,5 +63,6 @@ bpf_cookie                               # failed to open_and_load program: -524
> > >  xdp_do_redirect                          # prog_run_max_size unexpected error: -22 (errno 22)
> > >  send_signal                              # intermittently fails to receive signal
> > >  select_reuseport                         # intermittently fails on new s390x setup
> > > +tc_redirect/tc_redirect_dtime            # very flaky
> >
> > same for this, yes it's flaky, but this shouldn't be in this list (I'd
> > rather people actually fix the flakiness, of course). These configs
> > should be "known not working" test cases (e.g., like BPF
> > trampoline-based for s390x, that feature is just not implemented). But
> > flaky tests should go here, they should be ideally fixed and not be
> > blessed officially to be ignored.
>
> I can remove this change from the set. But really from my perspective
> the entire patch set's concern is not with cleaning up any of the lists
> -- it is about merging and integrating existing configuration from two
> others repositories into this one, while preserving what has been done
> and why in a way that can be followed when looking back at repository
> histories.
> My observation has been that at least on x86_64, none of the denied
> tests caused actual failures when run. And yet, that is best cleaned up
> subsequently if it were for me.

My point is that we shouldn't add them to selftests/bpf first just to
clean up later. We can leave those custom additions as is in CI repos
(either way we need to allow repos to augment "default" configs/lists)
and clean that up there.

Generally, allow/deny lists in selftests/bpf should be "authoritative"
in the sense that we know that those tests are not supposed to work
(right now or at all), we can even teach test_progs to ignore those by
default (now that denylist is collocated with test_progs). Anything
that's flaky shouldn't be added there, flakiness should be eliminated.
With those flaky tests I added in libbpf CI I was the only one
suffering from them, so sometimes I opted to just blacklist them for
my own sanity.

But now we should all share this pain and work together on improving tests! ;)

>
> Thanks,
> Daniel

Re: [PATCH bpf-next 1/3] selftests/bpf: Copy over libbpf configs

[Ilya Leoshkevich]

On Thu, 2022-07-14 at 11:20 -0700, Andrii Nakryiko wrote:
> On Thu, Jul 14, 2022 at 7:36 AM Daniel Müller <deso@posteo.net>
> wrote:
> > 
> > On Wed, Jul 13, 2022 at 09:48:32PM -0700, Andrii Nakryiko wrote:
> > > On Tue, Jul 12, 2022 at 4:01 PM Daniel Müller <deso@posteo.net>
> > > wrote:
> > > > 
> > > > On Tue, Jul 12, 2022 at 03:33:26PM -0700, sdf@google.com wrote:
> > > > > On 07/12, Daniel M�ller wrote:
> > > > > > On Tue, Jul 12, 2022 at 02:27:47PM -0700, Alexei
> > > > > > Starovoitov wrote:
> > > > > > > On Tue, Jul 12, 2022 at 2:21 PM Daniel M�ller
> > > > > > > <deso@posteo.net> wrote:
> > > > > > > > 
> > > > > > > > This change integrates the libbpf maintained
> > > > > > > > configurations and
> > > > > > > > black/white lists [0] into the repository, co-located
> > > > > > > > with the BPF
> > > > > > > > selftests themselves. The only differences from the
> > > > > > > > source is that we
> > > > > > > > replaced the terms blacklist & whitelist with denylist
> > > > > > > > and allowlist,
> > > > > > > > respectively.
> > > > > > > > 
> > > > > > > > [0]
> > > > > > > > https://github.com/libbpf/libbpf/tree/20f03302350a4143825cedcbd210c4d7112c1898/travis-ci/vmtest/configs
> > > > > > > > 
> > > > > > > > Signed-off-by: Daniel M�ller <deso@posteo.net>
> > > > > > > > ---
> > > > > > > >  .../bpf/configs/allowlist/ALLOWLIST-4.9.0     |    8 +
> > > > > > > >  .../bpf/configs/allowlist/ALLOWLIST-5.5.0     |   55 +
> > > > > > > >  .../selftests/bpf/configs/config-latest.s390x | 2711
> > > > > > > > +++++++++++++++
> > > > > > > >  .../bpf/configs/config-latest.x86_64          | 3073
> > > > > > +++++++++++++++++
> > > > > > > 
> > > > > > > Instead of checking in the full config please trim it to
> > > > > > > relevant dependencies like existing selftests/bpf/config.
> > > > > > > Otherwise every update/addition would trigger massive
> > > > > > > patches.
> > > > > 
> > > > > > Thanks for taking a look. Sure. Do we have some kind of
> > > > > > tooling for that
> > > > > > or are
> > > > > > there any suggestions on the best approach to minimize?
> > > > > 
> > > > > I would be interested to know as well if somebody knows some
> > > > > tricks on
> > > > > how to deal with kconfig. I've spent some time yesterday
> > > > > manually
> > > > > crafting various minimal bpf configs (for build tests),
> > > > > running make
> > > > > olddefconfig and then verifying that all my options are still
> > > > > present in
> > > > > the final config file.
> > > > > 
> > > > > It seems like kconfig tool can resolve some of the
> > > > > dependencies,
> > > > > but there is a lot of if/endif that can break in non-obvious
> > > > > ways.
> > > > > For example, putting CONFIG_TRACING=y and doing 'make
> > > > > olddefconfig'
> > > > > won't get you CONFIG_TRACING=y in the final .config
> > > > > 
> > > > > So the only thing, for me, that helped, was to manually go
> > > > > through
> > > > > the kconfig files trying to see what the dependencies are.
> > > > > I've tried scripts/kconfig/merge_config.sh, but it doesn't
> > > > > seem to bring anything new to the table..
> > > > > 
> > > > > So here is what I ended up with, I don't think it will help
> > > > > you that
> > > > > much, but at least can highlight the moving parts (I was
> > > > > thinking that
> > > > > maybe we can eventually put them in the CI as well to make
> > > > > sure all weird
> > > > > configurations are build-tested?):
> > > > 
> > > > [...]
> > > > 
> > > > I *think* that make savedefconfig [0] is the way to go, at
> > > > least for my use
> > > > case. That cuts down the config file to <350 lines. However, it
> > > > does change some
> > > > configurations from 'm' to 'y', which I can't say I quite
> > > > understand or would
> > > > have expected (but perhaps minimal implies no modules or
> > > > similar; I haven't
> > > > investigated).
> > > > I am still verifying that the result is working as expected,
> > > > though.
> > > 
> > > I think ideally we'd do defconfig first, then append whatever is
> > > in
> > > selftests/bpf/config, do olddefconfig to fill in all the
> > > unspecified
> > > options, and then use the result as the config. Yes, that
> > > requires
> > > that selftests/bpf/config has some of the dependent values
> > > specified,
> > > which is an annoying mostly one-time thing, but I think it's
> > > beneficial to all the new BPF users, because it *really* shows
> > > what
> > > needs to be added to kernel config to make everything work. We
> > > can
> > > also split it into BPF-specific and non-BPF (dependencies)
> > > configs, if
> > > that is cleaner.
> > 
> > Agreed, we should do that eventually. But let's not put everything
> > into
> > this patch set, which was never intended to rework everything we
> > have,
> > okay? It contains a few steps towards where we want to head.
> > 
> > If we start diverging massively now, while also moving
> > configurations
> > between multiple repositories, we end up with a mess of a history
> > that
> > will be hard to follow.
> > 
> > So unless there exist very strong arguments forcing us to do that
> > here
> > and now (such as us regressing on one front, which I don't see
> > here),
> > I'd rather we go about it at a later point after other check boxes
> > have
> > been ticked. What do you think?
> > 
> 
> You mean the part about checking in huge Kconfigs for x86 and s390x?
> I
> don't think we should do that as a first step. Yes it's an annoying
> (but also very important) part to figure out the minimal set of added
> configs on top of default config, but I think we should do that from
> the beginning instead of polluting Git history with massive configs.
> It will also keep selftests/bpf/config "honest" instead of putting it
> on new users to figure out other missed or dependent configs by
> themselves.
> 
> With s390x config, at least, I hope that Ilya can ease the pain,
> especially that he was the one who came up with that config in the
> first place (cc'ed Ilya).
> 
> 
> > > Also, I don't think we should move 4.9.0 and 5.5.0 lists here,
> > > let's
> > > keep them in libbpf CI, they are very specific there. Here we
> > > should
> > > only maintain the latest per-arch configs and allow/deny lists
> > > only.
> > 
> > Sounds good, will remove them.
> > 
> > Thanks,
> > Daniel

Hi!

For myself, I maintain this as a defconfig. I published the full config
for the CI, because it was the case for x86 already.

Here is what I currently have:

CONFIG_9P_FS=y
CONFIG_ALTIVEC=y
CONFIG_AUDIT=y
CONFIG_BLK_CGROUP=y
# CONFIG_BLK_DEBUG_FS is not set
# CONFIG_BLK_DEV_BSG is not set
CONFIG_BLK_DEV_INITRD=y
CONFIG_BLK_DEV_IO_TRACE=y
CONFIG_BLK_DEV_LOOP=y
CONFIG_BLK_DEV_RAM=y
# CONFIG_BLK_DEV_XPRAM is not set
CONFIG_BONDING=y
CONFIG_BPFILTER_UMH=y
CONFIG_BPFILTER=y
CONFIG_BPF_JIT_ALWAYS_ON=y
CONFIG_BPF_JIT=y
CONFIG_BPF_LIRC_MODE2=y
CONFIG_BPF_LSM=y
CONFIG_BPF_PRELOAD_UMD=y
CONFIG_BPF_PRELOAD=y
CONFIG_BPF_STREAM_PARSER=y
CONFIG_BPF_SYSCALL=y
# CONFIG_BPF_UNPRIV_DEFAULT_OFF is not set
CONFIG_CGROUP_BPF=y
CONFIG_CGROUP_CPUACCT=y
CONFIG_CGROUP_DEVICE=y
CONFIG_CGROUP_FREEZER=y
CONFIG_CGROUP_HUGETLB=y
CONFIG_CGROUP_NET_CLASSID=y
CONFIG_CGROUP_PERF=y
CONFIG_CGROUP_PIDS=y
CONFIG_CGROUP_SCHED=y
CONFIG_CGROUPS=y
CONFIG_CHECKPOINT_RESTORE=y
CONFIG_CLEANCACHE=y
# CONFIG_COMPAT_BRK is not set
CONFIG_COMPAT=y
# CONFIG_CPU_IDLE is not set
# CONFIG_CPU_ISOLATION is not set
CONFIG_CPU_LITTLE_ENDIAN=y
CONFIG_CPUSETS=y
CONFIG_CRASH_DUMP=y
CONFIG_CRYPTO_AES=y
CONFIG_CRYPTO_CBC=y
# CONFIG_CRYPTO_DEV_VIRTIO is not set
CONFIG_CRYPTO_HMAC=y
CONFIG_CRYPTO_SHA256=y
CONFIG_CRYPTO_USER_API_HASH=y
CONFIG_CRYPTO_USER_API_RNG=y
CONFIG_CRYPTO_USER_API_SKCIPHER=y
# CONFIG_CTCM is not set
# CONFIG_DASD is not set
# CONFIG_DCSSBLK is not set
CONFIG_DEBUG_ATOMIC_SLEEP=y
# CONFIG_DEBUG_FORCE_WEAK_PER_CPU is not set
CONFIG_DEBUG_INFO_BTF=y
CONFIG_DEBUG_INFO_DWARF4=y
CONFIG_DEBUG_INFO=y
CONFIG_DEBUG_LIST=y
CONFIG_DEBUG_LOCKDEP=y
CONFIG_DEBUG_NOTIFIERS=y
CONFIG_DEBUG_PAGEALLOC=y
CONFIG_DEBUG_SECTION_MISMATCH=y
CONFIG_DEBUG_SG=y
CONFIG_DETECT_HUNG_TASK=y
CONFIG_DEVKMEM=y
CONFIG_DEVTMPFS=y
# CONFIG_EADM_SCH is not set
# CONFIG_ETHERNET is not set
CONFIG_EXPERT=y
CONFIG_EXPOLINE=y
CONFIG_EXT4_FS_POSIX_ACL=y
CONFIG_EXT4_FS_SECURITY=y
CONFIG_EXT4_FS=y
CONFIG_FANOTIFY=y
CONFIG_FRAME_WARN=1024
CONFIG_FRONTSWAP=y
CONFIG_FTRACE_SYSCALLS=y
CONFIG_FUNCTION_PROFILER=y
CONFIG_GDB_SCRIPTS=y
CONFIG_GENEVE=y
CONFIG_HEADERS_INSTALL=y
CONFIG_HIBERNATION=y
CONFIG_HIGH_RES_TIMERS=y
# CONFIG_HMC_DRV is not set
CONFIG_HUGETLBFS=y
CONFIG_HVC_CONSOLE=y
# CONFIG_HVC_IUCV is not set
CONFIG_HW_RANDOM=y
CONFIG_HZ_100=y
CONFIG_IDLE_PAGE_TRACKING=y
CONFIG_IKCONFIG_PROC=y
CONFIG_IKCONFIG=y
CONFIG_IKHEADERS=y
CONFIG_IMA=y
CONFIG_INET6_ESP=y
CONFIG_INET_ESP=y
CONFIG_INET=y
CONFIG_IP_ADVANCED_ROUTER=y
CONFIG_IP_MULTICAST=y
CONFIG_IP_MULTIPLE_TABLES=y
CONFIG_IP_NF_FILTER=y
CONFIG_IP_NF_IPTABLES=y
CONFIG_IPV6_GRE=y
CONFIG_IPV6_SEG6_LWTUNNEL=y
CONFIG_IPVLAN=y
# CONFIG_IUCV is not set
CONFIG_JUMP_LABEL=y
CONFIG_KERNEL_UNCOMPRESSED=y
CONFIG_KEYS=y
CONFIG_KPROBES=y
CONFIG_KSM=y
CONFIG_KUNIT=y
CONFIG_LATENCYTOP=y
CONFIG_LIRC=y
CONFIG_LIVEPATCH=y
CONFIG_LOCK_STAT=y
CONFIG_LWTUNNEL=y
CONFIG_MACINTOSH_DRIVERS=y
CONFIG_MAC_PARTITION=y
CONFIG_MACVLAN=y
CONFIG_MACVTAP=y
CONFIG_MAGIC_SYSRQ=y
CONFIG_MEMCG_SWAP=y
CONFIG_MEMCG=y
CONFIG_MEMORY_HOTPLUG=y
CONFIG_MEMORY_HOTREMOVE=y
# CONFIG_MISC_FILESYSTEMS is not set
CONFIG_MODULES=y
CONFIG_MODULE_UNLOAD=y
# CONFIG_MONWRITER is not set
CONFIG_MPLS_IPTUNNEL=y
CONFIG_MPLS_ROUTING=y
CONFIG_MPLS=y
# CONFIG_MSDOS_PARTITION is not set
CONFIG_NAMESPACES=y
CONFIG_NET_9P_VIRTIO=y
CONFIG_NET_9P=y
CONFIG_NET_ACT_BPF=y
CONFIG_NET_ACT_GACT=y
CONFIG_NET_CLS_ACT=y
CONFIG_NET_CLS_BPF=y
CONFIG_NET_CLS_FLOWER=y
CONFIG_NETDEVICES=y
CONFIG_NETDEVSIM=y
CONFIG_NETFILTER_XT_MATCH_BPF=y
CONFIG_NETFILTER_XT_TARGET_MARK=y
CONFIG_NETFILTER=y
CONFIG_NET_FOU_IP_TUNNELS=y
CONFIG_NET_IPGRE_DEMUX=y
CONFIG_NET_IPGRE=y
CONFIG_NET_IPIP=y
CONFIG_NET_KEY=y
CONFIG_NET_MPLS_GSO=y
CONFIG_NET_SCHED=y
CONFIG_NET_SCH_FQ=y
CONFIG_NET_SCH_INGRESS=y
CONFIG_NET_VRF=y
CONFIG_NET=y
CONFIG_NF_CONNTRACK=y
CONFIG_NF_TABLES=y
CONFIG_NO_HZ_IDLE=y
CONFIG_NR_CPUS=256
CONFIG_NUMA=y
CONFIG_OPROFILE=y
CONFIG_PACKET=y
CONFIG_PAGE_EXTENSION=y
CONFIG_PANIC_ON_OOPS=y
CONFIG_PARTITION_ADVANCED=y
CONFIG_PCI_HOST_GENERIC=y
CONFIG_PCI=y
CONFIG_PM_DEBUG=y
CONFIG_POSIX_MQUEUE=y
CONFIG_PPC_4K_PAGES=y
CONFIG_PPC64=y
# CONFIG_PPC_QUEUED_SPINLOCKS is not set
CONFIG_PROC_KCORE=y
CONFIG_PROFILING=y
CONFIG_PROVE_LOCKING=y
CONFIG_PTDUMP_DEBUGFS=y
CONFIG_RC_CORE=y
CONFIG_RC_DEVICES=y
CONFIG_RC_LOOPBACK=y
# CONFIG_RC_MAP is not set
# CONFIG_RCU_TRACE is not set
# CONFIG_RD_BZIP2 is not set
# CONFIG_RD_GZIP is not set
# CONFIG_RD_LZ4 is not set
# CONFIG_RD_LZMA is not set
# CONFIG_RD_LZO is not set
# CONFIG_RD_XZ is not set
CONFIG_RT_GROUP_SCHED=y
# CONFIG_S390_PRNG is not set
CONFIG_S390_PTDUMP=y
# CONFIG_S390_TAPE is not set
# CONFIG_S390_VMUR is not set
CONFIG_SAMPLE_SECCOMP=y
CONFIG_SAMPLES=y
CONFIG_SCHED_TRACER=y
# CONFIG_SCLP_TTY is not set
# CONFIG_SCSI_PROC_FS is not set
CONFIG_SCSI_VIRTIO=y
CONFIG_SCSI=y
CONFIG_SECURITYFS=y
CONFIG_SECURITY_NETWORK=y
CONFIG_SECURITY=y
CONFIG_SERIAL_8250_CONSOLE=y
# CONFIG_SERIAL_8250_DEPRECATED_OPTIONS is not set
# CONFIG_SERIAL_8250_LPSS is not set
# CONFIG_SERIAL_8250_MID is not set
# CONFIG_SERIAL_8250_PCI is not set
# CONFIG_SERIAL_8250_PNP is not set
CONFIG_SERIAL_8250=y
CONFIG_SERIAL_AMBA_PL011_CONSOLE=y
CONFIG_SERIAL_AMBA_PL011=y
CONFIG_SMP=y
CONFIG_STACK_TRACER=y
CONFIG_STATIC_KEYS_SELFTEST=y
# CONFIG_STRICT_DEVMEM is not set
CONFIG_SYN_COOKIES=y
# CONFIG_SYSFS_SYSCALL is not set
CONFIG_SYSVIPC=y
CONFIG_TASK_DELAY_ACCT=y
CONFIG_TASK_IO_ACCOUNTING=y
CONFIG_TASKSTATS=y
CONFIG_TASK_XACCT=y
CONFIG_TCP_CONG_ADVANCED=y
# CONFIG_TCP_CONG_BIC is not set
CONFIG_TCP_CONG_DCTCP=y
# CONFIG_TCP_CONG_HTCP is not set
# CONFIG_TCP_CONG_WESTWOOD is not set
CONFIG_TEST_BPF=m
CONFIG_THREAD_SHIFT=13
CONFIG_TICK_CPU_ACCOUNTING=y
CONFIG_TLS=y
CONFIG_TMPFS_POSIX_ACL=y
CONFIG_TMPFS=y
# CONFIG_TN3215 is not set
# CONFIG_TN3270_FS is not set
CONFIG_TRACER_SNAPSHOT_PER_CPU_SWAP=y
CONFIG_TRANSPARENT_HUGEPAGE=y
CONFIG_TUN=y
CONFIG_UNIX=y
CONFIG_UNUSED_SYMBOLS=y
CONFIG_USELIB=y
CONFIG_USERFAULTFD=y
CONFIG_USER_NS=y
CONFIG_VETH=y
CONFIG_VIRTIO_BALLOON=y
CONFIG_VIRTIO_BLK=y
CONFIG_VIRTIO_NET=y
CONFIG_VIRTIO_PCI=y
CONFIG_VLAN_8021Q=y
# CONFIG_VMCP is not set
CONFIG_VSX=y
# CONFIG_VT is not set
CONFIG_VXLAN=y
CONFIG_XDP_SOCKETS=y
CONFIG_XFRM_USER=y

It's not a delta to a smaller "default" defconfig, but IMHO it's still
pretty concise. 

Hope that helps.

Best regards,
Ilya