[PATCH bpf-next 1/2] tools/bpftool: allow substituting custom vmlinux.h for the build

[PATCH bpf-next 1/2] tools/bpftool: allow substituting custom vmlinux.h for the build

[Andrii Nakryiko]

In some build contexts (e.g., Travis CI build for outdated kernel), vmlinux.h,
generated from available kernel, doesn't contain all the types necessary for
BPF program compilation. For such set up, the most maintainable way to deal
with this problem is to keep pre-generated (almost up-to-date) vmlinux.h
checked in and use it for compilation purposes. bpftool after that can deal
with kernel missing some of the features in runtime with no problems.

To that effect, allow to specify path to custom vmlinux.h to bpftool's
Makefile with VMLINUX_H variable.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
---
 tools/bpf/bpftool/Makefile | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
index 8c6563e56ffc..273da1615503 100644
--- a/tools/bpf/bpftool/Makefile
+++ b/tools/bpf/bpftool/Makefile
@@ -122,20 +122,24 @@ BPFTOOL_BOOTSTRAP := $(if $(OUTPUT),$(OUTPUT)bpftool-bootstrap,./bpftool-bootstr
 BOOTSTRAP_OBJS = $(addprefix $(OUTPUT),main.o common.o json_writer.o gen.o btf.o)
 OBJS = $(patsubst %.c,$(OUTPUT)%.o,$(SRCS)) $(OUTPUT)disasm.o
 
-VMLINUX_BTF_PATHS := $(if $(O),$(O)/vmlinux)				\
+VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux)				\
 		     $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux)	\
 		     ../../../vmlinux					\
 		     /sys/kernel/btf/vmlinux				\
 		     /boot/vmlinux-$(shell uname -r)
-VMLINUX_BTF := $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
 
-ifneq ($(VMLINUX_BTF),)
+ifneq ($(VMLINUX_BTF)$(VMLINUX_H),)
 ifeq ($(feature-clang-bpf-co-re),1)
 
 BUILD_BPF_SKELS := 1
 
 $(OUTPUT)vmlinux.h: $(VMLINUX_BTF) $(BPFTOOL_BOOTSTRAP)
+ifeq ($(VMLINUX_H),)
 	$(QUIET_GEN)$(BPFTOOL_BOOTSTRAP) btf dump file $< format c > $@
+else
+	$(Q)cp "$(VMLINUX_H)" $@
+endif
 
 $(OUTPUT)%.bpf.o: skeleton/%.bpf.c $(OUTPUT)vmlinux.h $(LIBBPF)
 	$(QUIET_CLANG)$(CLANG) \
-- 
2.24.1

[PATCH bpf-next 2/2] selftests/bpf: allow substituting custom vmlinux.h for selftests build

[Andrii Nakryiko]

Similarly to bpftool Makefile, allow to specify custom location of vmlinux.h
to be used during the build. This allows simpler testing setups with
checked-in pre-generated vmlinux.h.

Signed-off-by: Andrii Nakryiko <andriin@fb.com>
---
 tools/testing/selftests/bpf/Makefile | 9 +++++++--
 1 file changed, 7 insertions(+), 2 deletions(-)

diff --git a/tools/testing/selftests/bpf/Makefile b/tools/testing/selftests/bpf/Makefile
index 22aaec74ea0a..1f9c696b3edf 100644
--- a/tools/testing/selftests/bpf/Makefile
+++ b/tools/testing/selftests/bpf/Makefile
@@ -134,12 +134,12 @@ $(OUTPUT)/test_stub.o: test_stub.c $(BPFOBJ)
 	$(call msg,CC,,$@)
 	$(CC) -c $(CFLAGS) -o $@ $<
 
-VMLINUX_BTF_PATHS := $(if $(O),$(O)/vmlinux)				\
+VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux)				\
 		     $(if $(KBUILD_OUTPUT),$(KBUILD_OUTPUT)/vmlinux)	\
 		     ../../../../vmlinux				\
 		     /sys/kernel/btf/vmlinux				\
 		     /boot/vmlinux-$(shell uname -r)
-VMLINUX_BTF := $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
+VMLINUX_BTF ?= $(abspath $(firstword $(wildcard $(VMLINUX_BTF_PATHS))))
 
 $(OUTPUT)/runqslower: $(BPFOBJ)
 	$(Q)$(MAKE) $(submake_extras) -C $(TOOLSDIR)/bpf/runqslower	\
@@ -182,8 +182,13 @@ $(BUILD_DIR)/libbpf $(BUILD_DIR)/bpftool $(INCLUDE_DIR):
 	mkdir -p $@
 
 $(INCLUDE_DIR)/vmlinux.h: $(VMLINUX_BTF) | $(BPFTOOL) $(INCLUDE_DIR)
+ifeq ($(VMLINUX_H),)
 	$(call msg,GEN,,$@)
 	$(BPFTOOL) btf dump file $(VMLINUX_BTF) format c > $@
+else
+	$(call msg,CP,,$@)
+	cp "$(VMLINUX_H)" $@
+endif
 
 # Get Clang's default includes on this system, as opposed to those seen by
 # '-target bpf'. This fixes "missing" files on some architectures/distros,
-- 
2.24.1

Re: [PATCH bpf-next 1/2] tools/bpftool: allow substituting custom vmlinux.h for the build

[Yonghong Song]

On 6/29/20 5:47 PM, Andrii Nakryiko wrote:
> In some build contexts (e.g., Travis CI build for outdated kernel), vmlinux.h,
> generated from available kernel, doesn't contain all the types necessary for
> BPF program compilation. For such set up, the most maintainable way to deal
> with this problem is to keep pre-generated (almost up-to-date) vmlinux.h
> checked in and use it for compilation purposes. bpftool after that can deal
> with kernel missing some of the features in runtime with no problems.
> 
> To that effect, allow to specify path to custom vmlinux.h to bpftool's
> Makefile with VMLINUX_H variable.
> 
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>

Acked-by: Yonghong Song <yhs@fb.com>

Re: [PATCH bpf-next 2/2] selftests/bpf: allow substituting custom vmlinux.h for selftests build

[Yonghong Song]

On 6/29/20 5:47 PM, Andrii Nakryiko wrote:
> Similarly to bpftool Makefile, allow to specify custom location of vmlinux.h
> to be used during the build. This allows simpler testing setups with
> checked-in pre-generated vmlinux.h.
> 
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>

Acked-by: Yonghong Song <yhs@fb.com>

Re: [PATCH bpf-next 1/2] tools/bpftool: allow substituting custom vmlinux.h for the build

[Daniel Borkmann]

On 6/30/20 2:47 AM, Andrii Nakryiko wrote:
> In some build contexts (e.g., Travis CI build for outdated kernel), vmlinux.h,
> generated from available kernel, doesn't contain all the types necessary for
> BPF program compilation. For such set up, the most maintainable way to deal
> with this problem is to keep pre-generated (almost up-to-date) vmlinux.h
> checked in and use it for compilation purposes. bpftool after that can deal
> with kernel missing some of the features in runtime with no problems.
> 
> To that effect, allow to specify path to custom vmlinux.h to bpftool's
> Makefile with VMLINUX_H variable.
> 
> Signed-off-by: Andrii Nakryiko <andriin@fb.com>

Both applied, thanks!