This series contains mainly fixes/improvements for cross-compilation but not only, tested for arm, arm64, and intended for any arch. Also verified on native build (not cross compilation) for x86_64 and arm, arm64. Initial RFC link: https://lkml.org/lkml/2019/8/29/1665 Prev. version: https://lkml.org/lkml/2019/9/16/231 Besides the patches given here, the RFC also contains couple patches related to llvm clang arm: include: asm: swab: mask rev16 instruction for clang arm: include: asm: unified: mask .syntax unified for clang They are necessarily to verify arm 32 build. Also, couple more fixes were added but are not merged in bpf-next yet, they can be needed for verification/configuration steps, if not in your tree the fixes can be taken here: https://www.spinics.net/lists/netdev/msg601716.html https://www.spinics.net/lists/netdev/msg601714.html https://www.spinics.net/lists/linux-kbuild/msg23468.html Now, to build samples, SAMPLE_BPF should be enabled in config. The change touches not only cross-compilation and can have impact on other archs and build environments, so might be good idea to verify it in order to add appropriate changes, some warn options could be tuned also. All is tested on x86-64 with clang installed (has to be built containing targets for arm, arm64..., see llc --version, usually it's present already) Instructions to test native on x86_64 ================================================= Native build on x86_64 is done in usual way and shouldn't have difference except HOSTCC is now printed as CC wile building the samples. Instructions to test cross compilation on arm64 ================================================= #Toolchain used for test: gcc version 8.3.0 (GNU Toolchain for the A-profile Architecture 8.3-2019.03 (arm-rel-8.36)) # Get some arm64 FS, containing at least libelf I've used sdk for TI am65x got here: http://downloads.ti.com/processor-sdk-linux/esd/AM65X/latest/exports/\ ti-processor-sdk-linux-am65xx-evm-06.00.00.07-Linux-x86-Install.bin # Install this binary to some dir, say "sdk". # Configure kernel (use defconfig as no matter), but clean everything # before. make ARCH=arm64 -C tools/ clean make ARCH=arm64 -C samples/bpf clean make ARCH=arm64 clean make ARCH=arm64 defconfig # Enable SAMPLE_BPF and it's dependencies in config # The kernel version used in sdk doesn't correspond to checked one, # but for this verification only headers need to be syched, # so install them (can be enabled in config): make ARCH=arm64 headers_install # or on SDK if need keep them in sync (not necessarily to verify): make ARCH=arm64 INSTALL_HDR_PATH=/../sdk/\ ti-processor-sdk-linux-am65xx-evm-06.00.00.07/linux-devkit/sysroots/\ aarch64-linux/usr headers_install # Build samples make samples/bpf/ ARCH=arm64 CROSS_COMPILE="aarch64-linux-gnu-"\ SYSROOT="/../sdk/ti-processor-sdk-linux-am65xx-evm-06.00.00.07/\ linux-devkit/sysroots/aarch64-linux" Instructions to test cross compilation on arm ================================================= #Toolchains used for test: arm-linux-gnueabihf-gcc (Linaro GCC 7.2-2017.11) 7.2.1 20171011 or arm-linux-gnueabihf-gcc (GNU Toolchain for the A-profile Architecture 8.3-2019.03 \ (arm-rel-8.36)) 8.3.0 # Get some FS, I've used sdk for TI am52xx got here: http://downloads.ti.com/processor-sdk-linux/esd/AM57X/05_03_00_07/exports/\ ti-processor-sdk-linux-am57xx-evm-05.03.00.07-Linux-x86-Install.bin # Install this binary to some dir, say "sdk". # Configure kernel, but clean everything before. make ARCH=arm -C tools/ clean make ARCH=arm -C samples/bpf clean make ARCH=arm clean make ARCH=arm omap2plus_defconfig # The kernel version used in sdk doesn't correspond to checked one, but # headers only should be synched, # so install them (can be enabled in config): make ARCH=arm headers_install # or on SDK if need keep them in sync (not necessarily): make ARCH=arm INSTALL_HDR_PATH=/../sdk/\ ti-processor-sdk-linux-am57xx-evm-05.03.00.07/linux-devkit/sysroots/\ armv7ahf-neon-linux-gnueabi/usr headers_install # Build samples make samples/bpf/ ARCH=arm CROSS_COMPILE="arm-linux-gnueabihf-"\ SYSROOT="/../sdk/ti-processor-sdk-linux-am57xx-evm-05.03\ .00.07/linux-devkit/sysroots/armv7ahf-neon-linux-gnueabi" Based on bpf-next/master v4..v3: - renamed CLANG_EXTRA_CFLAGS on BPF_EXTRA_CFLAGS - used filter for ARCH_ARM_SELECTOR - omit "-fomit-frame-pointer" and use same flags for native and "cross" - used sample/bpf prefixes - use C instead of C++ compiler for test_libbpf target v3..v2: - renamed makefile.progs to makeifle.target, as more appropriate - left only __LINUX_ARM_ARCH__ for D options for arm - for host build - left options from KBUILD_HOST for compatibility reasons - split patch adding c/cxx/ld flags to libbpf by modules - moved readme change to separate patch - added patch setting options for cross-compile - fixed issue with option error for syscall_nrs.S, avoiding overlap for ccflags-y. v2..v1: - restructured patches order - split "samples: bpf: Makefile: base progs build on Makefile.progs" to make change more readable. It added couple nice extra patches. - removed redundant patch: "samples: bpf: Makefile: remove target for native build" - added fix: "samples: bpf: makefile: fix cookie_uid_helper_example obj build" - limited -D option filter only for arm - improved comments - added couple instructions to verify cross compilation for arm and arm64 arches based on TI am57xx and am65xx sdks. - corrected include a little order Ivan Khoronzhuk (15): samples/bpf: fix HDR_PROBE "echo" samples/bpf: fix cookie_uid_helper_example obj build samples/bpf: use --target from cross-compile samples/bpf: use own EXTRA_CFLAGS for clang commands samples/bpf: use __LINUX_ARM_ARCH__ selector for arm samples/bpf: drop unnecessarily inclusion for bpf_load samples/bpf add makefile.target for separate CC target build samples/bpf: base target programs rules on Makefile.target samples/bpf: use own flags but not HOSTCFLAGS samples/bpf: use target CC environment for HDR_PROBE libbpf: don't use cxx to test_libpf target libbpf: add C/LDFLAGS to libbpf.so and test_libpf targets samples/bpf: provide C/LDFLAGS to libbpf samples/bpf: add sysroot support samples/bpf: add preparation steps and sysroot info to readme samples/bpf/Makefile | 164 ++++++++++-------- samples/bpf/Makefile.target | 75 ++++++++ samples/bpf/README.rst | 41 ++++- tools/lib/bpf/Makefile | 23 +-- .../bpf/{test_libbpf.cpp => test_libbpf.c} | 14 +- 5 files changed, 218 insertions(+), 99 deletions(-) create mode 100644 samples/bpf/Makefile.target rename tools/lib/bpf/{test_libbpf.cpp => test_libbpf.c} (61%) -- 2.17.1
echo should be replaced with echo -e to handle '\n' correctly, but instead, replace it with printf as some systems can't handle echo -e. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index a11d7270583d..4f61725b1d86 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -201,7 +201,7 @@ endif # Don't evaluate probes and warnings if we need to run make recursively ifneq ($(src),) -HDR_PROBE := $(shell echo "\#include <linux/types.h>\n struct list_head { int a; }; int main() { return 0; }" | \ +HDR_PROBE := $(shell printf "\#include <linux/types.h>\n struct list_head { int a; }; int main() { return 0; }" | \ $(HOSTCC) $(KBUILD_HOSTCFLAGS) -x c - -o /dev/null 2>/dev/null && \ echo okay) -- 2.17.1
Don't list userspace "cookie_uid_helper_example" object in list for bpf objects. 'always' target is used for listing bpf programs, but 'cookie_uid_helper_example.o' is a user space ELF file, and covered by rule `per_socket_stats_example`, so shouldn't be in 'always'. Let us remove `always += cookie_uid_helper_example.o`, which avoids breaking cross compilation due to mismatched includes. Acked-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 1 - 1 file changed, 1 deletion(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 4f61725b1d86..045fa43842e6 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -145,7 +145,6 @@ always += sampleip_kern.o always += lwt_len_hist_kern.o always += xdp_tx_iptunnel_kern.o always += test_map_in_map_kern.o -always += cookie_uid_helper_example.o always += tcp_synrto_kern.o always += tcp_rwnd_kern.o always += tcp_bufs_kern.o -- 2.17.1
For cross compiling the target triple can be inherited from cross-compile prefix as it's done in CLANG_FLAGS from kernel makefile. So copy-paste this decision from kernel Makefile. Acked-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 045fa43842e6..9c8c9872004d 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -195,7 +195,7 @@ BTF_PAHOLE ?= pahole # Detect that we're cross compiling and use the cross compiler ifdef CROSS_COMPILE HOSTCC = $(CROSS_COMPILE)gcc -CLANG_ARCH_ARGS = -target $(ARCH) +CLANG_ARCH_ARGS = --target=$(notdir $(CROSS_COMPILE:%-=%)) endif # Don't evaluate probes and warnings if we need to run make recursively -- 2.17.1
It can overlap with CFLAGS used for libraries built with gcc if not now then in next patches. Correct it here for simplicity. Acked-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 9 +++++---- 1 file changed, 5 insertions(+), 4 deletions(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 9c8c9872004d..cf882e43648a 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -218,10 +218,10 @@ BTF_LLVM_PROBE := $(shell echo "int main() { return 0; }" | \ /bin/rm -f ./llvm_btf_verify.o) ifneq ($(BTF_LLVM_PROBE),) - EXTRA_CFLAGS += -g + BPF_EXTRA_CFLAGS += -g else ifneq ($(and $(BTF_LLC_PROBE),$(BTF_PAHOLE_PROBE),$(BTF_OBJCOPY_PROBE)),) - EXTRA_CFLAGS += -g + BPF_EXTRA_CFLAGS += -g LLC_FLAGS += -mattr=dwarfris DWARF2BTF = y endif @@ -280,8 +280,9 @@ $(obj)/hbm_edt_kern.o: $(src)/hbm.h $(src)/hbm_kern.h # useless for BPF samples. $(obj)/%.o: $(src)/%.c @echo " CLANG-bpf " $@ - $(Q)$(CLANG) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) $(EXTRA_CFLAGS) -I$(obj) \ - -I$(srctree)/tools/testing/selftests/bpf/ -I$(srctree)/tools/lib/bpf/ \ + $(Q)$(CLANG) $(NOSTDINC_FLAGS) $(LINUXINCLUDE) $(BPF_EXTRA_CFLAGS) \ + -I$(obj) -I$(srctree)/tools/testing/selftests/bpf/ \ + -I$(srctree)/tools/lib/bpf/ \ -D__KERNEL__ -D__BPF_TRACING__ -Wno-unused-value -Wno-pointer-sign \ -D__TARGET_ARCH_$(SRCARCH) -Wno-compare-distinct-pointer-types \ -Wno-gnu-variable-sized-type-not-at-end \ -- 2.17.1
For arm, -D__LINUX_ARM_ARCH__=X is min version used as instruction set selector and is absolutely required while parsing some parts of headers. It's present in KBUILD_CFLAGS but not in autoconf.h, so let's retrieve it from and add to programs cflags. In another case errors like "SMP is not supported" for armv7 and bunch of other errors are issued resulting to incorrect final object. --- samples/bpf/Makefile | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index cf882e43648a..9b33e7395eac 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -185,6 +185,14 @@ HOSTLDLIBS_map_perf_test += -lrt HOSTLDLIBS_test_overhead += -lrt HOSTLDLIBS_xdpsock += -pthread +ifeq ($(ARCH), arm) +# Strip all except -D__LINUX_ARM_ARCH__ option needed to handle linux +# headers when arm instruction set identification is requested. +ARM_ARCH_SELECTOR := $(filter -D__LINUX_ARM_ARCH__%, $(KBUILD_CFLAGS)) +BPF_EXTRA_CFLAGS := $(ARM_ARCH_SELECTOR) +KBUILD_HOSTCFLAGS += $(ARM_ARCH_SELECTOR) +endif + # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline: # make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang LLC ?= llc -- 2.17.1
Drop inclusion for bpf_load -I$(objtree)/usr/include as it is included for all objects anyway, with above line: KBUILD_HOSTCFLAGS += -I$(objtree)/usr/include Acked-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 9b33e7395eac..bb2d976e824e 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -176,7 +176,7 @@ KBUILD_HOSTCFLAGS += -I$(srctree)/tools/testing/selftests/bpf/ KBUILD_HOSTCFLAGS += -I$(srctree)/tools/lib/ -I$(srctree)/tools/include KBUILD_HOSTCFLAGS += -I$(srctree)/tools/perf -HOSTCFLAGS_bpf_load.o += -I$(objtree)/usr/include -Wno-unused-variable +HOSTCFLAGS_bpf_load.o += -Wno-unused-variable KBUILD_HOSTLDLIBS += $(LIBBPF) -lelf HOSTLDLIBS_tracex4 += -lrt -- 2.17.1
The Makefile.target is added only and will be used in sample/bpf/Makefile later in order to switch cross-compiling to CC from HOSTCC environment. The HOSTCC is supposed to build binaries and tools running on the host afterwards, in order to simplify build or so, like "fixdep" or else. In case of cross compiling "fixdep" is executed on host when the rest samples should run on target arch. In order to build binaries for target arch with CC and tools running on host with HOSTCC, lets add Makefile.target for simplicity, having definition and routines similar to ones, used in script/Makefile.host. This allows later add cross-compilation to samples/bpf with minimum changes. The tprog stands for target programs built with CC. Makefile.target contains only stuff needed for samples/bpf, potentially can be reused later and now needed only for unblocking tricky samples/bpf cross compilation. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile.target | 75 +++++++++++++++++++++++++++++++++++++ 1 file changed, 75 insertions(+) create mode 100644 samples/bpf/Makefile.target diff --git a/samples/bpf/Makefile.target b/samples/bpf/Makefile.target new file mode 100644 index 000000000000..7621f55e2947 --- /dev/null +++ b/samples/bpf/Makefile.target @@ -0,0 +1,75 @@ +# SPDX-License-Identifier: GPL-2.0 +# ========================================================================== +# Building binaries on the host system +# Binaries are not used during the compilation of the kernel, and intended +# to be build for target board, target board can be host of course. Added to +# build binaries to run not on host system. +# +# Sample syntax +# tprogs-y := xsk_example +# Will compile xsk_example.c and create an executable named xsk_example +# +# tprogs-y := xdpsock +# xdpsock-objs := xdpsock_1.o xdpsock_2.o +# Will compile xdpsock_1.c and xdpsock_2.c, and then link the executable +# xdpsock, based on xdpsock_1.o and xdpsock_2.o +# +# Derived from scripts/Makefile.host +# +__tprogs := $(sort $(tprogs-y)) + +# C code +# Executables compiled from a single .c file +tprog-csingle := $(foreach m,$(__tprogs), \ + $(if $($(m)-objs),,$(m))) + +# C executables linked based on several .o files +tprog-cmulti := $(foreach m,$(__tprogs),\ + $(if $($(m)-objs),$(m))) + +# Object (.o) files compiled from .c files +tprog-cobjs := $(sort $(foreach m,$(__tprogs),$($(m)-objs))) + +tprog-csingle := $(addprefix $(obj)/,$(tprog-csingle)) +tprog-cmulti := $(addprefix $(obj)/,$(tprog-cmulti)) +tprog-cobjs := $(addprefix $(obj)/,$(tprog-cobjs)) + +##### +# Handle options to gcc. Support building with separate output directory + +_tprogc_flags = $(TPROGS_CFLAGS) \ + $(TPROGCFLAGS_$(basetarget).o) + +# $(objtree)/$(obj) for including generated headers from checkin source files +ifeq ($(KBUILD_EXTMOD),) +ifdef building_out_of_srctree +_tprogc_flags += -I $(objtree)/$(obj) +endif +endif + +tprogc_flags = -Wp,-MD,$(depfile) $(_tprogc_flags) + +# Create executable from a single .c file +# tprog-csingle -> Executable +quiet_cmd_tprog-csingle = CC $@ + cmd_tprog-csingle = $(CC) $(tprogc_flags) $(TPROGS_LDFLAGS) -o $@ $< \ + $(TPROGS_LDLIBS) $(TPROGLDLIBS_$(@F)) +$(tprog-csingle): $(obj)/%: $(src)/%.c FORCE + $(call if_changed_dep,tprog-csingle) + +# Link an executable based on list of .o files, all plain c +# tprog-cmulti -> executable +quiet_cmd_tprog-cmulti = LD $@ + cmd_tprog-cmulti = $(CC) $(tprogc_flags) $(TPROGS_LDFLAGS) -o $@ \ + $(addprefix $(obj)/,$($(@F)-objs)) \ + $(TPROGS_LDLIBS) $(TPROGLDLIBS_$(@F)) +$(tprog-cmulti): $(tprog-cobjs) FORCE + $(call if_changed,tprog-cmulti) +$(call multi_depend, $(tprog-cmulti), , -objs) + +# Create .o file from a single .c file +# tprog-cobjs -> .o +quiet_cmd_tprog-cobjs = CC $@ + cmd_tprog-cobjs = $(CC) $(tprogc_flags) -c -o $@ $< +$(tprog-cobjs): $(obj)/%.o: $(src)/%.c FORCE + $(call if_changed_dep,tprog-cobjs) -- 2.17.1
The main reason for that - HOSTCC and CC have different aims. HOSTCC is used to build programs running on host, that can cross-comple target programs with CC. It was tested for arm and arm64 cross compilation, based on linaro toolchain, but should work for others. So, in order to split cross compilation (CC) with host build (HOSTCC), lets base samples on Makefile.target. It allows to cross-compile samples/bpf programs with CC while auxialry tools running on host built with HOSTCC. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 135 ++++++++++++++++++++++--------------------- 1 file changed, 69 insertions(+), 66 deletions(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index bb2d976e824e..91bfb421c278 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -4,55 +4,53 @@ BPF_SAMPLES_PATH ?= $(abspath $(srctree)/$(src)) TOOLS_PATH := $(BPF_SAMPLES_PATH)/../../tools # List of programs to build -hostprogs-y := test_lru_dist -hostprogs-y += sock_example -hostprogs-y += fds_example -hostprogs-y += sockex1 -hostprogs-y += sockex2 -hostprogs-y += sockex3 -hostprogs-y += tracex1 -hostprogs-y += tracex2 -hostprogs-y += tracex3 -hostprogs-y += tracex4 -hostprogs-y += tracex5 -hostprogs-y += tracex6 -hostprogs-y += tracex7 -hostprogs-y += test_probe_write_user -hostprogs-y += trace_output -hostprogs-y += lathist -hostprogs-y += offwaketime -hostprogs-y += spintest -hostprogs-y += map_perf_test -hostprogs-y += test_overhead -hostprogs-y += test_cgrp2_array_pin -hostprogs-y += test_cgrp2_attach -hostprogs-y += test_cgrp2_sock -hostprogs-y += test_cgrp2_sock2 -hostprogs-y += xdp1 -hostprogs-y += xdp2 -hostprogs-y += xdp_router_ipv4 -hostprogs-y += test_current_task_under_cgroup -hostprogs-y += trace_event -hostprogs-y += sampleip -hostprogs-y += tc_l2_redirect -hostprogs-y += lwt_len_hist -hostprogs-y += xdp_tx_iptunnel -hostprogs-y += test_map_in_map -hostprogs-y += per_socket_stats_example -hostprogs-y += xdp_redirect -hostprogs-y += xdp_redirect_map -hostprogs-y += xdp_redirect_cpu -hostprogs-y += xdp_monitor -hostprogs-y += xdp_rxq_info -hostprogs-y += syscall_tp -hostprogs-y += cpustat -hostprogs-y += xdp_adjust_tail -hostprogs-y += xdpsock -hostprogs-y += xdp_fwd -hostprogs-y += task_fd_query -hostprogs-y += xdp_sample_pkts -hostprogs-y += ibumad -hostprogs-y += hbm +tprogs-y := test_lru_dist +tprogs-y += sock_example +tprogs-y += fds_example +tprogs-y += sockex1 +tprogs-y += sockex2 +tprogs-y += sockex3 +tprogs-y += tracex1 +tprogs-y += tracex2 +tprogs-y += tracex3 +tprogs-y += tracex4 +tprogs-y += tracex5 +tprogs-y += tracex6 +tprogs-y += tracex7 +tprogs-y += test_probe_write_user +tprogs-y += trace_output +tprogs-y += lathist +tprogs-y += offwaketime +tprogs-y += spintest +tprogs-y += map_perf_test +tprogs-y += test_overhead +tprogs-y += test_cgrp2_array_pin +tprogs-y += test_cgrp2_attach +tprogs-y += test_cgrp2_sock +tprogs-y += test_cgrp2_sock2 +tprogs-y += xdp1 +tprogs-y += xdp2 +tprogs-y += xdp_router_ipv4 +tprogs-y += test_current_task_under_cgroup +tprogs-y += trace_event +tprogs-y += sampleip +tprogs-y += tc_l2_redirect +tprogs-y += lwt_len_hist +tprogs-y += xdp_tx_iptunnel +tprogs-y += test_map_in_map +tprogs-y += xdp_redirect_map +tprogs-y += xdp_redirect_cpu +tprogs-y += xdp_monitor +tprogs-y += xdp_rxq_info +tprogs-y += syscall_tp +tprogs-y += cpustat +tprogs-y += xdp_adjust_tail +tprogs-y += xdpsock +tprogs-y += xdp_fwd +tprogs-y += task_fd_query +tprogs-y += xdp_sample_pkts +tprogs-y += ibumad +tprogs-y += hbm # Libbpf dependencies LIBBPF = $(TOOLS_PATH)/lib/bpf/libbpf.a @@ -111,7 +109,7 @@ ibumad-objs := bpf_load.o ibumad_user.o $(TRACE_HELPERS) hbm-objs := bpf_load.o hbm.o $(CGROUP_HELPERS) # Tell kbuild to always build the programs -always := $(hostprogs-y) +always := $(tprogs-y) always += sockex1_kern.o always += sockex2_kern.o always += sockex3_kern.o @@ -170,29 +168,32 @@ always += ibumad_kern.o always += hbm_out_kern.o always += hbm_edt_kern.o -KBUILD_HOSTCFLAGS += -I$(objtree)/usr/include -KBUILD_HOSTCFLAGS += -I$(srctree)/tools/lib/bpf/ -KBUILD_HOSTCFLAGS += -I$(srctree)/tools/testing/selftests/bpf/ -KBUILD_HOSTCFLAGS += -I$(srctree)/tools/lib/ -I$(srctree)/tools/include -KBUILD_HOSTCFLAGS += -I$(srctree)/tools/perf - -HOSTCFLAGS_bpf_load.o += -Wno-unused-variable - -KBUILD_HOSTLDLIBS += $(LIBBPF) -lelf -HOSTLDLIBS_tracex4 += -lrt -HOSTLDLIBS_trace_output += -lrt -HOSTLDLIBS_map_perf_test += -lrt -HOSTLDLIBS_test_overhead += -lrt -HOSTLDLIBS_xdpsock += -pthread - ifeq ($(ARCH), arm) # Strip all except -D__LINUX_ARM_ARCH__ option needed to handle linux # headers when arm instruction set identification is requested. ARM_ARCH_SELECTOR := $(filter -D__LINUX_ARM_ARCH__%, $(KBUILD_CFLAGS)) BPF_EXTRA_CFLAGS := $(ARM_ARCH_SELECTOR) -KBUILD_HOSTCFLAGS += $(ARM_ARCH_SELECTOR) +TPROGS_CFLAGS += $(ARM_ARCH_SELECTOR) endif +TPROGS_LDLIBS := $(KBUILD_HOSTLDLIBS) +TPROGS_CFLAGS += $(KBUILD_HOSTCFLAGS) $(HOST_EXTRACFLAGS) +TPROGS_CFLAGS += -I$(objtree)/usr/include +TPROGS_CFLAGS += -I$(srctree)/tools/lib/bpf/ +TPROGS_CFLAGS += -I$(srctree)/tools/testing/selftests/bpf/ +TPROGS_CFLAGS += -I$(srctree)/tools/lib/ +TPROGS_CFLAGS += -I$(srctree)/tools/include +TPROGS_CFLAGS += -I$(srctree)/tools/perf + +TPROGCFLAGS_bpf_load.o += -Wno-unused-variable + +TPROGS_LDLIBS += $(LIBBPF) -lelf +TPROGLDLIBS_tracex4 += -lrt +TPROGLDLIBS_trace_output += -lrt +TPROGLDLIBS_map_perf_test += -lrt +TPROGLDLIBS_test_overhead += -lrt +TPROGLDLIBS_xdpsock += -pthread + # Allows pointing LLC/CLANG to a LLVM backend with bpf support, redefine on cmdline: # make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang LLC ?= llc @@ -283,6 +284,8 @@ $(obj)/hbm_out_kern.o: $(src)/hbm.h $(src)/hbm_kern.h $(obj)/hbm.o: $(src)/hbm.h $(obj)/hbm_edt_kern.o: $(src)/hbm.h $(src)/hbm_kern.h +-include $(BPF_SAMPLES_PATH)/Makefile.target + # asm/sysreg.h - inline assembly used by it is incompatible with llvm. # But, there is no easy way to fix it, so just exclude it since it is # useless for BPF samples. -- 2.17.1
While compiling natively, the host's cflags and ldflags are equal to ones used from HOSTCFLAGS and HOSTLDFLAGS. When cross compiling it should have own, used for target arch. While verification, for arm, arm64 and x86_64 the following flags were used always: -Wall -O2 -fomit-frame-pointer -Wmissing-prototypes -Wstrict-prototypes So, add them as they were verified and used before adding Makefile.target and lets omit "-fomit-frame-pointer" as were proposed while review, as no sense in such optimization for samples. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 91bfb421c278..57a15ff938a6 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -176,8 +176,10 @@ BPF_EXTRA_CFLAGS := $(ARM_ARCH_SELECTOR) TPROGS_CFLAGS += $(ARM_ARCH_SELECTOR) endif -TPROGS_LDLIBS := $(KBUILD_HOSTLDLIBS) -TPROGS_CFLAGS += $(KBUILD_HOSTCFLAGS) $(HOST_EXTRACFLAGS) +TPROGS_CFLAGS += -Wall -O2 +TPROGS_CFLAGS += -Wmissing-prototypes +TPROGS_CFLAGS += -Wstrict-prototypes + TPROGS_CFLAGS += -I$(objtree)/usr/include TPROGS_CFLAGS += -I$(srctree)/tools/lib/bpf/ TPROGS_CFLAGS += -I$(srctree)/tools/testing/selftests/bpf/ -- 2.17.1
No need in hacking HOSTCC to be cross-compiler any more, so drop this trick and use target CC for HDR_PROBE. Acked-by: Andrii Nakryiko <andriin@fb.com> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 57a15ff938a6..a6c33496e8ca 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -205,15 +205,14 @@ BTF_PAHOLE ?= pahole # Detect that we're cross compiling and use the cross compiler ifdef CROSS_COMPILE -HOSTCC = $(CROSS_COMPILE)gcc CLANG_ARCH_ARGS = --target=$(notdir $(CROSS_COMPILE:%-=%)) endif # Don't evaluate probes and warnings if we need to run make recursively ifneq ($(src),) HDR_PROBE := $(shell printf "\#include <linux/types.h>\n struct list_head { int a; }; int main() { return 0; }" | \ - $(HOSTCC) $(KBUILD_HOSTCFLAGS) -x c - -o /dev/null 2>/dev/null && \ - echo okay) + $(CC) $(TPROGS_CFLAGS) $(TPROGS_LDFLAGS) -x c - \ + -o /dev/null 2>/dev/null && echo okay) ifeq ($(HDR_PROBE),) $(warning WARNING: Detected possible issues with include path.) -- 2.17.1
No need to use C++ for test_libbpf target when libbpf is on C and it can be tested with C, after this change the CXXFLAGS in makefiles can be avoided, at least in bpf samples, when sysroot is used, passing same C/LDFLAGS as for lib. Add "return 0" in test_libbpf to void warn, but also remove spaces at start of the lines to keep same style and avoid warns while apply. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- tools/lib/bpf/Makefile | 18 +++++------------- .../lib/bpf/{test_libbpf.cpp => test_libbpf.c} | 14 ++++++++------ 2 files changed, 13 insertions(+), 19 deletions(-) rename tools/lib/bpf/{test_libbpf.cpp => test_libbpf.c} (61%) diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile index 1270955e4845..46280b5ad48d 100644 --- a/tools/lib/bpf/Makefile +++ b/tools/lib/bpf/Makefile @@ -52,7 +52,7 @@ ifndef VERBOSE endif FEATURE_USER = .libbpf -FEATURE_TESTS = libelf libelf-mmap bpf reallocarray cxx +FEATURE_TESTS = libelf libelf-mmap bpf reallocarray FEATURE_DISPLAY = libelf bpf INCLUDES = -I. -I$(srctree)/tools/include -I$(srctree)/tools/arch/$(ARCH)/include/uapi -I$(srctree)/tools/include/uapi @@ -142,15 +142,7 @@ GLOBAL_SYM_COUNT = $(shell readelf -s --wide $(BPF_IN) | \ VERSIONED_SYM_COUNT = $(shell readelf -s --wide $(OUTPUT)libbpf.so | \ grep -Eo '[^ ]+@LIBBPF_' | cut -d@ -f1 | sort -u | wc -l) -CMD_TARGETS = $(LIB_TARGET) $(PC_FILE) - -CXX_TEST_TARGET = $(OUTPUT)test_libbpf - -ifeq ($(feature-cxx), 1) - CMD_TARGETS += $(CXX_TEST_TARGET) -endif - -TARGETS = $(CMD_TARGETS) +CMD_TARGETS = $(LIB_TARGET) $(PC_FILE) $(OUTPUT)test_libbpf all: fixdep $(Q)$(MAKE) all_cmd @@ -190,8 +182,8 @@ $(OUTPUT)libbpf.so.$(LIBBPF_VERSION): $(BPF_IN) $(OUTPUT)libbpf.a: $(BPF_IN) $(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^ -$(OUTPUT)test_libbpf: test_libbpf.cpp $(OUTPUT)libbpf.a - $(QUIET_LINK)$(CXX) $(INCLUDES) $^ -lelf -o $@ +$(OUTPUT)test_libbpf: test_libbpf.c $(OUTPUT)libbpf.a + $(QUIET_LINK)$(CC) $(INCLUDES) $^ -lelf -o $@ $(OUTPUT)libbpf.pc: $(QUIET_GEN)sed -e "s|@PREFIX@|$(prefix)|" \ @@ -266,7 +258,7 @@ config-clean: $(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null clean: - $(call QUIET_CLEAN, libbpf) $(RM) $(TARGETS) $(CXX_TEST_TARGET) \ + $(call QUIET_CLEAN, libbpf) $(RM) $(CMD_TARGETS) \ *.o *~ *.a *.so *.so.$(LIBBPF_MAJOR_VERSION) .*.d .*.cmd \ *.pc LIBBPF-CFLAGS bpf_helper_defs.h $(call QUIET_CLEAN, core-gen) $(RM) $(OUTPUT)FEATURE-DUMP.libbpf diff --git a/tools/lib/bpf/test_libbpf.cpp b/tools/lib/bpf/test_libbpf.c similarity index 61% rename from tools/lib/bpf/test_libbpf.cpp rename to tools/lib/bpf/test_libbpf.c index fc134873bb6d..f0eb2727b766 100644 --- a/tools/lib/bpf/test_libbpf.cpp +++ b/tools/lib/bpf/test_libbpf.c @@ -7,12 +7,14 @@ int main(int argc, char *argv[]) { - /* libbpf.h */ - libbpf_set_print(NULL); + /* libbpf.h */ + libbpf_set_print(NULL); - /* bpf.h */ - bpf_prog_get_fd_by_id(0); + /* bpf.h */ + bpf_prog_get_fd_by_id(0); - /* btf.h */ - btf__new(NULL, 0); + /* btf.h */ + btf__new(NULL, 0); + + return 0; } -- 2.17.1
In case of C/LDFLAGS there is no way to pass them correctly to build command, for instance when --sysroot is used or external libraries are used, like -lelf, wich can be absent in toolchain. This can be used for samples/bpf cross-compiling allowing to get elf lib from sysroot. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- tools/lib/bpf/Makefile | 7 ++++--- 1 file changed, 4 insertions(+), 3 deletions(-) diff --git a/tools/lib/bpf/Makefile b/tools/lib/bpf/Makefile index 46280b5ad48d..75b538577c17 100644 --- a/tools/lib/bpf/Makefile +++ b/tools/lib/bpf/Makefile @@ -174,8 +174,9 @@ bpf_helper_defs.h: $(srctree)/include/uapi/linux/bpf.h $(OUTPUT)libbpf.so: $(OUTPUT)libbpf.so.$(LIBBPF_VERSION) $(OUTPUT)libbpf.so.$(LIBBPF_VERSION): $(BPF_IN) - $(QUIET_LINK)$(CC) --shared -Wl,-soname,libbpf.so.$(LIBBPF_MAJOR_VERSION) \ - -Wl,--version-script=$(VERSION_SCRIPT) $^ -lelf -o $@ + $(QUIET_LINK)$(CC) $(LDFLAGS) \ + --shared -Wl,-soname,libbpf.so.$(LIBBPF_MAJOR_VERSION) \ + -Wl,--version-script=$(VERSION_SCRIPT) $^ -lelf -o $@ @ln -sf $(@F) $(OUTPUT)libbpf.so @ln -sf $(@F) $(OUTPUT)libbpf.so.$(LIBBPF_MAJOR_VERSION) @@ -183,7 +184,7 @@ $(OUTPUT)libbpf.a: $(BPF_IN) $(QUIET_LINK)$(RM) $@; $(AR) rcs $@ $^ $(OUTPUT)test_libbpf: test_libbpf.c $(OUTPUT)libbpf.a - $(QUIET_LINK)$(CC) $(INCLUDES) $^ -lelf -o $@ + $(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) $(INCLUDES) $^ -lelf -o $@ $(OUTPUT)libbpf.pc: $(QUIET_GEN)sed -e "s|@PREFIX@|$(prefix)|" \ -- 2.17.1
In order to build lib using C/LD flags of target arch, provide them to libbpf make. Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index a6c33496e8ca..6b161326ac67 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -248,7 +248,8 @@ clean: $(LIBBPF): FORCE # Fix up variables inherited from Kbuild that tools/ build system won't like - $(MAKE) -C $(dir $@) RM='rm -rf' LDFLAGS= srctree=$(BPF_SAMPLES_PATH)/../../ O= + $(MAKE) -C $(dir $@) RM='rm -rf' EXTRA_CFLAGS="$(TPROGS_CFLAGS)" \ + LDFLAGS=$(TPROGS_LDFLAGS) srctree=$(BPF_SAMPLES_PATH)/../../ O= $(obj)/syscall_nrs.h: $(obj)/syscall_nrs.s FORCE $(call filechk,offsets,__SYSCALL_NRS_H__) -- 2.17.1
Basically it only enables that was added by previous couple fixes. Sysroot contains correct libs installed and its headers. Useful when working with NFC or virtual machine. Usage example: clean (on demand) make ARCH=arm -C samples/bpf clean make ARCH=arm -C tools clean make ARCH=arm clean configure and install headers: make ARCH=arm defconfig make ARCH=arm headers_install build samples/bpf: make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- samples/bpf/ \ SYSROOT="path/to/sysroot" Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> --- samples/bpf/Makefile | 5 +++++ 1 file changed, 5 insertions(+) diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile index 6b161326ac67..4df11ddb9c75 100644 --- a/samples/bpf/Makefile +++ b/samples/bpf/Makefile @@ -187,6 +187,11 @@ TPROGS_CFLAGS += -I$(srctree)/tools/lib/ TPROGS_CFLAGS += -I$(srctree)/tools/include TPROGS_CFLAGS += -I$(srctree)/tools/perf +ifdef SYSROOT +TPROGS_CFLAGS += --sysroot=$(SYSROOT) +TPROGS_LDFLAGS := -L$(SYSROOT)/usr/lib +endif + TPROGCFLAGS_bpf_load.o += -Wno-unused-variable TPROGS_LDLIBS += $(LIBBPF) -lelf -- 2.17.1
Add couple preparation steps: clean and configuration. Also add newly added sysroot support info to cross-compile section. --- samples/bpf/README.rst | 41 ++++++++++++++++++++++++++++++++++++----- 1 file changed, 36 insertions(+), 5 deletions(-) diff --git a/samples/bpf/README.rst b/samples/bpf/README.rst index 5f27e4faca50..cc1f00a1ee06 100644 --- a/samples/bpf/README.rst +++ b/samples/bpf/README.rst @@ -14,6 +14,20 @@ Compiling requires having installed: Note that LLVM's tool 'llc' must support target 'bpf', list version and supported targets with command: ``llc --version`` +Clean and configuration +----------------------- + +It can be needed to clean tools, samples or kernel before trying new arch or +after some changes (on demand):: + + make -C tools clean + make -C samples/bpf clean + make clean + +Configure kernel, defconfig for instance:: + + make defconfig + Kernel headers -------------- @@ -68,9 +82,26 @@ It is also possible to point make to the newly compiled 'llc' or Cross compiling samples ----------------------- In order to cross-compile, say for arm64 targets, export CROSS_COMPILE and ARCH -environment variables before calling make. This will direct make to build -samples for the cross target. +environment variables before calling make. But do this before clean, +cofiguration and header install steps described above. This will direct make to +build samples for the cross target:: + + export ARCH=arm64 + export CROSS_COMPILE="aarch64-linux-gnu-" + +Headers can be also installed on RFS of target board if need to keep them in +sync (not necessarily and it creates a local "usr/include" directory also):: + + make INSTALL_HDR_PATH=~/some_sysroot/usr headers_install + +Pointing LLC and CLANG is not necessarily if it's installed on HOST and have +in its targets appropriate arm64 arch (usually it has several arches). +Build samples:: + + make samples/bpf/ + +Or build samples with SYSROOT if some header or library is absent in toolchain, +say libelf, providing address to file system containing headers and libs, +can be RFS of target board:: -export ARCH=arm64 -export CROSS_COMPILE="aarch64-linux-gnu-" -make samples/bpf/ LLC=~/git/llvm/build/bin/llc CLANG=~/git/llvm/build/bin/clang + make samples/bpf/ SYSROOT=~/some_sysroot -- 2.17.1
On Wed, Oct 9, 2019 at 1:45 PM Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> wrote: > > echo should be replaced with echo -e to handle '\n' correctly, but > instead, replace it with printf as some systems can't handle echo -e. > > Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org> > --- Acked-by: Andrii Nakryiko <andriin@fb.com> > samples/bpf/Makefile | 2 +- > 1 file changed, 1 insertion(+), 1 deletion(-) > > diff --git a/samples/bpf/Makefile b/samples/bpf/Makefile > index a11d7270583d..4f61725b1d86 100644 > --- a/samples/bpf/Makefile > +++ b/samples/bpf/Makefile > @@ -201,7 +201,7 @@ endif > > # Don't evaluate probes and warnings if we need to run make recursively > ifneq ($(src),) > -HDR_PROBE := $(shell echo "\#include <linux/types.h>\n struct list_head { int a; }; int main() { return 0; }" | \ > +HDR_PROBE := $(shell printf "\#include <linux/types.h>\n struct list_head { int a; }; int main() { return 0; }" | \ > $(HOSTCC) $(KBUILD_HOSTCFLAGS) -x c - -o /dev/null 2>/dev/null && \ > echo okay) > > -- > 2.17.1 >
On Wed, Oct 9, 2019 at 1:43 PM Ivan Khoronzhuk
<ivan.khoronzhuk@linaro.org> wrote:
>
> No need to use C++ for test_libbpf target when libbpf is on C and it
> can be tested with C, after this change the CXXFLAGS in makefiles can
> be avoided, at least in bpf samples, when sysroot is used, passing
> same C/LDFLAGS as for lib.
>
> Add "return 0" in test_libbpf to void warn, but also remove spaces at
> start of the lines to keep same style and avoid warns while apply.
>
> Signed-off-by: Ivan Khoronzhuk <ivan.khoronzhuk@linaro.org>
> ---
Thanks for the clean up!
Acked-by: Andrii Nakryiko <andriin@fb.com>
[...]
On Wed, Oct 09, 2019 at 11:41:19PM +0300, Ivan Khoronzhuk wrote:
> This series contains mainly fixes/improvements for cross-compilation
> but not only, tested for arm, arm64, and intended for any arch.
> Also verified on native build (not cross compilation) for x86_64
> and arm, arm64.
[...]
There are multiple SOBs missing, please fix. Thanks!
[...]
5 files changed, 218 insertions(+), 99 deletions(-)
create mode 100644 samples/bpf/Makefile.target
rename tools/lib/bpf/{test_libbpf.cpp => test_libbpf.c} (61%)
Deleted branch mbox (was 9f35d1d0c8f0).
Commit 9f35d1d0c8f0 ("samples/bpf: Add preparation steps and sysroot info to readme")
author Signed-off-by missing
committer Signed-off-by missing
author email: ivan.khoronzhuk@linaro.org
committer email: daniel@iogearbox.net
Commit 1878c1de4607 ("samples/bpf: Use __LINUX_ARM_ARCH__ selector for arm")
author Signed-off-by missing
committer Signed-off-by missing
author email: ivan.khoronzhuk@linaro.org
committer email: daniel@iogearbox.net
Errors in tree with Signed-off-by, please fix!
On Fri, Oct 11, 2019 at 02:00:56AM +0200, Daniel Borkmann wrote:
>On Wed, Oct 09, 2019 at 11:41:19PM +0300, Ivan Khoronzhuk wrote:
>> This series contains mainly fixes/improvements for cross-compilation
>> but not only, tested for arm, arm64, and intended for any arch.
>> Also verified on native build (not cross compilation) for x86_64
>> and arm, arm64.
>[...]
>
>There are multiple SOBs missing, please fix. Thanks!
>
>[...]
> 5 files changed, 218 insertions(+), 99 deletions(-)
> create mode 100644 samples/bpf/Makefile.target
> rename tools/lib/bpf/{test_libbpf.cpp => test_libbpf.c} (61%)
>Deleted branch mbox (was 9f35d1d0c8f0).
>Commit 9f35d1d0c8f0 ("samples/bpf: Add preparation steps and sysroot info to readme")
> author Signed-off-by missing
> committer Signed-off-by missing
> author email: ivan.khoronzhuk@linaro.org
> committer email: daniel@iogearbox.net
>
>
>Commit 1878c1de4607 ("samples/bpf: Use __LINUX_ARM_ARCH__ selector for arm")
> author Signed-off-by missing
> committer Signed-off-by missing
> author email: ivan.khoronzhuk@linaro.org
> committer email: daniel@iogearbox.net
>
>
>Errors in tree with Signed-off-by, please fix!
Done.
--
Regards,
Ivan Khoronzhuk