[PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds

bpf.vger.kernel.org archive mirror
 help / color / mirror / Atom feed

* [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds
@ 2020-11-09 11:09 Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 1/6] tools: Factor HOSTCC, HOSTLD, HOSTAR definitions Jean-Philippe Brucker
                   ` (6 more replies)
  0 siblings, 7 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

A few fixes for cross and out-of-tree build of bpftool and runqslower.
These changes allow to build for different target architectures, using
the same source tree.

I sent [v1] ages ago but haven't found time to resend. No change except
rebasing on the latest bpf-next/master.

[v1] https://lore.kernel.org/bpf/20200827153629.3820891-1-jean-philippe@linaro.org/

Jean-Philippe Brucker (6):
  tools: Factor HOSTCC, HOSTLD, HOSTAR definitions
  tools/bpftool: Force clean of out-of-tree build
  tools/bpftool: Fix cross-build
  tools/runqslower: Use Makefile.include
  tools/runqslower: Enable out-of-tree build
  tools/runqslower: Build bpftool using HOSTCC

 tools/bpf/bpftool/Makefile        | 38 +++++++++++++----
 tools/bpf/resolve_btfids/Makefile |  9 ----
 tools/bpf/runqslower/Makefile     | 68 ++++++++++++++++++-------------
 tools/build/Makefile              |  4 --
 tools/objtool/Makefile            |  9 ----
 tools/perf/Makefile.perf          |  4 --
 tools/power/acpi/Makefile.config  |  1 -
 tools/scripts/Makefile.include    | 10 +++++
 8 files changed, 78 insertions(+), 65 deletions(-)

-- 
2.29.1


^ permalink raw reply	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 1/6] tools: Factor HOSTCC, HOSTLD, HOSTAR definitions
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 2/6] tools/bpftool: Force clean of out-of-tree build Jean-Philippe Brucker
                   ` (5 subsequent siblings)
  6 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker, Jiri Olsa, Rafael J . Wysocki,
	Josh Poimboeuf, Peter Zijlstra, Ingo Molnar,
	Arnaldo Carvalho de Melo, Mark Rutland, Alexander Shishkin,
	Namhyung Kim, Robert Moore, Erik Kaneda, Len Brown, linux-acpi,
	devel

Several Makefiles in tools/ need to define the host toolchain variables.
Move their definition to tools/scripts/Makefile.include

Acked-by: Jiri Olsa <jolsa@redhat.com>
Acked-by: Rafael J. Wysocki <rafael.j.wysocki@intel.com>
Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
v1: https://lore.kernel.org/bpf/20200827153629.3820891-2-jean-philippe@linaro.org/

Cc: Josh Poimboeuf <jpoimboe@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Arnaldo Carvalho de Melo <acme@kernel.org>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Alexander Shishkin <alexander.shishkin@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Robert Moore <robert.moore@intel.com>
Cc: Erik Kaneda <erik.kaneda@intel.com>
Cc: "Rafael J. Wysocki" <rafael.j.wysocki@intel.com>
Cc: Len Brown <lenb@kernel.org>
Cc: linux-acpi@vger.kernel.org
Cc: devel@acpica.org
---
 tools/bpf/resolve_btfids/Makefile |  9 ---------
 tools/build/Makefile              |  4 ----
 tools/objtool/Makefile            |  9 ---------
 tools/perf/Makefile.perf          |  4 ----
 tools/power/acpi/Makefile.config  |  1 -
 tools/scripts/Makefile.include    | 10 ++++++++++
 6 files changed, 10 insertions(+), 27 deletions(-)

diff --git a/tools/bpf/resolve_btfids/Makefile b/tools/bpf/resolve_btfids/Makefile
index 66cb92136de4..bf656432ad73 100644
--- a/tools/bpf/resolve_btfids/Makefile
+++ b/tools/bpf/resolve_btfids/Makefile
@@ -18,15 +18,6 @@ else
 endif
 
 # always use the host compiler
-ifneq ($(LLVM),)
-HOSTAR  ?= llvm-ar
-HOSTCC  ?= clang
-HOSTLD  ?= ld.lld
-else
-HOSTAR  ?= ar
-HOSTCC  ?= gcc
-HOSTLD  ?= ld
-endif
 AR       = $(HOSTAR)
 CC       = $(HOSTCC)
 LD       = $(HOSTLD)
diff --git a/tools/build/Makefile b/tools/build/Makefile
index 722f1700d96a..bae48e6fa995 100644
--- a/tools/build/Makefile
+++ b/tools/build/Makefile
@@ -15,10 +15,6 @@ endef
 $(call allow-override,CC,$(CROSS_COMPILE)gcc)
 $(call allow-override,LD,$(CROSS_COMPILE)ld)
 
-HOSTCC ?= gcc
-HOSTLD ?= ld
-HOSTAR ?= ar
-
 export HOSTCC HOSTLD HOSTAR
 
 ifeq ($(V),1)
diff --git a/tools/objtool/Makefile b/tools/objtool/Makefile
index 4ea9a833dde7..5cdb19036d7f 100644
--- a/tools/objtool/Makefile
+++ b/tools/objtool/Makefile
@@ -3,15 +3,6 @@ include ../scripts/Makefile.include
 include ../scripts/Makefile.arch
 
 # always use the host compiler
-ifneq ($(LLVM),)
-HOSTAR	?= llvm-ar
-HOSTCC	?= clang
-HOSTLD	?= ld.lld
-else
-HOSTAR	?= ar
-HOSTCC	?= gcc
-HOSTLD	?= ld
-endif
 AR	 = $(HOSTAR)
 CC	 = $(HOSTCC)
 LD	 = $(HOSTLD)
diff --git a/tools/perf/Makefile.perf b/tools/perf/Makefile.perf
index 7ce3f2e8b9c7..62f3deb1d3a8 100644
--- a/tools/perf/Makefile.perf
+++ b/tools/perf/Makefile.perf
@@ -175,10 +175,6 @@ endef
 
 LD += $(EXTRA_LDFLAGS)
 
-HOSTCC  ?= gcc
-HOSTLD  ?= ld
-HOSTAR  ?= ar
-
 PKG_CONFIG = $(CROSS_COMPILE)pkg-config
 LLVM_CONFIG ?= llvm-config
 
diff --git a/tools/power/acpi/Makefile.config b/tools/power/acpi/Makefile.config
index 54a2857c2510..331f6d30f472 100644
--- a/tools/power/acpi/Makefile.config
+++ b/tools/power/acpi/Makefile.config
@@ -54,7 +54,6 @@ INSTALL_SCRIPT = ${INSTALL_PROGRAM}
 CROSS = #/usr/i386-linux-uclibc/usr/bin/i386-uclibc-
 CROSS_COMPILE ?= $(CROSS)
 LD = $(CC)
-HOSTCC = gcc
 
 # check if compiler option is supported
 cc-supports = ${shell if $(CC) ${1} -S -o /dev/null -x c /dev/null > /dev/null 2>&1; then echo "$(1)"; fi;}
diff --git a/tools/scripts/Makefile.include b/tools/scripts/Makefile.include
index a7974638561c..1358e89cdf7d 100644
--- a/tools/scripts/Makefile.include
+++ b/tools/scripts/Makefile.include
@@ -59,6 +59,16 @@ $(call allow-override,LD,$(CROSS_COMPILE)ld)
 $(call allow-override,CXX,$(CROSS_COMPILE)g++)
 $(call allow-override,STRIP,$(CROSS_COMPILE)strip)
 
+ifneq ($(LLVM),)
+HOSTAR  ?= llvm-ar
+HOSTCC  ?= clang
+HOSTLD  ?= ld.lld
+else
+HOSTAR  ?= ar
+HOSTCC  ?= gcc
+HOSTLD  ?= ld
+endif
+
 ifeq ($(CC_NO_CLANG), 1)
 EXTRA_WARNINGS += -Wstrict-aliasing=3
 endif
-- 
2.29.1


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 2/6] tools/bpftool: Force clean of out-of-tree build
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 1/6] tools: Factor HOSTCC, HOSTLD, HOSTAR definitions Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build Jean-Philippe Brucker
                   ` (4 subsequent siblings)
  6 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

Cleaning a partial build can fail if the output directory for libbpf
wasn't created:

$ make -C tools/bpf/bpftool O=/tmp/bpf clean
/bin/sh: line 0: cd: /tmp/bpf/libbpf/: No such file or directory
tools/scripts/Makefile.include:17: *** output directory "/tmp/bpf/libbpf/" does not exist.  Stop.
make: *** [Makefile:36: /tmp/bpf/libbpf/libbpf.a-clean] Error 2

As a result make never gets around to clearing the leftover objects. Add
the libbpf output directory as clean dependency to ensure clean always
succeeds (similarly to the "descend" macro). The directory is later
removed by the clean recipe.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 tools/bpf/bpftool/Makefile | 8 +++++---
 1 file changed, 5 insertions(+), 3 deletions(-)

diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
index f60e6ad3a1df..1358c093b812 100644
--- a/tools/bpf/bpftool/Makefile
+++ b/tools/bpf/bpftool/Makefile
@@ -27,11 +27,13 @@ LIBBPF = $(LIBBPF_PATH)libbpf.a

 BPFTOOL_VERSION ?= $(shell make -rR --no-print-directory -sC ../../.. kernelversion)

-$(LIBBPF): FORCE
-	$(if $(LIBBPF_OUTPUT),@mkdir -p $(LIBBPF_OUTPUT))
+$(LIBBPF_OUTPUT):
+	$(QUIET_MKDIR)mkdir -p $@
+
+$(LIBBPF): FORCE | $(LIBBPF_OUTPUT)
 	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_OUTPUT) $(LIBBPF_OUTPUT)libbpf.a

-$(LIBBPF)-clean:
+$(LIBBPF)-clean: $(LIBBPF_OUTPUT)
 	$(call QUIET_CLEAN, libbpf)
 	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_OUTPUT) clean >/dev/null

-- 
2.29.1

^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 1/6] tools: Factor HOSTCC, HOSTLD, HOSTAR definitions Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 2/6] tools/bpftool: Force clean of out-of-tree build Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 20:17   ` Andrii Nakryiko
  2020-11-09 11:09 ` [PATCH bpf-next v2 4/6] tools/runqslower: Use Makefile.include Jean-Philippe Brucker
                   ` (3 subsequent siblings)
  6 siblings, 1 reply; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

The bpftool build first creates an intermediate binary, executed on the
host, to generate skeletons required by the final build. When
cross-building bpftool for an architecture different from the host, the
intermediate binary should be built using the host compiler (gcc) and
the final bpftool using the cross compiler (e.g. aarch64-linux-gnu-gcc).

Generate the intermediate objects into the bootstrap/ directory using
the host toolchain.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 tools/bpf/bpftool/Makefile | 32 +++++++++++++++++++++++++-------
 1 file changed, 25 insertions(+), 7 deletions(-)

diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
index 1358c093b812..0705c48e0ce0 100644
--- a/tools/bpf/bpftool/Makefile
+++ b/tools/bpf/bpftool/Makefile
@@ -19,24 +19,36 @@ BPF_DIR = $(srctree)/tools/lib/bpf/
 ifneq ($(OUTPUT),)
   LIBBPF_OUTPUT = $(OUTPUT)/libbpf/
   LIBBPF_PATH = $(LIBBPF_OUTPUT)
+  BOOTSTRAP_OUTPUT = $(OUTPUT)/bootstrap/
 else
   LIBBPF_PATH = $(BPF_DIR)
+  BOOTSTRAP_OUTPUT = $(CURDIR)/bootstrap/
 endif
 
 LIBBPF = $(LIBBPF_PATH)libbpf.a
+LIBBPF_BOOTSTRAP_OUTPUT = $(BOOTSTRAP_OUTPUT)libbpf/
+LIBBPF_BOOTSTRAP = $(LIBBPF_BOOTSTRAP_OUTPUT)libbpf.a
 
 BPFTOOL_VERSION ?= $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
 
-$(LIBBPF_OUTPUT):
+$(LIBBPF_OUTPUT) $(BOOTSTRAP_OUTPUT) $(LIBBPF_BOOTSTRAP_OUTPUT):
 	$(QUIET_MKDIR)mkdir -p $@
 
 $(LIBBPF): FORCE | $(LIBBPF_OUTPUT)
 	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_OUTPUT) $(LIBBPF_OUTPUT)libbpf.a
 
+$(LIBBPF_BOOTSTRAP): FORCE | $(LIBBPF_BOOTSTRAP_OUTPUT)
+	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) \
+		ARCH= CC=$(HOSTCC) LD=$(HOSTLD) $@
+
 $(LIBBPF)-clean: $(LIBBPF_OUTPUT)
 	$(call QUIET_CLEAN, libbpf)
 	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_OUTPUT) clean >/dev/null
 
+$(LIBBPF_BOOTSTRAP)-clean: $(LIBBPF_BOOTSTRAP_OUTPUT)
+	$(call QUIET_CLEAN, libbpf-bootstrap)
+	$(Q)$(MAKE) -C $(BPF_DIR) OUTPUT=$(LIBBPF_BOOTSTRAP_OUTPUT) clean >/dev/null
+
 prefix ?= /usr/local
 bash_compdir ?= /usr/share/bash-completion/completions
 
@@ -94,6 +106,7 @@ CFLAGS += -DCOMPAT_NEED_REALLOCARRAY
 endif
 
 LIBS = $(LIBBPF) -lelf -lz
+LIBS_BOOTSTRAP = $(LIBBPF_BOOTSTRAP) -lelf -lz
 ifeq ($(feature-libcap), 1)
 CFLAGS += -DUSE_LIBCAP
 LIBS += -lcap
@@ -120,9 +133,9 @@ CFLAGS += -DHAVE_LIBBFD_SUPPORT
 SRCS += $(BFD_SRCS)
 endif
 
-BPFTOOL_BOOTSTRAP := $(if $(OUTPUT),$(OUTPUT)bpftool-bootstrap,./bpftool-bootstrap)
+BPFTOOL_BOOTSTRAP := $(BOOTSTRAP_OUTPUT)bpftool
 
-BOOTSTRAP_OBJS = $(addprefix $(OUTPUT),main.o common.o json_writer.o gen.o btf.o)
+BOOTSTRAP_OBJS = $(addprefix $(BOOTSTRAP_OUTPUT),main.o common.o json_writer.o gen.o btf.o)
 OBJS = $(patsubst %.c,$(OUTPUT)%.o,$(SRCS)) $(OUTPUT)disasm.o
 
 VMLINUX_BTF_PATHS ?= $(if $(O),$(O)/vmlinux)				\
@@ -169,12 +182,16 @@ $(OUTPUT)disasm.o: $(srctree)/kernel/bpf/disasm.c
 
 $(OUTPUT)feature.o: | zdep
 
-$(BPFTOOL_BOOTSTRAP): $(BOOTSTRAP_OBJS) $(LIBBPF)
-	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(BOOTSTRAP_OBJS) $(LIBS)
+$(BPFTOOL_BOOTSTRAP): $(BOOTSTRAP_OBJS) $(LIBBPF_BOOTSTRAP)
+	$(QUIET_LINK)$(HOSTCC) $(CFLAGS) $(LDFLAGS) -o $@ $(BOOTSTRAP_OBJS) \
+		$(LIBS_BOOTSTRAP)
 
 $(OUTPUT)bpftool: $(OBJS) $(LIBBPF)
 	$(QUIET_LINK)$(CC) $(CFLAGS) $(LDFLAGS) -o $@ $(OBJS) $(LIBS)
 
+$(BOOTSTRAP_OUTPUT)%.o: %.c | $(BOOTSTRAP_OUTPUT)
+	$(QUIET_CC)$(HOSTCC) $(CFLAGS) -c -MMD -o $@ $<
+
 $(OUTPUT)%.o: %.c
 	$(QUIET_CC)$(CC) $(CFLAGS) -c -MMD -o $@ $<
 
@@ -182,11 +199,12 @@ feature-detect-clean:
 	$(call QUIET_CLEAN, feature-detect)
 	$(Q)$(MAKE) -C $(srctree)/tools/build/feature/ clean >/dev/null
 
-clean: $(LIBBPF)-clean feature-detect-clean
+clean: $(LIBBPF)-clean $(LIBBPF_BOOTSTRAP)-clean feature-detect-clean
 	$(call QUIET_CLEAN, bpftool)
 	$(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT)*.o $(OUTPUT)*.d
-	$(Q)$(RM) -- $(BPFTOOL_BOOTSTRAP) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
+	$(Q)$(RM) -- $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
 	$(Q)$(RM) -r -- $(OUTPUT)libbpf/
+	$(Q)$(RM) -r -- $(BOOTSTRAP_OUTPUT)
 	$(call QUIET_CLEAN, core-gen)
 	$(Q)$(RM) -- $(OUTPUT)FEATURE-DUMP.bpftool
 	$(Q)$(RM) -r -- $(OUTPUT)feature/
-- 
2.29.1


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 4/6] tools/runqslower: Use Makefile.include
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
                   ` (2 preceding siblings ...)
  2020-11-09 11:09 ` [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 11:09 ` [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build Jean-Philippe Brucker
                   ` (2 subsequent siblings)
  6 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

Makefile.include defines variables such as OUTPUT and CC for out-of-tree
build and cross-build. Include it into the runqslower Makefile and use
its $(QUIET*) helpers.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 tools/bpf/runqslower/Makefile | 24 +++++++++---------------
 1 file changed, 9 insertions(+), 15 deletions(-)

diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
index fb1337d69868..bcc4a7396713 100644
--- a/tools/bpf/runqslower/Makefile
+++ b/tools/bpf/runqslower/Makefile
@@ -1,4 +1,6 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
+include ../../scripts/Makefile.include
+
 OUTPUT := .output
 CLANG ?= clang
 LLC ?= llc
@@ -21,10 +23,8 @@ VMLINUX_BTF_PATH := $(or $(VMLINUX_BTF),$(firstword			       \
 abs_out := $(abspath $(OUTPUT))
 ifeq ($(V),1)
 Q =
-msg =
 else
 Q = @
-msg = @printf '  %-8s %s%s\n' "$(1)" "$(notdir $(2))" "$(if $(3), $(3))";
 MAKEFLAGS += --no-print-directory
 submake_extras := feature_display=0
 endif
@@ -37,12 +37,11 @@ all: runqslower
 runqslower: $(OUTPUT)/runqslower
 
 clean:
-	$(call msg,CLEAN)
+	$(call QUIET_CLEAN, runqslower)
 	$(Q)rm -rf $(OUTPUT) runqslower
 
 $(OUTPUT)/runqslower: $(OUTPUT)/runqslower.o $(BPFOBJ)
-	$(call msg,BINARY,$@)
-	$(Q)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
+	$(QUIET_LINK)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
 
 $(OUTPUT)/runqslower.o: runqslower.h $(OUTPUT)/runqslower.skel.h	      \
 			$(OUTPUT)/runqslower.bpf.o
@@ -50,31 +49,26 @@ $(OUTPUT)/runqslower.o: runqslower.h $(OUTPUT)/runqslower.skel.h	      \
 $(OUTPUT)/runqslower.bpf.o: $(OUTPUT)/vmlinux.h runqslower.h
 
 $(OUTPUT)/%.skel.h: $(OUTPUT)/%.bpf.o | $(BPFTOOL)
-	$(call msg,GEN-SKEL,$@)
-	$(Q)$(BPFTOOL) gen skeleton $< > $@
+	$(QUIET_GEN)$(BPFTOOL) gen skeleton $< > $@
 
 $(OUTPUT)/%.bpf.o: %.bpf.c $(BPFOBJ) | $(OUTPUT)
-	$(call msg,BPF,$@)
-	$(Q)$(CLANG) -g -O2 -target bpf $(INCLUDES)			      \
+	$(QUIET_GEN)$(CLANG) -g -O2 -target bpf $(INCLUDES)		      \
 		 -c $(filter %.c,$^) -o $@ &&				      \
 	$(LLVM_STRIP) -g $@
 
 $(OUTPUT)/%.o: %.c | $(OUTPUT)
-	$(call msg,CC,$@)
-	$(Q)$(CC) $(CFLAGS) $(INCLUDES) -c $(filter %.c,$^) -o $@
+	$(QUIET_CC)$(CC) $(CFLAGS) $(INCLUDES) -c $(filter %.c,$^) -o $@
 
 $(OUTPUT):
-	$(call msg,MKDIR,$@)
-	$(Q)mkdir -p $(OUTPUT)
+	$(QUIET_MKDIR)mkdir -p $(OUTPUT)
 
 $(OUTPUT)/vmlinux.h: $(VMLINUX_BTF_PATH) | $(OUTPUT) $(BPFTOOL)
-	$(call msg,GEN,$@)
 	$(Q)if [ ! -e "$(VMLINUX_BTF_PATH)" ] ; then \
 		echo "Couldn't find kernel BTF; set VMLINUX_BTF to"	       \
 			"specify its location." >&2;			       \
 		exit 1;\
 	fi
-	$(Q)$(BPFTOOL) btf dump file $(VMLINUX_BTF_PATH) format c > $@
+	$(QUIET_GEN)$(BPFTOOL) btf dump file $(VMLINUX_BTF_PATH) format c > $@
 
 $(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(OUTPUT)
 	$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC)			       \
-- 
2.29.1


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
                   ` (3 preceding siblings ...)
  2020-11-09 11:09 ` [PATCH bpf-next v2 4/6] tools/runqslower: Use Makefile.include Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 20:23   ` Andrii Nakryiko
  2020-11-09 11:09 ` [PATCH bpf-next v2 6/6] tools/runqslower: Build bpftool using HOSTCC Jean-Philippe Brucker
  2020-11-09 18:11 ` [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Andrii Nakryiko
  6 siblings, 1 reply; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

Enable out-of-tree build for runqslower. Only set OUTPUT=.output if it
wasn't already set by the user.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 tools/bpf/runqslower/Makefile | 45 +++++++++++++++++++++++------------
 1 file changed, 30 insertions(+), 15 deletions(-)

diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
index bcc4a7396713..861f4dcde960 100644
--- a/tools/bpf/runqslower/Makefile
+++ b/tools/bpf/runqslower/Makefile
@@ -1,15 +1,20 @@
 # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
 include ../../scripts/Makefile.include
 
-OUTPUT := .output
+ifeq ($(OUTPUT),)
+  OUTPUT = $(abspath .output)/
+endif
+
 CLANG ?= clang
 LLC ?= llc
 LLVM_STRIP ?= llvm-strip
-DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
+BPFTOOL_OUTPUT := $(OUTPUT)bpftool/
+DEFAULT_BPFTOOL := $(BPFTOOL_OUTPUT)bpftool
 BPFTOOL ?= $(DEFAULT_BPFTOOL)
 LIBBPF_SRC := $(abspath ../../lib/bpf)
-BPFOBJ := $(OUTPUT)/libbpf.a
-BPF_INCLUDE := $(OUTPUT)
+BPFOBJ_OUTPUT := $(OUTPUT)libbpf/
+BPFOBJ := $(BPFOBJ_OUTPUT)libbpf.a
+BPF_INCLUDE := $(BPFOBJ_OUTPUT)
 INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../lib)        \
        -I$(abspath ../../include/uapi)
 CFLAGS := -g -Wall
@@ -20,7 +25,6 @@ VMLINUX_BTF_PATHS := /sys/kernel/btf/vmlinux /boot/vmlinux-$(KERNEL_REL)
 VMLINUX_BTF_PATH := $(or $(VMLINUX_BTF),$(firstword			       \
 					  $(wildcard $(VMLINUX_BTF_PATHS))))
 
-abs_out := $(abspath $(OUTPUT))
 ifeq ($(V),1)
 Q =
 else
@@ -36,9 +40,13 @@ all: runqslower
 
 runqslower: $(OUTPUT)/runqslower
 
-clean:
+clean: $(DEFAULT_BPFTOOL)-clean $(BPFOBJ)-clean
 	$(call QUIET_CLEAN, runqslower)
-	$(Q)rm -rf $(OUTPUT) runqslower
+	$(Q)$(RM) -r $(BPFOBJ_OUTPUT) $(BPFTOOL_OUTPUT)
+	$(Q)$(RM) $(OUTPUT)*.o $(OUTPUT)*.d
+	$(Q)$(RM) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
+	$(Q)$(RM) $(OUTPUT)runqslower
+	$(Q)$(RM) -r .output
 
 $(OUTPUT)/runqslower: $(OUTPUT)/runqslower.o $(BPFOBJ)
 	$(QUIET_LINK)$(CC) $(CFLAGS) $^ -lelf -lz -o $@
@@ -59,8 +67,8 @@ $(OUTPUT)/%.bpf.o: %.bpf.c $(BPFOBJ) | $(OUTPUT)
 $(OUTPUT)/%.o: %.c | $(OUTPUT)
 	$(QUIET_CC)$(CC) $(CFLAGS) $(INCLUDES) -c $(filter %.c,$^) -o $@
 
-$(OUTPUT):
-	$(QUIET_MKDIR)mkdir -p $(OUTPUT)
+$(OUTPUT) $(BPFOBJ_OUTPUT) $(BPFTOOL_OUTPUT):
+	$(QUIET_MKDIR)mkdir -p $@
 
 $(OUTPUT)/vmlinux.h: $(VMLINUX_BTF_PATH) | $(OUTPUT) $(BPFTOOL)
 	$(Q)if [ ! -e "$(VMLINUX_BTF_PATH)" ] ; then \
@@ -70,10 +78,17 @@ $(OUTPUT)/vmlinux.h: $(VMLINUX_BTF_PATH) | $(OUTPUT) $(BPFTOOL)
 	fi
 	$(QUIET_GEN)$(BPFTOOL) btf dump file $(VMLINUX_BTF_PATH) format c > $@
 
-$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(OUTPUT)
-	$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC)			       \
-		    OUTPUT=$(abspath $(dir $@))/ $(abspath $@)
+$(BPFOBJ): $(wildcard $(LIBBPF_SRC)/*.[ch] $(LIBBPF_SRC)/Makefile) | $(BPFOBJ_OUTPUT)
+	$(Q)$(MAKE) $(submake_extras) -C $(LIBBPF_SRC) OUTPUT=$(BPFOBJ_OUTPUT) $@
+
+$(BPFOBJ)-clean: $(BPFOBJ_OUTPUT)
+	$(Q)$(MAKE) -C $(LIBBPF_SRC) OUTPUT=$(BPFOBJ_OUTPUT) clean
 
-$(DEFAULT_BPFTOOL):
-	$(Q)$(MAKE) $(submake_extras) -C ../bpftool			      \
-		    prefix= OUTPUT=$(abs_out)/ DESTDIR=$(abs_out) install
+$(DEFAULT_BPFTOOL): | $(BPFTOOL_OUTPUT)
+	$(Q)$(MAKE) $(submake_extras) -C ../bpftool OUTPUT=$(BPFTOOL_OUTPUT)
+
+$(DEFAULT_BPFTOOL)-clean: $(BPFTOOL_OUTPUT)
+ifeq ($(DEFAULT_BPFTOOL),$(BPFTOOL))
+	$(call QUIET_CLEAN,bpftool)
+	$(Q)$(MAKE) -C ../bpftool OUTPUT=$(BPFTOOL_OUTPUT) clean
+endif
-- 
2.29.1


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* [PATCH bpf-next v2 6/6] tools/runqslower: Build bpftool using HOSTCC
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
                   ` (4 preceding siblings ...)
  2020-11-09 11:09 ` [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build Jean-Philippe Brucker
@ 2020-11-09 11:09 ` Jean-Philippe Brucker
  2020-11-09 18:11 ` [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Andrii Nakryiko
  6 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-09 11:09 UTC (permalink / raw)
  To: ast, daniel
  Cc: bpf, kafai, songliubraving, yhs, andriin, john.fastabend,
	kpsingh, Jean-Philippe Brucker

When cross building runqslower for an other architecture, the
intermediate bpftool used to generate a skeleton must be built using the
host toolchain. Pass HOSTCC and HOSTLD, defined in Makefile.include, to
the bpftool Makefile.

Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
---
 tools/bpf/runqslower/Makefile | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
index 861f4dcde960..fa5c18b70dd0 100644
--- a/tools/bpf/runqslower/Makefile
+++ b/tools/bpf/runqslower/Makefile
@@ -85,7 +85,8 @@ $(BPFOBJ)-clean: $(BPFOBJ_OUTPUT)
 	$(Q)$(MAKE) -C $(LIBBPF_SRC) OUTPUT=$(BPFOBJ_OUTPUT) clean
 
 $(DEFAULT_BPFTOOL): | $(BPFTOOL_OUTPUT)
-	$(Q)$(MAKE) $(submake_extras) -C ../bpftool OUTPUT=$(BPFTOOL_OUTPUT)
+	$(Q)$(MAKE) $(submake_extras) -C ../bpftool OUTPUT=$(BPFTOOL_OUTPUT)   \
+		    CC=$(HOSTCC) LD=$(HOSTLD)
 
 $(DEFAULT_BPFTOOL)-clean: $(BPFTOOL_OUTPUT)
 ifeq ($(DEFAULT_BPFTOOL),$(BPFTOOL))
-- 
2.29.1


^ permalink raw reply related	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds
  2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
                   ` (5 preceding siblings ...)
  2020-11-09 11:09 ` [PATCH bpf-next v2 6/6] tools/runqslower: Build bpftool using HOSTCC Jean-Philippe Brucker
@ 2020-11-09 18:11 ` Andrii Nakryiko
  2020-11-10 12:42   ` Jean-Philippe Brucker
  6 siblings, 1 reply; 14+ messages in thread
From: Andrii Nakryiko @ 2020-11-09 18:11 UTC (permalink / raw)
  To: Jean-Philippe Brucker
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> A few fixes for cross and out-of-tree build of bpftool and runqslower.
> These changes allow to build for different target architectures, using
> the same source tree.
>
> I sent [v1] ages ago but haven't found time to resend. No change except
> rebasing on the latest bpf-next/master.
>
> [v1] https://lore.kernel.org/bpf/20200827153629.3820891-1-jean-philippe@linaro.org/
>

While you are looking at bpftool builds... Seems like it regressed
recently and doesn't honor -jX setting. Either way the build is
sequential (and rather slow). Do you mind checking if your changes
could fix the regression (I haven't had a chance to bisect the
offending change causing regression).


> Jean-Philippe Brucker (6):
>   tools: Factor HOSTCC, HOSTLD, HOSTAR definitions
>   tools/bpftool: Force clean of out-of-tree build
>   tools/bpftool: Fix cross-build
>   tools/runqslower: Use Makefile.include
>   tools/runqslower: Enable out-of-tree build
>   tools/runqslower: Build bpftool using HOSTCC
>
>  tools/bpf/bpftool/Makefile        | 38 +++++++++++++----
>  tools/bpf/resolve_btfids/Makefile |  9 ----
>  tools/bpf/runqslower/Makefile     | 68 ++++++++++++++++++-------------
>  tools/build/Makefile              |  4 --
>  tools/objtool/Makefile            |  9 ----
>  tools/perf/Makefile.perf          |  4 --
>  tools/power/acpi/Makefile.config  |  1 -
>  tools/scripts/Makefile.include    | 10 +++++
>  8 files changed, 78 insertions(+), 65 deletions(-)
>
> --
> 2.29.1
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build
  2020-11-09 11:09 ` [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build Jean-Philippe Brucker
@ 2020-11-09 20:17   ` Andrii Nakryiko
  2020-11-10 12:43     ` Jean-Philippe Brucker
  0 siblings, 1 reply; 14+ messages in thread
From: Andrii Nakryiko @ 2020-11-09 20:17 UTC (permalink / raw)
  To: Jean-Philippe Brucker
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> The bpftool build first creates an intermediate binary, executed on the
> host, to generate skeletons required by the final build. When
> cross-building bpftool for an architecture different from the host, the
> intermediate binary should be built using the host compiler (gcc) and
> the final bpftool using the cross compiler (e.g. aarch64-linux-gnu-gcc).
>
> Generate the intermediate objects into the bootstrap/ directory using
> the host toolchain.
>
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
>  tools/bpf/bpftool/Makefile | 32 +++++++++++++++++++++++++-------
>  1 file changed, 25 insertions(+), 7 deletions(-)
>
> diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
> index 1358c093b812..0705c48e0ce0 100644
> --- a/tools/bpf/bpftool/Makefile
> +++ b/tools/bpf/bpftool/Makefile
> @@ -19,24 +19,36 @@ BPF_DIR = $(srctree)/tools/lib/bpf/
>  ifneq ($(OUTPUT),)
>    LIBBPF_OUTPUT = $(OUTPUT)/libbpf/
>    LIBBPF_PATH = $(LIBBPF_OUTPUT)
> +  BOOTSTRAP_OUTPUT = $(OUTPUT)/bootstrap/
>  else

LIBBPF_OUTPUT is not set here, can you please fix that as well?

>    LIBBPF_PATH = $(BPF_DIR)
> +  BOOTSTRAP_OUTPUT = $(CURDIR)/bootstrap/
>  endif
>

[...]

> -clean: $(LIBBPF)-clean feature-detect-clean
> +clean: $(LIBBPF)-clean $(LIBBPF_BOOTSTRAP)-clean feature-detect-clean
>         $(call QUIET_CLEAN, bpftool)
>         $(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT)*.o $(OUTPUT)*.d
> -       $(Q)$(RM) -- $(BPFTOOL_BOOTSTRAP) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
> +       $(Q)$(RM) -- $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
>         $(Q)$(RM) -r -- $(OUTPUT)libbpf/
> +       $(Q)$(RM) -r -- $(BOOTSTRAP_OUTPUT)

Can you combine it with the previous line, maybe also specify more
explicitly $(LIBBPF_OUTPUT) instead of $(OUTPUT)libbpf/?

>         $(call QUIET_CLEAN, core-gen)
>         $(Q)$(RM) -- $(OUTPUT)FEATURE-DUMP.bpftool
>         $(Q)$(RM) -r -- $(OUTPUT)feature/
> --
> 2.29.1
>

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build
  2020-11-09 11:09 ` [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build Jean-Philippe Brucker
@ 2020-11-09 20:23   ` Andrii Nakryiko
  2020-11-10 12:43     ` Jean-Philippe Brucker
  0 siblings, 1 reply; 14+ messages in thread
From: Andrii Nakryiko @ 2020-11-09 20:23 UTC (permalink / raw)
  To: Jean-Philippe Brucker
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> Enable out-of-tree build for runqslower. Only set OUTPUT=.output if it
> wasn't already set by the user.
>
> Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> ---
>  tools/bpf/runqslower/Makefile | 45 +++++++++++++++++++++++------------
>  1 file changed, 30 insertions(+), 15 deletions(-)
>
> diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
> index bcc4a7396713..861f4dcde960 100644
> --- a/tools/bpf/runqslower/Makefile
> +++ b/tools/bpf/runqslower/Makefile
> @@ -1,15 +1,20 @@
>  # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
>  include ../../scripts/Makefile.include
>
> -OUTPUT := .output
> +ifeq ($(OUTPUT),)
> +  OUTPUT = $(abspath .output)/
> +endif

OUTPUT ?= .. didn't work?

> +
>  CLANG ?= clang
>  LLC ?= llc
>  LLVM_STRIP ?= llvm-strip
> -DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
> +BPFTOOL_OUTPUT := $(OUTPUT)bpftool/
> +DEFAULT_BPFTOOL := $(BPFTOOL_OUTPUT)bpftool
>  BPFTOOL ?= $(DEFAULT_BPFTOOL)
>  LIBBPF_SRC := $(abspath ../../lib/bpf)
> -BPFOBJ := $(OUTPUT)/libbpf.a
> -BPF_INCLUDE := $(OUTPUT)
> +BPFOBJ_OUTPUT := $(OUTPUT)libbpf/
> +BPFOBJ := $(BPFOBJ_OUTPUT)libbpf.a
> +BPF_INCLUDE := $(BPFOBJ_OUTPUT)
>  INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../lib)        \
>         -I$(abspath ../../include/uapi)
>  CFLAGS := -g -Wall
> @@ -20,7 +25,6 @@ VMLINUX_BTF_PATHS := /sys/kernel/btf/vmlinux /boot/vmlinux-$(KERNEL_REL)
>  VMLINUX_BTF_PATH := $(or $(VMLINUX_BTF),$(firstword                           \
>                                           $(wildcard $(VMLINUX_BTF_PATHS))))
>
> -abs_out := $(abspath $(OUTPUT))
>  ifeq ($(V),1)
>  Q =
>  else
> @@ -36,9 +40,13 @@ all: runqslower
>
>  runqslower: $(OUTPUT)/runqslower
>
> -clean:
> +clean: $(DEFAULT_BPFTOOL)-clean $(BPFOBJ)-clean

why separate targets for $(DEFAULT_BPFTOOL)-clean and $(BPFOBJ)-clean?
Are they intended to be called separately? I don't think
parallelization is that important for the clean up. Let's just keep
all the cleaning in one place, not spreading it across Makefile?

>         $(call QUIET_CLEAN, runqslower)
> -       $(Q)rm -rf $(OUTPUT) runqslower
> +       $(Q)$(RM) -r $(BPFOBJ_OUTPUT) $(BPFTOOL_OUTPUT)
> +       $(Q)$(RM) $(OUTPUT)*.o $(OUTPUT)*.d
> +       $(Q)$(RM) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
> +       $(Q)$(RM) $(OUTPUT)runqslower
> +       $(Q)$(RM) -r .output
>

[...]

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds
  2020-11-09 18:11 ` [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Andrii Nakryiko
@ 2020-11-10 12:42   ` Jean-Philippe Brucker
  2020-11-10 18:38     ` Andrii Nakryiko
  0 siblings, 1 reply; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-10 12:42 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 09, 2020 at 10:11:52AM -0800, Andrii Nakryiko wrote:
> On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > A few fixes for cross and out-of-tree build of bpftool and runqslower.
> > These changes allow to build for different target architectures, using
> > the same source tree.
> >
> > I sent [v1] ages ago but haven't found time to resend. No change except
> > rebasing on the latest bpf-next/master.
> >
> > [v1] https://lore.kernel.org/bpf/20200827153629.3820891-1-jean-philippe@linaro.org/
> >
> 
> While you are looking at bpftool builds... Seems like it regressed
> recently and doesn't honor -jX setting. Either way the build is
> sequential (and rather slow). Do you mind checking if your changes
> could fix the regression (I haven't had a chance to bisect the
> offending change causing regression).

I bisected it to ba2fd563b740 ("tools/bpftool: Support passing
BPFTOOL_VERSION to make"), in v5.9. As BPFTOOL_VERSION became a recursivly
expanded variable, the shell function is evaluated on every expansion of
CFLAGS. I'll add the fix to v3:

-BPFTOOL_VERSION ?= $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
+ifeq ($(BPFTOOL_VERSION),)
+BPFTOOL_VERSION := $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
+endif

Thanks,
Jean

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build
  2020-11-09 20:17   ` Andrii Nakryiko
@ 2020-11-10 12:43     ` Jean-Philippe Brucker
  0 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-10 12:43 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 09, 2020 at 12:17:49PM -0800, Andrii Nakryiko wrote:
> On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > The bpftool build first creates an intermediate binary, executed on the
> > host, to generate skeletons required by the final build. When
> > cross-building bpftool for an architecture different from the host, the
> > intermediate binary should be built using the host compiler (gcc) and
> > the final bpftool using the cross compiler (e.g. aarch64-linux-gnu-gcc).
> >
> > Generate the intermediate objects into the bootstrap/ directory using
> > the host toolchain.
> >
> > Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> > ---
> >  tools/bpf/bpftool/Makefile | 32 +++++++++++++++++++++++++-------
> >  1 file changed, 25 insertions(+), 7 deletions(-)
> >
> > diff --git a/tools/bpf/bpftool/Makefile b/tools/bpf/bpftool/Makefile
> > index 1358c093b812..0705c48e0ce0 100644
> > --- a/tools/bpf/bpftool/Makefile
> > +++ b/tools/bpf/bpftool/Makefile
> > @@ -19,24 +19,36 @@ BPF_DIR = $(srctree)/tools/lib/bpf/
> >  ifneq ($(OUTPUT),)
> >    LIBBPF_OUTPUT = $(OUTPUT)/libbpf/
> >    LIBBPF_PATH = $(LIBBPF_OUTPUT)
> > +  BOOTSTRAP_OUTPUT = $(OUTPUT)/bootstrap/
> >  else
> 
> LIBBPF_OUTPUT is not set here, can you please fix that as well?

Ok. I'll set it to "", so the clean receipe doesn't remove the source tree

> 
> >    LIBBPF_PATH = $(BPF_DIR)
> > +  BOOTSTRAP_OUTPUT = $(CURDIR)/bootstrap/
> >  endif
> >
> 
> [...]
> 
> > -clean: $(LIBBPF)-clean feature-detect-clean
> > +clean: $(LIBBPF)-clean $(LIBBPF_BOOTSTRAP)-clean feature-detect-clean
> >         $(call QUIET_CLEAN, bpftool)
> >         $(Q)$(RM) -- $(OUTPUT)bpftool $(OUTPUT)*.o $(OUTPUT)*.d
> > -       $(Q)$(RM) -- $(BPFTOOL_BOOTSTRAP) $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
> > +       $(Q)$(RM) -- $(OUTPUT)*.skel.h $(OUTPUT)vmlinux.h
> >         $(Q)$(RM) -r -- $(OUTPUT)libbpf/
> > +       $(Q)$(RM) -r -- $(BOOTSTRAP_OUTPUT)
> 
> Can you combine it with the previous line, maybe also specify more
> explicitly $(LIBBPF_OUTPUT) instead of $(OUTPUT)libbpf/?

Sure

Thanks,
Jean

> 
> >         $(call QUIET_CLEAN, core-gen)
> >         $(Q)$(RM) -- $(OUTPUT)FEATURE-DUMP.bpftool
> >         $(Q)$(RM) -r -- $(OUTPUT)feature/
> > --
> > 2.29.1
> >

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build
  2020-11-09 20:23   ` Andrii Nakryiko
@ 2020-11-10 12:43     ` Jean-Philippe Brucker
  0 siblings, 0 replies; 14+ messages in thread
From: Jean-Philippe Brucker @ 2020-11-10 12:43 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Mon, Nov 09, 2020 at 12:23:28PM -0800, Andrii Nakryiko wrote:
> On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
> <jean-philippe@linaro.org> wrote:
> >
> > Enable out-of-tree build for runqslower. Only set OUTPUT=.output if it
> > wasn't already set by the user.
> >
> > Signed-off-by: Jean-Philippe Brucker <jean-philippe@linaro.org>
> > ---
> >  tools/bpf/runqslower/Makefile | 45 +++++++++++++++++++++++------------
> >  1 file changed, 30 insertions(+), 15 deletions(-)
> >
> > diff --git a/tools/bpf/runqslower/Makefile b/tools/bpf/runqslower/Makefile
> > index bcc4a7396713..861f4dcde960 100644
> > --- a/tools/bpf/runqslower/Makefile
> > +++ b/tools/bpf/runqslower/Makefile
> > @@ -1,15 +1,20 @@
> >  # SPDX-License-Identifier: (LGPL-2.1 OR BSD-2-Clause)
> >  include ../../scripts/Makefile.include
> >
> > -OUTPUT := .output
> > +ifeq ($(OUTPUT),)
> > +  OUTPUT = $(abspath .output)/
> > +endif
> 
> OUTPUT ?= .. didn't work?

Yes it should, I'll change it. The difference is when make is invoked with
OUTPUT explicitly set to "", then ?= wouldn't override it. But I didn't
see any external Makefile passing OUTPUT= to this one, and the user-facing
option is O=

> 
> > +
> >  CLANG ?= clang
> >  LLC ?= llc
> >  LLVM_STRIP ?= llvm-strip
> > -DEFAULT_BPFTOOL := $(OUTPUT)/sbin/bpftool
> > +BPFTOOL_OUTPUT := $(OUTPUT)bpftool/
> > +DEFAULT_BPFTOOL := $(BPFTOOL_OUTPUT)bpftool
> >  BPFTOOL ?= $(DEFAULT_BPFTOOL)
> >  LIBBPF_SRC := $(abspath ../../lib/bpf)
> > -BPFOBJ := $(OUTPUT)/libbpf.a
> > -BPF_INCLUDE := $(OUTPUT)
> > +BPFOBJ_OUTPUT := $(OUTPUT)libbpf/
> > +BPFOBJ := $(BPFOBJ_OUTPUT)libbpf.a
> > +BPF_INCLUDE := $(BPFOBJ_OUTPUT)
> >  INCLUDES := -I$(OUTPUT) -I$(BPF_INCLUDE) -I$(abspath ../../lib)        \
> >         -I$(abspath ../../include/uapi)
> >  CFLAGS := -g -Wall
> > @@ -20,7 +25,6 @@ VMLINUX_BTF_PATHS := /sys/kernel/btf/vmlinux /boot/vmlinux-$(KERNEL_REL)
> >  VMLINUX_BTF_PATH := $(or $(VMLINUX_BTF),$(firstword                           \
> >                                           $(wildcard $(VMLINUX_BTF_PATHS))))
> >
> > -abs_out := $(abspath $(OUTPUT))
> >  ifeq ($(V),1)
> >  Q =
> >  else
> > @@ -36,9 +40,13 @@ all: runqslower
> >
> >  runqslower: $(OUTPUT)/runqslower
> >
> > -clean:
> > +clean: $(DEFAULT_BPFTOOL)-clean $(BPFOBJ)-clean
> 
> why separate targets for $(DEFAULT_BPFTOOL)-clean and $(BPFOBJ)-clean?
> Are they intended to be called separately? I don't think
> parallelization is that important for the clean up. Let's just keep
> all the cleaning in one place, not spreading it across Makefile?

Ok

Thanks,
Jean

^ permalink raw reply	[flat|nested] 14+ messages in thread

* Re: [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds
  2020-11-10 12:42   ` Jean-Philippe Brucker
@ 2020-11-10 18:38     ` Andrii Nakryiko
  0 siblings, 0 replies; 14+ messages in thread
From: Andrii Nakryiko @ 2020-11-10 18:38 UTC (permalink / raw)
  To: Jean-Philippe Brucker
  Cc: Alexei Starovoitov, Daniel Borkmann, bpf, Martin Lau, Song Liu,
	Yonghong Song, Andrii Nakryiko, john fastabend, KP Singh

On Tue, Nov 10, 2020 at 4:42 AM Jean-Philippe Brucker
<jean-philippe@linaro.org> wrote:
>
> On Mon, Nov 09, 2020 at 10:11:52AM -0800, Andrii Nakryiko wrote:
> > On Mon, Nov 9, 2020 at 3:11 AM Jean-Philippe Brucker
> > <jean-philippe@linaro.org> wrote:
> > >
> > > A few fixes for cross and out-of-tree build of bpftool and runqslower.
> > > These changes allow to build for different target architectures, using
> > > the same source tree.
> > >
> > > I sent [v1] ages ago but haven't found time to resend. No change except
> > > rebasing on the latest bpf-next/master.
> > >
> > > [v1] https://lore.kernel.org/bpf/20200827153629.3820891-1-jean-philippe@linaro.org/
> > >
> >
> > While you are looking at bpftool builds... Seems like it regressed
> > recently and doesn't honor -jX setting. Either way the build is
> > sequential (and rather slow). Do you mind checking if your changes
> > could fix the regression (I haven't had a chance to bisect the
> > offending change causing regression).
>
> I bisected it to ba2fd563b740 ("tools/bpftool: Support passing
> BPFTOOL_VERSION to make"), in v5.9. As BPFTOOL_VERSION became a recursivly
> expanded variable, the shell function is evaluated on every expansion of
> CFLAGS. I'll add the fix to v3:
>
> -BPFTOOL_VERSION ?= $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
> +ifeq ($(BPFTOOL_VERSION),)
> +BPFTOOL_VERSION := $(shell make -rR --no-print-directory -sC ../../.. kernelversion)
> +endif
>

Great, thanks!

> Thanks,
> Jean

^ permalink raw reply	[flat|nested] 14+ messages in thread

end of thread, other threads:[~2020-11-10 18:38 UTC | newest]

Thread overview: 14+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-11-09 11:09 [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 1/6] tools: Factor HOSTCC, HOSTLD, HOSTAR definitions Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 2/6] tools/bpftool: Force clean of out-of-tree build Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 3/6] tools/bpftool: Fix cross-build Jean-Philippe Brucker
2020-11-09 20:17   ` Andrii Nakryiko
2020-11-10 12:43     ` Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 4/6] tools/runqslower: Use Makefile.include Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 5/6] tools/runqslower: Enable out-of-tree build Jean-Philippe Brucker
2020-11-09 20:23   ` Andrii Nakryiko
2020-11-10 12:43     ` Jean-Philippe Brucker
2020-11-09 11:09 ` [PATCH bpf-next v2 6/6] tools/runqslower: Build bpftool using HOSTCC Jean-Philippe Brucker
2020-11-09 18:11 ` [PATCH bpf-next v2 0/6] tools/bpftool: Fix cross and out-of-tree builds Andrii Nakryiko
2020-11-10 12:42   ` Jean-Philippe Brucker
2020-11-10 18:38     ` Andrii Nakryiko

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).