All of lore.kernel.org
 help / color / mirror / Atom feed
* [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-10-17 10:37 ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: linux-kernel, Masahiro Yamada, Albert Ou, Borislav Petkov,
	Christophe Leroy, Dave Hansen, H. Peter Anvin, Ingo Molnar,
	Michael Ellerman, Nathan Chancellor, Nicholas Piggin,
	Nick Desaulniers, Nicolas Schier, Palmer Dabbelt, Paul Walmsley,
	Thomas Bogendoerfer, Thomas Gleixner, linux-mips, linux-riscv,
	linuxppc-dev, x86

The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
'true' command.

Remove the meaningless code.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 arch/mips/Makefile.postlink    | 3 ---
 arch/powerpc/Makefile.postlink | 3 ---
 arch/riscv/Makefile.postlink   | 3 ---
 arch/x86/Makefile.postlink     | 3 ---
 scripts/Makefile.modfinal      | 5 +----
 5 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/arch/mips/Makefile.postlink b/arch/mips/Makefile.postlink
index 34e3bd71f3b0..6cfdc149d3bc 100644
--- a/arch/mips/Makefile.postlink
+++ b/arch/mips/Makefile.postlink
@@ -31,9 +31,6 @@ ifeq ($(CONFIG_RELOCATABLE),y)
 	$(call if_changed,relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/powerpc/Makefile.postlink b/arch/powerpc/Makefile.postlink
index 1f860b3c9bec..ae5a4256b03d 100644
--- a/arch/powerpc/Makefile.postlink
+++ b/arch/powerpc/Makefile.postlink
@@ -35,9 +35,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_check)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	rm -f .tmp_symbols.txt
 
diff --git a/arch/riscv/Makefile.postlink b/arch/riscv/Makefile.postlink
index a46fc578b30b..829b9abc91f6 100644
--- a/arch/riscv/Makefile.postlink
+++ b/arch/riscv/Makefile.postlink
@@ -36,9 +36,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_strip)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/x86/Makefile.postlink b/arch/x86/Makefile.postlink
index 936093d29160..fef2e977cc7d 100644
--- a/arch/x86/Makefile.postlink
+++ b/arch/x86/Makefile.postlink
@@ -34,9 +34,6 @@ ifeq ($(CONFIG_X86_NEED_RELOCS),y)
 	$(call cmd,strip_relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@rm -f $(OUT_RELOCS)/vmlinux.relocs
 
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index b3a6aa8fbe8c..8568d256d6fb 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -28,14 +28,11 @@ quiet_cmd_cc_o_c = CC [M]  $@
 %.mod.o: %.mod.c FORCE
 	$(call if_changed_dep,cc_o_c)
 
-ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
-
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T scripts/module.lds -o $@ $(filter %.o, $^);		\
-	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
+		-T scripts/module.lds -o $@ $(filter %.o, $^)
 
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-10-17 10:37 ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: linux-kernel, Masahiro Yamada, Albert Ou, Borislav Petkov,
	Christophe Leroy, Dave Hansen, H. Peter Anvin, Ingo Molnar,
	Michael Ellerman, Nathan Chancellor, Nicholas Piggin,
	Nick Desaulniers, Nicolas Schier, Palmer Dabbelt, Paul Walmsley,
	Thomas Bogendoerfer, Thomas Gleixner, linux-mips, linux-riscv,
	linuxppc-dev, x86

The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
'true' command.

Remove the meaningless code.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 arch/mips/Makefile.postlink    | 3 ---
 arch/powerpc/Makefile.postlink | 3 ---
 arch/riscv/Makefile.postlink   | 3 ---
 arch/x86/Makefile.postlink     | 3 ---
 scripts/Makefile.modfinal      | 5 +----
 5 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/arch/mips/Makefile.postlink b/arch/mips/Makefile.postlink
index 34e3bd71f3b0..6cfdc149d3bc 100644
--- a/arch/mips/Makefile.postlink
+++ b/arch/mips/Makefile.postlink
@@ -31,9 +31,6 @@ ifeq ($(CONFIG_RELOCATABLE),y)
 	$(call if_changed,relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/powerpc/Makefile.postlink b/arch/powerpc/Makefile.postlink
index 1f860b3c9bec..ae5a4256b03d 100644
--- a/arch/powerpc/Makefile.postlink
+++ b/arch/powerpc/Makefile.postlink
@@ -35,9 +35,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_check)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	rm -f .tmp_symbols.txt
 
diff --git a/arch/riscv/Makefile.postlink b/arch/riscv/Makefile.postlink
index a46fc578b30b..829b9abc91f6 100644
--- a/arch/riscv/Makefile.postlink
+++ b/arch/riscv/Makefile.postlink
@@ -36,9 +36,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_strip)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/x86/Makefile.postlink b/arch/x86/Makefile.postlink
index 936093d29160..fef2e977cc7d 100644
--- a/arch/x86/Makefile.postlink
+++ b/arch/x86/Makefile.postlink
@@ -34,9 +34,6 @@ ifeq ($(CONFIG_X86_NEED_RELOCS),y)
 	$(call cmd,strip_relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@rm -f $(OUT_RELOCS)/vmlinux.relocs
 
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index b3a6aa8fbe8c..8568d256d6fb 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -28,14 +28,11 @@ quiet_cmd_cc_o_c = CC [M]  $@
 %.mod.o: %.mod.c FORCE
 	$(call if_changed_dep,cc_o_c)
 
-ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
-
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T scripts/module.lds -o $@ $(filter %.o, $^);		\
-	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
+		-T scripts/module.lds -o $@ $(filter %.o, $^)
 
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
-- 
2.40.1


_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-10-17 10:37 ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: x86, Albert Ou, Nicolas Schier, Masahiro Yamada, Dave Hansen,
	Nick Desaulniers, linux-kernel, linux-mips, Nathan Chancellor,
	Ingo Molnar, Borislav Petkov, Nicholas Piggin, Paul Walmsley,
	H. Peter Anvin, Palmer Dabbelt, Thomas Gleixner,
	Thomas Bogendoerfer, linuxppc-dev, linux-riscv

The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
'true' command.

Remove the meaningless code.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 arch/mips/Makefile.postlink    | 3 ---
 arch/powerpc/Makefile.postlink | 3 ---
 arch/riscv/Makefile.postlink   | 3 ---
 arch/x86/Makefile.postlink     | 3 ---
 scripts/Makefile.modfinal      | 5 +----
 5 files changed, 1 insertion(+), 16 deletions(-)

diff --git a/arch/mips/Makefile.postlink b/arch/mips/Makefile.postlink
index 34e3bd71f3b0..6cfdc149d3bc 100644
--- a/arch/mips/Makefile.postlink
+++ b/arch/mips/Makefile.postlink
@@ -31,9 +31,6 @@ ifeq ($(CONFIG_RELOCATABLE),y)
 	$(call if_changed,relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/powerpc/Makefile.postlink b/arch/powerpc/Makefile.postlink
index 1f860b3c9bec..ae5a4256b03d 100644
--- a/arch/powerpc/Makefile.postlink
+++ b/arch/powerpc/Makefile.postlink
@@ -35,9 +35,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_check)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	rm -f .tmp_symbols.txt
 
diff --git a/arch/riscv/Makefile.postlink b/arch/riscv/Makefile.postlink
index a46fc578b30b..829b9abc91f6 100644
--- a/arch/riscv/Makefile.postlink
+++ b/arch/riscv/Makefile.postlink
@@ -36,9 +36,6 @@ ifdef CONFIG_RELOCATABLE
 	$(call if_changed,relocs_strip)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@true
 
diff --git a/arch/x86/Makefile.postlink b/arch/x86/Makefile.postlink
index 936093d29160..fef2e977cc7d 100644
--- a/arch/x86/Makefile.postlink
+++ b/arch/x86/Makefile.postlink
@@ -34,9 +34,6 @@ ifeq ($(CONFIG_X86_NEED_RELOCS),y)
 	$(call cmd,strip_relocs)
 endif
 
-%.ko: FORCE
-	@true
-
 clean:
 	@rm -f $(OUT_RELOCS)/vmlinux.relocs
 
diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index b3a6aa8fbe8c..8568d256d6fb 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -28,14 +28,11 @@ quiet_cmd_cc_o_c = CC [M]  $@
 %.mod.o: %.mod.c FORCE
 	$(call if_changed_dep,cc_o_c)
 
-ARCH_POSTLINK := $(wildcard $(srctree)/arch/$(SRCARCH)/Makefile.postlink)
-
 quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T scripts/module.lds -o $@ $(filter %.o, $^);		\
-	$(if $(ARCH_POSTLINK), $(MAKE) -f $(ARCH_POSTLINK) $@, true)
+		-T scripts/module.lds -o $@ $(filter %.o, $^)
 
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` Masahiro Yamada
  (?)
  (?)
@ 2023-10-17 10:37 ` Masahiro Yamada
  2023-10-17 12:14   ` Alan Maguire
                     ` (5 more replies)
  -1 siblings, 6 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: linux-kernel, Masahiro Yamada, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

scripts/pahole-flags.sh is executed so many times.

You can check how many times it is invoked during the build, as follows:

  $ cat <<EOF >> scripts/pahole-flags.sh
  > echo "scripts/pahole-flags.sh was executed" >&2
  > EOF

  $ make -s
  scripts/pahole-flags.sh was executed
  scripts/pahole-flags.sh was executed
  scripts/pahole-flags.sh was executed
  scripts/pahole-flags.sh was executed
  scripts/pahole-flags.sh was executed
    [ lots of repeated lines suppressed... ]

This scripts is exectuted more than 20 times during the kernel build
because PAHOLE_FLAGS is a recursively expanded variable and exported
to sub-processes.

With the GNU Make >= 4.4, it is executed more than 60 times because
exported variables are also passed to other $(shell ) invocations.
Without careful coding, it is known to cause an exponential fork
explosion. [1]

The use of $(shell ) in an exported recursive variable is likely wrong
because $(shell ) is always evaluated due to the 'export' keyword, and
the evaluation can occur multiple times by the nature of recursive
variables.

Convert the shell script to a Makefile, which is included only when
CONFIG_DEBUG_INFO_BTF=y.

[1]: https://savannah.gnu.org/bugs/index.php?64746

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 Makefile                |  4 +---
 scripts/Makefile.btf    | 19 +++++++++++++++++++
 scripts/pahole-flags.sh | 30 ------------------------------
 3 files changed, 20 insertions(+), 33 deletions(-)
 create mode 100644 scripts/Makefile.btf
 delete mode 100755 scripts/pahole-flags.sh

diff --git a/Makefile b/Makefile
index fed9a6cc3665..eaddec67e5e1 100644
--- a/Makefile
+++ b/Makefile
@@ -513,8 +513,6 @@ LZ4		= lz4c
 XZ		= xz
 ZSTD		= zstd
 
-PAHOLE_FLAGS	= $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
-
 CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
 		  -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
 NOSTDINC_FLAGS :=
@@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE
 export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
 export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE
 export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL
-export PAHOLE_FLAGS
 
 # Files to ignore in find ... statements
 
@@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
 # include additional Makefiles when needed
 include-y			:= scripts/Makefile.extrawarn
 include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
+include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
 include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
 include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
 include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan
diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
new file mode 100644
index 000000000000..82377e470aed
--- /dev/null
+++ b/scripts/Makefile.btf
@@ -0,0 +1,19 @@
+# SPDX-License-Identifier: GPL-2.0
+
+pahole-ver := $(CONFIG_PAHOLE_VERSION)
+pahole-flags-y :=
+
+# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
+ifeq ($(call test-le, $(pahole-ver), 121),y)
+pahole-flags-$(call test-ge, $(pahole-ver), 118)	+= --skip_encoding_btf_vars
+endif
+
+pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
+
+pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
+
+pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
+
+pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
+
+export PAHOLE_FLAGS := $(pahole-flags-y)
diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
deleted file mode 100755
index 728d55190d97..000000000000
--- a/scripts/pahole-flags.sh
+++ /dev/null
@@ -1,30 +0,0 @@
-#!/bin/sh
-# SPDX-License-Identifier: GPL-2.0
-
-extra_paholeopt=
-
-if ! [ -x "$(command -v ${PAHOLE})" ]; then
-	exit 0
-fi
-
-pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE})
-
-if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
-	# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
-	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
-fi
-if [ "${pahole_ver}" -ge "121" ]; then
-	extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
-fi
-if [ "${pahole_ver}" -ge "122" ]; then
-	extra_paholeopt="${extra_paholeopt} -j"
-fi
-if [ "${pahole_ver}" -ge "124" ]; then
-	# see PAHOLE_HAS_LANG_EXCLUDE
-	extra_paholeopt="${extra_paholeopt} --lang_exclude=rust"
-fi
-if [ "${pahole_ver}" -ge "125" ]; then
-	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized"
-fi
-
-echo ${extra_paholeopt}
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 3/4] kbuild: skip module BTF with one-time check for vmlinux
  2023-10-17 10:37 ` Masahiro Yamada
                   ` (2 preceding siblings ...)
  (?)
@ 2023-10-17 10:37 ` Masahiro Yamada
  2023-10-17 12:37   ` Nicolas Schier
  -1 siblings, 1 reply; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: linux-kernel, Masahiro Yamada, Nathan Chancellor,
	Nick Desaulniers, Nicolas Schier

When CONFIG_DEBUG_INFO_BTF_MODULES is enabled, vmlinux presence is
checked in every module build, resulting in repetitive warning
messages if vmlinux is missing.

Check vmlinux and print a warning just once.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/Makefile.modfinal | 19 ++++++++++++-------
 1 file changed, 12 insertions(+), 7 deletions(-)

diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 8568d256d6fb..9fd7a26e4fe9 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -14,6 +14,15 @@ include $(srctree)/scripts/Makefile.lib
 
 # find all modules listed in modules.order
 modules := $(call read-file, $(MODORDER))
+vmlinux :=
+
+ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+ifneq ($(wildcard vmlinux),)
+vmlinux := vmlinux
+else
+$(warning Skipping BTF generation due to unavailability of vmlinux)
+endif
+endif
 
 __modfinal: $(modules:%.o=%.ko)
 	@:
@@ -36,12 +45,8 @@ quiet_cmd_ld_ko_o = LD [M]  $@
 
 quiet_cmd_btf_ko = BTF [M] $@
       cmd_btf_ko = 							\
-	if [ ! -f vmlinux ]; then					\
-		printf "Skipping BTF generation for %s due to unavailability of vmlinux\n" $@ 1>&2; \
-	else								\
 		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
-		$(RESOLVE_BTFIDS) -b vmlinux $@; 			\
-	fi;
+		$(RESOLVE_BTFIDS) -b vmlinux $@
 
 # Same as newer-prereqs, but allows to exclude specified extra dependencies
 newer_prereqs_except = $(filter-out $(PHONY) $(1),$?)
@@ -52,9 +57,9 @@ if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
 	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
 
 # Re-generate module BTFs if either module's .ko or vmlinux changed
-%.ko: %.o %.mod.o scripts/module.lds $(and $(CONFIG_DEBUG_INFO_BTF_MODULES),$(KBUILD_BUILTIN),vmlinux) FORCE
+%.ko: %.o %.mod.o scripts/module.lds $(vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,vmlinux)
-ifdef CONFIG_DEBUG_INFO_BTF_MODULES
+ifdef vmlinux
 	+$(if $(newer-prereqs),$(call cmd,btf_ko))
 endif
 
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 23+ messages in thread
* [PATCH 4/4] kbuild: refactor module BTF rule
  2023-10-17 10:37 ` Masahiro Yamada
                   ` (3 preceding siblings ...)
  (?)
@ 2023-10-17 10:37 ` Masahiro Yamada
  2023-10-17 12:58   ` Nicolas Schier
  -1 siblings, 1 reply; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-17 10:37 UTC (permalink / raw)
  To: linux-kbuild
  Cc: linux-kernel, Masahiro Yamada, Nathan Chancellor,
	Nick Desaulniers, Nicolas Schier

newer_prereqs_except and if_changed_except are ugly hacks of the
newer_prereqs and if_changed in scripts/Kbuild.include.

Remove.

Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
---

 scripts/Makefile.modfinal | 23 +++++------------------
 1 file changed, 5 insertions(+), 18 deletions(-)

diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
index 9fd7a26e4fe9..6ab10dba05c7 100644
--- a/scripts/Makefile.modfinal
+++ b/scripts/Makefile.modfinal
@@ -19,6 +19,9 @@ vmlinux :=
 ifdef CONFIG_DEBUG_INFO_BTF_MODULES
 ifneq ($(wildcard vmlinux),)
 vmlinux := vmlinux
+cmd_btf = ; \
+	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
+	$(RESOLVE_BTFIDS) -b vmlinux $@
 else
 $(warning Skipping BTF generation due to unavailability of vmlinux)
 endif
@@ -41,27 +44,11 @@ quiet_cmd_ld_ko_o = LD [M]  $@
       cmd_ld_ko_o +=							\
 	$(LD) -r $(KBUILD_LDFLAGS)					\
 		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
-		-T scripts/module.lds -o $@ $(filter %.o, $^)
+		-T scripts/module.lds -o $@ $(filter %.o, $^)		\
+	$(cmd_btf)
 
-quiet_cmd_btf_ko = BTF [M] $@
-      cmd_btf_ko = 							\
-		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
-		$(RESOLVE_BTFIDS) -b vmlinux $@
-
-# Same as newer-prereqs, but allows to exclude specified extra dependencies
-newer_prereqs_except = $(filter-out $(PHONY) $(1),$?)
-
-# Same as if_changed, but allows to exclude specified extra dependencies
-if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
-	$(cmd);                                                              \
-	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
-
-# Re-generate module BTFs if either module's .ko or vmlinux changed
 %.ko: %.o %.mod.o scripts/module.lds $(vmlinux) FORCE
 	+$(call if_changed_except,ld_ko_o,vmlinux)
-ifdef vmlinux
-	+$(if $(newer-prereqs),$(call cmd,btf_ko))
-endif
 
 targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o)
 
-- 
2.40.1


^ permalink raw reply related	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
  2023-10-17 10:37 ` Masahiro Yamada
  (?)
@ 2023-10-17 12:09   ` Nicolas Schier
  -1 siblings, 0 replies; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:09 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Albert Ou, Borislav Petkov,
	Christophe Leroy, Dave Hansen, H. Peter Anvin, Ingo Molnar,
	Michael Ellerman, Nathan Chancellor, Nicholas Piggin,
	Nick Desaulniers, Nicolas Schier, Palmer Dabbelt, Paul Walmsley,
	Thomas Bogendoerfer, Thomas Gleixner, linux-mips, linux-riscv,
	linuxppc-dev, x86

On Tue, Oct 17, 2023 at 07:37:39PM +0900, Masahiro Yamada wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  arch/mips/Makefile.postlink    | 3 ---
>  arch/powerpc/Makefile.postlink | 3 ---
>  arch/riscv/Makefile.postlink   | 3 ---
>  arch/x86/Makefile.postlink     | 3 ---
>  scripts/Makefile.modfinal      | 5 +----
>  5 files changed, 1 insertion(+), 16 deletions(-)

Reviewed-by: Nicolas Schier <n.schier@avm.de>

_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-10-17 12:09   ` Nicolas Schier
  0 siblings, 0 replies; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:09 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Albert Ou, Borislav Petkov,
	Christophe Leroy, Dave Hansen, H. Peter Anvin, Ingo Molnar,
	Michael Ellerman, Nathan Chancellor, Nicholas Piggin,
	Nick Desaulniers, Nicolas Schier, Palmer Dabbelt, Paul Walmsley,
	Thomas Bogendoerfer, Thomas Gleixner, linux-mips, linux-riscv,
	linuxppc-dev, x86

On Tue, Oct 17, 2023 at 07:37:39PM +0900, Masahiro Yamada wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  arch/mips/Makefile.postlink    | 3 ---
>  arch/powerpc/Makefile.postlink | 3 ---
>  arch/riscv/Makefile.postlink   | 3 ---
>  arch/x86/Makefile.postlink     | 3 ---
>  scripts/Makefile.modfinal      | 5 +----
>  5 files changed, 1 insertion(+), 16 deletions(-)

Reviewed-by: Nicolas Schier <n.schier@avm.de>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-10-17 12:09   ` Nicolas Schier
  0 siblings, 0 replies; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:09 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: x86, Albert Ou, Nicolas Schier, linux-kbuild, Dave Hansen,
	Nick Desaulniers, linux-kernel, linux-mips, Nathan Chancellor,
	Ingo Molnar, Borislav Petkov, Nicholas Piggin, Paul Walmsley,
	H. Peter Anvin, Palmer Dabbelt, Thomas Gleixner,
	Thomas Bogendoerfer, linuxppc-dev, linux-riscv

On Tue, Oct 17, 2023 at 07:37:39PM +0900, Masahiro Yamada wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  arch/mips/Makefile.postlink    | 3 ---
>  arch/powerpc/Makefile.postlink | 3 ---
>  arch/riscv/Makefile.postlink   | 3 ---
>  arch/x86/Makefile.postlink     | 3 ---
>  scripts/Makefile.modfinal      | 5 +----
>  5 files changed, 1 insertion(+), 16 deletions(-)

Reviewed-by: Nicolas Schier <n.schier@avm.de>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
@ 2023-10-17 12:14   ` Alan Maguire
  2023-10-17 12:15   ` Jiri Olsa
                     ` (4 subsequent siblings)
  5 siblings, 0 replies; 23+ messages in thread
From: Alan Maguire @ 2023-10-17 12:14 UTC (permalink / raw)
  To: Masahiro Yamada, linux-kbuild
  Cc: linux-kernel, Alex Gaynor, Alexei Starovoitov, Alice Ryhl,
	Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On 17/10/2023 11:37, Masahiro Yamada wrote:
> scripts/pahole-flags.sh is executed so many times.
> 
> You can check how many times it is invoked during the build, as follows:
> 
>   $ cat <<EOF >> scripts/pahole-flags.sh
>   > echo "scripts/pahole-flags.sh was executed" >&2
>   > EOF
> 
>   $ make -s
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>     [ lots of repeated lines suppressed... ]
> 
> This scripts is exectuted more than 20 times during the kernel build
> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
> 
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]
> 
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
> 
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
> 
> [1]: https://savannah.gnu.org/bugs/index.php?64746
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

Looks like a neat change to me. I tested with CONFIG_DEBUG_INFO_BTF=y
builds for full kernel build and 'make M=modpath' module-only builds,
and both work well with correct pahole flags. Feel free to add
either/both of

Reviewed-by: Alan Maguire <alan.maguire@oracle.com>
Tested-by: Alan Maguire <alan.maguire@oracle.com>

> ---
> 
>  Makefile                |  4 +---
>  scripts/Makefile.btf    | 19 +++++++++++++++++++
>  scripts/pahole-flags.sh | 30 ------------------------------
>  3 files changed, 20 insertions(+), 33 deletions(-)
>  create mode 100644 scripts/Makefile.btf
>  delete mode 100755 scripts/pahole-flags.sh
> 
> diff --git a/Makefile b/Makefile
> index fed9a6cc3665..eaddec67e5e1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -513,8 +513,6 @@ LZ4		= lz4c
>  XZ		= xz
>  ZSTD		= zstd
>  
> -PAHOLE_FLAGS	= $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
> -
>  CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
>  		  -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
>  NOSTDINC_FLAGS :=
> @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE
>  export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
>  export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE
>  export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL
> -export PAHOLE_FLAGS
>  
>  # Files to ignore in find ... statements
>  
> @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
>  # include additional Makefiles when needed
>  include-y			:= scripts/Makefile.extrawarn
>  include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
> +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
>  include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
>  include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
>  include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> new file mode 100644
> index 000000000000..82377e470aed
> --- /dev/null
> +++ b/scripts/Makefile.btf
> @@ -0,0 +1,19 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +pahole-ver := $(CONFIG_PAHOLE_VERSION)
> +pahole-flags-y :=
> +
> +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> +ifeq ($(call test-le, $(pahole-ver), 121),y)
> +pahole-flags-$(call test-ge, $(pahole-ver), 118)	+= --skip_encoding_btf_vars
> +endif
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +
> +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> +
> +export PAHOLE_FLAGS := $(pahole-flags-y)
> diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> deleted file mode 100755
> index 728d55190d97..000000000000
> --- a/scripts/pahole-flags.sh
> +++ /dev/null
> @@ -1,30 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0
> -
> -extra_paholeopt=
> -
> -if ! [ -x "$(command -v ${PAHOLE})" ]; then
> -	exit 0
> -fi
> -
> -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE})
> -
> -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
> -	# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
> -fi
> -if [ "${pahole_ver}" -ge "121" ]; then
> -	extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
> -fi
> -if [ "${pahole_ver}" -ge "122" ]; then
> -	extra_paholeopt="${extra_paholeopt} -j"
> -fi
> -if [ "${pahole_ver}" -ge "124" ]; then
> -	# see PAHOLE_HAS_LANG_EXCLUDE
> -	extra_paholeopt="${extra_paholeopt} --lang_exclude=rust"
> -fi
> -if [ "${pahole_ver}" -ge "125" ]; then
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized"
> -fi
> -
> -echo ${extra_paholeopt}

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
  2023-10-17 12:14   ` Alan Maguire
@ 2023-10-17 12:15   ` Jiri Olsa
  2023-10-17 12:29   ` Nicolas Schier
                     ` (3 subsequent siblings)
  5 siblings, 0 replies; 23+ messages in thread
From: Jiri Olsa @ 2023-10-17 12:15 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On Tue, Oct 17, 2023 at 07:37:40PM +0900, Masahiro Yamada wrote:
> scripts/pahole-flags.sh is executed so many times.
> 
> You can check how many times it is invoked during the build, as follows:
> 
>   $ cat <<EOF >> scripts/pahole-flags.sh
>   > echo "scripts/pahole-flags.sh was executed" >&2
>   > EOF
> 
>   $ make -s
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>     [ lots of repeated lines suppressed... ]
> 
> This scripts is exectuted more than 20 times during the kernel build
> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
> 
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]

nice :-\

> 
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
> 
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.

looks good.. could you please resend this patch with bpf-next in subject
so CI tests would trigger for it?

thanks,
jirka

> 
> [1]: https://savannah.gnu.org/bugs/index.php?64746
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  Makefile                |  4 +---
>  scripts/Makefile.btf    | 19 +++++++++++++++++++
>  scripts/pahole-flags.sh | 30 ------------------------------
>  3 files changed, 20 insertions(+), 33 deletions(-)
>  create mode 100644 scripts/Makefile.btf
>  delete mode 100755 scripts/pahole-flags.sh
> 
> diff --git a/Makefile b/Makefile
> index fed9a6cc3665..eaddec67e5e1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -513,8 +513,6 @@ LZ4		= lz4c
>  XZ		= xz
>  ZSTD		= zstd
>  
> -PAHOLE_FLAGS	= $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
> -
>  CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
>  		  -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
>  NOSTDINC_FLAGS :=
> @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE
>  export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
>  export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE
>  export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL
> -export PAHOLE_FLAGS
>  
>  # Files to ignore in find ... statements
>  
> @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
>  # include additional Makefiles when needed
>  include-y			:= scripts/Makefile.extrawarn
>  include-$(CONFIG_DEBUG_INFO)	+= scripts/Makefile.debug
> +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
>  include-$(CONFIG_KASAN)		+= scripts/Makefile.kasan
>  include-$(CONFIG_KCSAN)		+= scripts/Makefile.kcsan
>  include-$(CONFIG_KMSAN)		+= scripts/Makefile.kmsan
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> new file mode 100644
> index 000000000000..82377e470aed
> --- /dev/null
> +++ b/scripts/Makefile.btf
> @@ -0,0 +1,19 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +pahole-ver := $(CONFIG_PAHOLE_VERSION)
> +pahole-flags-y :=
> +
> +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> +ifeq ($(call test-le, $(pahole-ver), 121),y)
> +pahole-flags-$(call test-ge, $(pahole-ver), 118)	+= --skip_encoding_btf_vars
> +endif
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 121)	+= --btf_gen_floats
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 122)	+= -j
> +
> +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)		+= --lang_exclude=rust
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 125)	+= --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> +
> +export PAHOLE_FLAGS := $(pahole-flags-y)
> diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> deleted file mode 100755
> index 728d55190d97..000000000000
> --- a/scripts/pahole-flags.sh
> +++ /dev/null
> @@ -1,30 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0
> -
> -extra_paholeopt=
> -
> -if ! [ -x "$(command -v ${PAHOLE})" ]; then
> -	exit 0
> -fi
> -
> -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE})
> -
> -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
> -	# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
> -fi
> -if [ "${pahole_ver}" -ge "121" ]; then
> -	extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
> -fi
> -if [ "${pahole_ver}" -ge "122" ]; then
> -	extra_paholeopt="${extra_paholeopt} -j"
> -fi
> -if [ "${pahole_ver}" -ge "124" ]; then
> -	# see PAHOLE_HAS_LANG_EXCLUDE
> -	extra_paholeopt="${extra_paholeopt} --lang_exclude=rust"
> -fi
> -if [ "${pahole_ver}" -ge "125" ]; then
> -	extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized"
> -fi
> -
> -echo ${extra_paholeopt}
> -- 
> 2.40.1
> 

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
  2023-10-17 12:14   ` Alan Maguire
  2023-10-17 12:15   ` Jiri Olsa
@ 2023-10-17 12:29   ` Nicolas Schier
  2023-10-17 15:21   ` Miguel Ojeda
                     ` (2 subsequent siblings)
  5 siblings, 0 replies; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:29 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

> Subject: kbuild: avoid too many execution of scripts/pahole-flags.sh
executions?

On Tue, Oct 17, 2023 at 07:37:40PM +0900, Masahiro Yamada wrote:
> scripts/pahole-flags.sh is executed so many times.
> 
> You can check how many times it is invoked during the build, as follows:
> 
>   $ cat <<EOF >> scripts/pahole-flags.sh
>   > echo "scripts/pahole-flags.sh was executed" >&2
>   > EOF
> 
>   $ make -s
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>     [ lots of repeated lines suppressed... ]
> 
> This scripts is exectuted more than 20 times during the kernel build

executed

> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
> 
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]
> 
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
> 
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
> 
> [1]: https://savannah.gnu.org/bugs/index.php?64746
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---

Thanks, looks good to me!

Reviewed-by: Nicolas Schier <n.schier@avm.de>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 3/4] kbuild: skip module BTF with one-time check for vmlinux
  2023-10-17 10:37 ` [PATCH 3/4] kbuild: skip module BTF with one-time check for vmlinux Masahiro Yamada
@ 2023-10-17 12:37   ` Nicolas Schier
  0 siblings, 0 replies; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:37 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Nathan Chancellor, Nick Desaulniers

On Tue, Oct 17, 2023 at 07:37:41PM +0900, Masahiro Yamada wrote:
> When CONFIG_DEBUG_INFO_BTF_MODULES is enabled, vmlinux presence is
> checked in every module build, resulting in repetitive warning
> messages if vmlinux is missing.
> 
> Check vmlinux and print a warning just once.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---

Reviewed-by: Nicolas Schier <n.schier@avm.de>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 4/4] kbuild: refactor module BTF rule
  2023-10-17 10:37 ` [PATCH 4/4] kbuild: refactor module BTF rule Masahiro Yamada
@ 2023-10-17 12:58   ` Nicolas Schier
  2023-10-18 15:16     ` Masahiro Yamada
  0 siblings, 1 reply; 23+ messages in thread
From: Nicolas Schier @ 2023-10-17 12:58 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier

On Tue, Oct 17, 2023 at 07:37:42PM +0900, Masahiro Yamada wrote:
> newer_prereqs_except and if_changed_except are ugly hacks of the
> newer_prereqs and if_changed in scripts/Kbuild.include.

newer-prereqs

> 
> Remove.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> 
>  scripts/Makefile.modfinal | 23 +++++------------------
>  1 file changed, 5 insertions(+), 18 deletions(-)
> 
> diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> index 9fd7a26e4fe9..6ab10dba05c7 100644
> --- a/scripts/Makefile.modfinal
> +++ b/scripts/Makefile.modfinal
> @@ -19,6 +19,9 @@ vmlinux :=
>  ifdef CONFIG_DEBUG_INFO_BTF_MODULES
>  ifneq ($(wildcard vmlinux),)
>  vmlinux := vmlinux
> +cmd_btf = ; \

while reading, I stumpled over this semicolon, but probably it's a good
reminder that cmd_btf is only a cmd extension.

> +	LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
> +	$(RESOLVE_BTFIDS) -b vmlinux $@
>  else
>  $(warning Skipping BTF generation due to unavailability of vmlinux)
>  endif
> @@ -41,27 +44,11 @@ quiet_cmd_ld_ko_o = LD [M]  $@
>        cmd_ld_ko_o +=							\
>  	$(LD) -r $(KBUILD_LDFLAGS)					\
>  		$(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)		\
> -		-T scripts/module.lds -o $@ $(filter %.o, $^)
> +		-T scripts/module.lds -o $@ $(filter %.o, $^)		\
> +	$(cmd_btf)
>  
> -quiet_cmd_btf_ko = BTF [M] $@
> -      cmd_btf_ko = 							\
> -		LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
> -		$(RESOLVE_BTFIDS) -b vmlinux $@
> -
> -# Same as newer-prereqs, but allows to exclude specified extra dependencies
> -newer_prereqs_except = $(filter-out $(PHONY) $(1),$?)
> -
> -# Same as if_changed, but allows to exclude specified extra dependencies
> -if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
> -	$(cmd);                                                              \
> -	printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
> -
> -# Re-generate module BTFs if either module's .ko or vmlinux changed
>  %.ko: %.o %.mod.o scripts/module.lds $(vmlinux) FORCE
>  	+$(call if_changed_except,ld_ko_o,vmlinux)

This should probably be:

  +$(call if_changed,ld_ko_o)


> -ifdef vmlinux
> -	+$(if $(newer-prereqs),$(call cmd,btf_ko))
> -endif
>  
>  targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o)
>  
> -- 
> 2.40.1
> 

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
                     ` (2 preceding siblings ...)
  2023-10-17 12:29   ` Nicolas Schier
@ 2023-10-17 15:21   ` Miguel Ojeda
  2023-10-18 12:14     ` Masahiro Yamada
  2023-10-17 19:56   ` Andrii Nakryiko
  2023-10-23  0:30   ` Martin Rodriguez Reboredo
  5 siblings, 1 reply; 23+ messages in thread
From: Miguel Ojeda @ 2023-10-17 15:21 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On Tue, Oct 17, 2023 at 12:38 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
>
> [1]: https://savannah.gnu.org/bugs/index.php?64746
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>

The field in `MAINTAINERS` should be removed:

    F: scripts/pahole-flags.sh

But other than that, it looks good to me! I tried it for a given
config and it does call `pahole` with the same flags.

Tested-by: Miguel Ojeda <ojeda@kernel.org>
Acked-by: Miguel Ojeda <ojeda@kernel.org>

Cheers,
Miguel

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
                     ` (3 preceding siblings ...)
  2023-10-17 15:21   ` Miguel Ojeda
@ 2023-10-17 19:56   ` Andrii Nakryiko
  2023-10-18  5:00     ` Masahiro Yamada
  2023-10-23  0:30   ` Martin Rodriguez Reboredo
  5 siblings, 1 reply; 23+ messages in thread
From: Andrii Nakryiko @ 2023-10-17 19:56 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On Tue, Oct 17, 2023 at 3:38 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
>
> scripts/pahole-flags.sh is executed so many times.
>
> You can check how many times it is invoked during the build, as follows:
>
>   $ cat <<EOF >> scripts/pahole-flags.sh
>   > echo "scripts/pahole-flags.sh was executed" >&2
>   > EOF
>
>   $ make -s
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>   scripts/pahole-flags.sh was executed
>     [ lots of repeated lines suppressed... ]
>
> This scripts is exectuted more than 20 times during the kernel build
> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
>
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]
>
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
>
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
>
> [1]: https://savannah.gnu.org/bugs/index.php?64746
>
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
>
>  Makefile                |  4 +---
>  scripts/Makefile.btf    | 19 +++++++++++++++++++
>  scripts/pahole-flags.sh | 30 ------------------------------
>  3 files changed, 20 insertions(+), 33 deletions(-)
>  create mode 100644 scripts/Makefile.btf
>  delete mode 100755 scripts/pahole-flags.sh
>
> diff --git a/Makefile b/Makefile
> index fed9a6cc3665..eaddec67e5e1 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -513,8 +513,6 @@ LZ4         = lz4c
>  XZ             = xz
>  ZSTD           = zstd
>
> -PAHOLE_FLAGS   = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)

What if we just used := here? Wouldn't it avoid unnecessary multiple executions?

I don't make Makefile.btf approach, just curious why := doesn't work,
if it doesn't.

> -
>  CHECKFLAGS     := -D__linux__ -Dlinux -D__STDC__ -Dunix -D__unix__ \
>                   -Wbitwise -Wno-return-void -Wno-unknown-attribute $(CF)
>  NOSTDINC_FLAGS :=
> @@ -605,7 +603,6 @@ export KBUILD_RUSTFLAGS RUSTFLAGS_KERNEL RUSTFLAGS_MODULE
>  export KBUILD_AFLAGS AFLAGS_KERNEL AFLAGS_MODULE
>  export KBUILD_AFLAGS_MODULE KBUILD_CFLAGS_MODULE KBUILD_RUSTFLAGS_MODULE KBUILD_LDFLAGS_MODULE
>  export KBUILD_AFLAGS_KERNEL KBUILD_CFLAGS_KERNEL KBUILD_RUSTFLAGS_KERNEL
> -export PAHOLE_FLAGS
>
>  # Files to ignore in find ... statements
>
> @@ -1002,6 +999,7 @@ KBUILD_CPPFLAGS += $(call cc-option,-fmacro-prefix-map=$(srctree)/=)
>  # include additional Makefiles when needed
>  include-y                      := scripts/Makefile.extrawarn
>  include-$(CONFIG_DEBUG_INFO)   += scripts/Makefile.debug
> +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf
>  include-$(CONFIG_KASAN)                += scripts/Makefile.kasan
>  include-$(CONFIG_KCSAN)                += scripts/Makefile.kcsan
>  include-$(CONFIG_KMSAN)                += scripts/Makefile.kmsan
> diff --git a/scripts/Makefile.btf b/scripts/Makefile.btf
> new file mode 100644
> index 000000000000..82377e470aed
> --- /dev/null
> +++ b/scripts/Makefile.btf
> @@ -0,0 +1,19 @@
> +# SPDX-License-Identifier: GPL-2.0
> +
> +pahole-ver := $(CONFIG_PAHOLE_VERSION)
> +pahole-flags-y :=
> +
> +# pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> +ifeq ($(call test-le, $(pahole-ver), 121),y)
> +pahole-flags-$(call test-ge, $(pahole-ver), 118)       += --skip_encoding_btf_vars
> +endif
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 121)       += --btf_gen_floats
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 122)       += -j
> +
> +pahole-flags-$(CONFIG_PAHOLE_HAS_LANG_EXCLUDE)         += --lang_exclude=rust
> +
> +pahole-flags-$(call test-ge, $(pahole-ver), 125)       += --skip_encoding_btf_inconsistent_proto --btf_gen_optimized
> +
> +export PAHOLE_FLAGS := $(pahole-flags-y)
> diff --git a/scripts/pahole-flags.sh b/scripts/pahole-flags.sh
> deleted file mode 100755
> index 728d55190d97..000000000000
> --- a/scripts/pahole-flags.sh
> +++ /dev/null
> @@ -1,30 +0,0 @@
> -#!/bin/sh
> -# SPDX-License-Identifier: GPL-2.0
> -
> -extra_paholeopt=
> -
> -if ! [ -x "$(command -v ${PAHOLE})" ]; then
> -       exit 0
> -fi
> -
> -pahole_ver=$($(dirname $0)/pahole-version.sh ${PAHOLE})
> -
> -if [ "${pahole_ver}" -ge "118" ] && [ "${pahole_ver}" -le "121" ]; then
> -       # pahole 1.18 through 1.21 can't handle zero-sized per-CPU vars
> -       extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_vars"
> -fi
> -if [ "${pahole_ver}" -ge "121" ]; then
> -       extra_paholeopt="${extra_paholeopt} --btf_gen_floats"
> -fi
> -if [ "${pahole_ver}" -ge "122" ]; then
> -       extra_paholeopt="${extra_paholeopt} -j"
> -fi
> -if [ "${pahole_ver}" -ge "124" ]; then
> -       # see PAHOLE_HAS_LANG_EXCLUDE
> -       extra_paholeopt="${extra_paholeopt} --lang_exclude=rust"
> -fi
> -if [ "${pahole_ver}" -ge "125" ]; then
> -       extra_paholeopt="${extra_paholeopt} --skip_encoding_btf_inconsistent_proto --btf_gen_optimized"
> -fi
> -
> -echo ${extra_paholeopt}
> --
> 2.40.1
>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 19:56   ` Andrii Nakryiko
@ 2023-10-18  5:00     ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-18  5:00 UTC (permalink / raw)
  To: Andrii Nakryiko
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On Wed, Oct 18, 2023 at 4:57 AM Andrii Nakryiko
<andrii.nakryiko@gmail.com> wrote:
>
> On Tue, Oct 17, 2023 at 3:38 AM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > scripts/pahole-flags.sh is executed so many times.
> >
> > You can check how many times it is invoked during the build, as follows:
> >
> >   $ cat <<EOF >> scripts/pahole-flags.sh
> >   > echo "scripts/pahole-flags.sh was executed" >&2
> >   > EOF
> >
> >   $ make -s
> >   scripts/pahole-flags.sh was executed
> >   scripts/pahole-flags.sh was executed
> >   scripts/pahole-flags.sh was executed
> >   scripts/pahole-flags.sh was executed
> >   scripts/pahole-flags.sh was executed
> >     [ lots of repeated lines suppressed... ]
> >
> > This scripts is exectuted more than 20 times during the kernel build
> > because PAHOLE_FLAGS is a recursively expanded variable and exported
> > to sub-processes.
> >
> > With the GNU Make >= 4.4, it is executed more than 60 times because
> > exported variables are also passed to other $(shell ) invocations.
> > Without careful coding, it is known to cause an exponential fork
> > explosion. [1]
> >
> > The use of $(shell ) in an exported recursive variable is likely wrong
> > because $(shell ) is always evaluated due to the 'export' keyword, and
> > the evaluation can occur multiple times by the nature of recursive
> > variables.
> >
> > Convert the shell script to a Makefile, which is included only when
> > CONFIG_DEBUG_INFO_BTF=y.
> >
> > [1]: https://savannah.gnu.org/bugs/index.php?64746
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > ---
> >
> >  Makefile                |  4 +---
> >  scripts/Makefile.btf    | 19 +++++++++++++++++++
> >  scripts/pahole-flags.sh | 30 ------------------------------
> >  3 files changed, 20 insertions(+), 33 deletions(-)
> >  create mode 100644 scripts/Makefile.btf
> >  delete mode 100755 scripts/pahole-flags.sh
> >
> > diff --git a/Makefile b/Makefile
> > index fed9a6cc3665..eaddec67e5e1 100644
> > --- a/Makefile
> > +++ b/Makefile
> > @@ -513,8 +513,6 @@ LZ4         = lz4c
> >  XZ             = xz
> >  ZSTD           = zstd
> >
> > -PAHOLE_FLAGS   = $(shell PAHOLE=$(PAHOLE) $(srctree)/scripts/pahole-flags.sh)
>
> What if we just used := here? Wouldn't it avoid unnecessary multiple executions?


Yeah, := is less silly than =.


But, I do not like to run the script for non-build targets
such as 'make clean', 'make help', etc.

Also, when building with CONFIG_DEBUG_INFO_BTF=n,
the shell is forked to compute PAHOLE_FLAGS,
which we know are unnecessary.





--
Best Regards

Masahiro Yamada

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 15:21   ` Miguel Ojeda
@ 2023-10-18 12:14     ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-18 12:14 UTC (permalink / raw)
  To: Miguel Ojeda
  Cc: linux-kbuild, linux-kernel, Alex Gaynor, Alexei Starovoitov,
	Alice Ryhl, Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On Wed, Oct 18, 2023 at 12:21 AM Miguel Ojeda
<miguel.ojeda.sandonis@gmail.com> wrote:
>
> On Tue, Oct 17, 2023 at 12:38 PM Masahiro Yamada <masahiroy@kernel.org> wrote:
> >
> > Convert the shell script to a Makefile, which is included only when
> > CONFIG_DEBUG_INFO_BTF=y.
> >
> > [1]: https://savannah.gnu.org/bugs/index.php?64746
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
>
> The field in `MAINTAINERS` should be removed:
>
>     F: scripts/pahole-flags.sh


I will replace it with

F: scripts/Makefile.btf


Thanks.



>
> But other than that, it looks good to me! I tried it for a given
> config and it does call `pahole` with the same flags.
>
> Tested-by: Miguel Ojeda <ojeda@kernel.org>
> Acked-by: Miguel Ojeda <ojeda@kernel.org>
>
> Cheers,
> Miguel



-- 
Best Regards
Masahiro Yamada

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 4/4] kbuild: refactor module BTF rule
  2023-10-17 12:58   ` Nicolas Schier
@ 2023-10-18 15:16     ` Masahiro Yamada
  0 siblings, 0 replies; 23+ messages in thread
From: Masahiro Yamada @ 2023-10-18 15:16 UTC (permalink / raw)
  To: Masahiro Yamada, linux-kbuild, linux-kernel, Nathan Chancellor,
	Nick Desaulniers
  Cc: Nicolas Schier

On Tue, Oct 17, 2023 at 9:58 PM Nicolas Schier <n.schier@avm.de> wrote:
>
> On Tue, Oct 17, 2023 at 07:37:42PM +0900, Masahiro Yamada wrote:
> > newer_prereqs_except and if_changed_except are ugly hacks of the
> > newer_prereqs and if_changed in scripts/Kbuild.include.
>
> newer-prereqs

Yes.


>
> >
> > Remove.
> >
> > Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> > ---
> >
> >  scripts/Makefile.modfinal | 23 +++++------------------
> >  1 file changed, 5 insertions(+), 18 deletions(-)
> >
> > diff --git a/scripts/Makefile.modfinal b/scripts/Makefile.modfinal
> > index 9fd7a26e4fe9..6ab10dba05c7 100644
> > --- a/scripts/Makefile.modfinal
> > +++ b/scripts/Makefile.modfinal
> > @@ -19,6 +19,9 @@ vmlinux :=
> >  ifdef CONFIG_DEBUG_INFO_BTF_MODULES
> >  ifneq ($(wildcard vmlinux),)
> >  vmlinux := vmlinux
> > +cmd_btf = ; \
>
> while reading, I stumpled over this semicolon, but probably it's a good
> reminder that cmd_btf is only a cmd extension.


A semicolon is needed as a command separator, but
the trailing semicolon after the last command.

I usually prepend a semicolon to conditional commands.


>
> > +     LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
> > +     $(RESOLVE_BTFIDS) -b vmlinux $@
> >  else
> >  $(warning Skipping BTF generation due to unavailability of vmlinux)
> >  endif
> > @@ -41,27 +44,11 @@ quiet_cmd_ld_ko_o = LD [M]  $@
> >        cmd_ld_ko_o +=                                                 \
> >       $(LD) -r $(KBUILD_LDFLAGS)                                      \
> >               $(KBUILD_LDFLAGS_MODULE) $(LDFLAGS_MODULE)              \
> > -             -T scripts/module.lds -o $@ $(filter %.o, $^)
> > +             -T scripts/module.lds -o $@ $(filter %.o, $^)           \
> > +     $(cmd_btf)
> >
> > -quiet_cmd_btf_ko = BTF [M] $@
> > -      cmd_btf_ko =                                                   \
> > -             LLVM_OBJCOPY="$(OBJCOPY)" $(PAHOLE) -J $(PAHOLE_FLAGS) --btf_base vmlinux $@; \
> > -             $(RESOLVE_BTFIDS) -b vmlinux $@
> > -
> > -# Same as newer-prereqs, but allows to exclude specified extra dependencies
> > -newer_prereqs_except = $(filter-out $(PHONY) $(1),$?)
> > -
> > -# Same as if_changed, but allows to exclude specified extra dependencies
> > -if_changed_except = $(if $(call newer_prereqs_except,$(2))$(cmd-check),      \
> > -     $(cmd);                                                              \
> > -     printf '%s\n' 'savedcmd_$@ := $(make-cmd)' > $(dot-target).cmd, @:)
> > -
> > -# Re-generate module BTFs if either module's .ko or vmlinux changed
> >  %.ko: %.o %.mod.o scripts/module.lds $(vmlinux) FORCE
> >       +$(call if_changed_except,ld_ko_o,vmlinux)
>
> This should probably be:
>
>   +$(call if_changed,ld_ko_o)


Right. Thanks for catching it.


>
> > -ifdef vmlinux
> > -     +$(if $(newer-prereqs),$(call cmd,btf_ko))
> > -endif
> >
> >  targets += $(modules:%.o=%.ko) $(modules:%.o=%.mod.o)
> >
> > --
> > 2.40.1
> >



-- 
Best Regards
Masahiro Yamada

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh
  2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
                     ` (4 preceding siblings ...)
  2023-10-17 19:56   ` Andrii Nakryiko
@ 2023-10-23  0:30   ` Martin Rodriguez Reboredo
  5 siblings, 0 replies; 23+ messages in thread
From: Martin Rodriguez Reboredo @ 2023-10-23  0:30 UTC (permalink / raw)
  To: Masahiro Yamada, linux-kbuild
  Cc: linux-kernel, Alex Gaynor, Alexei Starovoitov, Alice Ryhl,
	Andreas Hindborg, Andrii Nakryiko, Benno Lossin,
	Björn Roy Baron, Boqun Feng, Daniel Borkmann, Gary Guo,
	Hao Luo, Jiri Olsa, John Fastabend, KP Singh, Martin KaFai Lau,
	Miguel Ojeda, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Song Liu, Stanislav Fomichev,
	Wedson Almeida Filho, Yonghong Song, bpf, rust-for-linux

On 10/17/23 07:37, Masahiro Yamada wrote:
> scripts/pahole-flags.sh is executed so many times.
> 
> You can check how many times it is invoked during the build, as follows:
> 
>    $ cat <<EOF >> scripts/pahole-flags.sh
>    > echo "scripts/pahole-flags.sh was executed" >&2
>    > EOF
> 
>    $ make -s
>    scripts/pahole-flags.sh was executed
>    scripts/pahole-flags.sh was executed
>    scripts/pahole-flags.sh was executed
>    scripts/pahole-flags.sh was executed
>    scripts/pahole-flags.sh was executed
>      [ lots of repeated lines suppressed... ]
> 
> This scripts is exectuted more than 20 times during the kernel build
> because PAHOLE_FLAGS is a recursively expanded variable and exported
> to sub-processes.
> 
> With the GNU Make >= 4.4, it is executed more than 60 times because
> exported variables are also passed to other $(shell ) invocations.
> Without careful coding, it is known to cause an exponential fork
> explosion. [1]
> 
> The use of $(shell ) in an exported recursive variable is likely wrong
> because $(shell ) is always evaluated due to the 'export' keyword, and
> the evaluation can occur multiple times by the nature of recursive
> variables.
> 
> Convert the shell script to a Makefile, which is included only when
> CONFIG_DEBUG_INFO_BTF=y.
> 
> [1]: https://savannah.gnu.org/bugs/index.php?64746
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
> [...]
> +include-$(CONFIG_DEBUG_INFO_BTF)+= scripts/Makefile.btf

Would have used a tab.

> [...]
Reviewed-by: Martin Rodriguez Reboredo <yakoyoku@gmail.com>

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
  2023-10-17 10:37 ` Masahiro Yamada
  (?)
@ 2023-11-12  0:55   ` patchwork-bot+linux-riscv
  -1 siblings, 0 replies; 23+ messages in thread
From: patchwork-bot+linux-riscv @ 2023-11-12  0:55 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-riscv, linux-kbuild, linux-kernel, aou, bp,
	christophe.leroy, dave.hansen, hpa, mingo, mpe, nathan, npiggin,
	ndesaulniers, nicolas, palmer, paul.walmsley, tsbogend, tglx,
	linux-mips, linuxppc-dev, x86

Hello:

This patch was applied to riscv/linux.git (fixes)
by Masahiro Yamada <masahiroy@kernel.org>:

On Tue, 17 Oct 2023 19:37:39 +0900 you wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> 
> [...]

Here is the summary with links:
  - [1/4] kbuild: remove ARCH_POSTLINK from module builds
    https://git.kernel.org/riscv/c/7f6d8f7e43fb

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-11-12  0:55   ` patchwork-bot+linux-riscv
  0 siblings, 0 replies; 23+ messages in thread
From: patchwork-bot+linux-riscv @ 2023-11-12  0:55 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: linux-riscv, linux-kbuild, linux-kernel, aou, bp,
	christophe.leroy, dave.hansen, hpa, mingo, mpe, nathan, npiggin,
	ndesaulniers, nicolas, palmer, paul.walmsley, tsbogend, tglx,
	linux-mips, linuxppc-dev, x86

Hello:

This patch was applied to riscv/linux.git (fixes)
by Masahiro Yamada <masahiroy@kernel.org>:

On Tue, 17 Oct 2023 19:37:39 +0900 you wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> 
> [...]

Here is the summary with links:
  - [1/4] kbuild: remove ARCH_POSTLINK from module builds
    https://git.kernel.org/riscv/c/7f6d8f7e43fb

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



_______________________________________________
linux-riscv mailing list
linux-riscv@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-riscv

^ permalink raw reply	[flat|nested] 23+ messages in thread
* Re: [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds
@ 2023-11-12  0:55   ` patchwork-bot+linux-riscv
  0 siblings, 0 replies; 23+ messages in thread
From: patchwork-bot+linux-riscv @ 2023-11-12  0:55 UTC (permalink / raw)
  To: Masahiro Yamada
  Cc: x86, aou, nicolas, linux-kbuild, dave.hansen, ndesaulniers,
	linux-kernel, linux-mips, nathan, mingo, bp, npiggin,
	paul.walmsley, hpa, palmer, linux-riscv, tsbogend, linuxppc-dev,
	tglx

Hello:

This patch was applied to riscv/linux.git (fixes)
by Masahiro Yamada <masahiroy@kernel.org>:

On Tue, 17 Oct 2023 19:37:39 +0900 you wrote:
> The '%.ko' rule in arch/*/Makefile.postlink does nothing but call the
> 'true' command.
> 
> Remove the meaningless code.
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> 
> [...]

Here is the summary with links:
  - [1/4] kbuild: remove ARCH_POSTLINK from module builds
    https://git.kernel.org/riscv/c/7f6d8f7e43fb

You are awesome, thank you!
-- 
Deet-doot-dot, I am a bot.
https://korg.docs.kernel.org/patchwork/pwbot.html



^ permalink raw reply	[flat|nested] 23+ messages in thread
end of thread, other threads:[~2023-11-12  0:57 UTC | newest]

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-10-17 10:37 [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds Masahiro Yamada
2023-10-17 10:37 ` Masahiro Yamada
2023-10-17 10:37 ` Masahiro Yamada
2023-10-17 10:37 ` [PATCH 2/4] kbuild: avoid too many execution of scripts/pahole-flags.sh Masahiro Yamada
2023-10-17 12:14   ` Alan Maguire
2023-10-17 12:15   ` Jiri Olsa
2023-10-17 12:29   ` Nicolas Schier
2023-10-17 15:21   ` Miguel Ojeda
2023-10-18 12:14     ` Masahiro Yamada
2023-10-17 19:56   ` Andrii Nakryiko
2023-10-18  5:00     ` Masahiro Yamada
2023-10-23  0:30   ` Martin Rodriguez Reboredo
2023-10-17 10:37 ` [PATCH 3/4] kbuild: skip module BTF with one-time check for vmlinux Masahiro Yamada
2023-10-17 12:37   ` Nicolas Schier
2023-10-17 10:37 ` [PATCH 4/4] kbuild: refactor module BTF rule Masahiro Yamada
2023-10-17 12:58   ` Nicolas Schier
2023-10-18 15:16     ` Masahiro Yamada
2023-10-17 12:09 ` [PATCH 1/4] kbuild: remove ARCH_POSTLINK from module builds Nicolas Schier
2023-10-17 12:09   ` Nicolas Schier
2023-10-17 12:09   ` Nicolas Schier
2023-11-12  0:55 ` patchwork-bot+linux-riscv
2023-11-12  0:55   ` patchwork-bot+linux-riscv
2023-11-12  0:55   ` patchwork-bot+linux-riscv

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.