linux-kbuild.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image
@ 2023-09-17 15:21 Thomas Weißschuh
  2023-09-18 16:20 ` Willy Tarreau
  2023-09-19 11:47 ` Nicolas Schier
  0 siblings, 2 replies; 3+ messages in thread
From: Thomas Weißschuh @ 2023-09-17 15:21 UTC (permalink / raw)
  To: Masahiro Yamada, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Willy Tarreau, Shuah Khan
  Cc: linux-kbuild, linux-kernel, linux-kselftest, Thomas Weißschuh

When the initramfs is embedded into the kernel each rebuild of it will
trigger a full kernel relink and all the expensive postprocessing steps.

Currently nolibc-test and therefore the initramfs are always rebuild,
even without source changes, leading to lots of slow kernel relinks.

Instead of linking the initramfs into the kernel assemble it manually
and pass it explicitly to qemu.
This avoids all of the kernel relinks.

Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
---
Currently the nolibc testsuite embeds the test executable into a kernel
image with CONFIG_INITRAMFS_SOURCE.
This forces a full kernel relink everytime the test executable is
updated.

This relinking step dominates the test cycle.
It is slower than building and running the test in qemu together.

With a bit of Makefile-shuffling the relinking can be avoided.
---
Changes in v2:
- avoid need to modify top-level Makefile
- drop patch removing "rerun" target
- add kernel-standalone target
- Link to v1: https://lore.kernel.org/r/20230916-nolibc-initramfs-v1-0-4416ecedca6d@weissschuh.net
---
 tools/testing/selftests/nolibc/Makefile | 42 ++++++++++++++++++++-------------
 1 file changed, 26 insertions(+), 16 deletions(-)

diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
index 689658f81a19..ee6a9ad28cfd 100644
--- a/tools/testing/selftests/nolibc/Makefile
+++ b/tools/testing/selftests/nolibc/Makefile
@@ -131,18 +131,20 @@ REPORT  ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{if (!f) printf("\n"); f++
 
 help:
 	@echo "Supported targets under selftests/nolibc:"
-	@echo "  all          call the \"run\" target below"
-	@echo "  help         this help"
-	@echo "  sysroot      create the nolibc sysroot here (uses \$$ARCH)"
-	@echo "  nolibc-test  build the executable (uses \$$CC and \$$CROSS_COMPILE)"
-	@echo "  libc-test    build an executable using the compiler's default libc instead"
-	@echo "  run-user     runs the executable under QEMU (uses \$$XARCH, \$$TEST)"
-	@echo "  initramfs    prepare the initramfs with nolibc-test"
-	@echo "  defconfig    create a fresh new default config (uses \$$XARCH)"
-	@echo "  kernel       (re)build the kernel with the initramfs (uses \$$XARCH)"
-	@echo "  run          runs the kernel in QEMU after building it (uses \$$XARCH, \$$TEST)"
-	@echo "  rerun        runs a previously prebuilt kernel in QEMU (uses \$$XARCH, \$$TEST)"
-	@echo "  clean        clean the sysroot, initramfs, build and output files"
+	@echo "  all               call the \"run\" target below"
+	@echo "  help              this help"
+	@echo "  sysroot           create the nolibc sysroot here (uses \$$ARCH)"
+	@echo "  nolibc-test       build the executable (uses \$$CC and \$$CROSS_COMPILE)"
+	@echo "  libc-test         build an executable using the compiler's default libc instead"
+	@echo "  run-user          runs the executable under QEMU (uses \$$XARCH, \$$TEST)"
+	@echo "  initramfs.cpio    prepare the initramfs archive with nolibc-test"
+	@echo "  initramfs         prepare the initramfs tree with nolibc-test"
+	@echo "  defconfig         create a fresh new default config (uses \$$XARCH)"
+	@echo "  kernel            (re)build the kernel (uses \$$XARCH)"
+	@echo "  kernel-standalone (re)build the kernel with the initramfs (uses \$$XARCH)"
+	@echo "  run               runs the kernel in QEMU after building it (uses \$$XARCH, \$$TEST)"
+	@echo "  rerun             runs a previously prebuilt kernel in QEMU (uses \$$XARCH, \$$TEST)"
+	@echo "  clean             clean the sysroot, initramfs, build and output files"
 	@echo ""
 	@echo "The output file is \"run.out\". Test ranges may be passed using \$$TEST."
 	@echo ""
@@ -195,6 +197,9 @@ run-user: nolibc-test
 	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
+initramfs.cpio: kernel nolibc-test
+	$(QUIET_GEN)echo 'file /init nolibc-test 755 0 0' | $(srctree)/usr/gen_init_cpio - > initramfs.cpio
+
 initramfs: nolibc-test
 	$(QUIET_MKDIR)mkdir -p initramfs
 	$(call QUIET_INSTALL, initramfs/init)
@@ -203,17 +208,20 @@ initramfs: nolibc-test
 defconfig:
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
 
-kernel: initramfs
+kernel:
+	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME)
+
+kernel-standalone: initramfs
 	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
 
 # run the tests after building the kernel
-run: kernel
-	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
+run: kernel initramfs.cpio
+	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -initrd initramfs.cpio -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
 # re-run the tests from an existing kernel
 rerun:
-	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
+	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -initrd initramfs.cpio -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
 	$(Q)$(REPORT) $(CURDIR)/run.out
 
 # report with existing test log
@@ -227,6 +235,8 @@ clean:
 	$(Q)rm -f nolibc-test
 	$(call QUIET_CLEAN, libc-test)
 	$(Q)rm -f libc-test
+	$(call QUIET_CLEAN, initramfs.cpio)
+	$(Q)rm -rf initramfs.cpio
 	$(call QUIET_CLEAN, initramfs)
 	$(Q)rm -rf initramfs
 	$(call QUIET_CLEAN, run.out)

---
base-commit: 3f79a57865b33f49fdae6655510bd27c8e6610e0
change-id: 20230916-nolibc-initramfs-4fd00eac3256

Best regards,
-- 
Thomas Weißschuh <linux@weissschuh.net>


^ permalink raw reply related	[flat|nested] 3+ messages in thread

* Re: [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image
  2023-09-17 15:21 [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image Thomas Weißschuh
@ 2023-09-18 16:20 ` Willy Tarreau
  2023-09-19 11:47 ` Nicolas Schier
  1 sibling, 0 replies; 3+ messages in thread
From: Willy Tarreau @ 2023-09-18 16:20 UTC (permalink / raw)
  To: Thomas Weißschuh
  Cc: Masahiro Yamada, Nathan Chancellor, Nick Desaulniers,
	Nicolas Schier, Shuah Khan, linux-kbuild, linux-kernel,
	linux-kselftest

Hi Thomas,

On Sun, Sep 17, 2023 at 05:21:38PM +0200, Thomas Weißschuh wrote:
> When the initramfs is embedded into the kernel each rebuild of it will
> trigger a full kernel relink and all the expensive postprocessing steps.
> 
> Currently nolibc-test and therefore the initramfs are always rebuild,
> even without source changes, leading to lots of slow kernel relinks.
> 
> Instead of linking the initramfs into the kernel assemble it manually
> and pass it explicitly to qemu.
> This avoids all of the kernel relinks.
> 
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>

OK, let's try it!

thanks,
Willy

^ permalink raw reply	[flat|nested] 3+ messages in thread

* Re: [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image
  2023-09-17 15:21 [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image Thomas Weißschuh
  2023-09-18 16:20 ` Willy Tarreau
@ 2023-09-19 11:47 ` Nicolas Schier
  1 sibling, 0 replies; 3+ messages in thread
From: Nicolas Schier @ 2023-09-19 11:47 UTC (permalink / raw)
  To: Thomas Weißschuh
  Cc: Masahiro Yamada, Nathan Chancellor, Nick Desaulniers,
	Willy Tarreau, Shuah Khan, linux-kbuild, linux-kernel,
	linux-kselftest

On Sun, Sep 17, 2023 at 05:21:38PM +0200 Thomas Weißschuh wrote:
> When the initramfs is embedded into the kernel each rebuild of it will
> trigger a full kernel relink and all the expensive postprocessing steps.
> 
> Currently nolibc-test and therefore the initramfs are always rebuild,
> even without source changes, leading to lots of slow kernel relinks.
> 
> Instead of linking the initramfs into the kernel assemble it manually
> and pass it explicitly to qemu.
> This avoids all of the kernel relinks.
> 
> Signed-off-by: Thomas Weißschuh <linux@weissschuh.net>
> ---
> Currently the nolibc testsuite embeds the test executable into a kernel
> image with CONFIG_INITRAMFS_SOURCE.
> This forces a full kernel relink everytime the test executable is
> updated.
> 
> This relinking step dominates the test cycle.
> It is slower than building and running the test in qemu together.
> 
> With a bit of Makefile-shuffling the relinking can be avoided.
> ---
> Changes in v2:
> - avoid need to modify top-level Makefile
> - drop patch removing "rerun" target
> - add kernel-standalone target
> - Link to v1: https://lore.kernel.org/r/20230916-nolibc-initramfs-v1-0-4416ecedca6d@weissschuh.net
> ---

Thanks, seems to work as described (and I am surprised how fast the 'run'
target is) and patch looks good to me.

Reviewed-by: Nicolas Schier <nicolas@fjasle.eu>


>  tools/testing/selftests/nolibc/Makefile | 42 ++++++++++++++++++++-------------
>  1 file changed, 26 insertions(+), 16 deletions(-)
> 
> diff --git a/tools/testing/selftests/nolibc/Makefile b/tools/testing/selftests/nolibc/Makefile
> index 689658f81a19..ee6a9ad28cfd 100644
> --- a/tools/testing/selftests/nolibc/Makefile
> +++ b/tools/testing/selftests/nolibc/Makefile
> @@ -131,18 +131,20 @@ REPORT  ?= awk '/\[OK\][\r]*$$/{p++} /\[FAIL\][\r]*$$/{if (!f) printf("\n"); f++
>  
>  help:
>  	@echo "Supported targets under selftests/nolibc:"
> -	@echo "  all          call the \"run\" target below"
> -	@echo "  help         this help"
> -	@echo "  sysroot      create the nolibc sysroot here (uses \$$ARCH)"
> -	@echo "  nolibc-test  build the executable (uses \$$CC and \$$CROSS_COMPILE)"
> -	@echo "  libc-test    build an executable using the compiler's default libc instead"
> -	@echo "  run-user     runs the executable under QEMU (uses \$$XARCH, \$$TEST)"
> -	@echo "  initramfs    prepare the initramfs with nolibc-test"
> -	@echo "  defconfig    create a fresh new default config (uses \$$XARCH)"
> -	@echo "  kernel       (re)build the kernel with the initramfs (uses \$$XARCH)"
> -	@echo "  run          runs the kernel in QEMU after building it (uses \$$XARCH, \$$TEST)"
> -	@echo "  rerun        runs a previously prebuilt kernel in QEMU (uses \$$XARCH, \$$TEST)"
> -	@echo "  clean        clean the sysroot, initramfs, build and output files"
> +	@echo "  all               call the \"run\" target below"
> +	@echo "  help              this help"
> +	@echo "  sysroot           create the nolibc sysroot here (uses \$$ARCH)"
> +	@echo "  nolibc-test       build the executable (uses \$$CC and \$$CROSS_COMPILE)"
> +	@echo "  libc-test         build an executable using the compiler's default libc instead"
> +	@echo "  run-user          runs the executable under QEMU (uses \$$XARCH, \$$TEST)"
> +	@echo "  initramfs.cpio    prepare the initramfs archive with nolibc-test"
> +	@echo "  initramfs         prepare the initramfs tree with nolibc-test"
> +	@echo "  defconfig         create a fresh new default config (uses \$$XARCH)"
> +	@echo "  kernel            (re)build the kernel (uses \$$XARCH)"
> +	@echo "  kernel-standalone (re)build the kernel with the initramfs (uses \$$XARCH)"
> +	@echo "  run               runs the kernel in QEMU after building it (uses \$$XARCH, \$$TEST)"
> +	@echo "  rerun             runs a previously prebuilt kernel in QEMU (uses \$$XARCH, \$$TEST)"
> +	@echo "  clean             clean the sysroot, initramfs, build and output files"
>  	@echo ""
>  	@echo "The output file is \"run.out\". Test ranges may be passed using \$$TEST."
>  	@echo ""
> @@ -195,6 +197,9 @@ run-user: nolibc-test
>  	$(Q)qemu-$(QEMU_ARCH) ./nolibc-test > "$(CURDIR)/run.out" || :
>  	$(Q)$(REPORT) $(CURDIR)/run.out
>  
> +initramfs.cpio: kernel nolibc-test
> +	$(QUIET_GEN)echo 'file /init nolibc-test 755 0 0' | $(srctree)/usr/gen_init_cpio - > initramfs.cpio
> +
>  initramfs: nolibc-test
>  	$(QUIET_MKDIR)mkdir -p initramfs
>  	$(call QUIET_INSTALL, initramfs/init)
> @@ -203,17 +208,20 @@ initramfs: nolibc-test
>  defconfig:
>  	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) mrproper $(DEFCONFIG) prepare
>  
> -kernel: initramfs
> +kernel:
> +	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME)
> +
> +kernel-standalone: initramfs
>  	$(Q)$(MAKE) -C $(srctree) ARCH=$(ARCH) CC=$(CC) CROSS_COMPILE=$(CROSS_COMPILE) $(IMAGE_NAME) CONFIG_INITRAMFS_SOURCE=$(CURDIR)/initramfs
>  
>  # run the tests after building the kernel
> -run: kernel
> -	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> +run: kernel initramfs.cpio
> +	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -initrd initramfs.cpio -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
>  	$(Q)$(REPORT) $(CURDIR)/run.out
>  
>  # re-run the tests from an existing kernel
>  rerun:
> -	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
> +	$(Q)qemu-system-$(QEMU_ARCH) -display none -no-reboot -kernel "$(srctree)/$(IMAGE)" -initrd initramfs.cpio -serial stdio $(QEMU_ARGS) > "$(CURDIR)/run.out"
>  	$(Q)$(REPORT) $(CURDIR)/run.out
>  
>  # report with existing test log
> @@ -227,6 +235,8 @@ clean:
>  	$(Q)rm -f nolibc-test
>  	$(call QUIET_CLEAN, libc-test)
>  	$(Q)rm -f libc-test
> +	$(call QUIET_CLEAN, initramfs.cpio)
> +	$(Q)rm -rf initramfs.cpio
>  	$(call QUIET_CLEAN, initramfs)
>  	$(Q)rm -rf initramfs
>  	$(call QUIET_CLEAN, run.out)
> 
> ---
> base-commit: 3f79a57865b33f49fdae6655510bd27c8e6610e0
> change-id: 20230916-nolibc-initramfs-4fd00eac3256
> 
> Best regards,
> -- 
> Thomas Weißschuh <linux@weissschuh.net>

-- 
epost|xmpp: nicolas@fjasle.eu          irc://oftc.net/nsc
↳ gpg: 18ed 52db e34f 860e e9fb  c82b 7d97 0932 55a0 ce7f
     -- frykten for herren er opphav til kunnskap --

^ permalink raw reply	[flat|nested] 3+ messages in thread

end of thread, other threads:[~2023-09-19 11:48 UTC | newest]

Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-09-17 15:21 [PATCH RFC v2] selftests/nolibc: don't embed initramfs into kernel image Thomas Weißschuh
2023-09-18 16:20 ` Willy Tarreau
2023-09-19 11:47 ` Nicolas Schier

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).