Re: [PATCH v3 6/7] kbuild: use obj-y instead extra-y for objects placed at the head

[Jiri Slaby <jirislaby@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 2709 bytes --]

Hi,

following an IRC discussion with many parties...

On 24. 09. 22, 20:19, Masahiro Yamada wrote:
> The objects placed at the head of vmlinux need special treatments:
> 
>   - arch/$(SRCARCH)/Makefile adds them to head-y in order to place
>     them before other archives in the linker command line.
> 
>   - arch/$(SRCARCH)/kernel/Makefile adds them to extra-y instead of
>     obj-y to avoid them going into built-in.a.
> 
> This commit gets rid of the latter.
> 
> Create vmlinux.a to collect all the objects that are unconditionally
> linked to vmlinux. The objects listed in head-y are moved to the head
> of vmlinux.a by using 'ar m'.
> 
> With this, arch/$(SRCARCH)/kernel/Makefile can consistently use obj-y
> for builtin objects.
> 
> There is no *.o that is directly linked to vmlinux. Drop unneeded code
> in scripts/clang-tools/gen_compile_commands.py.
> 
> $(AR) mPi needs 'T' to workaround the llvm-ar bug. The fix was suggested
> by Nathan Chancellor [1].
> 
> [1]: https://lore.kernel.org/llvm/YyjjT5gQ2hGMH0ni@dev-arch.thelio-3990X/
> 
> Signed-off-by: Masahiro Yamada <masahiroy@kernel.org>
> ---
...
> --- a/scripts/Makefile.vmlinux_o
> +++ b/scripts/Makefile.vmlinux_o
> @@ -18,7 +18,7 @@ quiet_cmd_gen_initcalls_lds = GEN     $@
>   	$(PERL) $(real-prereqs) > $@
>   
>   .tmp_initcalls.lds: $(srctree)/scripts/generate_initcall_order.pl \
> -		$(KBUILD_VMLINUX_OBJS) $(KBUILD_VMLINUX_LIBS) FORCE
> +		vmlinux.a $(KBUILD_VMLINUX_LIBS) FORCE

There is a slight problem with this. The kernel built with gcc-LTO does 
not boot. But as I understand it, it's not limited to gcc-LTO only.

On x86, startup_64() is supposed to be at offset >zero< of the image 
(see .Lrelocated()). It was ensured by putting head64.o to the beginning 
of vmlinux (by KBUILD_VMLINUX_OBJS on the LD command-line above). The 
patch above instead packs head64.o into vmlinux.a and then moves it 
using "ar -m" to the beginning (it's in 7/7 of the series IIRC).

The problem is that .o files listed on the LD command line explicitly 
are taken as spelled. But unpacking .a inside LD gives no guarantees on 
the order of packed objects. To quote: "that it happens to work 
sometimes is pure luck." (Correct me guys, if I misunderstood you.)

For x86, the most ideal fix seems to be to fix it in the linker script. 
By putting startup_64() to a different section and handle it in the ld 
script specially -- see the attachment. It should always have been put 
this way, the command line order is only a workaround. But this might 
need more fixes on other archs too -- I haven't take a look.

Ideas, comments? I'll send the attachment as a PATCH later (if there are 
no better suggestions).

thanks,
-- 
js
suse labs

[-- Attachment #2: 0001-head-fix.patch --]
[-- Type: text/x-patch, Size: 1731 bytes --]

From 8565e13d5d29eb32bed9674240593315f3bdd7f5 Mon Sep 17 00:00:00 2001
From: Jiri Slaby <jslaby@suse.cz>
Date: Mon, 24 Oct 2022 11:17:04 +0200
Subject: [PATCH] head fix

Signed-off-by: Jiri Slaby <jslaby@suse.cz>
---
 arch/x86/kernel/head_64.S     | 4 +++-
 arch/x86/kernel/vmlinux.lds.S | 1 +
 include/linux/init.h          | 1 +
 3 files changed, 5 insertions(+), 1 deletion(-)

diff --git a/arch/x86/kernel/head_64.S b/arch/x86/kernel/head_64.S
index d860d437631b..417bcd9da3df 100644
--- a/arch/x86/kernel/head_64.S
+++ b/arch/x86/kernel/head_64.S
@@ -39,8 +39,8 @@ L4_START_KERNEL = l4_index(__START_KERNEL_map)
 L3_START_KERNEL = pud_index(__START_KERNEL_map)
 
 	.text
-	__HEAD
 	.code64
+	__HEAD_FIRST
 SYM_CODE_START_NOALIGN(startup_64)
 	UNWIND_HINT_EMPTY
 	/*
@@ -126,6 +126,8 @@ SYM_CODE_START_NOALIGN(startup_64)
 	jmp 1f
 SYM_CODE_END(startup_64)
 
+	__HEAD
+
 SYM_CODE_START(secondary_startup_64)
 	UNWIND_HINT_EMPTY
 	ANNOTATE_NOENDBR
diff --git a/arch/x86/kernel/vmlinux.lds.S b/arch/x86/kernel/vmlinux.lds.S
index 15f29053cec4..a10a9c50bc3f 100644
--- a/arch/x86/kernel/vmlinux.lds.S
+++ b/arch/x86/kernel/vmlinux.lds.S
@@ -126,6 +126,7 @@ SECTIONS
 		_text = .;
 		_stext = .;
 		/* bootstrapping code */
+		KEEP(*(.head.first.text))
 		HEAD_TEXT
 		TEXT_TEXT
 		SCHED_TEXT
diff --git a/include/linux/init.h b/include/linux/init.h
index ca827e2fb0da..3a11d19e39cf 100644
--- a/include/linux/init.h
+++ b/include/linux/init.h
@@ -92,6 +92,7 @@
 #define __memexitconst   __section(".memexit.rodata")
 
 /* For assembly routines */
+#define __HEAD_FIRST	.section	".head.first.text","ax"
 #define __HEAD		.section	".head.text","ax"
 #define __INIT		.section	".init.text","ax"
 #define __FINIT		.previous
-- 
2.38.0