[PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Ricardo Ribalda]

When upreving llvm I realised that kexec stopped working on my test
platform.

The reason seems to be that due to PGO there are multiple .text sections
on the purgatory, and kexec does not supports that.

Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
---
Changes in v7:
- Fix $SUBJECT of riscv patch
- Rename PGO as Profile-guided optimization
- Link to v6: https://lore.kernel.org/r/20230321-kexec_clang16-v6-0-a2255e81ab45@chromium.org

Changes in v6:
- Replace linker script with Makefile rule. Thanks Nick
- Link to v5: https://lore.kernel.org/r/20230321-kexec_clang16-v5-0-5563bf7c4173@chromium.org

Changes in v5:
- Add warning when multiple text sections are found. Thanks Simon!
- Add Fixes tag.
- Link to v4: https://lore.kernel.org/r/20230321-kexec_clang16-v4-0-1340518f98e9@chromium.org

Changes in v4:
- Add Cc: stable
- Add linker script for x86
- Add a warning when the kernel image has overlapping sections.
- Link to v3: https://lore.kernel.org/r/20230321-kexec_clang16-v3-0-5f016c8d0e87@chromium.org

Changes in v3:
- Fix initial value. Thanks Ross!
- Link to v2: https://lore.kernel.org/r/20230321-kexec_clang16-v2-0-d10e5d517869@chromium.org

Changes in v2:
- Fix if condition. Thanks Steven!.
- Update Philipp email. Thanks Baoquan.
- Link to v1: https://lore.kernel.org/r/20230321-kexec_clang16-v1-0-a768fc2c7c4d@chromium.org

---
Ricardo Ribalda (4):
      kexec: Support purgatories with .text.hot sections
      x86/purgatory: Remove PGO flags
      powerpc/purgatory: Remove PGO flags
      riscv/purgatory: Remove PGO flags

 arch/powerpc/purgatory/Makefile |  5 +++++
 arch/riscv/purgatory/Makefile   |  5 +++++
 arch/x86/purgatory/Makefile     |  5 +++++
 kernel/kexec_file.c             | 14 +++++++++++++-
 4 files changed, 28 insertions(+), 1 deletion(-)
---
base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f
change-id: 20230321-kexec_clang16-4510c23d129c

Best regards,
-- 
Ricardo Ribalda Delgado <ribalda@chromium.org>

[PATCH v7 1/4] kexec: Support purgatories with .text.hot sections

[Ricardo Ribalda]

Clang16 links the purgatory text in two sections when PGO is in use:

  [ 1] .text             PROGBITS         0000000000000000  00000040
       00000000000011a1  0000000000000000  AX       0     0     16
  [ 2] .rela.text        RELA             0000000000000000  00003498
       0000000000000648  0000000000000018   I      24     1     8
  ...
  [17] .text.hot.        PROGBITS         0000000000000000  00003220
       000000000000020b  0000000000000000  AX       0     0     1
  [18] .rela.text.hot.   RELA             0000000000000000  00004428
       0000000000000078  0000000000000018   I      24    17     8

And both of them have their range [sh_addr ... sh_addr+sh_size] on the
area pointed by `e_entry`.

This causes that image->start is calculated twice, once for .text and
another time for .text.hot. The second calculation leaves image->start
in a random location.

Because of this, the system crashes immediately after:

kexec_core: Starting new kernel

Cc: stable@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Reviewed-by: Ross Zwisler <zwisler@google.com>
Reviewed-by: Steven Rostedt (Google) <rostedt@goodmis.org>
Reviewed-by: Philipp Rudo <prudo@redhat.com>
Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
---
 kernel/kexec_file.c | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index f989f5f1933b..69ee4a29136f 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -901,10 +901,22 @@ static int kexec_purgatory_setup_sechdrs(struct purgatory_info *pi,
 		}
 
 		offset = ALIGN(offset, align);
+
+		/*
+		 * Check if the segment contains the entry point, if so,
+		 * calculate the value of image->start based on it.
+		 * If the compiler has produced more than one .text section
+		 * (Eg: .text.hot), they are generally after the main .text
+		 * section, and they shall not be used to calculate
+		 * image->start. So do not re-calculate image->start if it
+		 * is not set to the initial value, and warn the user so they
+		 * have a chance to fix their purgatory's linker script.
+		 */
 		if (sechdrs[i].sh_flags & SHF_EXECINSTR &&
 		    pi->ehdr->e_entry >= sechdrs[i].sh_addr &&
 		    pi->ehdr->e_entry < (sechdrs[i].sh_addr
-					 + sechdrs[i].sh_size)) {
+					 + sechdrs[i].sh_size) &&
+		    !WARN_ON(kbuf->image->start != pi->ehdr->e_entry)) {
 			kbuf->image->start -= sechdrs[i].sh_addr;
 			kbuf->image->start += kbuf->mem + offset;
 		}

-- 
2.40.1.698.g37aff9b760-goog

[PATCH v7 2/4] x86/purgatory: Remove PGO flags

[Ricardo Ribalda]

If profile-guided optimization is enabled, the purgatory ends up with
multiple .text sections.
This is not supported by kexec and crashes the system.

Cc: stable@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
---
 arch/x86/purgatory/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/x86/purgatory/Makefile b/arch/x86/purgatory/Makefile
index 82fec66d46d2..42abd6af1198 100644
--- a/arch/x86/purgatory/Makefile
+++ b/arch/x86/purgatory/Makefile
@@ -14,6 +14,11 @@ $(obj)/sha256.o: $(srctree)/lib/crypto/sha256.c FORCE
 
 CFLAGS_sha256.o := -D__DISABLE_EXPORTS
 
+# When profile-guided optimization is enabled, llvm emits two different
+# overlapping text sections, which is not supported by kexec. Remove profile
+# optimization flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib

-- 
2.40.1.698.g37aff9b760-goog

[PATCH v7 3/4] powerpc/purgatory: Remove PGO flags

[Ricardo Ribalda]

If profile-guided optimization is enabled, the purgatory ends up with
multiple .text sections.
This is not supported by kexec and crashes the system.

Cc: stable@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
---
 arch/powerpc/purgatory/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/powerpc/purgatory/Makefile b/arch/powerpc/purgatory/Makefile
index 6f5e2727963c..78473d69cd2b 100644
--- a/arch/powerpc/purgatory/Makefile
+++ b/arch/powerpc/purgatory/Makefile
@@ -5,6 +5,11 @@ KCSAN_SANITIZE := n
 
 targets += trampoline_$(BITS).o purgatory.ro
 
+# When profile-guided optimization is enabled, llvm emits two different
+# overlapping text sections, which is not supported by kexec. Remove profile
+# optimization flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
+
 LDFLAGS_purgatory.ro := -e purgatory_start -r --no-undefined
 
 $(obj)/purgatory.ro: $(obj)/trampoline_$(BITS).o FORCE

-- 
2.40.1.698.g37aff9b760-goog

[PATCH v7 4/4] riscv/purgatory: Remove PGO flags

[Ricardo Ribalda]

If profile-guided optimization is enabled, the purgatory ends up with
multiple .text sections.
This is not supported by kexec and crashes the system.

Cc: stable@vger.kernel.org
Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
Acked-by: Palmer Dabbelt <palmer@rivosinc.com>
Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
---
 arch/riscv/purgatory/Makefile | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/arch/riscv/purgatory/Makefile b/arch/riscv/purgatory/Makefile
index 5730797a6b40..bd2e27f82532 100644
--- a/arch/riscv/purgatory/Makefile
+++ b/arch/riscv/purgatory/Makefile
@@ -35,6 +35,11 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
 CFLAGS_string.o := -D__DISABLE_EXPORTS
 CFLAGS_ctype.o := -D__DISABLE_EXPORTS
 
+# When profile-guided optimization is enabled, llvm emits two different
+# overlapping text sections, which is not supported by kexec. Remove profile
+# optimization flags.
+KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=% -fprofile-use=%,$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib

-- 
2.40.1.698.g37aff9b760-goog

Re: [PATCH v7 3/4] powerpc/purgatory: Remove PGO flags

[Michael Ellerman]

Ricardo Ribalda <ribalda@chromium.org> writes:
> If profile-guided optimization is enabled, the purgatory ends up with
> multiple .text sections.
> This is not supported by kexec and crashes the system.
>
> Cc: stable@vger.kernel.org
> Fixes: 930457057abe ("kernel/kexec_file.c: split up __kexec_load_puragory")
> Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
> ---
>  arch/powerpc/purgatory/Makefile | 5 +++++
>  1 file changed, 5 insertions(+)

Acked-by: Michael Ellerman <mpe@ellerman.id.au> (powerpc)

cheers

Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Song Liu]

Hi Ricardo and folks,

On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@chromium.org> wrote:
>
> When upreving llvm I realised that kexec stopped working on my test
> platform.
>
> The reason seems to be that due to PGO there are multiple .text sections
> on the purgatory, and kexec does not supports that.
>
> Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>

We are seeing WARNINGs like the following while kexec'ing a PGO and
LTO enabled kernel:

WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
kexec_load_purgatory+0x37f/0x390

AFAICT, the warning was added by this set, and it was triggered when
we have many .text sections
in purgatory.ro. The kexec was actually successful. So I wonder
whether we really need the
WARNING here. If we disable LTO (PGO is still enabled), we don't see
the WARNING any more.

I also tested an older kernel (5.19 based), where we also see many
.text sections with LTO. It
kexec()'ed fine. (It doesn't have the WARN_ON() in
kexec_purgatory_setup_sechdrs).

Please help us fix this properly (as I really don't know much about kexec).

Thanks in advance,
Song


Here is readelf -S output on purgatory.ro.

With LTO:

readelf -W -S purgatory.ro
There are 48 section headers, starting at offset 0x4a10:

Section Headers:
  [Nr] Name              Type            Address          Off    Size
 ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000
000000 00      0   0  0
  [ 1] .text             PROGBITS        0000000000000000 000040
0000d0 00  AX  0   0 16
  [ 2] .data             PROGBITS        0000000000000000 001000
001000 00  WA  0   0 4096
  [ 3] .rela.text        RELA            0000000000000000 003788
000228 18   I 45   1  8
  [ 4] .rodata           PROGBITS        0000000000000000 002000
0000e0 00   A  0   0 16
  [ 5] .rela.rodata      RELA            0000000000000000 0039b0
000030 18   I 45   4  8
  [ 6] .bss              NOBITS          0000000000000000 0020e0
001000 00  WA  0   0 4096
  [ 7] .text.purgatory   PROGBITS        0000000000000000 0020e0
0000df 00  AX  0   0 16
  [ 8] .rela.text.purgatory RELA            0000000000000000 0039e0
000060 18   I 45   7  8
  [ 9] .text.warn        PROGBITS        0000000000000000 0021c0
000001 00  AX  0   0 16
  [10] .kexec-purgatory  PROGBITS        0000000000000000 0021d0
000120 00  WA  0   0 16
  [11] .comment          PROGBITS        0000000000000000 003a40
000046 01  MS  0   0  1
  [12] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 003a86
000005 00   E  0   0  1
  [13] .text.sha256_update PROGBITS        0000000000000000 0022f0
0008eb 00  AX  0   0 16
  [14] .rela.text.sha256_update RELA            0000000000000000
003a90 000060 18   I 45  13  8
  [15] .text.sha224_update PROGBITS        0000000000000000 002be0
00000c 00  AX  0   0 16
  [16] .rela.text.sha224_update RELA            0000000000000000
003af0 000018 18   I 45  15  8
  [17] .text.sha256_final PROGBITS        0000000000000000 002bf0
0000cd 00  AX  0   0 16
  [18] .rela.text.sha256_final RELA            0000000000000000 003b08
000030 18   I 45  17  8
  [19] .text.sha224_final PROGBITS        0000000000000000 002cc0
0000bd 00  AX  0   0 16
  [20] .rela.text.sha224_final RELA            0000000000000000 003b38
000030 18   I 45  19  8
  [21] .text.sha256      PROGBITS        0000000000000000 002d80
00011d 00  AX  0   0 16
  [22] .rela.text.sha256 RELA            0000000000000000 003b68
000030 18   I 45  21  8
  [23] .modinfo          PROGBITS        0000000000000000 002e9d
000039 00   A  0   0  1
  [24] .rodata.SHA256_K  PROGBITS        0000000000000000 002ee0
000100 00   A  0   0 16
  [25] .rodata.__sha256_final.padding PROGBITS        0000000000000000
002fe0 000040 00   A  0   0 16
  [26] .text.memcmp      PROGBITS        0000000000000000 003020
00000b 00  AX  0   0 16
  [27] .text.bcmp        PROGBITS        0000000000000000 003030
00000b 00  AX  0   0 16
  [28] .text.strcmp      PROGBITS        0000000000000000 003040
000041 00  AX  0   0 16
  [29] .text.strncmp     PROGBITS        0000000000000000 003090
00003a 00  AX  0   0 16
  [30] .text.strnlen     PROGBITS        0000000000000000 0030d0
000039 00  AX  0   0 16
  [31] .text.atou        PROGBITS        0000000000000000 003110
000035 00  AX  0   0 16
  [32] .text.simple_strtoull PROGBITS        0000000000000000 003150
0000b6 00  AX  0   0 16
  [33] .text.simple_strtol PROGBITS        0000000000000000 003210
0001b6 00  AX  0   0 16
  [34] .text.strlen      PROGBITS        0000000000000000 0033d0
00001c 00  AX  0   0 16
  [35] .text.strstr      PROGBITS        0000000000000000 0033f0
00005f 00  AX  0   0 16
  [36] .text.strchr      PROGBITS        0000000000000000 003450
000022 00  AX  0   0 16
  [37] .text.kstrtoull   PROGBITS        0000000000000000 003480
000142 00  AX  0   0 16
  [38] .text.boot_kstrtoul PROGBITS        0000000000000000 0035d0
00000c 00  AX  0   0 16
  [39] .rela.text.boot_kstrtoul RELA            0000000000000000
003b98 000018 18   I 45  38  8
  [40] .text.memset      PROGBITS        0000000000000000 0035e0
00001f 00  AX  0   0 16
  [41] .text.memmove     PROGBITS        0000000000000000 003600
0000a6 00  AX  0   0 16
  [42] .text.memcpy      PROGBITS        0000000000000000 0036b0
0000a6 00  AX  0   0 16
  [43] .rodata.str1.1    PROGBITS        0000000000000000 003756
000032 01 AMS  0   0  1
  [44] .note.GNU-stack   PROGBITS        0000000000000000 003bb0
000000 00      0   0  1
  [45] .symtab           SYMTAB          0000000000000000 003bb0
000948 18     47  68  8
  [46] .shstrtab         STRTAB          0000000000000000 0044f8
0002cd 00      0   0  1
  [47] .strtab           STRTAB          0000000000000000 0047c5
000248 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)


Without LTO:

readelf -W -S purgatory.ro
There are 16 section headers, starting at offset 0x4130:

Section Headers:
  [Nr] Name              Type            Address          Off    Size
 ES Flg Lk Inf Al
  [ 0]                   NULL            0000000000000000 000000
000000 00      0   0  0
  [ 1] .text             PROGBITS        0000000000000000 000040
00131b 00  AX  0   0 16
  [ 2] .rela.text        RELA            0000000000000000 003290
000480 18   I 13   1  8
  [ 3] .kexec-purgatory  PROGBITS        0000000000000000 001360
000120 00  WA  0   0 16
  [ 4] .comment          PROGBITS        0000000000000000 003710
000046 01  MS  0   0  1
  [ 5] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 003756
000005 00   E  0   0  1
  [ 6] .data             PROGBITS        0000000000000000 002000
001000 00  WA  0   0 4096
  [ 7] .rodata           PROGBITS        0000000000000000 003000
000220 00   A  0   0 16
  [ 8] .rela.rodata      RELA            0000000000000000 003760
000030 18   I 13   7  8
  [ 9] .bss              NOBITS          0000000000000000 003220
001000 00  WA  0   0 4096
  [10] .modinfo          PROGBITS        0000000000000000 003220
000039 00   A  0   0  1
  [11] .rodata.str1.1    PROGBITS        0000000000000000 003259
000032 01 AMS  0   0  1
  [12] .note.GNU-stack   PROGBITS        0000000000000000 003790
000000 00      0   0  1
  [13] .symtab           SYMTAB          0000000000000000 003790
0006d8 18     15  43  8
  [14] .shstrtab         STRTAB          0000000000000000 003e68
00009c 00      0   0  1
  [15] .strtab           STRTAB          0000000000000000 003f04
00022b 00      0   0  1
Key to Flags:
  W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
  L (link order), O (extra OS processing required), G (group), T (TLS),
  C (compressed), x (unknown), o (OS specific), E (exclude),
  l (large), p (processor specific)

[...]

> Ricardo Ribalda (4):
>       kexec: Support purgatories with .text.hot sections
>       x86/purgatory: Remove PGO flags
>       powerpc/purgatory: Remove PGO flags
>       riscv/purgatory: Remove PGO flags
>
>  arch/powerpc/purgatory/Makefile |  5 +++++
>  arch/riscv/purgatory/Makefile   |  5 +++++
>  arch/x86/purgatory/Makefile     |  5 +++++
>  kernel/kexec_file.c             | 14 +++++++++++++-
>  4 files changed, 28 insertions(+), 1 deletion(-)
> ---
> base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f
> change-id: 20230321-kexec_clang16-4510c23d129c
>
> Best regards,
> --
> Ricardo Ribalda Delgado <ribalda@chromium.org>
>

Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Ricardo Ribalda]

Hi Song

On Fri, 8 Sept 2023 at 01:08, Song Liu <song@kernel.org> wrote:
>
> Hi Ricardo and folks,
>
> On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@chromium.org> wrote:
> >
> > When upreving llvm I realised that kexec stopped working on my test
> > platform.
> >
> > The reason seems to be that due to PGO there are multiple .text sections
> > on the purgatory, and kexec does not supports that.
> >
> > Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
>
> We are seeing WARNINGs like the following while kexec'ing a PGO and
> LTO enabled kernel:
>
> WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
> kexec_load_purgatory+0x37f/0x390
>
> AFAICT, the warning was added by this set, and it was triggered when
> we have many .text sections
> in purgatory.ro. The kexec was actually successful. So I wonder
> whether we really need the
> WARNING here. If we disable LTO (PGO is still enabled), we don't see
> the WARNING any more.
>
> I also tested an older kernel (5.19 based), where we also see many
> .text sections with LTO. It
> kexec()'ed fine. (It doesn't have the WARN_ON() in
> kexec_purgatory_setup_sechdrs).

You have been "lucky" that the code has chosen the correct start
address, you need to modify the linker script of your kernel to
disable PGO.
You need to backport a patch like this:
https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJvAeQ@mail.gmail.com/T/#md68b7f832216b0c56bbec0c9b07332e180b9ba2b

(assuming x86)

Regards

>
> Please help us fix this properly (as I really don't know much about kexec).
>
> Thanks in advance,
> Song
>
>
> Here is readelf -S output on purgatory.ro.
>
> With LTO:
>
> readelf -W -S purgatory.ro
> There are 48 section headers, starting at offset 0x4a10:
>
> Section Headers:
>   [Nr] Name              Type            Address          Off    Size
>  ES Flg Lk Inf Al
>   [ 0]                   NULL            0000000000000000 000000
> 000000 00      0   0  0
>   [ 1] .text             PROGBITS        0000000000000000 000040
> 0000d0 00  AX  0   0 16
>   [ 2] .data             PROGBITS        0000000000000000 001000
> 001000 00  WA  0   0 4096
>   [ 3] .rela.text        RELA            0000000000000000 003788
> 000228 18   I 45   1  8
>   [ 4] .rodata           PROGBITS        0000000000000000 002000
> 0000e0 00   A  0   0 16
>   [ 5] .rela.rodata      RELA            0000000000000000 0039b0
> 000030 18   I 45   4  8
>   [ 6] .bss              NOBITS          0000000000000000 0020e0
> 001000 00  WA  0   0 4096
>   [ 7] .text.purgatory   PROGBITS        0000000000000000 0020e0
> 0000df 00  AX  0   0 16
>   [ 8] .rela.text.purgatory RELA            0000000000000000 0039e0
> 000060 18   I 45   7  8
>   [ 9] .text.warn        PROGBITS        0000000000000000 0021c0
> 000001 00  AX  0   0 16
>   [10] .kexec-purgatory  PROGBITS        0000000000000000 0021d0
> 000120 00  WA  0   0 16
>   [11] .comment          PROGBITS        0000000000000000 003a40
> 000046 01  MS  0   0  1
>   [12] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 003a86
> 000005 00   E  0   0  1
>   [13] .text.sha256_update PROGBITS        0000000000000000 0022f0
> 0008eb 00  AX  0   0 16
>   [14] .rela.text.sha256_update RELA            0000000000000000
> 003a90 000060 18   I 45  13  8
>   [15] .text.sha224_update PROGBITS        0000000000000000 002be0
> 00000c 00  AX  0   0 16
>   [16] .rela.text.sha224_update RELA            0000000000000000
> 003af0 000018 18   I 45  15  8
>   [17] .text.sha256_final PROGBITS        0000000000000000 002bf0
> 0000cd 00  AX  0   0 16
>   [18] .rela.text.sha256_final RELA            0000000000000000 003b08
> 000030 18   I 45  17  8
>   [19] .text.sha224_final PROGBITS        0000000000000000 002cc0
> 0000bd 00  AX  0   0 16
>   [20] .rela.text.sha224_final RELA            0000000000000000 003b38
> 000030 18   I 45  19  8
>   [21] .text.sha256      PROGBITS        0000000000000000 002d80
> 00011d 00  AX  0   0 16
>   [22] .rela.text.sha256 RELA            0000000000000000 003b68
> 000030 18   I 45  21  8
>   [23] .modinfo          PROGBITS        0000000000000000 002e9d
> 000039 00   A  0   0  1
>   [24] .rodata.SHA256_K  PROGBITS        0000000000000000 002ee0
> 000100 00   A  0   0 16
>   [25] .rodata.__sha256_final.padding PROGBITS        0000000000000000
> 002fe0 000040 00   A  0   0 16
>   [26] .text.memcmp      PROGBITS        0000000000000000 003020
> 00000b 00  AX  0   0 16
>   [27] .text.bcmp        PROGBITS        0000000000000000 003030
> 00000b 00  AX  0   0 16
>   [28] .text.strcmp      PROGBITS        0000000000000000 003040
> 000041 00  AX  0   0 16
>   [29] .text.strncmp     PROGBITS        0000000000000000 003090
> 00003a 00  AX  0   0 16
>   [30] .text.strnlen     PROGBITS        0000000000000000 0030d0
> 000039 00  AX  0   0 16
>   [31] .text.atou        PROGBITS        0000000000000000 003110
> 000035 00  AX  0   0 16
>   [32] .text.simple_strtoull PROGBITS        0000000000000000 003150
> 0000b6 00  AX  0   0 16
>   [33] .text.simple_strtol PROGBITS        0000000000000000 003210
> 0001b6 00  AX  0   0 16
>   [34] .text.strlen      PROGBITS        0000000000000000 0033d0
> 00001c 00  AX  0   0 16
>   [35] .text.strstr      PROGBITS        0000000000000000 0033f0
> 00005f 00  AX  0   0 16
>   [36] .text.strchr      PROGBITS        0000000000000000 003450
> 000022 00  AX  0   0 16
>   [37] .text.kstrtoull   PROGBITS        0000000000000000 003480
> 000142 00  AX  0   0 16
>   [38] .text.boot_kstrtoul PROGBITS        0000000000000000 0035d0
> 00000c 00  AX  0   0 16
>   [39] .rela.text.boot_kstrtoul RELA            0000000000000000
> 003b98 000018 18   I 45  38  8
>   [40] .text.memset      PROGBITS        0000000000000000 0035e0
> 00001f 00  AX  0   0 16
>   [41] .text.memmove     PROGBITS        0000000000000000 003600
> 0000a6 00  AX  0   0 16
>   [42] .text.memcpy      PROGBITS        0000000000000000 0036b0
> 0000a6 00  AX  0   0 16
>   [43] .rodata.str1.1    PROGBITS        0000000000000000 003756
> 000032 01 AMS  0   0  1
>   [44] .note.GNU-stack   PROGBITS        0000000000000000 003bb0
> 000000 00      0   0  1
>   [45] .symtab           SYMTAB          0000000000000000 003bb0
> 000948 18     47  68  8
>   [46] .shstrtab         STRTAB          0000000000000000 0044f8
> 0002cd 00      0   0  1
>   [47] .strtab           STRTAB          0000000000000000 0047c5
> 000248 00      0   0  1
> Key to Flags:
>   W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
>   L (link order), O (extra OS processing required), G (group), T (TLS),
>   C (compressed), x (unknown), o (OS specific), E (exclude),
>   l (large), p (processor specific)
>
>
> Without LTO:
>
> readelf -W -S purgatory.ro
> There are 16 section headers, starting at offset 0x4130:
>
> Section Headers:
>   [Nr] Name              Type            Address          Off    Size
>  ES Flg Lk Inf Al
>   [ 0]                   NULL            0000000000000000 000000
> 000000 00      0   0  0
>   [ 1] .text             PROGBITS        0000000000000000 000040
> 00131b 00  AX  0   0 16
>   [ 2] .rela.text        RELA            0000000000000000 003290
> 000480 18   I 13   1  8
>   [ 3] .kexec-purgatory  PROGBITS        0000000000000000 001360
> 000120 00  WA  0   0 16
>   [ 4] .comment          PROGBITS        0000000000000000 003710
> 000046 01  MS  0   0  1
>   [ 5] .llvm_addrsig     LOOS+0xfff4c03  0000000000000000 003756
> 000005 00   E  0   0  1
>   [ 6] .data             PROGBITS        0000000000000000 002000
> 001000 00  WA  0   0 4096
>   [ 7] .rodata           PROGBITS        0000000000000000 003000
> 000220 00   A  0   0 16
>   [ 8] .rela.rodata      RELA            0000000000000000 003760
> 000030 18   I 13   7  8
>   [ 9] .bss              NOBITS          0000000000000000 003220
> 001000 00  WA  0   0 4096
>   [10] .modinfo          PROGBITS        0000000000000000 003220
> 000039 00   A  0   0  1
>   [11] .rodata.str1.1    PROGBITS        0000000000000000 003259
> 000032 01 AMS  0   0  1
>   [12] .note.GNU-stack   PROGBITS        0000000000000000 003790
> 000000 00      0   0  1
>   [13] .symtab           SYMTAB          0000000000000000 003790
> 0006d8 18     15  43  8
>   [14] .shstrtab         STRTAB          0000000000000000 003e68
> 00009c 00      0   0  1
>   [15] .strtab           STRTAB          0000000000000000 003f04
> 00022b 00      0   0  1
> Key to Flags:
>   W (write), A (alloc), X (execute), M (merge), S (strings), I (info),
>   L (link order), O (extra OS processing required), G (group), T (TLS),
>   C (compressed), x (unknown), o (OS specific), E (exclude),
>   l (large), p (processor specific)
>
> [...]
>
> > Ricardo Ribalda (4):
> >       kexec: Support purgatories with .text.hot sections
> >       x86/purgatory: Remove PGO flags
> >       powerpc/purgatory: Remove PGO flags
> >       riscv/purgatory: Remove PGO flags
> >
> >  arch/powerpc/purgatory/Makefile |  5 +++++
> >  arch/riscv/purgatory/Makefile   |  5 +++++
> >  arch/x86/purgatory/Makefile     |  5 +++++
> >  kernel/kexec_file.c             | 14 +++++++++++++-
> >  4 files changed, 28 insertions(+), 1 deletion(-)
> > ---
> > base-commit: 58390c8ce1bddb6c623f62e7ed36383e7fa5c02f
> > change-id: 20230321-kexec_clang16-4510c23d129c
> >
> > Best regards,
> > --
> > Ricardo Ribalda Delgado <ribalda@chromium.org>
> >



-- 
Ricardo Ribalda

Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Song Liu]

Hi Ricardo,

Thanks for your kind reply.

On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda <ribalda@chromium.org> wrote:
>
> Hi Song
>
> On Fri, 8 Sept 2023 at 01:08, Song Liu <song@kernel.org> wrote:
> >
> > Hi Ricardo and folks,
> >
> > On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@chromium.org> wrote:
> > >
> > > When upreving llvm I realised that kexec stopped working on my test
> > > platform.
> > >
> > > The reason seems to be that due to PGO there are multiple .text sections
> > > on the purgatory, and kexec does not supports that.
> > >
> > > Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
> >
> > We are seeing WARNINGs like the following while kexec'ing a PGO and
> > LTO enabled kernel:
> >
> > WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
> > kexec_load_purgatory+0x37f/0x390
> >
> > AFAICT, the warning was added by this set, and it was triggered when
> > we have many .text sections
> > in purgatory.ro. The kexec was actually successful. So I wonder
> > whether we really need the
> > WARNING here. If we disable LTO (PGO is still enabled), we don't see
> > the WARNING any more.
> >
> > I also tested an older kernel (5.19 based), where we also see many
> > .text sections with LTO. It
> > kexec()'ed fine. (It doesn't have the WARN_ON() in
> > kexec_purgatory_setup_sechdrs).
>
> You have been "lucky" that the code has chosen the correct start
> address, you need to modify the linker script of your kernel to
> disable PGO.
> You need to backport a patch like this:
> https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJvAeQ@mail.gmail.com/T/#md68b7f832216b0c56bbec0c9b07332e180b9ba2b

We already have this commit in our branch. AFAICT, the issue was
triggered by LTO. So something like the following seems fixes it
(I haven't finished the end-to-end test yet). Does this change make
sense to you?

Thanks again,
Song

diff --git i/arch/x86/purgatory/Makefile w/arch/x86/purgatory/Makefile
index 8f71aaa04cc2..dc306fa7197d 100644
--- i/arch/x86/purgatory/Makefile
+++ w/arch/x86/purgatory/Makefile
@@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
 # optimization flags.
 KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
-fprofile-use=%,$(KBUILD_CFLAGS))

+# When LTO is enabled, llvm emits many text sections, which is not supported
+# by kexec. Remove -flto=* flags.
+KBUILD_CFLAGS := $(filter-out -flto=%,$(KBUILD_CFLAGS))
+
 # When linking purgatory.ro with -r unresolved symbols are not checked,
 # also link a purgatory.chk binary without -r to check for unresolved symbols.
 PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib

Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Ricardo Ribalda]

Hi Song

On Fri, 8 Sept 2023 at 23:48, Song Liu <song@kernel.org> wrote:
>
> Hi Ricardo,
>
> Thanks for your kind reply.
>
> On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda <ribalda@chromium.org> wrote:
> >
> > Hi Song
> >
> > On Fri, 8 Sept 2023 at 01:08, Song Liu <song@kernel.org> wrote:
> > >
> > > Hi Ricardo and folks,
> > >
> > > On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@chromium.org> wrote:
> > > >
> > > > When upreving llvm I realised that kexec stopped working on my test
> > > > platform.
> > > >
> > > > The reason seems to be that due to PGO there are multiple .text sections
> > > > on the purgatory, and kexec does not supports that.
> > > >
> > > > Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
> > >
> > > We are seeing WARNINGs like the following while kexec'ing a PGO and
> > > LTO enabled kernel:
> > >
> > > WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
> > > kexec_load_purgatory+0x37f/0x390
> > >
> > > AFAICT, the warning was added by this set, and it was triggered when
> > > we have many .text sections
> > > in purgatory.ro. The kexec was actually successful. So I wonder
> > > whether we really need the
> > > WARNING here. If we disable LTO (PGO is still enabled), we don't see
> > > the WARNING any more.
> > >
> > > I also tested an older kernel (5.19 based), where we also see many
> > > .text sections with LTO. It
> > > kexec()'ed fine. (It doesn't have the WARN_ON() in
> > > kexec_purgatory_setup_sechdrs).
> >
> > You have been "lucky" that the code has chosen the correct start
> > address, you need to modify the linker script of your kernel to
> > disable PGO.
> > You need to backport a patch like this:
> > https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJvAeQ@mail.gmail.com/T/#md68b7f832216b0c56bbec0c9b07332e180b9ba2b
>
> We already have this commit in our branch. AFAICT, the issue was
> triggered by LTO. So something like the following seems fixes it
> (I haven't finished the end-to-end test yet). Does this change make
> sense to you?

if the end-to-end works, please send it as a patch to the mailing list.

Thanks! :)

>
> Thanks again,
> Song
>
> diff --git i/arch/x86/purgatory/Makefile w/arch/x86/purgatory/Makefile
> index 8f71aaa04cc2..dc306fa7197d 100644
> --- i/arch/x86/purgatory/Makefile
> +++ w/arch/x86/purgatory/Makefile
> @@ -19,6 +19,10 @@ CFLAGS_sha256.o := -D__DISABLE_EXPORTS
>  # optimization flags.
>  KBUILD_CFLAGS := $(filter-out -fprofile-sample-use=%
> -fprofile-use=%,$(KBUILD_CFLAGS))
>
> +# When LTO is enabled, llvm emits many text sections, which is not supported
> +# by kexec. Remove -flto=* flags.
> +KBUILD_CFLAGS := $(filter-out -flto=%,$(KBUILD_CFLAGS))
> +
>  # When linking purgatory.ro with -r unresolved symbols are not checked,
>  # also link a purgatory.chk binary without -r to check for unresolved symbols.
>  PURGATORY_LDFLAGS := -e purgatory_start -z nodefaultlib



-- 
Ricardo Ribalda

Re: [PATCH v7 0/4] kexec: Fix kexec_file_load for llvm16 with PGO

[Song Liu]

On Fri, Sep 8, 2023 at 2:52 PM Ricardo Ribalda <ribalda@chromium.org> wrote:
>
> Hi Song
>
> On Fri, 8 Sept 2023 at 23:48, Song Liu <song@kernel.org> wrote:
> >
> > Hi Ricardo,
> >
> > Thanks for your kind reply.
> >
> > On Fri, Sep 8, 2023 at 2:18 PM Ricardo Ribalda <ribalda@chromium.org> wrote:
> > >
> > > Hi Song
> > >
> > > On Fri, 8 Sept 2023 at 01:08, Song Liu <song@kernel.org> wrote:
> > > >
> > > > Hi Ricardo and folks,
> > > >
> > > > On Fri, May 19, 2023 at 7:48 AM Ricardo Ribalda <ribalda@chromium.org> wrote:
> > > > >
> > > > > When upreving llvm I realised that kexec stopped working on my test
> > > > > platform.
> > > > >
> > > > > The reason seems to be that due to PGO there are multiple .text sections
> > > > > on the purgatory, and kexec does not supports that.
> > > > >
> > > > > Signed-off-by: Ricardo Ribalda <ribalda@chromium.org>
> > > >
> > > > We are seeing WARNINGs like the following while kexec'ing a PGO and
> > > > LTO enabled kernel:
> > > >
> > > > WARNING: CPU: 26 PID: 110894 at kernel/kexec_file.c:919
> > > > kexec_load_purgatory+0x37f/0x390
> > > >
> > > > AFAICT, the warning was added by this set, and it was triggered when
> > > > we have many .text sections
> > > > in purgatory.ro. The kexec was actually successful. So I wonder
> > > > whether we really need the
> > > > WARNING here. If we disable LTO (PGO is still enabled), we don't see
> > > > the WARNING any more.
> > > >
> > > > I also tested an older kernel (5.19 based), where we also see many
> > > > .text sections with LTO. It
> > > > kexec()'ed fine. (It doesn't have the WARN_ON() in
> > > > kexec_purgatory_setup_sechdrs).
> > >
> > > You have been "lucky" that the code has chosen the correct start
> > > address, you need to modify the linker script of your kernel to
> > > disable PGO.
> > > You need to backport a patch like this:
> > > https://lore.kernel.org/lkml/CAPhsuW5_qAvV0N3o+hOiAnb1=buJ1pLzqYW9D+Bwft6hxJvAeQ@mail.gmail.com/T/#md68b7f832216b0c56bbec0c9b07332e180b9ba2b
> >
> > We already have this commit in our branch. AFAICT, the issue was
> > triggered by LTO. So something like the following seems fixes it
> > (I haven't finished the end-to-end test yet). Does this change make
> > sense to you?
>
> if the end-to-end works, please send it as a patch to the mailing list.
>
> Thanks! :)

OK, it works (AFAICT). Sending the patch.

Thanks,
Song