All of lore.kernel.org
 help / color / mirror / Atom feed
* Kernel Crash Dump (kdump) broken with 6.5
@ 2023-07-17 14:45 Sachin Sant
  2023-07-17 16:38 ` Mahesh J Salgaonkar
  2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  0 siblings, 2 replies; 12+ messages in thread
From: Sachin Sant @ 2023-07-17 14:45 UTC (permalink / raw)
  To: linuxppc-dev, Nicholas Piggin
  Cc: Sourabh Jain, Hari Bathini, Mahesh J Salgaonkar

Kdump seems to be broken with 6.5 for ppc64le.

[ 14.200412] systemd[1]: Starting dracut pre-pivot and cleanup hook...
[[0;32m OK [0m] Started dracut pre-pivot and cleanup hook.
Starting Kdump Vmcore Save Service...
[ 14.231669] systemd[1]: Started dracut pre-pivot and cleanup hook.
[ 14.231801] systemd[1]: Starting Kdump Vmcore Save Service...
[ 14.341035] kdump.sh[297]: kdump: saving to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
[ 14.350053] EXT4-fs (sda2): re-mounted e971a335-1ef8-4295-ab4e-3940f28e53fc r/w. Quota mode: none.
[ 14.345979] kdump.sh[297]: kdump: saving vmcore-dmesg.txt to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
[ 14.348742] kdump.sh[331]: Cannot open /proc/vmcore: No such file or directory
[ 14.348845] kdump.sh[297]: kdump: saving vmcore-dmesg.txt failed
[ 14.349014] kdump.sh[297]: kdump: saving vmcore
[ 14.443422] kdump.sh[332]: open_dump_memory: Can't open the dump memory(/proc/vmcore). No such file or directory
[ 14.456413] kdump.sh[332]: makedumpfile Failed.
[ 14.456662] kdump.sh[297]: kdump: saving vmcore failed, _exitcode:1
[ 14.456822] kdump.sh[297]: kdump: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
[ 14.487002] kdump.sh[297]: kdump: saving vmcore failed
[[0;1;31mFAILED[0m] Failed to start Kdump Vmcore Save Service.

6.4 was good. Git bisect points to following patch

commit 606787fed7268feb256957872586370b56af697a
    powerpc/64s: Remove support for ELFv1 little endian userspace

Reverting this patch allows a successful capture of vmcore.

Does this change require any corresponding change to kdump
and/or kexec tools?

- Sachin


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-17 14:45 Kernel Crash Dump (kdump) broken with 6.5 Sachin Sant
@ 2023-07-17 16:38 ` Mahesh J Salgaonkar
  2023-07-18 13:19   ` Michael Ellerman
  2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  1 sibling, 1 reply; 12+ messages in thread
From: Mahesh J Salgaonkar @ 2023-07-17 16:38 UTC (permalink / raw)
  To: Sachin Sant; +Cc: Hari Bathini, linuxppc-dev, Sourabh Jain, Nicholas Piggin

On 2023-07-17 20:15:53 Mon, Sachin Sant wrote:
> Kdump seems to be broken with 6.5 for ppc64le.
> 
> [ 14.200412] systemd[1]: Starting dracut pre-pivot and cleanup hook...
> [[0;32m OK [0m] Started dracut pre-pivot and cleanup hook.
> Starting Kdump Vmcore Save Service...
> [ 14.231669] systemd[1]: Started dracut pre-pivot and cleanup hook.
> [ 14.231801] systemd[1]: Starting Kdump Vmcore Save Service...
> [ 14.341035] kdump.sh[297]: kdump: saving to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> [ 14.350053] EXT4-fs (sda2): re-mounted e971a335-1ef8-4295-ab4e-3940f28e53fc r/w. Quota mode: none.
> [ 14.345979] kdump.sh[297]: kdump: saving vmcore-dmesg.txt to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> [ 14.348742] kdump.sh[331]: Cannot open /proc/vmcore: No such file or directory
> [ 14.348845] kdump.sh[297]: kdump: saving vmcore-dmesg.txt failed
> [ 14.349014] kdump.sh[297]: kdump: saving vmcore
> [ 14.443422] kdump.sh[332]: open_dump_memory: Can't open the dump memory(/proc/vmcore). No such file or directory
> [ 14.456413] kdump.sh[332]: makedumpfile Failed.
> [ 14.456662] kdump.sh[297]: kdump: saving vmcore failed, _exitcode:1
> [ 14.456822] kdump.sh[297]: kdump: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> [ 14.487002] kdump.sh[297]: kdump: saving vmcore failed
> [[0;1;31mFAILED[0m] Failed to start Kdump Vmcore Save Service.

Thanks Sachin for catching this.

> 
> 6.4 was good. Git bisect points to following patch
> 
> commit 606787fed7268feb256957872586370b56af697a
>     powerpc/64s: Remove support for ELFv1 little endian userspace
> 
> Reverting this patch allows a successful capture of vmcore.
> 
> Does this change require any corresponding change to kdump
> and/or kexec tools?

Need to investigate that. It looks like vmcore_elf64_check_arch(&ehdr)
check from fs/proc/vmcore.c is failing after above commit.

static int __init parse_crash_elf64_headers(void)
{
[...]

        /* Do some basic Verification. */
        if (memcmp(ehdr.e_ident, ELFMAG, SELFMAG) != 0 ||
                (ehdr.e_type != ET_CORE) ||
                !vmcore_elf64_check_arch(&ehdr) ||
[...]

It looks like ehdr->e_flags are not set properly while generating vmcore
ELF header. I see that in kexec_file_load, ehdr->e_flags left set to 0
irrespective of IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) is true or false.

-Mahesh

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-17 16:38 ` Mahesh J Salgaonkar
@ 2023-07-18 13:19   ` Michael Ellerman
  2023-07-19 18:03     ` Mahesh J Salgaonkar
  0 siblings, 1 reply; 12+ messages in thread
From: Michael Ellerman @ 2023-07-18 13:19 UTC (permalink / raw)
  To: mahesh, Sachin Sant
  Cc: linuxppc-dev, Sourabh Jain, Hari Bathini, Nicholas Piggin

Mahesh J Salgaonkar <mahesh@linux.ibm.com> writes:
> On 2023-07-17 20:15:53 Mon, Sachin Sant wrote:
>> Kdump seems to be broken with 6.5 for ppc64le.
>> 
>> [ 14.200412] systemd[1]: Starting dracut pre-pivot and cleanup hook...
>> [[0;32m OK [0m] Started dracut pre-pivot and cleanup hook.
>> Starting Kdump Vmcore Save Service...
>> [ 14.231669] systemd[1]: Started dracut pre-pivot and cleanup hook.
>> [ 14.231801] systemd[1]: Starting Kdump Vmcore Save Service...
>> [ 14.341035] kdump.sh[297]: kdump: saving to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>> [ 14.350053] EXT4-fs (sda2): re-mounted e971a335-1ef8-4295-ab4e-3940f28e53fc r/w. Quota mode: none.
>> [ 14.345979] kdump.sh[297]: kdump: saving vmcore-dmesg.txt to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>> [ 14.348742] kdump.sh[331]: Cannot open /proc/vmcore: No such file or directory
>> [ 14.348845] kdump.sh[297]: kdump: saving vmcore-dmesg.txt failed
>> [ 14.349014] kdump.sh[297]: kdump: saving vmcore
>> [ 14.443422] kdump.sh[332]: open_dump_memory: Can't open the dump memory(/proc/vmcore). No such file or directory
>> [ 14.456413] kdump.sh[332]: makedumpfile Failed.
>> [ 14.456662] kdump.sh[297]: kdump: saving vmcore failed, _exitcode:1
>> [ 14.456822] kdump.sh[297]: kdump: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>> [ 14.487002] kdump.sh[297]: kdump: saving vmcore failed
>> [[0;1;31mFAILED[0m] Failed to start Kdump Vmcore Save Service.
>
> Thanks Sachin for catching this.
>
>> 
>> 6.4 was good. Git bisect points to following patch
>> 
>> commit 606787fed7268feb256957872586370b56af697a
>>     powerpc/64s: Remove support for ELFv1 little endian userspace
>> 
>> Reverting this patch allows a successful capture of vmcore.
>> 
>> Does this change require any corresponding change to kdump
>> and/or kexec tools?
>
> Need to investigate that. It looks like vmcore_elf64_check_arch(&ehdr)
> check from fs/proc/vmcore.c is failing after above commit.
>
> static int __init parse_crash_elf64_headers(void)
> {
> [...]
>
>         /* Do some basic Verification. */
>         if (memcmp(ehdr.e_ident, ELFMAG, SELFMAG) != 0 ||
>                 (ehdr.e_type != ET_CORE) ||
>                 !vmcore_elf64_check_arch(&ehdr) ||
> [...]

Where vmcore_elf64_check_arch() calls elf_check_arch(), which was
modified by the commit, so that makes sense.

> It looks like ehdr->e_flags are not set properly while generating vmcore
> ELF header. I see that in kexec_file_load, ehdr->e_flags left set to 0
> irrespective of IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) is true or false.

Does initialising it in crash_prepare_elf64_headers() fix the issue?

cheers

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-17 14:45 Kernel Crash Dump (kdump) broken with 6.5 Sachin Sant
@ 2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  1 sibling, 0 replies; 12+ messages in thread
From: Linux regression tracking #adding (Thorsten Leemhuis) @ 2023-07-19 16:19 UTC (permalink / raw)
  To: Sachin Sant, linuxppc-dev, Nicholas Piggin
  Cc: Sourabh Jain, Hari Bathini, Mahesh J Salgaonkar,
	Linux kernel regressions list

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 17.07.23 16:45, Sachin Sant wrote:
> Kdump seems to be broken with 6.5 for ppc64le.
> [...]
> 
> 6.4 was good. Git bisect points to following patch
> 
> commit 606787fed7268feb256957872586370b56af697a
>     powerpc/64s: Remove support for ELFv1 little endian userspace
> 
> Reverting this patch allows a successful capture of vmcore.

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 606787fed7268feb256957872586370b56af69
#regzbot title powerpc/64s: Crash Dump (kdump) broken with 6.5
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
@ 2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  0 siblings, 0 replies; 12+ messages in thread
From: Linux regression tracking #adding (Thorsten Leemhuis) @ 2023-07-19 16:19 UTC (permalink / raw)
  To: Sachin Sant, linuxppc-dev, Nicholas Piggin
  Cc: Linux kernel regressions list, Hari Bathini, Sourabh Jain,
	Mahesh J Salgaonkar

[CCing the regression list, as it should be in the loop for regressions:
https://docs.kernel.org/admin-guide/reporting-regressions.html]

[TLDR: I'm adding this report to the list of tracked Linux kernel
regressions; the text you find below is based on a few templates
paragraphs you might have encountered already in similar form.
See link in footer if these mails annoy you.]

On 17.07.23 16:45, Sachin Sant wrote:
> Kdump seems to be broken with 6.5 for ppc64le.
> [...]
> 
> 6.4 was good. Git bisect points to following patch
> 
> commit 606787fed7268feb256957872586370b56af697a
>     powerpc/64s: Remove support for ELFv1 little endian userspace
> 
> Reverting this patch allows a successful capture of vmcore.

Thanks for the report. To be sure the issue doesn't fall through the
cracks unnoticed, I'm adding it to regzbot, the Linux kernel regression
tracking bot:

#regzbot ^introduced 606787fed7268feb256957872586370b56af69
#regzbot title powerpc/64s: Crash Dump (kdump) broken with 6.5
#regzbot ignore-activity

This isn't a regression? This issue or a fix for it are already
discussed somewhere else? It was fixed already? You want to clarify when
the regression started to happen? Or point out I got the title or
something else totally wrong? Then just reply and tell me -- ideally
while also telling regzbot about it, as explained by the page listed in
the footer of this mail.

Developers: When fixing the issue, remember to add 'Link:' tags pointing
to the report (the parent of this mail). See page linked in footer for
details.

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-18 13:19   ` Michael Ellerman
@ 2023-07-19 18:03     ` Mahesh J Salgaonkar
  2023-07-20  8:39       ` Sachin Sant
                         ` (2 more replies)
  0 siblings, 3 replies; 12+ messages in thread
From: Mahesh J Salgaonkar @ 2023-07-19 18:03 UTC (permalink / raw)
  To: Michael Ellerman
  Cc: Hari Bathini, linuxppc-dev, Sourabh Jain, Sachin Sant, Nicholas Piggin

On 2023-07-18 23:19:23 Tue, Michael Ellerman wrote:
> Mahesh J Salgaonkar <mahesh@linux.ibm.com> writes:
> > On 2023-07-17 20:15:53 Mon, Sachin Sant wrote:
> >> Kdump seems to be broken with 6.5 for ppc64le.
> >> 
> >> [ 14.200412] systemd[1]: Starting dracut pre-pivot and cleanup hook...
> >> [[0;32m OK [0m] Started dracut pre-pivot and cleanup hook.
> >> Starting Kdump Vmcore Save Service...
> >> [ 14.231669] systemd[1]: Started dracut pre-pivot and cleanup hook.
> >> [ 14.231801] systemd[1]: Starting Kdump Vmcore Save Service...
> >> [ 14.341035] kdump.sh[297]: kdump: saving to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> >> [ 14.350053] EXT4-fs (sda2): re-mounted e971a335-1ef8-4295-ab4e-3940f28e53fc r/w. Quota mode: none.
> >> [ 14.345979] kdump.sh[297]: kdump: saving vmcore-dmesg.txt to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> >> [ 14.348742] kdump.sh[331]: Cannot open /proc/vmcore: No such file or directory
> >> [ 14.348845] kdump.sh[297]: kdump: saving vmcore-dmesg.txt failed
> >> [ 14.349014] kdump.sh[297]: kdump: saving vmcore
> >> [ 14.443422] kdump.sh[332]: open_dump_memory: Can't open the dump memory(/proc/vmcore). No such file or directory
> >> [ 14.456413] kdump.sh[332]: makedumpfile Failed.
> >> [ 14.456662] kdump.sh[297]: kdump: saving vmcore failed, _exitcode:1
> >> [ 14.456822] kdump.sh[297]: kdump: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
> >> [ 14.487002] kdump.sh[297]: kdump: saving vmcore failed
> >> [[0;1;31mFAILED[0m] Failed to start Kdump Vmcore Save Service.
> >
> > Thanks Sachin for catching this.
> >
> >> 
> >> 6.4 was good. Git bisect points to following patch
> >> 
> >> commit 606787fed7268feb256957872586370b56af697a
> >>     powerpc/64s: Remove support for ELFv1 little endian userspace
> >> 
> >> Reverting this patch allows a successful capture of vmcore.
> >> 
> >> Does this change require any corresponding change to kdump
> >> and/or kexec tools?
> >
> > Need to investigate that. It looks like vmcore_elf64_check_arch(&ehdr)
> > check from fs/proc/vmcore.c is failing after above commit.
> >
> > static int __init parse_crash_elf64_headers(void)
> > {
> > [...]
> >
> >         /* Do some basic Verification. */
> >         if (memcmp(ehdr.e_ident, ELFMAG, SELFMAG) != 0 ||
> >                 (ehdr.e_type != ET_CORE) ||
> >                 !vmcore_elf64_check_arch(&ehdr) ||
> > [...]
> 
> Where vmcore_elf64_check_arch() calls elf_check_arch(), which was
> modified by the commit, so that makes sense.
> 
> > It looks like ehdr->e_flags are not set properly while generating vmcore
> > ELF header. I see that in kexec_file_load, ehdr->e_flags left set to 0
> > irrespective of IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) is true or false.
> 
> Does initialising it in crash_prepare_elf64_headers() fix the issue?

Yes, the bellow change fixes the issue. Can't use
IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)i check in common code. I see that
fs/proc/kcore.c uses ELF_CORE_EFLAGS to set e_flags. Will send out
formal patch.

From 2d12fe7dff5dfe9035a75b1fe8d7da7da3000b90 Mon Sep 17 00:00:00 2001
From: Mahesh Salgaonkar <mahesh@linux.ibm.com>
Date: Wed, 19 Jul 2023 20:36:37 +0530
Subject: [PATCH] kdump fix

Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
---
 kernel/kexec_file.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/kernel/kexec_file.c b/kernel/kexec_file.c
index 881ba0d1714cc..be51560509128 100644
--- a/kernel/kexec_file.c
+++ b/kernel/kexec_file.c
@@ -1279,6 +1279,7 @@ int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
 	ehdr->e_phoff = sizeof(Elf64_Ehdr);
 	ehdr->e_ehsize = sizeof(Elf64_Ehdr);
 	ehdr->e_phentsize = sizeof(Elf64_Phdr);
+	ehdr->e_flags = ELF_CORE_EFLAGS;
 
 	/* Prepare one phdr of type PT_NOTE for each present CPU */
 	for_each_present_cpu(cpu) {
-- 
2.41.0

-- 
Mahesh J Salgaonkar

^ permalink raw reply related	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-19 18:03     ` Mahesh J Salgaonkar
@ 2023-07-20  8:39       ` Sachin Sant
  2023-07-20 19:08         ` kernel test robot
  2023-07-21  3:50         ` kernel test robot
  2 siblings, 0 replies; 12+ messages in thread
From: Sachin Sant @ 2023-07-20  8:39 UTC (permalink / raw)
  To: Mahesh J Salgaonkar, ajd
  Cc: linuxppc-dev, Hari Bathini, Sourabh Jain, Nicholas Piggin



> On 19-Jul-2023, at 11:33 PM, Mahesh J Salgaonkar <mahesh@linux.ibm.com> wrote:
> 
> On 2023-07-18 23:19:23 Tue, Michael Ellerman wrote:
>> Mahesh J Salgaonkar <mahesh@linux.ibm.com> writes:
>>> On 2023-07-17 20:15:53 Mon, Sachin Sant wrote:
>>>> Kdump seems to be broken with 6.5 for ppc64le.
>>>> 
>>>> [ 14.200412] systemd[1]: Starting dracut pre-pivot and cleanup hook...
>>>> [[0;32m OK [0m] Started dracut pre-pivot and cleanup hook.
>>>> Starting Kdump Vmcore Save Service...
>>>> [ 14.231669] systemd[1]: Started dracut pre-pivot and cleanup hook.
>>>> [ 14.231801] systemd[1]: Starting Kdump Vmcore Save Service...
>>>> [ 14.341035] kdump.sh[297]: kdump: saving to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>>>> [ 14.350053] EXT4-fs (sda2): re-mounted e971a335-1ef8-4295-ab4e-3940f28e53fc r/w. Quota mode: none.
>>>> [ 14.345979] kdump.sh[297]: kdump: saving vmcore-dmesg.txt to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>>>> [ 14.348742] kdump.sh[331]: Cannot open /proc/vmcore: No such file or directory
>>>> [ 14.348845] kdump.sh[297]: kdump: saving vmcore-dmesg.txt failed
>>>> [ 14.349014] kdump.sh[297]: kdump: saving vmcore
>>>> [ 14.443422] kdump.sh[332]: open_dump_memory: Can't open the dump memory(/proc/vmcore). No such file or directory
>>>> [ 14.456413] kdump.sh[332]: makedumpfile Failed.
>>>> [ 14.456662] kdump.sh[297]: kdump: saving vmcore failed, _exitcode:1
>>>> [ 14.456822] kdump.sh[297]: kdump: saving the /run/initramfs/kexec-dmesg.log to /sysroot//var/crash//127.0.0.1-2023-07-14-13:32:34/
>>>> [ 14.487002] kdump.sh[297]: kdump: saving vmcore failed
>>>> [[0;1;31mFAILED[0m] Failed to start Kdump Vmcore Save Service.
>>> 
>>> Thanks Sachin for catching this.
>>> 
>>>> 
>>>> 6.4 was good. Git bisect points to following patch
>>>> 
>>>> commit 606787fed7268feb256957872586370b56af697a
>>>>    powerpc/64s: Remove support for ELFv1 little endian userspace
>>>> 
>>>> Reverting this patch allows a successful capture of vmcore.
>>>> 
>>>> Does this change require any corresponding change to kdump
>>>> and/or kexec tools?
>>> 
>>> Need to investigate that. It looks like vmcore_elf64_check_arch(&ehdr)
>>> check from fs/proc/vmcore.c is failing after above commit.
>>> 
>>> static int __init parse_crash_elf64_headers(void)
>>> {
>>> [...]
>>> 
>>>        /* Do some basic Verification. */
>>>        if (memcmp(ehdr.e_ident, ELFMAG, SELFMAG) != 0 ||
>>>                (ehdr.e_type != ET_CORE) ||
>>>                !vmcore_elf64_check_arch(&ehdr) ||
>>> [...]
>> 
>> Where vmcore_elf64_check_arch() calls elf_check_arch(), which was
>> modified by the commit, so that makes sense.
>> 
>>> It looks like ehdr->e_flags are not set properly while generating vmcore
>>> ELF header. I see that in kexec_file_load, ehdr->e_flags left set to 0
>>> irrespective of IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2) is true or false.
>> 
>> Does initialising it in crash_prepare_elf64_headers() fix the issue?
> 
> Yes, the bellow change fixes the issue. Can't use
> IS_ENABLED(CONFIG_PPC64_ELF_ABI_V2)i check in common code. I see that
> fs/proc/kcore.c uses ELF_CORE_EFLAGS to set e_flags. Will send out
> formal patch.
> 
> From 2d12fe7dff5dfe9035a75b1fe8d7da7da3000b90 Mon Sep 17 00:00:00 2001
> From: Mahesh Salgaonkar <mahesh@linux.ibm.com>
> Date: Wed, 19 Jul 2023 20:36:37 +0530
> Subject: [PATCH] kdump fix
> 
> Signed-off-by: Mahesh Salgaonkar <mahesh@linux.ibm.com>
> ---

Thanks Mahesh for the fix. With this patch applied I am able to capture kdump.

Tested-by: Sachin Sant <sachinp@linux.ibm.com>

I also see that Andrew has posted a patch [0] to revert commit 606787fed726

If the said commit is reverted, do we still need this change?

- Sachin

[0] https://lists.ozlabs.org/pipermail/linuxppc-dev/2023-July/260928.html


^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] kdump fix
  2023-07-19 18:03     ` Mahesh J Salgaonkar
@ 2023-07-20 19:08         ` kernel test robot
  2023-07-20 19:08         ` kernel test robot
  2023-07-21  3:50         ` kernel test robot
  2 siblings, 0 replies; 12+ messages in thread
From: kernel test robot @ 2023-07-20 19:08 UTC (permalink / raw)
  To: Mahesh J Salgaonkar, Michael Ellerman
  Cc: llvm, oe-kbuild-all, Hari Bathini, linuxppc-dev, Sourabh Jain,
	Sachin Sant, Nicholas Piggin

Hi Mahesh,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Mahesh-J-Salgaonkar/kdump-fix/20230720-021115
base:   linus/master
patch link:    https://lore.kernel.org/r/2ogfzwjumrd44kxv7njfpot6fhtkzpqu77qv3bspfixdmsxcwc%40umt35y2hmslm
patch subject: [PATCH] kdump fix
config: x86_64-randconfig-r021-20230720 (https://download.01.org/0day-ci/archive/20230721/202307210220.4FoNS76V-lkp@intel.com/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307210220.4FoNS76V-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307210220.4FoNS76V-lkp@intel.com/

All errors (new ones prefixed by >>):

>> kernel/kexec_file.c:1282:18: error: use of undeclared identifier 'ELF_CORE_EFLAGS'
           ehdr->e_flags = ELF_CORE_EFLAGS;
                           ^
   1 error generated.


vim +/ELF_CORE_EFLAGS +1282 kernel/kexec_file.c

  1236	
  1237	int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
  1238				  void **addr, unsigned long *sz)
  1239	{
  1240		Elf64_Ehdr *ehdr;
  1241		Elf64_Phdr *phdr;
  1242		unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
  1243		unsigned char *buf;
  1244		unsigned int cpu, i;
  1245		unsigned long long notes_addr;
  1246		unsigned long mstart, mend;
  1247	
  1248		/* extra phdr for vmcoreinfo ELF note */
  1249		nr_phdr = nr_cpus + 1;
  1250		nr_phdr += mem->nr_ranges;
  1251	
  1252		/*
  1253		 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
  1254		 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
  1255		 * I think this is required by tools like gdb. So same physical
  1256		 * memory will be mapped in two ELF headers. One will contain kernel
  1257		 * text virtual addresses and other will have __va(physical) addresses.
  1258		 */
  1259	
  1260		nr_phdr++;
  1261		elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
  1262		elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
  1263	
  1264		buf = vzalloc(elf_sz);
  1265		if (!buf)
  1266			return -ENOMEM;
  1267	
  1268		ehdr = (Elf64_Ehdr *)buf;
  1269		phdr = (Elf64_Phdr *)(ehdr + 1);
  1270		memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
  1271		ehdr->e_ident[EI_CLASS] = ELFCLASS64;
  1272		ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
  1273		ehdr->e_ident[EI_VERSION] = EV_CURRENT;
  1274		ehdr->e_ident[EI_OSABI] = ELF_OSABI;
  1275		memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
  1276		ehdr->e_type = ET_CORE;
  1277		ehdr->e_machine = ELF_ARCH;
  1278		ehdr->e_version = EV_CURRENT;
  1279		ehdr->e_phoff = sizeof(Elf64_Ehdr);
  1280		ehdr->e_ehsize = sizeof(Elf64_Ehdr);
  1281		ehdr->e_phentsize = sizeof(Elf64_Phdr);
> 1282		ehdr->e_flags = ELF_CORE_EFLAGS;

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] kdump fix
@ 2023-07-20 19:08         ` kernel test robot
  0 siblings, 0 replies; 12+ messages in thread
From: kernel test robot @ 2023-07-20 19:08 UTC (permalink / raw)
  To: Mahesh J Salgaonkar, Michael Ellerman
  Cc: llvm, Sourabh Jain, Nicholas Piggin, Sachin Sant, oe-kbuild-all,
	linuxppc-dev, Hari Bathini

Hi Mahesh,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Mahesh-J-Salgaonkar/kdump-fix/20230720-021115
base:   linus/master
patch link:    https://lore.kernel.org/r/2ogfzwjumrd44kxv7njfpot6fhtkzpqu77qv3bspfixdmsxcwc%40umt35y2hmslm
patch subject: [PATCH] kdump fix
config: x86_64-randconfig-r021-20230720 (https://download.01.org/0day-ci/archive/20230721/202307210220.4FoNS76V-lkp@intel.com/config)
compiler: clang version 15.0.7 (https://github.com/llvm/llvm-project.git 8dfdcc7b7bf66834a761bd8de445840ef68e4d1a)
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307210220.4FoNS76V-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307210220.4FoNS76V-lkp@intel.com/

All errors (new ones prefixed by >>):

>> kernel/kexec_file.c:1282:18: error: use of undeclared identifier 'ELF_CORE_EFLAGS'
           ehdr->e_flags = ELF_CORE_EFLAGS;
                           ^
   1 error generated.


vim +/ELF_CORE_EFLAGS +1282 kernel/kexec_file.c

  1236	
  1237	int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
  1238				  void **addr, unsigned long *sz)
  1239	{
  1240		Elf64_Ehdr *ehdr;
  1241		Elf64_Phdr *phdr;
  1242		unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
  1243		unsigned char *buf;
  1244		unsigned int cpu, i;
  1245		unsigned long long notes_addr;
  1246		unsigned long mstart, mend;
  1247	
  1248		/* extra phdr for vmcoreinfo ELF note */
  1249		nr_phdr = nr_cpus + 1;
  1250		nr_phdr += mem->nr_ranges;
  1251	
  1252		/*
  1253		 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
  1254		 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
  1255		 * I think this is required by tools like gdb. So same physical
  1256		 * memory will be mapped in two ELF headers. One will contain kernel
  1257		 * text virtual addresses and other will have __va(physical) addresses.
  1258		 */
  1259	
  1260		nr_phdr++;
  1261		elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
  1262		elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
  1263	
  1264		buf = vzalloc(elf_sz);
  1265		if (!buf)
  1266			return -ENOMEM;
  1267	
  1268		ehdr = (Elf64_Ehdr *)buf;
  1269		phdr = (Elf64_Phdr *)(ehdr + 1);
  1270		memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
  1271		ehdr->e_ident[EI_CLASS] = ELFCLASS64;
  1272		ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
  1273		ehdr->e_ident[EI_VERSION] = EV_CURRENT;
  1274		ehdr->e_ident[EI_OSABI] = ELF_OSABI;
  1275		memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
  1276		ehdr->e_type = ET_CORE;
  1277		ehdr->e_machine = ELF_ARCH;
  1278		ehdr->e_version = EV_CURRENT;
  1279		ehdr->e_phoff = sizeof(Elf64_Ehdr);
  1280		ehdr->e_ehsize = sizeof(Elf64_Ehdr);
  1281		ehdr->e_phentsize = sizeof(Elf64_Phdr);
> 1282		ehdr->e_flags = ELF_CORE_EFLAGS;

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] kdump fix
  2023-07-19 18:03     ` Mahesh J Salgaonkar
@ 2023-07-21  3:50         ` kernel test robot
  2023-07-20 19:08         ` kernel test robot
  2023-07-21  3:50         ` kernel test robot
  2 siblings, 0 replies; 12+ messages in thread
From: kernel test robot @ 2023-07-21  3:50 UTC (permalink / raw)
  To: Mahesh J Salgaonkar, Michael Ellerman
  Cc: oe-kbuild-all, Hari Bathini, linuxppc-dev, Sourabh Jain,
	Sachin Sant, Nicholas Piggin

Hi Mahesh,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Mahesh-J-Salgaonkar/kdump-fix/20230720-021115
base:   linus/master
patch link:    https://lore.kernel.org/r/2ogfzwjumrd44kxv7njfpot6fhtkzpqu77qv3bspfixdmsxcwc%40umt35y2hmslm
patch subject: [PATCH] kdump fix
config: x86_64-allyesconfig (https://download.01.org/0day-ci/archive/20230721/202307211100.UDh0FQ50-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307211100.UDh0FQ50-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307211100.UDh0FQ50-lkp@intel.com/

All errors (new ones prefixed by >>):

   kernel/kexec_file.c: In function 'crash_prepare_elf64_headers':
>> kernel/kexec_file.c:1282:25: error: 'ELF_CORE_EFLAGS' undeclared (first use in this function)
    1282 |         ehdr->e_flags = ELF_CORE_EFLAGS;
         |                         ^~~~~~~~~~~~~~~
   kernel/kexec_file.c:1282:25: note: each undeclared identifier is reported only once for each function it appears in


vim +/ELF_CORE_EFLAGS +1282 kernel/kexec_file.c

  1236	
  1237	int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
  1238				  void **addr, unsigned long *sz)
  1239	{
  1240		Elf64_Ehdr *ehdr;
  1241		Elf64_Phdr *phdr;
  1242		unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
  1243		unsigned char *buf;
  1244		unsigned int cpu, i;
  1245		unsigned long long notes_addr;
  1246		unsigned long mstart, mend;
  1247	
  1248		/* extra phdr for vmcoreinfo ELF note */
  1249		nr_phdr = nr_cpus + 1;
  1250		nr_phdr += mem->nr_ranges;
  1251	
  1252		/*
  1253		 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
  1254		 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
  1255		 * I think this is required by tools like gdb. So same physical
  1256		 * memory will be mapped in two ELF headers. One will contain kernel
  1257		 * text virtual addresses and other will have __va(physical) addresses.
  1258		 */
  1259	
  1260		nr_phdr++;
  1261		elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
  1262		elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
  1263	
  1264		buf = vzalloc(elf_sz);
  1265		if (!buf)
  1266			return -ENOMEM;
  1267	
  1268		ehdr = (Elf64_Ehdr *)buf;
  1269		phdr = (Elf64_Phdr *)(ehdr + 1);
  1270		memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
  1271		ehdr->e_ident[EI_CLASS] = ELFCLASS64;
  1272		ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
  1273		ehdr->e_ident[EI_VERSION] = EV_CURRENT;
  1274		ehdr->e_ident[EI_OSABI] = ELF_OSABI;
  1275		memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
  1276		ehdr->e_type = ET_CORE;
  1277		ehdr->e_machine = ELF_ARCH;
  1278		ehdr->e_version = EV_CURRENT;
  1279		ehdr->e_phoff = sizeof(Elf64_Ehdr);
  1280		ehdr->e_ehsize = sizeof(Elf64_Ehdr);
  1281		ehdr->e_phentsize = sizeof(Elf64_Phdr);
> 1282		ehdr->e_flags = ELF_CORE_EFLAGS;

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: [PATCH] kdump fix
@ 2023-07-21  3:50         ` kernel test robot
  0 siblings, 0 replies; 12+ messages in thread
From: kernel test robot @ 2023-07-21  3:50 UTC (permalink / raw)
  To: Mahesh J Salgaonkar, Michael Ellerman
  Cc: Nicholas Piggin, Sourabh Jain, Sachin Sant, oe-kbuild-all,
	linuxppc-dev, Hari Bathini

Hi Mahesh,

kernel test robot noticed the following build errors:

[auto build test ERROR on linus/master]
[also build test ERROR on v6.5-rc2 next-20230720]
[If your patch is applied to the wrong git tree, kindly drop us a note.
And when submitting patch, we suggest to use '--base' as documented in
https://git-scm.com/docs/git-format-patch#_base_tree_information]

url:    https://github.com/intel-lab-lkp/linux/commits/Mahesh-J-Salgaonkar/kdump-fix/20230720-021115
base:   linus/master
patch link:    https://lore.kernel.org/r/2ogfzwjumrd44kxv7njfpot6fhtkzpqu77qv3bspfixdmsxcwc%40umt35y2hmslm
patch subject: [PATCH] kdump fix
config: x86_64-allyesconfig (https://download.01.org/0day-ci/archive/20230721/202307211100.UDh0FQ50-lkp@intel.com/config)
compiler: gcc-12 (Debian 12.2.0-14) 12.2.0
reproduce: (https://download.01.org/0day-ci/archive/20230721/202307211100.UDh0FQ50-lkp@intel.com/reproduce)

If you fix the issue in a separate patch/commit (i.e. not just a new version of
the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <lkp@intel.com>
| Closes: https://lore.kernel.org/oe-kbuild-all/202307211100.UDh0FQ50-lkp@intel.com/

All errors (new ones prefixed by >>):

   kernel/kexec_file.c: In function 'crash_prepare_elf64_headers':
>> kernel/kexec_file.c:1282:25: error: 'ELF_CORE_EFLAGS' undeclared (first use in this function)
    1282 |         ehdr->e_flags = ELF_CORE_EFLAGS;
         |                         ^~~~~~~~~~~~~~~
   kernel/kexec_file.c:1282:25: note: each undeclared identifier is reported only once for each function it appears in


vim +/ELF_CORE_EFLAGS +1282 kernel/kexec_file.c

  1236	
  1237	int crash_prepare_elf64_headers(struct crash_mem *mem, int need_kernel_map,
  1238				  void **addr, unsigned long *sz)
  1239	{
  1240		Elf64_Ehdr *ehdr;
  1241		Elf64_Phdr *phdr;
  1242		unsigned long nr_cpus = num_possible_cpus(), nr_phdr, elf_sz;
  1243		unsigned char *buf;
  1244		unsigned int cpu, i;
  1245		unsigned long long notes_addr;
  1246		unsigned long mstart, mend;
  1247	
  1248		/* extra phdr for vmcoreinfo ELF note */
  1249		nr_phdr = nr_cpus + 1;
  1250		nr_phdr += mem->nr_ranges;
  1251	
  1252		/*
  1253		 * kexec-tools creates an extra PT_LOAD phdr for kernel text mapping
  1254		 * area (for example, ffffffff80000000 - ffffffffa0000000 on x86_64).
  1255		 * I think this is required by tools like gdb. So same physical
  1256		 * memory will be mapped in two ELF headers. One will contain kernel
  1257		 * text virtual addresses and other will have __va(physical) addresses.
  1258		 */
  1259	
  1260		nr_phdr++;
  1261		elf_sz = sizeof(Elf64_Ehdr) + nr_phdr * sizeof(Elf64_Phdr);
  1262		elf_sz = ALIGN(elf_sz, ELF_CORE_HEADER_ALIGN);
  1263	
  1264		buf = vzalloc(elf_sz);
  1265		if (!buf)
  1266			return -ENOMEM;
  1267	
  1268		ehdr = (Elf64_Ehdr *)buf;
  1269		phdr = (Elf64_Phdr *)(ehdr + 1);
  1270		memcpy(ehdr->e_ident, ELFMAG, SELFMAG);
  1271		ehdr->e_ident[EI_CLASS] = ELFCLASS64;
  1272		ehdr->e_ident[EI_DATA] = ELFDATA2LSB;
  1273		ehdr->e_ident[EI_VERSION] = EV_CURRENT;
  1274		ehdr->e_ident[EI_OSABI] = ELF_OSABI;
  1275		memset(ehdr->e_ident + EI_PAD, 0, EI_NIDENT - EI_PAD);
  1276		ehdr->e_type = ET_CORE;
  1277		ehdr->e_machine = ELF_ARCH;
  1278		ehdr->e_version = EV_CURRENT;
  1279		ehdr->e_phoff = sizeof(Elf64_Ehdr);
  1280		ehdr->e_ehsize = sizeof(Elf64_Ehdr);
  1281		ehdr->e_phentsize = sizeof(Elf64_Phdr);
> 1282		ehdr->e_flags = ELF_CORE_EFLAGS;

-- 
0-DAY CI Kernel Test Service
https://github.com/intel/lkp-tests/wiki

^ permalink raw reply	[flat|nested] 12+ messages in thread

* Re: Kernel Crash Dump (kdump) broken with 6.5
  2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
  (?)
@ 2023-07-30 16:13   ` Linux regression tracking #update (Thorsten Leemhuis)
  -1 siblings, 0 replies; 12+ messages in thread
From: Linux regression tracking #update (Thorsten Leemhuis) @ 2023-07-30 16:13 UTC (permalink / raw)
  To: linuxppc-dev; +Cc: Linux kernel regressions list

On 19.07.23 18:19, Linux regression tracking #adding (Thorsten Leemhuis)
wrote:
> On 17.07.23 16:45, Sachin Sant wrote:
>> Kdump seems to be broken with 6.5 for ppc64le.
>> [...]
>>
>> 6.4 was good. Git bisect points to following patch
>>
>> commit 606787fed7268feb256957872586370b56af697a
>>     powerpc/64s: Remove support for ELFv1 little endian userspace
>>
>> Reverting this patch allows a successful capture of vmcore.

Was fixed by revert:

#regzbot fix: 106ea7ffd56
#regzbot ignore-activity

Ciao, Thorsten (wearing his 'the Linux kernel's regression tracker' hat)
--
Everything you wanna know about Linux kernel regression tracking:
https://linux-regtracking.leemhuis.info/about/#tldr
That page also explains what to do if mails like this annoy you.

^ permalink raw reply	[flat|nested] 12+ messages in thread

end of thread, other threads:[~2023-07-30 16:14 UTC | newest]

Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-07-17 14:45 Kernel Crash Dump (kdump) broken with 6.5 Sachin Sant
2023-07-17 16:38 ` Mahesh J Salgaonkar
2023-07-18 13:19   ` Michael Ellerman
2023-07-19 18:03     ` Mahesh J Salgaonkar
2023-07-20  8:39       ` Sachin Sant
2023-07-20 19:08       ` [PATCH] kdump fix kernel test robot
2023-07-20 19:08         ` kernel test robot
2023-07-21  3:50       ` kernel test robot
2023-07-21  3:50         ` kernel test robot
2023-07-19 16:19 ` Kernel Crash Dump (kdump) broken with 6.5 Linux regression tracking #adding (Thorsten Leemhuis)
2023-07-19 16:19   ` Linux regression tracking #adding (Thorsten Leemhuis)
2023-07-30 16:13   ` Linux regression tracking #update (Thorsten Leemhuis)

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.