[PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[nikhil.gupta]

[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #1: Type: text/plain; charset=y, Size: 3421 bytes --]

From: Nikhil Gupta <nikhil.gupta@nxp.com>

elfcorehdr_addr is fixed address passed to Second kernel which may be conflicted
with potential reserved memory in Second kernel,so fdt_reserve_elfcorehdr() ahead
of fdt_init_reserved_mem() can relieve this situation.

Signed-off-by: Nikhil Gupta <nikhil.gupta@nxp.com>
---
elfcorehdr addr is calculated in kexec tool based on reserved crashkernel memory space. (Last page is used in top down order)
So It may potentially conflict the reserved memory in 2nd kernel.
Below is log:
load_crashdump_segments: page_offset:   ffffc00000000000
…..
load_crashdump_segments: elfcorehdr 0xf3fff000-0xf3fff3ff
read_1st_dtb: found /sys/firmware/fdt
get_cells_size: #address-cells:2 #size-cells:2
cells_size_fitted: f3fff000-f3fff3ff
cells_size_fitted: d4000000-f3ffffff

Overlaps seen on LS1043A SoC:

[    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d677ffff] arm64_memblock_init+0x258/0x2c8
[    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f2000000-0x00000000f2ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
[    0.000000] OF: fdt: elfcorehdr is overlapped

panic in elfcorehdr_read.
[    0.443984] Unable to handle kernel paging request at virtual address ffff000037fff000
[    0.451942] Mem abort info:
[    0.454740]   ESR = 0x96000006
[    0.457806]   EC = 0x25: DABT (current EL), IL = 32 bits
[    0.463142]   SET = 0, FnV = 0
[    0.466202]   EA = 0, S1PTW = 0
[    0.469353] Data abort info:
[    0.472243]   ISV = 0, ISS = 0x00000006
[    0.476094]   CM = 0, WnR = 0
[    0.479072] swapper pgtable: 4k pages, 48-bit VAs, pgdp=00000000d9373000
[    0.485806] [ffff000037fff000] pgd=00000000f7bf7003, pud=00000000f7bf6003, pmd=0000000000000000
[    0.494553] Internal error: Oops: 96000006 [#1] PREEMPT SMP
[    0.500146] Modules linked in:
[    0.503211] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 5.5.0-rc3-00062-gbf8d1cd #5
[    0.510725] Hardware name: LS1043A RDB Board (DT)
[    0.515446] pstate: 80000005 (Nzcv daif -PAN -UAO)
[    0.520260] pc : __memcpy+0x78/0x180
[    0.523847] lr : elfcorehdr_read+0x28/0x38
[    0.611262] Call trace:
[    0.613713]  __memcpy+0x78/0x180
[    0.616950]  vmcore_init+0x70/0x534
 drivers/of/fdt.c | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/drivers/of/fdt.c b/drivers/of/fdt.c
index ad85ff6474ff..ec315b060cd5 100644
--- a/drivers/of/fdt.c
+++ b/drivers/of/fdt.c
@@ -648,8 +648,8 @@ void __init early_init_fdt_scan_reserved_mem(void)
 	}
 
 	fdt_scan_reserved_mem();
-	fdt_init_reserved_mem();
 	fdt_reserve_elfcorehdr();
+	fdt_init_reserved_mem();
 }
 
 /**
-- 
2.17.1

Re: [PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[Rob Herring]

On Thu, Jan 27, 2022 at 10:23 PM <nikhil.gupta@nxp.com> wrote:
>
> From: Nikhil Gupta <nikhil.gupta@nxp.com>
>
> elfcorehdr_addr is fixed address passed to Second kernel which may be conflicted
> with potential reserved memory in Second kernel,so fdt_reserve_elfcorehdr() ahead
> of fdt_init_reserved_mem() can relieve this situation.

How does changing the order help exactly? If you have an overlap, that
seems bad no matter what. What if the overlapping reserved area is
used by another core or firmware?

Rob

RE: [EXT] Re: [PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[Nikhil Gupta]

Hello Rob,

This issue will come in scenarios where the memory has been reserved using fdt_init_reserved_mem()
The memory address which is reserved overlaps with the address of elfcorehdr which has a fixed address,
So the memory for elfcorehdr needs to be reserved first.

crashkernel memory range : 0x00000000d4000000 - 0x00000000f4000000 (512 MB)
elfcorehdr addr is calculated in kexec tool based on reserved crashkernel memory space. (Last page is used in top down order)
elfcorehdr conflict with memory [    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff] reserved by fdt_init_reserved_mem()

Without Patch:

[    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d677ffff] arm64_memblock_init+0x258/0x2c8
[    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
[    0.000000] memblock_reserve: [0x00000000f2000000-0x00000000f2ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
[    0.000000] OF: fdt: elfcorehdr is overlapped

With Patch:

[    0.000000] memblock_remove: [0x0001000000000000-0x0000fffffffffffe] arm64_memblock_init+0xa0/0x2d8
[    0.000000] memblock_remove: [0x00008000c0000000-0x00008000bffffffe] arm64_memblock_init+0xfc/0x2d8
[    0.000000] memblock_remove: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1c8/0x2d8
[    0.000000] memblock_add: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1d4/0x2d8
[    0.000000] memblock_reserve: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1e0/0x2d8
[    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d615ffff] arm64_memblock_init+0x260/0x2d8
[    0.000000] memblock_reserve: [0x00000000f3fff000-0x00000000f3fff3ff] early_init_fdt_scan_reserved_mem+0x370/0x3c8
[    0.000000] OF: fdt: Reserving 1 KiB of memory at 0xf3fff000 for elfcorehdr
[    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
[    0.000000] memblock_reserve: [0x00000000f3800000-0x00000000f3bfffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
[    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff] memblock_alloc_range_nid+0xdc/0x150
[    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c

Kind Regards,
Nikhil

-----Original Message-----
From: Rob Herring <robh+dt@kernel.org> 
Sent: Friday, January 28, 2022 9:03 PM
To: Nikhil Gupta <nikhil.gupta@nxp.com>
Cc: linux-arm-kernel <linux-arm-kernel@lists.infradead.org>; Frank Rowand <frowand.list@gmail.com>; devicetree@vger.kernel.org; linux-kernel@vger.kernel.org; Priyanka Jain <priyanka.jain@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>
Subject: [EXT] Re: [PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

Caution: EXT Email

On Thu, Jan 27, 2022 at 10:23 PM <nikhil.gupta@nxp.com> wrote:
>
> From: Nikhil Gupta <nikhil.gupta@nxp.com>
>
> elfcorehdr_addr is fixed address passed to Second kernel which may be 
> conflicted with potential reserved memory in Second kernel,so 
> fdt_reserve_elfcorehdr() ahead of fdt_init_reserved_mem() can relieve this situation.

How does changing the order help exactly? If you have an overlap, that seems bad no matter what. What if the overlapping reserved area is used by another core or firmware?

Rob

Re: [EXT] Re: [PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[Rob Herring]

On Tue, Feb 1, 2022 at 9:37 AM Nikhil Gupta <nikhil.gupta@nxp.com> wrote:
>

Please don't top post and fix your email client quoting on replies.
IOW, don't use Outlook. While MS loves Linux now, they don't make an
email client that works with maillists.

> Hello Rob,
>
> This issue will come in scenarios where the memory has been reserved using fdt_init_reserved_mem()
> The memory address which is reserved overlaps with the address of elfcorehdr which has a fixed address,
> So the memory for elfcorehdr needs to be reserved first.
>
> crashkernel memory range : 0x00000000d4000000 - 0x00000000f4000000 (512 MB)
> elfcorehdr addr is calculated in kexec tool based on reserved crashkernel memory space. (Last page is used in top down order)
> elfcorehdr conflict with memory [    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff] reserved by fdt_init_reserved_mem()
>
> Without Patch:
>
> [    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d677ffff] arm64_memblock_init+0x258/0x2c8
> [    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> [    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff] memblock_alloc_range_nid+0xdc/0x150
> [    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> [    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff] memblock_alloc_range_nid+0xdc/0x150
> [    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> [    0.000000] memblock_reserve: [0x00000000f2000000-0x00000000f2ffffff] memblock_alloc_range_nid+0xdc/0x150
> [    0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id fsl,bman-fbpr
> [    0.000000] OF: fdt: elfcorehdr is overlapped
>
> With Patch:
>
> [    0.000000] memblock_remove: [0x0001000000000000-0x0000fffffffffffe] arm64_memblock_init+0xa0/0x2d8
> [    0.000000] memblock_remove: [0x00008000c0000000-0x00008000bffffffe] arm64_memblock_init+0xfc/0x2d8
> [    0.000000] memblock_remove: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1c8/0x2d8
> [    0.000000] memblock_add: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1d4/0x2d8
> [    0.000000] memblock_reserve: [0x00000000d6160000-0x00000000d83befff] arm64_memblock_init+0x1e0/0x2d8
> [    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d615ffff] arm64_memblock_init+0x260/0x2d8
> [    0.000000] memblock_reserve: [0x00000000f3fff000-0x00000000f3fff3ff] early_init_fdt_scan_reserved_mem+0x370/0x3c8
> [    0.000000] OF: fdt: Reserving 1 KiB of memory at 0xf3fff000 for elfcorehdr
> [    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
> [    0.000000] memblock_reserve: [0x00000000f3800000-0x00000000f3bfffff] memblock_alloc_range_nid+0xdc/0x150
> [    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
> [    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff] memblock_alloc_range_nid+0xdc/0x150
> [    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000 from=0x0000000000000000 max_addr=0x0001000000000000 early_init_dt_alloc_reserved_memory_arch+0x38/0x8c

Digging a bit more, it seems this never worked? It is also not fixable
in this way before commit 57beb9bd18fc ("arm64: kdump: Remove custom
linux,elfcorehdr handling"). So if you want a fix prior to v5.15, it
will need a different fix.

Rob

RE: [EXT] Re: [PATCH] of/fdt: move elfcorehdr reservation early for crash dump kernel

[Nikhil Gupta]

> -----Original Message-----
> From: Rob Herring <robh+dt@kernel.org>
> Sent: Thursday, February 10, 2022 5:03 AM
> To: Nikhil Gupta <nikhil.gupta@nxp.com>
> Cc: linux-arm-kernel <linux-arm-kernel@lists.infradead.org>; Frank Rowand
> <frowand.list@gmail.com>; devicetree@vger.kernel.org; linux-
> kernel@vger.kernel.org; Priyanka Jain <priyanka.jain@nxp.com>; Aisheng Dong
> <aisheng.dong@nxp.com>
> Subject: Re: [EXT] Re: [PATCH] of/fdt: move elfcorehdr reservation early for
> crash dump kernel
> 
> Caution: EXT Email
> 
> On Tue, Feb 1, 2022 at 9:37 AM Nikhil Gupta <nikhil.gupta@nxp.com> wrote:
> >
> 
> Please don't top post and fix your email client quoting on replies.
> IOW, don't use Outlook. While MS loves Linux now, they don't make an email
> client that works with maillists.

Thanks, will take care !
> 
> > Hello Rob,
> >
> > This issue will come in scenarios where the memory has been reserved
> > using fdt_init_reserved_mem() The memory address which is reserved
> > overlaps with the address of elfcorehdr which has a fixed address, So the
> memory for elfcorehdr needs to be reserved first.
> >
> > crashkernel memory range : 0x00000000d4000000 - 0x00000000f4000000
> > (512 MB) elfcorehdr addr is calculated in kexec tool based on reserved
> crashkernel memory space. (Last page is used in top down order)
> > elfcorehdr conflict with memory [    0.000000] memblock_reserve:
> [0x00000000f3c00000-0x00000000f3ffffff] reserved by fdt_init_reserved_mem()
> >
> > Without Patch:
> >
> > [    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d677ffff]
> arm64_memblock_init+0x258/0x2c8
> > [    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> > [    0.000000] memblock_reserve: [0x00000000f3c00000-0x00000000f3ffffff]
> memblock_alloc_range_nid+0xdc/0x150
> > [    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> > [    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff]
> memblock_alloc_range_nid+0xdc/0x150
> > [    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x9c/0x16c
> > [    0.000000] memblock_reserve: [0x00000000f2000000-0x00000000f2ffffff]
> memblock_alloc_range_nid+0xdc/0x150
> > [    0.000000] OF: reserved mem: initialized node bman-fbpr, compatible id
> fsl,bman-fbpr
> > [    0.000000] OF: fdt: elfcorehdr is overlapped
> >
> > With Patch:
> >
> > [    0.000000] memblock_remove: [0x0001000000000000-0x0000fffffffffffe]
> arm64_memblock_init+0xa0/0x2d8
> > [    0.000000] memblock_remove: [0x00008000c0000000-0x00008000bffffffe]
> arm64_memblock_init+0xfc/0x2d8
> > [    0.000000] memblock_remove: [0x00000000d6160000-0x00000000d83befff]
> arm64_memblock_init+0x1c8/0x2d8
> > [    0.000000] memblock_add: [0x00000000d6160000-0x00000000d83befff]
> arm64_memblock_init+0x1d4/0x2d8
> > [    0.000000] memblock_reserve: [0x00000000d6160000-0x00000000d83befff]
> arm64_memblock_init+0x1e0/0x2d8
> > [    0.000000] memblock_reserve: [0x00000000d4010000-0x00000000d615ffff]
> arm64_memblock_init+0x260/0x2d8
> > [    0.000000] memblock_reserve: [0x00000000f3fff000-0x00000000f3fff3ff]
> early_init_fdt_scan_reserved_mem+0x370/0x3c8
> > [    0.000000] OF: fdt: Reserving 1 KiB of memory at 0xf3fff000 for elfcorehdr
> > [    0.000000] memblock_phys_alloc_range: 4194304 bytes align=0x400000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
> > [    0.000000] memblock_reserve: [0x00000000f3800000-0x00000000f3bfffff]
> memblock_alloc_range_nid+0xdc/0x150
> > [    0.000000] memblock_phys_alloc_range: 33554432 bytes align=0x2000000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
> > [    0.000000] memblock_reserve: [0x00000000f0000000-0x00000000f1ffffff]
> memblock_alloc_range_nid+0xdc/0x150
> > [    0.000000] memblock_phys_alloc_range: 16777216 bytes align=0x1000000
> from=0x0000000000000000 max_addr=0x0001000000000000
> early_init_dt_alloc_reserved_memory_arch+0x38/0x8c
> 
> Digging a bit more, it seems this never worked? It is also not fixable in this way
> before commit 57beb9bd18fc ("arm64: kdump: Remove custom linux,elfcorehdr
> handling"). So if you want a fix prior to v5.15, it will need a different fix.
> 
> Rob

Yes, you are right. I face this issue earlier as well with previous kernel. For that , a different similar patch is needed on another file.
I will float the patch to fix issue in previous kernels once this patch gets reviewed and approved.

Do you have any other query on this patch or is it fine?

Thanks
Nikhil