Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Baoquan He]

On 02/09/22 at 09:26am, Huang Shijie wrote:
> The following interrelated ranges are needed by the kdump crash tool:
> 	MODULES_VADDR ~ MODULES_END,
> 	VMALLOC_START ~ VMALLOC_END,
> 	VMEMMAP_START ~ VMEMMAP_END
> 
> Since these values change from time to time, it is preferable to export
> them via vmcoreinfo than to change the crash's code frequently.

Add Kazu and Lianbo to CC since they take care of Crash utility now,
please check if this is necessary for Crash.

> 
> Signed-off-by: Huang Shijie <shijie@os.amperecomputing.com>
> ---
>  Documentation/admin-guide/kdump/vmcoreinfo.rst | 8 ++++++++
>  arch/arm64/kernel/crash_core.c                 | 6 ++++++
>  2 files changed, 14 insertions(+)
> 
> diff --git a/Documentation/admin-guide/kdump/vmcoreinfo.rst b/Documentation/admin-guide/kdump/vmcoreinfo.rst
> index 3861a25faae1..a339af45a22e 100644
> --- a/Documentation/admin-guide/kdump/vmcoreinfo.rst
> +++ b/Documentation/admin-guide/kdump/vmcoreinfo.rst
> @@ -494,6 +494,14 @@ architecture which is used to lookup the page-tables for the Virtual
>  addresses in the higher VA range (refer to ARMv8 ARM document for
>  more details).
>  
> +MODULES_VADDR|MODULES_END|VMALLOC_START|VMALLOC_END|VMEMMAP_START|VMEMMAP_END
> +-------------
> +
> +Used to get the correct ranges:
> +	MODULES_VADDR ~ MODULES_END-1 : Kernel module space.
> +	VMALLOC_START ~ VMALLOC_END-1 : vmalloc() / ioremap() space.
> +	VMEMMAP_START ~ VMEMMAP_END-1 : vmemmap region, used for struct page array.
> +
>  arm
>  ===
>  
> diff --git a/arch/arm64/kernel/crash_core.c b/arch/arm64/kernel/crash_core.c
> index 314391a156ee..2b65aae332ce 100644
> --- a/arch/arm64/kernel/crash_core.c
> +++ b/arch/arm64/kernel/crash_core.c
> @@ -20,6 +20,12 @@ void arch_crash_save_vmcoreinfo(void)
>  {
>  	VMCOREINFO_NUMBER(VA_BITS);
>  	/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */
> +	vmcoreinfo_append_str("NUMBER(MODULES_VADDR)=0x%lx\n", MODULES_VADDR);
> +	vmcoreinfo_append_str("NUMBER(MODULES_END)=0x%lx\n", MODULES_END);
> +	vmcoreinfo_append_str("NUMBER(VMALLOC_START)=0x%lx\n", VMALLOC_START);
> +	vmcoreinfo_append_str("NUMBER(VMALLOC_END)=0x%lx\n", VMALLOC_END);
> +	vmcoreinfo_append_str("NUMBER(VMEMMAP_START)=0x%lx\n", VMEMMAP_START);
> +	vmcoreinfo_append_str("NUMBER(VMEMMAP_END)=0x%lx\n", VMEMMAP_END);
>  	vmcoreinfo_append_str("NUMBER(kimage_voffset)=0x%llx\n",
>  						kimage_voffset);
>  	vmcoreinfo_append_str("NUMBER(PHYS_OFFSET)=0x%llx\n",
> -- 
> 2.30.2
> 

[PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Huang Shijie]

The following interrelated ranges are needed by the kdump crash tool:
	MODULES_VADDR ~ MODULES_END,
	VMALLOC_START ~ VMALLOC_END,
	VMEMMAP_START ~ VMEMMAP_END

Since these values change from time to time, it is preferable to export
them via vmcoreinfo than to change the crash's code frequently.

Signed-off-by: Huang Shijie <shijie@os.amperecomputing.com>
---
 Documentation/admin-guide/kdump/vmcoreinfo.rst | 8 ++++++++
 arch/arm64/kernel/crash_core.c                 | 6 ++++++
 2 files changed, 14 insertions(+)

diff --git a/Documentation/admin-guide/kdump/vmcoreinfo.rst b/Documentation/admin-guide/kdump/vmcoreinfo.rst
index 3861a25faae1..a339af45a22e 100644
--- a/Documentation/admin-guide/kdump/vmcoreinfo.rst
+++ b/Documentation/admin-guide/kdump/vmcoreinfo.rst
@@ -494,6 +494,14 @@ architecture which is used to lookup the page-tables for the Virtual
 addresses in the higher VA range (refer to ARMv8 ARM document for
 more details).
 
+MODULES_VADDR|MODULES_END|VMALLOC_START|VMALLOC_END|VMEMMAP_START|VMEMMAP_END
+-------------
+
+Used to get the correct ranges:
+	MODULES_VADDR ~ MODULES_END-1 : Kernel module space.
+	VMALLOC_START ~ VMALLOC_END-1 : vmalloc() / ioremap() space.
+	VMEMMAP_START ~ VMEMMAP_END-1 : vmemmap region, used for struct page array.
+
 arm
 ===
 
diff --git a/arch/arm64/kernel/crash_core.c b/arch/arm64/kernel/crash_core.c
index 314391a156ee..2b65aae332ce 100644
--- a/arch/arm64/kernel/crash_core.c
+++ b/arch/arm64/kernel/crash_core.c
@@ -20,6 +20,12 @@ void arch_crash_save_vmcoreinfo(void)
 {
 	VMCOREINFO_NUMBER(VA_BITS);
 	/* Please note VMCOREINFO_NUMBER() uses "%d", not "%x" */
+	vmcoreinfo_append_str("NUMBER(MODULES_VADDR)=0x%lx\n", MODULES_VADDR);
+	vmcoreinfo_append_str("NUMBER(MODULES_END)=0x%lx\n", MODULES_END);
+	vmcoreinfo_append_str("NUMBER(VMALLOC_START)=0x%lx\n", VMALLOC_START);
+	vmcoreinfo_append_str("NUMBER(VMALLOC_END)=0x%lx\n", VMALLOC_END);
+	vmcoreinfo_append_str("NUMBER(VMEMMAP_START)=0x%lx\n", VMEMMAP_START);
+	vmcoreinfo_append_str("NUMBER(VMEMMAP_END)=0x%lx\n", VMEMMAP_END);
 	vmcoreinfo_append_str("NUMBER(kimage_voffset)=0x%llx\n",
 						kimage_voffset);
 	vmcoreinfo_append_str("NUMBER(PHYS_OFFSET)=0x%llx\n",
-- 
2.30.2

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Will Deacon]

On Wed, Feb 09, 2022 at 09:26:42AM +0000, Huang Shijie wrote:
> The following interrelated ranges are needed by the kdump crash tool:
> 	MODULES_VADDR ~ MODULES_END,
> 	VMALLOC_START ~ VMALLOC_END,
> 	VMEMMAP_START ~ VMEMMAP_END
> 
> Since these values change from time to time, it is preferable to export
> them via vmcoreinfo than to change the crash's code frequently.

Please can you explain _why_ they are needed?

Will

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Huang Shijie]

Hi Will,
On Tue, Feb 15, 2022 at 04:44:23PM +0000, Will Deacon wrote:
> On Wed, Feb 09, 2022 at 09:26:42AM +0000, Huang Shijie wrote:
> > The following interrelated ranges are needed by the kdump crash tool:
> > 	MODULES_VADDR ~ MODULES_END,
> > 	VMALLOC_START ~ VMALLOC_END,
> > 	VMEMMAP_START ~ VMEMMAP_END
> > 
> > Since these values change from time to time, it is preferable to export
> > them via vmcoreinfo than to change the crash's code frequently.
> 
> Please can you explain _why_ they are needed?

The current Crash code is still based at kernel v4.9.
   The virtual memory layout looks like this:
   +--------------------------------------------------------------------+
   |    KASAN     |   MODULE   |   VMALLOC   | .... |     VMEMMAP       |
   +--------------------------------------------------------------------+

The Crash uses MODULES range to set the VMALLOC ranges.
If the ranges are wrong, Crash will _NOT_ works well for some latest kernel
,such as v5.11 later. (Please correct me if I am wrong).
It seems the VMEMMAP range is less important.


I list all the changes since v4.9 to now:

1.) The current crash code is based at kernel v4.9.
   The virtual memory layout looks like this:
   +--------------------------------------------------------------------+
   |    KASAN     |   MODULE   |   VMALLOC   | .... |     VMEMMAP       |
   +--------------------------------------------------------------------+

    The macros are:
    #define MODULES_VADDR	(VA_START + KASAN_SHADOW_SIZE)
    #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)

    #define VMALLOC_START	(MODULES_END)
    #define VMALLOC_END		(PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)

    #define VMEMMAP_START	(PAGE_OFFSET - VMEMMAP_SIZE)

2.) In the kernel v5.0, the patch will add a new BFP JIT region:
       "91fc957c9b1d arm64/bpf: don't allocate BPF JIT programs in module memory"

   The virtual memory layout looks like this:
   +--------------------------------------------------------------------+
   | KASAN | BPF_JIT | MODULE | VMALLOC |     ....     | VMEMMAP        |
   +--------------------------------------------------------------------+

    The macros are:
    #define MODULES_VADDR	(BPF_JIT_REGION_END)
    #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)

    #define VMALLOC_START	(MODULES_END)
    #define VMALLOC_END		(PAGE_OFFSET - PUD_SIZE - VMEMMAP_SIZE - SZ_64K)

    #define VMEMMAP_START	(PAGE_OFFSET - VMEMMAP_SIZE)

    The layout does not changed until v5.4.

3.) In the kernel v5.4, several patches changes the layout, such as:
       "ce3aaed87344 arm64: mm: Modify calculation of VMEMMAP_SIZE"
       "14c127c957c1 arm64: mm: Flip kernel VA space"
    and the virtual memory layout looks like this:

   +--------------------------------------------------------------------+
   | KASAN | BPF_JIT | MODULE | VMALLOC |     ....     | VMEMMAP        |
   +--------------------------------------------------------------------+

    The macros are:
    #define MODULES_VADDR	(BPF_JIT_REGION_END)
    #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)

    #define VMALLOC_START	(MODULES_END)
    #define VMALLOC_END		(- PUD_SIZE - VMEMMAP_SIZE - SZ_64K)

    #define VMEMMAP_START	(-VMEMMAP_SIZE - SZ_2M)

4.) In the kernel v5.11, several patches changes the layout, such as:
       "9ad7c6d5e75b arm64: mm: tidy up top of kernel VA space"
       "f4693c2716b3 arm64: mm: extend linear region for 52-bit VA configurations"
    and the virtual memory layout looks like this:

   +--------------------------------------------------------------------+
   |   BPF_JIT | MODULE  |  VMALLOC   |    ....    |     VMEMMAP        |
   +--------------------------------------------------------------------+

    The macros are:
    #define MODULES_VADDR	(BPF_JIT_REGION_END)
    #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)

    #define VMALLOC_START	(MODULES_END)
    #define VMALLOC_END		(VMEMMAP_START - SZ_256M)

    #define VMEMMAP_START	(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
    #define VMEMMAP_END		(VMEMMAP_START + VMEMMAP_SIZE)

5.) In the kernel v5.16, after the patch
      "b89ddf4cca43 arm64/bpf: Remove 128MB limit for BPF JIT programs"
    the virtual memory layout looks like this:

   +--------------------------------------------------------------------+
   |      MODULE     |   VMALLOC   |     ....     |      VMEMMAP        |
   +--------------------------------------------------------------------+

    The macros are:
    #define MODULES_VADDR	(_PAGE_END(VA_BITS_MIN))
    #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)

    #define VMALLOC_START	(MODULES_END)
    #define VMALLOC_END		(VMEMMAP_START - SZ_256M)

    #define VMEMMAP_START	(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
    #define VMEMMAP_END		(VMEMMAP_START + VMEMMAP_SIZE)


BTW:I am currently coding a patch for the Crash to update all the ranges to
the latest kernel version v5.17-rc4.

Thanks
Huang Shijie

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Will Deacon]

On Wed, Feb 16, 2022 at 09:28:49AM +0000, Huang Shijie wrote:
> Hi Will,
> On Tue, Feb 15, 2022 at 04:44:23PM +0000, Will Deacon wrote:
> > On Wed, Feb 09, 2022 at 09:26:42AM +0000, Huang Shijie wrote:
> > > The following interrelated ranges are needed by the kdump crash tool:
> > > 	MODULES_VADDR ~ MODULES_END,
> > > 	VMALLOC_START ~ VMALLOC_END,
> > > 	VMEMMAP_START ~ VMEMMAP_END
> > > 
> > > Since these values change from time to time, it is preferable to export
> > > them via vmcoreinfo than to change the crash's code frequently.
> > 
> > Please can you explain _why_ they are needed?
> 
> The current Crash code is still based at kernel v4.9.
>    The virtual memory layout looks like this:
>    +--------------------------------------------------------------------+
>    |    KASAN     |   MODULE   |   VMALLOC   | .... |     VMEMMAP       |
>    +--------------------------------------------------------------------+
> 
> The Crash uses MODULES range to set the VMALLOC ranges.
> If the ranges are wrong, Crash will _NOT_ works well for some latest kernel
> ,such as v5.11 later. (Please correct me if I am wrong).
> It seems the VMEMMAP range is less important.

[...]

> 5.) In the kernel v5.16, after the patch
>       "b89ddf4cca43 arm64/bpf: Remove 128MB limit for BPF JIT programs"
>     the virtual memory layout looks like this:
> 
>    +--------------------------------------------------------------------+
>    |      MODULE     |   VMALLOC   |     ....     |      VMEMMAP        |
>    +--------------------------------------------------------------------+
> 
>     The macros are:
>     #define MODULES_VADDR	(_PAGE_END(VA_BITS_MIN))
>     #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)
> 
>     #define VMALLOC_START	(MODULES_END)
>     #define VMALLOC_END		(VMEMMAP_START - SZ_256M)
> 
>     #define VMEMMAP_START	(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
>     #define VMEMMAP_END		(VMEMMAP_START + VMEMMAP_SIZE)
> 
> 
> BTW:I am currently coding a patch for the Crash to update all the ranges to
> the latest kernel version v5.17-rc4.

Thanks for digging up all of the kernel memory map changes and taking the
time to explain the macros. However, all I'm really after is something in
the commit message of the patch which explains what is broken without this
patch. What does crash use this information for, and what doesn't work at
the moment?

Cheers,

Will

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Huang Shijie]

Hi Will,

CC Kazu and Lianbo.
On Wed, Feb 16, 2022 at 12:40:27PM +0000, Will Deacon wrote:
> On Wed, Feb 16, 2022 at 09:28:49AM +0000, Huang Shijie wrote:
> > Hi Will,
> > On Tue, Feb 15, 2022 at 04:44:23PM +0000, Will Deacon wrote:
> > > On Wed, Feb 09, 2022 at 09:26:42AM +0000, Huang Shijie wrote:
> > > > The following interrelated ranges are needed by the kdump crash tool:
> > > > 	MODULES_VADDR ~ MODULES_END,
> > > > 	VMALLOC_START ~ VMALLOC_END,
> > > > 	VMEMMAP_START ~ VMEMMAP_END
> > > > 
> > > > Since these values change from time to time, it is preferable to export
> > > > them via vmcoreinfo than to change the crash's code frequently.
> > > 
> > > Please can you explain _why_ they are needed?
> > 
> > The current Crash code is still based at kernel v4.9.
> >    The virtual memory layout looks like this:
> >    +--------------------------------------------------------------------+
> >    |    KASAN     |   MODULE   |   VMALLOC   | .... |     VMEMMAP       |
> >    +--------------------------------------------------------------------+
> > 
> > The Crash uses MODULES range to set the VMALLOC ranges.
> > If the ranges are wrong, Crash will _NOT_ works well for some latest kernel
> > ,such as v5.11 later. (Please correct me if I am wrong).
> > It seems the VMEMMAP range is less important.
> 
> [...]
> 
> > 5.) In the kernel v5.16, after the patch
> >       "b89ddf4cca43 arm64/bpf: Remove 128MB limit for BPF JIT programs"
> >     the virtual memory layout looks like this:
> > 
> >    +--------------------------------------------------------------------+
> >    |      MODULE     |   VMALLOC   |     ....     |      VMEMMAP        |
> >    +--------------------------------------------------------------------+
> > 
> >     The macros are:
> >     #define MODULES_VADDR	(_PAGE_END(VA_BITS_MIN))
> >     #define MODULES_END		(MODULES_VADDR + MODULES_VSIZE)
> > 
> >     #define VMALLOC_START	(MODULES_END)
> >     #define VMALLOC_END		(VMEMMAP_START - SZ_256M)
> > 
> >     #define VMEMMAP_START	(-(UL(1) << (VA_BITS - VMEMMAP_SHIFT)))
> >     #define VMEMMAP_END		(VMEMMAP_START + VMEMMAP_SIZE)
> > 
> > 
> > BTW:I am currently coding a patch for the Crash to update all the ranges to
> > the latest kernel version v5.17-rc4.
> 
> Thanks for digging up all of the kernel memory map changes and taking the
> time to explain the macros. However, all I'm really after is something in
> the commit message of the patch which explains what is broken without this
This kernel patch does not break anything.
It just makes the Crash easy to maintain.

> patch. What does crash use this information for, and what doesn't work at
> the moment?
I know two cases now:
	1.) The Crash uses MODULES/VMALLOC/VMEMMAP ranges in
            arm64_IS_VMALLOC_ADDR():
		https://github.com/crash-utility/crash/blob/master/arm64.c#L4104

	    If arm64_IS_VMALLOC_ADDR() does not work correctly, the internal
	    code may get wrong results. 

	2.) The "help -m" gets wrong output about MODULES/VMALLOC/VMEMMAP ranges.
	    The guy who use the Use the Crash to debug a vmcore, will get the wrong
	    information of the kernel panic.

Thanks
Huang Shijie

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Huang Shijie]

On Thu, Feb 17, 2022 at 10:02:03AM +0000, Huang Shijie wrote:
> > Thanks for digging up all of the kernel memory map changes and taking the
> > time to explain the macros. However, all I'm really after is something in
> > the commit message of the patch which explains what is broken without this
> This kernel patch does not break anything.
I mean this kernel patch is not a bugfix, it is just make the ARM64 ecosystem
(including the Crash) a little better..

The Crash can just parse out the MODULES/VMALLOC/VMALLOC ranges from the vmcore file,
no need to calculate them in a very _hard_ way. 

Thanks
Huang Shijie

Re: [PATCH] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges

[Will Deacon]

On Wed, 9 Feb 2022 09:26:42 +0000, Huang Shijie wrote:
> The following interrelated ranges are needed by the kdump crash tool:
> 	MODULES_VADDR ~ MODULES_END,
> 	VMALLOC_START ~ VMALLOC_END,
> 	VMEMMAP_START ~ VMEMMAP_END
> 
> Since these values change from time to time, it is preferable to export
> them via vmcoreinfo than to change the crash's code frequently.
> 
> [...]

Applied to arm64 (for-next/mm), thanks!

[1/1] arm64: crash_core: Export MODULES, VMALLOC, and VMEMMAP ranges
      https://git.kernel.org/arm64/c/2369f171d5c5

Cheers,
-- 
Will

https://fixes.arm64.dev
https://next.arm64.dev
https://will.arm64.dev