[[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[Thomas Bogendoerfer]

Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
and NUMA enabled loongson3 by doing memblock_set_current_limit()
before max_low_pfn has been evaluated. Both platforms need to do the
memblock_set_current_limit() in platform specific code. For
consistency the call to memblock_set_current_limit() is moved
to the common bootmem_init(), where max_low_pfn is calculated
for non NUMA enabled platforms.

Fixes: bcec54bf3118 ("mips: switch to NO_BOOTMEM")
Signed-off-by: Thomas Bogendoerfer <tbogendoerfer@suse.de>
---
 arch/mips/kernel/setup.c               | 18 +++++++++---------
 arch/mips/loongson64/loongson-3/numa.c |  1 +
 arch/mips/sgi-ip27/ip27-memory.c       |  1 +
 3 files changed, 11 insertions(+), 9 deletions(-)

diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index ea09ed6a80a9..b5167b198231 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -576,6 +576,15 @@ static void __init bootmem_init(void)
 	 * Reserve initrd memory if needed.
 	 */
 	finalize_initrd();
+
+	/*
+	 * Prevent memblock from allocating high memory.
+	 * This cannot be done before max_low_pfn is detected, so up
+	 * to this point is possible to only reserve physical memory
+	 * with memblock_reserve; memblock_alloc* can be used
+	 * only after this point
+	 */
+	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
 }
 
 #endif	/* CONFIG_SGI_IP27 */
@@ -854,15 +863,6 @@ static void __init arch_mem_init(char **cmdline_p)
 
 	bootmem_init();
 
-	/*
-	 * Prevent memblock from allocating high memory.
-	 * This cannot be done before max_low_pfn is detected, so up
-	 * to this point is possible to only reserve physical memory
-	 * with memblock_reserve; memblock_alloc* can be used
-	 * only after this point
-	 */
-	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
-
 #ifdef CONFIG_PROC_VMCORE
 	if (setup_elfcorehdr && setup_elfcorehdr_size) {
 		printk(KERN_INFO "kdump reserved memory at %lx-%lx\n",
diff --git a/arch/mips/loongson64/loongson-3/numa.c b/arch/mips/loongson64/loongson-3/numa.c
index 622761878cd1..5593a8e3cd88 100644
--- a/arch/mips/loongson64/loongson-3/numa.c
+++ b/arch/mips/loongson64/loongson-3/numa.c
@@ -265,6 +265,7 @@ void __init paging_init(void)
 	zones_size[ZONE_DMA32] = MAX_DMA32_PFN;
 #endif
 	zones_size[ZONE_NORMAL] = max_low_pfn;
+	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
 	free_area_init_nodes(zones_size);
 }
 
diff --git a/arch/mips/sgi-ip27/ip27-memory.c b/arch/mips/sgi-ip27/ip27-memory.c
index d8b8444d6795..0d0deeae1f47 100644
--- a/arch/mips/sgi-ip27/ip27-memory.c
+++ b/arch/mips/sgi-ip27/ip27-memory.c
@@ -468,6 +468,7 @@ void __init paging_init(void)
 			max_low_pfn = end_pfn;
 	}
 	zones_size[ZONE_NORMAL] = max_low_pfn;
+	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
 	free_area_init_nodes(zones_size);
 }
 
-- 
2.13.7

Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[Mike Rapoport]

On Thu, Nov 08, 2018 at 03:44:28PM +0100, Thomas Bogendoerfer wrote:
> Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
> and NUMA enabled loongson3 by doing memblock_set_current_limit()
> before max_low_pfn has been evaluated. Both platforms need to do the
> memblock_set_current_limit() in platform specific code. For
> consistency the call to memblock_set_current_limit() is moved
> to the common bootmem_init(), where max_low_pfn is calculated
> for non NUMA enabled platforms.

As far as I can tell loongsoon3 initially sets max_low_pfn in
node_mem_init(0) and then re-evaluates it in paging_init(). So it seems
that in this case the limit would be wrong for a system with more than one
node, but it would be set to some usable value.

As for SGI-IP27, the initialization of max_low_pfn as late as in
paging_init() seems to be broken because it's value is used in
arch_mem_init() and in finalize_initrd() anyway.

AFAIU, both platforms set max_low_pfn to last available pfn, so it seems we
can simply do

	max_low_pfn = PFN_PHYS(memblock_end_of_DRAM())

in the prom_meminit() function for both platforms and drop the loop
evaluating max_low_pfn in paging_init().
 
> Fixes: bcec54bf3118 ("mips: switch to NO_BOOTMEM")
> Signed-off-by: Thomas Bogendoerfer <tbogendoerfer@suse.de>
> ---
>  arch/mips/kernel/setup.c               | 18 +++++++++---------
>  arch/mips/loongson64/loongson-3/numa.c |  1 +
>  arch/mips/sgi-ip27/ip27-memory.c       |  1 +
>  3 files changed, 11 insertions(+), 9 deletions(-)
> 
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index ea09ed6a80a9..b5167b198231 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -576,6 +576,15 @@ static void __init bootmem_init(void)
>  	 * Reserve initrd memory if needed.
>  	 */
>  	finalize_initrd();
> +
> +	/*
> +	 * Prevent memblock from allocating high memory.
> +	 * This cannot be done before max_low_pfn is detected, so up
> +	 * to this point is possible to only reserve physical memory
> +	 * with memblock_reserve; memblock_alloc* can be used
> +	 * only after this point
> +	 */
> +	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
>  }
> 
>  #endif	/* CONFIG_SGI_IP27 */
> @@ -854,15 +863,6 @@ static void __init arch_mem_init(char **cmdline_p)
> 
>  	bootmem_init();
> 
> -	/*
> -	 * Prevent memblock from allocating high memory.
> -	 * This cannot be done before max_low_pfn is detected, so up
> -	 * to this point is possible to only reserve physical memory
> -	 * with memblock_reserve; memblock_alloc* can be used
> -	 * only after this point
> -	 */
> -	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
> -
>  #ifdef CONFIG_PROC_VMCORE
>  	if (setup_elfcorehdr && setup_elfcorehdr_size) {
>  		printk(KERN_INFO "kdump reserved memory at %lx-%lx\n",
> diff --git a/arch/mips/loongson64/loongson-3/numa.c b/arch/mips/loongson64/loongson-3/numa.c
> index 622761878cd1..5593a8e3cd88 100644
> --- a/arch/mips/loongson64/loongson-3/numa.c
> +++ b/arch/mips/loongson64/loongson-3/numa.c
> @@ -265,6 +265,7 @@ void __init paging_init(void)
>  	zones_size[ZONE_DMA32] = MAX_DMA32_PFN;
>  #endif
>  	zones_size[ZONE_NORMAL] = max_low_pfn;
> +	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
>  	free_area_init_nodes(zones_size);
>  }
> 
> diff --git a/arch/mips/sgi-ip27/ip27-memory.c b/arch/mips/sgi-ip27/ip27-memory.c
> index d8b8444d6795..0d0deeae1f47 100644
> --- a/arch/mips/sgi-ip27/ip27-memory.c
> +++ b/arch/mips/sgi-ip27/ip27-memory.c
> @@ -468,6 +468,7 @@ void __init paging_init(void)
>  			max_low_pfn = end_pfn;
>  	}
>  	zones_size[ZONE_NORMAL] = max_low_pfn;
> +	memblock_set_current_limit(PFN_PHYS(max_low_pfn));
>  	free_area_init_nodes(zones_size);
>  }
> 
> -- 
> 2.13.7
> 

-- 
Sincerely yours,
Mike.

Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[Thomas Bogendoerfer]

On Thu, 8 Nov 2018 18:18:23 +0200
Mike Rapoport <rppt@linux.ibm.com> wrote:

> On Thu, Nov 08, 2018 at 03:44:28PM +0100, Thomas Bogendoerfer wrote:
> > Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
> > and NUMA enabled loongson3 by doing memblock_set_current_limit()
> > before max_low_pfn has been evaluated. Both platforms need to do the
> > memblock_set_current_limit() in platform specific code. For
> > consistency the call to memblock_set_current_limit() is moved
> > to the common bootmem_init(), where max_low_pfn is calculated
> > for non NUMA enabled platforms.
> [..]
> 
> As for SGI-IP27, the initialization of max_low_pfn as late as in
> paging_init() seems to be broken because it's value is used in
> arch_mem_init() and in finalize_initrd() anyway.

well, the patch is tested on real hardware and the first caller of
a memblock_alloc* function is in a function called by free_area_init_nodes().

> AFAIU, both platforms set max_low_pfn to last available pfn, so it seems we
> can simply do
> 
> 	max_low_pfn = PFN_PHYS(memblock_end_of_DRAM())
> 
> in the prom_meminit() function for both platforms and drop the loop
> evaluating max_low_pfn in paging_init().

sounds like a better plan. I'll prepare a new patch.

Thomas.

-- 
SUSE Linux GmbH
GF: Felix Imendörffer, Jane Smithard, Graham Norton
HRB 21284 (AG Nürnberg)

Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[Mike Rapoport]

On November 8, 2018 6:52:17 PM GMT+02:00, Thomas Bogendoerfer <tbogendoerfer@suse.de> wrote:
>On Thu, 8 Nov 2018 18:18:23 +0200
>Mike Rapoport <rppt@linux.ibm.com> wrote:
>
>> On Thu, Nov 08, 2018 at 03:44:28PM +0100, Thomas Bogendoerfer wrote:
>> > Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
>> > and NUMA enabled loongson3 by doing memblock_set_current_limit()
>> > before max_low_pfn has been evaluated. Both platforms need to do
>the
>> > memblock_set_current_limit() in platform specific code. For
>> > consistency the call to memblock_set_current_limit() is moved
>> > to the common bootmem_init(), where max_low_pfn is calculated
>> > for non NUMA enabled platforms.
>> [..]
>> 
>> As for SGI-IP27, the initialization of max_low_pfn as late as in
>> paging_init() seems to be broken because it's value is used in
>> arch_mem_init() and in finalize_initrd() anyway.
>
>well, the patch is tested on real hardware and the first caller of
>a memblock_alloc* function is in a function called by
>free_area_init_nodes().
 
Then, apparently, I've missed something else.
The Onyx2 I worked on is dead for a couple of years now ;-)

>> AFAIU, both platforms set max_low_pfn to last available pfn, so it
>seems we
>> can simply do
>> 
>> 	max_low_pfn = PFN_PHYS(memblock_end_of_DRAM())
>>

Should have been PHYS_PFN, sorry.

>> in the prom_meminit() function for both platforms and drop the loop
>> evaluating max_low_pfn in paging_init().
>
>sounds like a better plan. I'll prepare a new patch.
>
>Thomas.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[陈华才]

Hi,

It seems the patch below can solve many problems after switched to NO_BOOTMEM, because the memory allocation behavior is more similar as before.

diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
index 070234b..7a449d9 100644
--- a/arch/mips/kernel/setup.c
+++ b/arch/mips/kernel/setup.c
@@ -839,6 +839,7 @@ static void __init arch_mem_init(char **cmdline_p)

        /* call board setup routine */
        plat_mem_setup();
+       memblock_set_bottom_up(true);

        /*
         * Make sure all kernel memory is in the maps.  The "UP" and
diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
index 0f852e1..15e103c 100644
--- a/arch/mips/kernel/traps.c
+++ b/arch/mips/kernel/traps.c
@@ -2260,10 +2260,8 @@ void __init trap_init(void)
                unsigned long size = 0x200 + VECTORSPACING*64;
                phys_addr_t ebase_pa;

-               memblock_set_bottom_up(true);
                ebase = (unsigned long)
                        memblock_alloc_from(size, 1 << fls(size), 0);
-               memblock_set_bottom_up(false);

                /*
                 * Try to ensure ebase resides in KSeg0 if possible.
@@ -2307,6 +2305,7 @@ void __init trap_init(void)
        if (board_ebase_setup)
                board_ebase_setup();
        per_cpu_trap_init(true);
+       memblock_set_bottom_up(false);

        /*
         * Copy the generic exception handlers to their final destination.

 
------------------ Original ------------------
From:  "Mike Rapoport"<rppt@linux.ibm.com>;
Date:  Fri, Nov 9, 2018 02:01 AM
To:  "Thomas Bogendoerfer"<tbogendoerfer@suse.de>;
Cc:  "Ralf Baechle"<ralf@linux-mips.org>; "Paul Burton"<paul.burton@mips.com>; "James Hogan"<jhogan@kernel.org>; "Huacai Chen"<chenhc@lemote.com>; "linux-mips"<linux-mips@linux-mips.org>; "linux-kernel"<linux-kernel@vger.kernel.org>; "rppt"<rppt@linux.vnet.ibm.com>;
Subject:  Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA
 


On November 8, 2018 6:52:17 PM GMT+02:00, Thomas Bogendoerfer <tbogendoerfer@suse.de> wrote:
>On Thu, 8 Nov 2018 18:18:23 +0200
>Mike Rapoport <rppt@linux.ibm.com> wrote:
>
>> On Thu, Nov 08, 2018 at 03:44:28PM +0100, Thomas Bogendoerfer wrote:
>> > Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
>> > and NUMA enabled loongson3 by doing memblock_set_current_limit()
>> > before max_low_pfn has been evaluated. Both platforms need to do
>the
>> > memblock_set_current_limit() in platform specific code. For
>> > consistency the call to memblock_set_current_limit() is moved
>> > to the common bootmem_init(), where max_low_pfn is calculated
>> > for non NUMA enabled platforms.
>> [..]
>> 
>> As for SGI-IP27, the initialization of max_low_pfn as late as in
>> paging_init() seems to be broken because it's value is used in
>> arch_mem_init() and in finalize_initrd() anyway.
>
>well, the patch is tested on real hardware and the first caller of
>a memblock_alloc* function is in a function called by
>free_area_init_nodes().
 
Then, apparently, I've missed something else.
The Onyx2 I worked on is dead for a couple of years now ;-)

>> AFAIU, both platforms set max_low_pfn to last available pfn, so it
>seems we
>> can simply do
>> 
>> max_low_pfn = PFN_PHYS(memblock_end_of_DRAM())
>>

Should have been PHYS_PFN, sorry.

>> in the prom_meminit() function for both platforms and drop the loop
>> evaluating max_low_pfn in paging_init().
>
>sounds like a better plan. I'll prepare a new patch.
>
>Thomas.

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA

[Mike Rapoport]

On Fri, Nov 09, 2018 at 10:54:26AM +0800, 陈华才 wrote:
> Hi,
> 
> It seems the patch below can solve many problems after switched to NO_BOOTMEM, because the memory allocation behavior is more similar as before.

Yes, this should work.
Still, simplifying the max_low_pfn evaluation has it's value regardless.
 
> diff --git a/arch/mips/kernel/setup.c b/arch/mips/kernel/setup.c
> index 070234b..7a449d9 100644
> --- a/arch/mips/kernel/setup.c
> +++ b/arch/mips/kernel/setup.c
> @@ -839,6 +839,7 @@ static void __init arch_mem_init(char **cmdline_p)
> 
>         /* call board setup routine */
>         plat_mem_setup();
> +       memblock_set_bottom_up(true);
> 
>         /*
>          * Make sure all kernel memory is in the maps.  The "UP" and
> diff --git a/arch/mips/kernel/traps.c b/arch/mips/kernel/traps.c
> index 0f852e1..15e103c 100644
> --- a/arch/mips/kernel/traps.c
> +++ b/arch/mips/kernel/traps.c
> @@ -2260,10 +2260,8 @@ void __init trap_init(void)
>                 unsigned long size = 0x200 + VECTORSPACING*64;
>                 phys_addr_t ebase_pa;
> 
> -               memblock_set_bottom_up(true);
>                 ebase = (unsigned long)
>                         memblock_alloc_from(size, 1 << fls(size), 0);
> -               memblock_set_bottom_up(false);
> 
>                 /*
>                  * Try to ensure ebase resides in KSeg0 if possible.
> @@ -2307,6 +2305,7 @@ void __init trap_init(void)
>         if (board_ebase_setup)
>                 board_ebase_setup();
>         per_cpu_trap_init(true);
> +       memblock_set_bottom_up(false);
> 
>         /*
>          * Copy the generic exception handlers to their final destination.
> 
>  
> ------------------ Original ------------------
> From:  "Mike Rapoport"<rppt@linux.ibm.com>;
> Date:  Fri, Nov 9, 2018 02:01 AM
> To:  "Thomas Bogendoerfer"<tbogendoerfer@suse.de>;
> Cc:  "Ralf Baechle"<ralf@linux-mips.org>; "Paul Burton"<paul.burton@mips.com>; "James Hogan"<jhogan@kernel.org>; "Huacai Chen"<chenhc@lemote.com>; "linux-mips"<linux-mips@linux-mips.org>; "linux-kernel"<linux-kernel@vger.kernel.org>; "rppt"<rppt@linux.vnet.ibm.com>;
> Subject:  Re: [[PATCH]] mips: Fix switch to NO_BOOTMEM for SGI-IP27/loongons3 NUMA
>  
> 
> 
> On November 8, 2018 6:52:17 PM GMT+02:00, Thomas Bogendoerfer <tbogendoerfer@suse.de> wrote:
> >On Thu, 8 Nov 2018 18:18:23 +0200
> >Mike Rapoport <rppt@linux.ibm.com> wrote:
> >
> >> On Thu, Nov 08, 2018 at 03:44:28PM +0100, Thomas Bogendoerfer wrote:
> >> > Commit bcec54bf3118 ("mips: switch to NO_BOOTMEM") broke SGI-IP27
> >> > and NUMA enabled loongson3 by doing memblock_set_current_limit()
> >> > before max_low_pfn has been evaluated. Both platforms need to do
> >the
> >> > memblock_set_current_limit() in platform specific code. For
> >> > consistency the call to memblock_set_current_limit() is moved
> >> > to the common bootmem_init(), where max_low_pfn is calculated
> >> > for non NUMA enabled platforms.
> >> [..]
> >> 
> >> As for SGI-IP27, the initialization of max_low_pfn as late as in
> >> paging_init() seems to be broken because it's value is used in
> >> arch_mem_init() and in finalize_initrd() anyway.
> >
> >well, the patch is tested on real hardware and the first caller of
> >a memblock_alloc* function is in a function called by
> >free_area_init_nodes().
>  
> Then, apparently, I've missed something else.
> The Onyx2 I worked on is dead for a couple of years now ;-)
> 
> >> AFAIU, both platforms set max_low_pfn to last available pfn, so it
> >seems we
> >> can simply do
> >> 
> >> max_low_pfn = PFN_PHYS(memblock_end_of_DRAM())
> >>
> 
> Should have been PHYS_PFN, sorry.
> 
> >> in the prom_meminit() function for both platforms and drop the loop
> >> evaluating max_low_pfn in paging_init().
> >
> >sounds like a better plan. I'll prepare a new patch.
> >
> >Thomas.
> 
> -- 
> Sent from my Android device with K-9 Mail. Please excuse my brevity.

-- 
Sincerely yours,
Mike.