[PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Barry Song]

hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
done yet. so all reserved memory will be located at node0.

Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma")
Cc: Matthias Brugger <matthias.bgg@gmail.com>
Acked-by: Roman Gushchin <guro@fb.com>
Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
---
 -v2: add Fixes tag according to Matthias Brugger's comment

 arch/arm64/mm/init.c | 10 +++++-----
 1 file changed, 5 insertions(+), 5 deletions(-)

diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index e631e6425165..41914b483d54 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
 	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
 
 	dma_contiguous_reserve(arm64_dma32_phys_limit);
-
-#ifdef CONFIG_ARM64_4K_PAGES
-	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
-#endif
-
 }
 
 void __init bootmem_init(void)
@@ -424,6 +419,11 @@ void __init bootmem_init(void)
 	min_low_pfn = min;
 
 	arm64_numa_init();
+
+#ifdef CONFIG_ARM64_4K_PAGES
+	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
+#endif
+
 	/*
 	 * Sparsemem tries to allocate bootmem in memory_present(), so must be
 	 * done after the fixed reservations.
-- 
2.23.0

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Anshuman Khandual]

On 06/17/2020 03:49 AM, Barry Song wrote:
> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
> done yet. so all reserved memory will be located at node0.
> 
> Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma")
> Cc: Matthias Brugger <matthias.bgg@gmail.com>
> Acked-by: Roman Gushchin <guro@fb.com>
> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> ---
>  -v2: add Fixes tag according to Matthias Brugger's comment
> 
>  arch/arm64/mm/init.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index e631e6425165..41914b483d54 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
>  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
>  
>  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> -
> -#ifdef CONFIG_ARM64_4K_PAGES
> -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> -#endif
> -
>  }
>  
>  void __init bootmem_init(void)
> @@ -424,6 +419,11 @@ void __init bootmem_init(void)
>  	min_low_pfn = min;
>  
>  	arm64_numa_init();
> +
> +#ifdef CONFIG_ARM64_4K_PAGES
> +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> +#endif

arm64_numa_init() calls numa_init() which initializes node_online_map
that gets used in hugetlb_cma_reserve() while allocating required CMA
size across online nodes.

Reviewed-by: Anshuman Khandual <anshuman.khandual@arm.com>

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Matthias Brugger]

On 17/06/2020 00:19, Barry Song wrote:
> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
> done yet. so all reserved memory will be located at node0.
> 
> Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma")
> Cc: Matthias Brugger <matthias.bgg@gmail.com>
> Acked-by: Roman Gushchin <guro@fb.com>
> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>

Reviewed-by: Matthias Brugger <matthias.bgg@gmail.com>

> ---
>  -v2: add Fixes tag according to Matthias Brugger's comment
> 
>  arch/arm64/mm/init.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index e631e6425165..41914b483d54 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
>  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
>  
>  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> -
> -#ifdef CONFIG_ARM64_4K_PAGES
> -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> -#endif
> -
>  }
>  
>  void __init bootmem_init(void)
> @@ -424,6 +419,11 @@ void __init bootmem_init(void)
>  	min_low_pfn = min;
>  
>  	arm64_numa_init();
> +
> +#ifdef CONFIG_ARM64_4K_PAGES
> +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> +#endif
> +
>  	/*
>  	 * Sparsemem tries to allocate bootmem in memory_present(), so must be
>  	 * done after the fixed reservations.
> 

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Will Deacon]

On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
> done yet. so all reserved memory will be located at node0.
> 
> Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages using cma")

Damn, wasn't CC'd on that :/

> Cc: Matthias Brugger <matthias.bgg@gmail.com>
> Acked-by: Roman Gushchin <guro@fb.com>
> Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> ---
>  -v2: add Fixes tag according to Matthias Brugger's comment
> 
>  arch/arm64/mm/init.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
> 
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index e631e6425165..41914b483d54 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
>  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
>  
>  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> -
> -#ifdef CONFIG_ARM64_4K_PAGES
> -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> -#endif

Why is this dependent on CONFIG_ARM64_4K_PAGES? We unconditionally
select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.

> -
>  }
>  
>  void __init bootmem_init(void)
> @@ -424,6 +419,11 @@ void __init bootmem_init(void)
>  	min_low_pfn = min;
>  
>  	arm64_numa_init();
> +
> +#ifdef CONFIG_ARM64_4K_PAGES
> +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> +#endif

A comment here wouldn't hurt, as it does look a lot more natural next
to dma_contiguous_reserve().

Will

RE: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Will Deacon [mailto:will@kernel.org]
> Sent: Wednesday, June 17, 2020 10:18 PM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> Cc: catalin.marinas@arm.com; nsaenzjulienne@suse.de;
> steve.capper@arm.com; rppt@linux.ibm.com; akpm@linux-foundation.org;
> linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Linuxarm
> <linuxarm@huawei.com>; Matthias Brugger <matthias.bgg@gmail.com>;
> Roman Gushchin <guro@fb.com>
> Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> 
> On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
> > done yet. so all reserved memory will be located at node0.
> >
> > Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages
> using cma")
> 
> Damn, wasn't CC'd on that :/
> 
> > Cc: Matthias Brugger <matthias.bgg@gmail.com>
> > Acked-by: Roman Gushchin <guro@fb.com>
> > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> > ---
> >  -v2: add Fixes tag according to Matthias Brugger's comment
> >
> >  arch/arm64/mm/init.c | 10 +++++-----
> >  1 file changed, 5 insertions(+), 5 deletions(-)
> >
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index e631e6425165..41914b483d54 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> >
> >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > -
> > -#ifdef CONFIG_ARM64_4K_PAGES
> > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > -#endif
> 
> Why is this dependent on CONFIG_ARM64_4K_PAGES? We unconditionally
> select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.

Roman, would you like to answer this question? Have you found any problem if system
doesn't set 4K_PAGES?

> 
> > -
> >  }
> >
> >  void __init bootmem_init(void)
> > @@ -424,6 +419,11 @@ void __init bootmem_init(void)
> >  	min_low_pfn = min;
> >
> >  	arm64_numa_init();
> > +
> > +#ifdef CONFIG_ARM64_4K_PAGES
> > +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > +#endif
> 
> A comment here wouldn't hurt, as it does look a lot more natural next
> to dma_contiguous_reserve().

Will add some comment here.

> 
> Will

barry

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Roman Gushchin]

On Wed, Jun 17, 2020 at 11:38:03AM +0000, Song Bao Hua (Barry Song) wrote:
> 
> 
> > -----Original Message-----
> > From: Will Deacon [mailto:will@kernel.org]
> > Sent: Wednesday, June 17, 2020 10:18 PM
> > To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> > Cc: catalin.marinas@arm.com; nsaenzjulienne@suse.de;
> > steve.capper@arm.com; rppt@linux.ibm.com; akpm@linux-foundation.org;
> > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; Linuxarm
> > <linuxarm@huawei.com>; Matthias Brugger <matthias.bgg@gmail.com>;
> > Roman Gushchin <guro@fb.com>
> > Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> > 
> > On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > > hugetlb_cma_reserve() is called at the wrong place. numa_init has not been
> > > done yet. so all reserved memory will be located at node0.
> > >
> > > Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic hugepages
> > using cma")
> > 
> > Damn, wasn't CC'd on that :/
> > 
> > > Cc: Matthias Brugger <matthias.bgg@gmail.com>
> > > Acked-by: Roman Gushchin <guro@fb.com>
> > > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> > > ---
> > >  -v2: add Fixes tag according to Matthias Brugger's comment
> > >
> > >  arch/arm64/mm/init.c | 10 +++++-----
> > >  1 file changed, 5 insertions(+), 5 deletions(-)
> > >
> > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > index e631e6425165..41914b483d54 100644
> > > --- a/arch/arm64/mm/init.c
> > > +++ b/arch/arm64/mm/init.c
> > > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> > >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> > >
> > >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > > -
> > > -#ifdef CONFIG_ARM64_4K_PAGES
> > > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > -#endif
> > 
> > Why is this dependent on CONFIG_ARM64_4K_PAGES? We unconditionally
> > select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.
> 
> Roman, would you like to answer this question? Have you found any problem if system
> doesn't set 4K_PAGES?

No, I was just following the code in arch/arm64/mm/hugetlbpage.c where all
related to PUD-sized pages is guarded by CONFIG_ARM64_4K_PAGES.
Actually I did all my testing on x86-64, I don't even have any arm hardware.

I'm totally fine with removing this #ifdef if it's not needed.

Thanks!

> 
> > 
> > > -
> > >  }
> > >
> > >  void __init bootmem_init(void)
> > > @@ -424,6 +419,11 @@ void __init bootmem_init(void)
> > >  	min_low_pfn = min;
> > >
> > >  	arm64_numa_init();
> > > +
> > > +#ifdef CONFIG_ARM64_4K_PAGES
> > > +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > +#endif
> > 
> > A comment here wouldn't hurt, as it does look a lot more natural next
> > to dma_contiguous_reserve().
> 
> Will add some comment here.
> 
> > 
> > Will
> 
> barry

RE: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Roman Gushchin [mailto:guro@fb.com]
> Sent: Thursday, June 18, 2020 6:20 AM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> Cc: Will Deacon <will@kernel.org>; catalin.marinas@arm.com;
> nsaenzjulienne@suse.de; steve.capper@arm.com; rppt@linux.ibm.com;
> akpm@linux-foundation.org; linux-arm-kernel@lists.infradead.org;
> linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; Matthias
> Brugger <matthias.bgg@gmail.com>
> Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> 
> On Wed, Jun 17, 2020 at 11:38:03AM +0000, Song Bao Hua (Barry Song)
> wrote:
> >
> >
> > > -----Original Message-----
> > > From: Will Deacon [mailto:will@kernel.org]
> > > Sent: Wednesday, June 17, 2020 10:18 PM
> > > To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> > > Cc: catalin.marinas@arm.com; nsaenzjulienne@suse.de;
> > > steve.capper@arm.com; rppt@linux.ibm.com;
> akpm@linux-foundation.org;
> > > linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org;
> Linuxarm
> > > <linuxarm@huawei.com>; Matthias Brugger <matthias.bgg@gmail.com>;
> > > Roman Gushchin <guro@fb.com>
> > > Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> > >
> > > On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > > > hugetlb_cma_reserve() is called at the wrong place. numa_init has not
> been
> > > > done yet. so all reserved memory will be located at node0.
> > > >
> > > > Fixes: cf11e85fc08c ("mm: hugetlb: optionally allocate gigantic
> hugepages
> > > using cma")
> > >
> > > Damn, wasn't CC'd on that :/
> > >
> > > > Cc: Matthias Brugger <matthias.bgg@gmail.com>
> > > > Acked-by: Roman Gushchin <guro@fb.com>
> > > > Signed-off-by: Barry Song <song.bao.hua@hisilicon.com>
> > > > ---
> > > >  -v2: add Fixes tag according to Matthias Brugger's comment
> > > >
> > > >  arch/arm64/mm/init.c | 10 +++++-----
> > > >  1 file changed, 5 insertions(+), 5 deletions(-)
> > > >
> > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > > index e631e6425165..41914b483d54 100644
> > > > --- a/arch/arm64/mm/init.c
> > > > +++ b/arch/arm64/mm/init.c
> > > > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> > > >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> > > >
> > > >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > > > -
> > > > -#ifdef CONFIG_ARM64_4K_PAGES
> > > > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > > -#endif
> > >
> > > Why is this dependent on CONFIG_ARM64_4K_PAGES? We unconditionally
> > > select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.
> >
> > Roman, would you like to answer this question? Have you found any
> problem if system
> > doesn't set 4K_PAGES?
> 
> No, I was just following the code in arch/arm64/mm/hugetlbpage.c where all
> related to PUD-sized pages is guarded by CONFIG_ARM64_4K_PAGES.
> Actually I did all my testing on x86-64, I don't even have any arm hardware.
> 
> I'm totally fine with removing this #ifdef if it's not needed.

At this moment, I would suggest we should keep this "ifdef". Otherwise, hugetlb_cma_reserve() won't be really useful.

For example, while setting PAGE size to 64KB. I got this error in hugetlb_cma_reserve():
hugetlb_cma: cma area should be at least 4194304 MiB
This is absolutely unreasonable.

Supporting hugetlb_cma_reserve() for page sizes other than 4k is a different issue. 
It might be addressed in a separate patch later.

> 
> Thanks!
> 
> >
> > >
> > > > -
> > > >  }
> > > >
> > > >  void __init bootmem_init(void)
> > > > @@ -424,6 +419,11 @@ void __init bootmem_init(void)
> > > >  	min_low_pfn = min;
> > > >
> > > >  	arm64_numa_init();
> > > > +
> > > > +#ifdef CONFIG_ARM64_4K_PAGES
> > > > +	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > > +#endif
> > >
> > > A comment here wouldn't hurt, as it does look a lot more natural next
> > > to dma_contiguous_reserve().
> >
> > Will add some comment here.
> >
> > >
> > > Will
> >
> > barry

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Will Deacon]

On Wed, Jun 17, 2020 at 09:43:51PM +0000, Song Bao Hua (Barry Song) wrote:
> > From: Roman Gushchin [mailto:guro@fb.com]
> > On Wed, Jun 17, 2020 at 11:38:03AM +0000, Song Bao Hua (Barry Song)
> > > > From: Will Deacon [mailto:will@kernel.org]
> > > > On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > > > > hugetlb_cma_reserve() is called at the wrong place. numa_init has not
> > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > > > index e631e6425165..41914b483d54 100644
> > > > > --- a/arch/arm64/mm/init.c
> > > > > +++ b/arch/arm64/mm/init.c
> > > > > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> > > > >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> > > > >
> > > > >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > > > > -
> > > > > -#ifdef CONFIG_ARM64_4K_PAGES
> > > > > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > > > -#endif
> > > >
> > > > Why is this dependent on CONFIG_ARM64_4K_PAGES? We unconditionally
> > > > select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.
> > >
> > > Roman, would you like to answer this question? Have you found any
> > problem if system
> > > doesn't set 4K_PAGES?
> > 
> > No, I was just following the code in arch/arm64/mm/hugetlbpage.c where all
> > related to PUD-sized pages is guarded by CONFIG_ARM64_4K_PAGES.
> > Actually I did all my testing on x86-64, I don't even have any arm hardware.
> > 
> > I'm totally fine with removing this #ifdef if it's not needed.
> 
> At this moment, I would suggest we should keep this "ifdef". Otherwise, hugetlb_cma_reserve() won't be really useful.
> 
> For example, while setting PAGE size to 64KB. I got this error in hugetlb_cma_reserve():
> hugetlb_cma: cma area should be at least 4194304 MiB
> This is absolutely unreasonable.

Maybe one for RaspberryPi 5, huh? ;)

But ok, I'll take your patch as-is and add a comment about NUMA.

Thanks,

Will

RE: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Will Deacon [mailto:will@kernel.org]
> Sent: Thursday, June 18, 2020 7:20 PM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> Cc: Roman Gushchin <guro@fb.com>; catalin.marinas@arm.com;
> nsaenzjulienne@suse.de; steve.capper@arm.com; rppt@linux.ibm.com;
> akpm@linux-foundation.org; linux-arm-kernel@lists.infradead.org;
> linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; Matthias
> Brugger <matthias.bgg@gmail.com>
> Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> 
> On Wed, Jun 17, 2020 at 09:43:51PM +0000, Song Bao Hua (Barry Song)
> wrote:
> > > From: Roman Gushchin [mailto:guro@fb.com]
> > > On Wed, Jun 17, 2020 at 11:38:03AM +0000, Song Bao Hua (Barry Song)
> > > > > From: Will Deacon [mailto:will@kernel.org]
> > > > > On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > > > > > hugetlb_cma_reserve() is called at the wrong place. numa_init has not
> > > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > > > > index e631e6425165..41914b483d54 100644
> > > > > > --- a/arch/arm64/mm/init.c
> > > > > > +++ b/arch/arm64/mm/init.c
> > > > > > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> > > > > >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> > > > > >
> > > > > >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > > > > > -
> > > > > > -#ifdef CONFIG_ARM64_4K_PAGES
> > > > > > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > > > > -#endif
> > > > >
> > > > > Why is this dependent on CONFIG_ARM64_4K_PAGES? We
> unconditionally
> > > > > select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.
> > > >
> > > > Roman, would you like to answer this question? Have you found any
> > > problem if system
> > > > doesn't set 4K_PAGES?
> > >
> > > No, I was just following the code in arch/arm64/mm/hugetlbpage.c where
> all
> > > related to PUD-sized pages is guarded by CONFIG_ARM64_4K_PAGES.
> > > Actually I did all my testing on x86-64, I don't even have any arm
> hardware.
> > >
> > > I'm totally fine with removing this #ifdef if it's not needed.
> >
> > At this moment, I would suggest we should keep this "ifdef". Otherwise,
> hugetlb_cma_reserve() won't be really useful.
> >
> > For example, while setting PAGE size to 64KB. I got this error in
> hugetlb_cma_reserve():
> > hugetlb_cma: cma area should be at least 4194304 MiB
> > This is absolutely unreasonable.
> 
> Maybe one for RaspberryPi 5, huh? ;)
> 
> But ok, I'll take your patch as-is and add a comment about NUMA.

Have you seen the v3 with comment? I've already sent.

> 
> Thanks,
> 
> Will

Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init

[Will Deacon]

On Thu, Jun 18, 2020 at 07:43:43AM +0000, Song Bao Hua (Barry Song) wrote:
> 
> 
> > -----Original Message-----
> > From: Will Deacon [mailto:will@kernel.org]
> > Sent: Thursday, June 18, 2020 7:20 PM
> > To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>
> > Cc: Roman Gushchin <guro@fb.com>; catalin.marinas@arm.com;
> > nsaenzjulienne@suse.de; steve.capper@arm.com; rppt@linux.ibm.com;
> > akpm@linux-foundation.org; linux-arm-kernel@lists.infradead.org;
> > linux-kernel@vger.kernel.org; Linuxarm <linuxarm@huawei.com>; Matthias
> > Brugger <matthias.bgg@gmail.com>
> > Subject: Re: [PATCH v2] arm64: mm: reserve hugetlb CMA after numa_init
> > 
> > On Wed, Jun 17, 2020 at 09:43:51PM +0000, Song Bao Hua (Barry Song)
> > wrote:
> > > > From: Roman Gushchin [mailto:guro@fb.com]
> > > > On Wed, Jun 17, 2020 at 11:38:03AM +0000, Song Bao Hua (Barry Song)
> > > > > > From: Will Deacon [mailto:will@kernel.org]
> > > > > > On Wed, Jun 17, 2020 at 10:19:24AM +1200, Barry Song wrote:
> > > > > > > hugetlb_cma_reserve() is called at the wrong place. numa_init has not
> > > > > > > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > > > > > > index e631e6425165..41914b483d54 100644
> > > > > > > --- a/arch/arm64/mm/init.c
> > > > > > > +++ b/arch/arm64/mm/init.c
> > > > > > > @@ -404,11 +404,6 @@ void __init arm64_memblock_init(void)
> > > > > > >  	high_memory = __va(memblock_end_of_DRAM() - 1) + 1;
> > > > > > >
> > > > > > >  	dma_contiguous_reserve(arm64_dma32_phys_limit);
> > > > > > > -
> > > > > > > -#ifdef CONFIG_ARM64_4K_PAGES
> > > > > > > -	hugetlb_cma_reserve(PUD_SHIFT - PAGE_SHIFT);
> > > > > > > -#endif
> > > > > >
> > > > > > Why is this dependent on CONFIG_ARM64_4K_PAGES? We
> > unconditionally
> > > > > > select ARCH_HAS_GIGANTIC_PAGE so this seems unnecessary.
> > > > >
> > > > > Roman, would you like to answer this question? Have you found any
> > > > problem if system
> > > > > doesn't set 4K_PAGES?
> > > >
> > > > No, I was just following the code in arch/arm64/mm/hugetlbpage.c where
> > all
> > > > related to PUD-sized pages is guarded by CONFIG_ARM64_4K_PAGES.
> > > > Actually I did all my testing on x86-64, I don't even have any arm
> > hardware.
> > > >
> > > > I'm totally fine with removing this #ifdef if it's not needed.
> > >
> > > At this moment, I would suggest we should keep this "ifdef". Otherwise,
> > hugetlb_cma_reserve() won't be really useful.
> > >
> > > For example, while setting PAGE size to 64KB. I got this error in
> > hugetlb_cma_reserve():
> > > hugetlb_cma: cma area should be at least 4194304 MiB
> > > This is absolutely unreasonable.
> > 
> > Maybe one for RaspberryPi 5, huh? ;)
> > 
> > But ok, I'll take your patch as-is and add a comment about NUMA.
> 
> Have you seen the v3 with comment? I've already sent.

Thanks, just saw that (I'm going through morning email atm :)

Will