[PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Wei Li]

For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
do not free the reserved memory for the page map, decrease the section
size can reduce the waste of reserved memory.

Signed-off-by: Wei Li <liwei213@huawei.com>
Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
---
 arch/arm64/include/asm/sparsemem.h | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
index 1f43fcc79738..8963bd3def28 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -7,7 +7,7 @@

 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
+#define SECTION_SIZE_BITS	27
 #endif

 #endif
--
2.15.0

RE: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: liwei (CM)
> Sent: Friday, December 4, 2020 2:45 PM
> To: catalin.marinas@arm.com; rppt@linux.ibm.com; will@kernel.org; liwei (CM)
> <liwei213@huawei.com>
> Cc: fengbaopeng <fengbaopeng2@hisilicon.com>; nsaenzjulienne@suse.de;
> steve.capper@arm.com; Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>;
> linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; butao
> <butao@hisilicon.com>
> Subject: [PATCH] arm64: mm: decrease the section size to reduce the memory
> reserved for the page map
> 
> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> do not free the reserved memory for the page map, decrease the section
> size can reduce the waste of reserved memory.
> 
> Signed-off-by: Wei Li <liwei213@huawei.com>
> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> ---

Reviewed-by: Barry Song <song.bao.hua@hisilicon.com>

When page size is 4K, for each 1GB memory, we need 16MB vmemmap;
For each 128MB memory, we need 2MB vmemmap.

So while we have memory hole like 928MB(1GB-64MB),if SECTION_SIZE_BITS
is 30, we waste 15MB; if SECTION_SIZE_BITS is 27, we waste 1MB only.

>  arch/arm64/include/asm/sparsemem.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h
> b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..8963bd3def28 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,7 @@
> 
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> +#define SECTION_SIZE_BITS	27
>  #endif
> 
>  #endif
> --
> 2.15.0

Thanks
Barry

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Will Deacon]

On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> do not free the reserved memory for the page map, decrease the section
> size can reduce the waste of reserved memory.
> 
> Signed-off-by: Wei Li <liwei213@huawei.com>
> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> ---
>  arch/arm64/include/asm/sparsemem.h | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..8963bd3def28 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,7 @@
> 
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> +#define SECTION_SIZE_BITS	27

We chose '30' to avoid running out of bits in the page flags. What changed?

With this patch, I can trigger:

./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
#error Allocator MAX_ORDER exceeds SECTION_SIZE

if I bump up NR_CPUS and NODES_SHIFT.

Will

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Mike Rapoport]

On Fri, Dec 04, 2020 at 11:13:47AM +0000, Will Deacon wrote:
> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > do not free the reserved memory for the page map, decrease the section
> > size can reduce the waste of reserved memory.
> > 
> > Signed-off-by: Wei Li <liwei213@huawei.com>
> > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > ---
> >  arch/arm64/include/asm/sparsemem.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> > index 1f43fcc79738..8963bd3def28 100644
> > --- a/arch/arm64/include/asm/sparsemem.h
> > +++ b/arch/arm64/include/asm/sparsemem.h
> > @@ -7,7 +7,7 @@
> > 
> >  #ifdef CONFIG_SPARSEMEM
> >  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> > -#define SECTION_SIZE_BITS	30
> > +#define SECTION_SIZE_BITS	27
> 
> We chose '30' to avoid running out of bits in the page flags. What changed?

I think that for 64-bit there are still plenty of free bits. I didn't
check now, but when I played with SPARSEMEM on m68k there were 8 bits
for section out of 32.

> With this patch, I can trigger:
> 
> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
> #error Allocator MAX_ORDER exceeds SECTION_SIZE
> 
> if I bump up NR_CPUS and NODES_SHIFT.

I don't think it's related to NR_CPUS and NODES_SHIFT. 
This seems rather 64K pages that cause this.

Not that is shouldn't be addressed.

> Will

-- 
Sincerely yours,
Mike.

RE: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Mike Rapoport [mailto:rppt@linux.ibm.com]
> Sent: Saturday, December 5, 2020 12:44 AM
> To: Will Deacon <will@kernel.org>
> Cc: liwei (CM) <liwei213@huawei.com>; catalin.marinas@arm.com; fengbaopeng
> <fengbaopeng2@hisilicon.com>; nsaenzjulienne@suse.de; steve.capper@arm.com;
> Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>;
> linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; butao
> <butao@hisilicon.com>
> Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory
> reserved for the page map
> 
> On Fri, Dec 04, 2020 at 11:13:47AM +0000, Will Deacon wrote:
> > On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > > do not free the reserved memory for the page map, decrease the section
> > > size can reduce the waste of reserved memory.
> > >
> > > Signed-off-by: Wei Li <liwei213@huawei.com>
> > > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > > ---
> > >  arch/arm64/include/asm/sparsemem.h | 2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > >
> > > diff --git a/arch/arm64/include/asm/sparsemem.h
> b/arch/arm64/include/asm/sparsemem.h
> > > index 1f43fcc79738..8963bd3def28 100644
> > > --- a/arch/arm64/include/asm/sparsemem.h
> > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > @@ -7,7 +7,7 @@
> > >
> > >  #ifdef CONFIG_SPARSEMEM
> > >  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> > > -#define SECTION_SIZE_BITS	30
> > > +#define SECTION_SIZE_BITS	27
> >
> > We chose '30' to avoid running out of bits in the page flags. What changed?
> 
> I think that for 64-bit there are still plenty of free bits. I didn't
> check now, but when I played with SPARSEMEM on m68k there were 8 bits
> for section out of 32.
> 
> > With this patch, I can trigger:
> >
> > ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
> SECTION_SIZE
> > #error Allocator MAX_ORDER exceeds SECTION_SIZE
> >
> > if I bump up NR_CPUS and NODES_SHIFT.
> 
> I don't think it's related to NR_CPUS and NODES_SHIFT.
> This seems rather 64K pages that cause this.
> 
> Not that is shouldn't be addressed.

Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS.
Other cases will use vmemmap_populate_basepages().
The original patch should be only addressing the issue in 4K pages:
https://lore.kernel.org/lkml/20200812010655.96339-1-liwei213@huawei.com/

would we do something like the below?
#ifdef CONFIG_ARM64_4K_PAGE
#define SECTION_SIZE_BITS	27
#else
#define SECTION_SIZE_BITS	30
#endif

> 
> > Will
> 
> --
> Sincerely yours,
> Mike.

Thanks
Barry

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Anshuman Khandual]

On 12/7/20 7:10 AM, Song Bao Hua (Barry Song) wrote:
> 
> 
>> -----Original Message-----
>> From: Mike Rapoport [mailto:rppt@linux.ibm.com]
>> Sent: Saturday, December 5, 2020 12:44 AM
>> To: Will Deacon <will@kernel.org>
>> Cc: liwei (CM) <liwei213@huawei.com>; catalin.marinas@arm.com; fengbaopeng
>> <fengbaopeng2@hisilicon.com>; nsaenzjulienne@suse.de; steve.capper@arm.com;
>> Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>;
>> linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; butao
>> <butao@hisilicon.com>
>> Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory
>> reserved for the page map
>>
>> On Fri, Dec 04, 2020 at 11:13:47AM +0000, Will Deacon wrote:
>>> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
>>>> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
>>>> do not free the reserved memory for the page map, decrease the section
>>>> size can reduce the waste of reserved memory.
>>>>
>>>> Signed-off-by: Wei Li <liwei213@huawei.com>
>>>> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
>>>> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
>>>> ---
>>>>  arch/arm64/include/asm/sparsemem.h | 2 +-
>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>
>>>> diff --git a/arch/arm64/include/asm/sparsemem.h
>> b/arch/arm64/include/asm/sparsemem.h
>>>> index 1f43fcc79738..8963bd3def28 100644
>>>> --- a/arch/arm64/include/asm/sparsemem.h
>>>> +++ b/arch/arm64/include/asm/sparsemem.h
>>>> @@ -7,7 +7,7 @@
>>>>
>>>>  #ifdef CONFIG_SPARSEMEM
>>>>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
>>>> -#define SECTION_SIZE_BITS	30
>>>> +#define SECTION_SIZE_BITS	27
>>>
>>> We chose '30' to avoid running out of bits in the page flags. What changed?
>>
>> I think that for 64-bit there are still plenty of free bits. I didn't
>> check now, but when I played with SPARSEMEM on m68k there were 8 bits
>> for section out of 32.
>>
>>> With this patch, I can trigger:
>>>
>>> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
>> SECTION_SIZE
>>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>>>
>>> if I bump up NR_CPUS and NODES_SHIFT.
>>
>> I don't think it's related to NR_CPUS and NODES_SHIFT.
>> This seems rather 64K pages that cause this.
>>
>> Not that is shouldn't be addressed.
> 
> Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS.
> Other cases will use vmemmap_populate_basepages().
> The original patch should be only addressing the issue in 4K pages:
> https://lore.kernel.org/lkml/20200812010655.96339-1-liwei213@huawei.com/
> 
> would we do something like the below?
> #ifdef CONFIG_ARM64_4K_PAGE
> #define SECTION_SIZE_BITS	27
> #else
> #define SECTION_SIZE_BITS	30
> #endif

This is bit arbitrary. Probably 27 can be further reduced for 4K page size.
Instead, we should make SECTION_SIZE_BITS explicitly depend upon MAX_ORDER.
IOW section size should be the same as the highest order page in the buddy.
CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64. A quick test shows
SECTION_SIZE_BITS would be 22 on 4K pages and 29 for 64K pages. As a fall
back SECTION_SIZE_BITS can still be 30 in case CONFIG_FORCE_MAX_ZONEORDER
is not defined.

--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -7,7 +7,7 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS       CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS      30
+#define SECTION_SIZE_BITS      (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
 #endif
 
 #endif

A similar approach exists on ia64 platform as well.

RE: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Song Bao Hua (Barry Song)]

> -----Original Message-----
> From: Anshuman Khandual [mailto:anshuman.khandual@arm.com]
> Sent: Monday, December 7, 2020 8:31 PM
> To: Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>; Mike Rapoport
> <rppt@linux.ibm.com>; Will Deacon <will@kernel.org>
> Cc: steve.capper@arm.com; catalin.marinas@arm.com;
> linux-kernel@vger.kernel.org; nsaenzjulienne@suse.de; liwei (CM)
> <liwei213@huawei.com>; butao <butao@hisilicon.com>;
> linux-arm-kernel@lists.infradead.org; fengbaopeng
> <fengbaopeng2@hisilicon.com>
> Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory
> reserved for the page map
> 
> 
> 
> On 12/7/20 7:10 AM, Song Bao Hua (Barry Song) wrote:
> >
> >
> >> -----Original Message-----
> >> From: Mike Rapoport [mailto:rppt@linux.ibm.com]
> >> Sent: Saturday, December 5, 2020 12:44 AM
> >> To: Will Deacon <will@kernel.org>
> >> Cc: liwei (CM) <liwei213@huawei.com>; catalin.marinas@arm.com; fengbaopeng
> >> <fengbaopeng2@hisilicon.com>; nsaenzjulienne@suse.de;
> steve.capper@arm.com;
> >> Song Bao Hua (Barry Song) <song.bao.hua@hisilicon.com>;
> >> linux-arm-kernel@lists.infradead.org; linux-kernel@vger.kernel.org; butao
> >> <butao@hisilicon.com>
> >> Subject: Re: [PATCH] arm64: mm: decrease the section size to reduce the memory
> >> reserved for the page map
> >>
> >> On Fri, Dec 04, 2020 at 11:13:47AM +0000, Will Deacon wrote:
> >>> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> >>>> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> >>>> do not free the reserved memory for the page map, decrease the section
> >>>> size can reduce the waste of reserved memory.
> >>>>
> >>>> Signed-off-by: Wei Li <liwei213@huawei.com>
> >>>> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> >>>> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> >>>> ---
> >>>>  arch/arm64/include/asm/sparsemem.h | 2 +-
> >>>>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>>>
> >>>> diff --git a/arch/arm64/include/asm/sparsemem.h
> >> b/arch/arm64/include/asm/sparsemem.h
> >>>> index 1f43fcc79738..8963bd3def28 100644
> >>>> --- a/arch/arm64/include/asm/sparsemem.h
> >>>> +++ b/arch/arm64/include/asm/sparsemem.h
> >>>> @@ -7,7 +7,7 @@
> >>>>
> >>>>  #ifdef CONFIG_SPARSEMEM
> >>>>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> >>>> -#define SECTION_SIZE_BITS	30
> >>>> +#define SECTION_SIZE_BITS	27
> >>>
> >>> We chose '30' to avoid running out of bits in the page flags. What changed?
> >>
> >> I think that for 64-bit there are still plenty of free bits. I didn't
> >> check now, but when I played with SPARSEMEM on m68k there were 8 bits
> >> for section out of 32.
> >>
> >>> With this patch, I can trigger:
> >>>
> >>> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
> >> SECTION_SIZE
> >>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
> >>>
> >>> if I bump up NR_CPUS and NODES_SHIFT.
> >>
> >> I don't think it's related to NR_CPUS and NODES_SHIFT.
> >> This seems rather 64K pages that cause this.
> >>
> >> Not that is shouldn't be addressed.
> >
> > Right now, only 4K PAGES will define ARM64_SWAPPER_USES_SECTION_MAPS.
> > Other cases will use vmemmap_populate_basepages().
> > The original patch should be only addressing the issue in 4K pages:
> > https://lore.kernel.org/lkml/20200812010655.96339-1-liwei213@huawei.com/
> >
> > would we do something like the below?
> > #ifdef CONFIG_ARM64_4K_PAGE
> > #define SECTION_SIZE_BITS	27
> > #else
> > #define SECTION_SIZE_BITS	30
> > #endif
> 
> This is bit arbitrary. Probably 27 can be further reduced for 4K page size.
> Instead, we should make SECTION_SIZE_BITS explicitly depend upon MAX_ORDER.
> IOW section size should be the same as the highest order page in the buddy.
> CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64. A quick test shows
> SECTION_SIZE_BITS would be 22 on 4K pages and 29 for 64K pages. As a fall
> back SECTION_SIZE_BITS can still be 30 in case CONFIG_FORCE_MAX_ZONEORDER
> is not defined.

This will break the pmd mapping for vmemmap. As for each 128M(2^27), we can
have 2MB pmd mapping.

> 
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,7 @@
> 
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS       CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS      30
> +#define SECTION_SIZE_BITS      (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
>  #endif
> 
>  #endif
> 
> A similar approach exists on ia64 platform as well.

Thanks
Barry

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Ard Biesheuvel]

(+ Marc)

On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
>
> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > do not free the reserved memory for the page map, decrease the section
> > size can reduce the waste of reserved memory.
> >
> > Signed-off-by: Wei Li <liwei213@huawei.com>
> > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > ---
> >  arch/arm64/include/asm/sparsemem.h | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> >
> > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> > index 1f43fcc79738..8963bd3def28 100644
> > --- a/arch/arm64/include/asm/sparsemem.h
> > +++ b/arch/arm64/include/asm/sparsemem.h
> > @@ -7,7 +7,7 @@
> >
> >  #ifdef CONFIG_SPARSEMEM
> >  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
> > -#define SECTION_SIZE_BITS    30
> > +#define SECTION_SIZE_BITS    27
>
> We chose '30' to avoid running out of bits in the page flags. What changed?
>
> With this patch, I can trigger:
>
> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds SECTION_SIZE
> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>
> if I bump up NR_CPUS and NODES_SHIFT.
>

Does this mean we will run into problems with the GICv3 ITS LPI tables
again if we are forced to reduce MAX_ORDER to fit inside
SECTION_SIZE_BITS?

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Marc Zyngier]

On 2020-12-07 09:09, Ard Biesheuvel wrote:
> (+ Marc)
> 
> On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
>> 
>> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
>> > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
>> > do not free the reserved memory for the page map, decrease the section
>> > size can reduce the waste of reserved memory.
>> >
>> > Signed-off-by: Wei Li <liwei213@huawei.com>
>> > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
>> > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
>> > ---
>> >  arch/arm64/include/asm/sparsemem.h | 2 +-
>> >  1 file changed, 1 insertion(+), 1 deletion(-)
>> >
>> > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
>> > index 1f43fcc79738..8963bd3def28 100644
>> > --- a/arch/arm64/include/asm/sparsemem.h
>> > +++ b/arch/arm64/include/asm/sparsemem.h
>> > @@ -7,7 +7,7 @@
>> >
>> >  #ifdef CONFIG_SPARSEMEM
>> >  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
>> > -#define SECTION_SIZE_BITS    30
>> > +#define SECTION_SIZE_BITS    27
>> 
>> We chose '30' to avoid running out of bits in the page flags. What 
>> changed?
>> 
>> With this patch, I can trigger:
>> 
>> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds 
>> SECTION_SIZE
>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>> 
>> if I bump up NR_CPUS and NODES_SHIFT.
>> 
> 
> Does this mean we will run into problems with the GICv3 ITS LPI tables
> again if we are forced to reduce MAX_ORDER to fit inside
> SECTION_SIZE_BITS?

Most probably. We are already massively constraint on platforms
such as TX1, and dividing the max allocatable range by 8 isn't
going to make it work any better...

         M.
-- 
Jazz is not dead. It just smells funny...

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Mike Rapoport]

On Mon, Dec 07, 2020 at 09:35:06AM +0000, Marc Zyngier wrote:
> On 2020-12-07 09:09, Ard Biesheuvel wrote:
> > (+ Marc)
> > 
> > On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
> > > 
> > > On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > > > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > > > do not free the reserved memory for the page map, decrease the section
> > > > size can reduce the waste of reserved memory.
> > > >
> > > > Signed-off-by: Wei Li <liwei213@huawei.com>
> > > > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > > > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > > > ---
> > > >  arch/arm64/include/asm/sparsemem.h | 2 +-
> > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > >
> > > > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> > > > index 1f43fcc79738..8963bd3def28 100644
> > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > @@ -7,7 +7,7 @@
> > > >
> > > >  #ifdef CONFIG_SPARSEMEM
> > > >  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
> > > > -#define SECTION_SIZE_BITS    30
> > > > +#define SECTION_SIZE_BITS    27
> > > 
> > > We chose '30' to avoid running out of bits in the page flags. What
> > > changed?
> > > 
> > > With this patch, I can trigger:
> > > 
> > > ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
> > > SECTION_SIZE
> > > #error Allocator MAX_ORDER exceeds SECTION_SIZE
> > > 
> > > if I bump up NR_CPUS and NODES_SHIFT.
> > > 
> > 
> > Does this mean we will run into problems with the GICv3 ITS LPI tables
> > again if we are forced to reduce MAX_ORDER to fit inside
> > SECTION_SIZE_BITS?
> 
> Most probably. We are already massively constraint on platforms
> such as TX1, and dividing the max allocatable range by 8 isn't
> going to make it work any better...

I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is
reduced it should accomodate the existing MAX_ORDER.

My two pennies.

>         M.
> -- 
> Jazz is not dead. It just smells funny...

-- 
Sincerely yours,
Mike.

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Ard Biesheuvel]

On Mon, 7 Dec 2020 at 10:42, Mike Rapoport <rppt@linux.ibm.com> wrote:
>
> On Mon, Dec 07, 2020 at 09:35:06AM +0000, Marc Zyngier wrote:
> > On 2020-12-07 09:09, Ard Biesheuvel wrote:
> > > (+ Marc)
> > >
> > > On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
> > > >
> > > > On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > > > > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > > > > do not free the reserved memory for the page map, decrease the section
> > > > > size can reduce the waste of reserved memory.
> > > > >
> > > > > Signed-off-by: Wei Li <liwei213@huawei.com>
> > > > > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > > > > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > > > > ---
> > > > >  arch/arm64/include/asm/sparsemem.h | 2 +-
> > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > >
> > > > > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> > > > > index 1f43fcc79738..8963bd3def28 100644
> > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > @@ -7,7 +7,7 @@
> > > > >
> > > > >  #ifdef CONFIG_SPARSEMEM
> > > > >  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
> > > > > -#define SECTION_SIZE_BITS    30
> > > > > +#define SECTION_SIZE_BITS    27
> > > >
> > > > We chose '30' to avoid running out of bits in the page flags. What
> > > > changed?
> > > >
> > > > With this patch, I can trigger:
> > > >
> > > > ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
> > > > SECTION_SIZE
> > > > #error Allocator MAX_ORDER exceeds SECTION_SIZE
> > > >
> > > > if I bump up NR_CPUS and NODES_SHIFT.
> > > >
> > >
> > > Does this mean we will run into problems with the GICv3 ITS LPI tables
> > > again if we are forced to reduce MAX_ORDER to fit inside
> > > SECTION_SIZE_BITS?
> >
> > Most probably. We are already massively constraint on platforms
> > such as TX1, and dividing the max allocatable range by 8 isn't
> > going to make it work any better...
>
> I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is
> reduced it should accomodate the existing MAX_ORDER.
>
> My two pennies.
>

But include/linux/mmzone.h:1170 has this:

#if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
#error Allocator MAX_ORDER exceeds SECTION_SIZE
#endif

and Will managed to trigger it after applying this patch.

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Mike Rapoport]

On Mon, Dec 07, 2020 at 10:49:26AM +0100, Ard Biesheuvel wrote:
> On Mon, 7 Dec 2020 at 10:42, Mike Rapoport <rppt@linux.ibm.com> wrote:
> >
> > On Mon, Dec 07, 2020 at 09:35:06AM +0000, Marc Zyngier wrote:
> > > On 2020-12-07 09:09, Ard Biesheuvel wrote:
> > > > (+ Marc)
> > > >
> > > > On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
> > > > >
> > > > > On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
> > > > > > For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
> > > > > > do not free the reserved memory for the page map, decrease the section
> > > > > > size can reduce the waste of reserved memory.
> > > > > >
> > > > > > Signed-off-by: Wei Li <liwei213@huawei.com>
> > > > > > Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
> > > > > > Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
> > > > > > ---
> > > > > >  arch/arm64/include/asm/sparsemem.h | 2 +-
> > > > > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > > > >
> > > > > > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> > > > > > index 1f43fcc79738..8963bd3def28 100644
> > > > > > --- a/arch/arm64/include/asm/sparsemem.h
> > > > > > +++ b/arch/arm64/include/asm/sparsemem.h
> > > > > > @@ -7,7 +7,7 @@
> > > > > >
> > > > > >  #ifdef CONFIG_SPARSEMEM
> > > > > >  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
> > > > > > -#define SECTION_SIZE_BITS    30
> > > > > > +#define SECTION_SIZE_BITS    27
> > > > >
> > > > > We chose '30' to avoid running out of bits in the page flags. What
> > > > > changed?
> > > > >
> > > > > With this patch, I can trigger:
> > > > >
> > > > > ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
> > > > > SECTION_SIZE
> > > > > #error Allocator MAX_ORDER exceeds SECTION_SIZE
> > > > >
> > > > > if I bump up NR_CPUS and NODES_SHIFT.
> > > > >
> > > >
> > > > Does this mean we will run into problems with the GICv3 ITS LPI tables
> > > > again if we are forced to reduce MAX_ORDER to fit inside
> > > > SECTION_SIZE_BITS?
> > >
> > > Most probably. We are already massively constraint on platforms
> > > such as TX1, and dividing the max allocatable range by 8 isn't
> > > going to make it work any better...
> >
> > I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is
> > reduced it should accomodate the existing MAX_ORDER.
> >
> > My two pennies.
> >
> 
> But include/linux/mmzone.h:1170 has this:
> 
> #if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
> #error Allocator MAX_ORDER exceeds SECTION_SIZE
> #endif
> 
> and Will managed to trigger it after applying this patch.

Right, because with 64K pages section size of 27 bits is not enough to
accomodate MAX_ORDER (2^13 pages of 64K).

Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER
into account either statically with 

#ifdef ARM64_4K_PAGES
#define SECTION_SIZE_BITS <a number>
#elif ARM64_16K_PAGES
#define SECTION_SIZE_BITS <a larger number>
#elif ARM64_64K_PAGES
#define SECTION_SIZE_BITS <even larger number>
#else
#error "and what is the page size?"
#endif

or dynamically, like e.g. ia64 does:

#ifdef CONFIG_FORCE_MAX_ZONEORDER
#if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS)
#undef SECTION_SIZE_BITS
#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
#endif


-- 
Sincerely yours,
Mike.

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Anshuman Khandual]

On 12/7/20 3:34 PM, Mike Rapoport wrote:
> On Mon, Dec 07, 2020 at 10:49:26AM +0100, Ard Biesheuvel wrote:
>> On Mon, 7 Dec 2020 at 10:42, Mike Rapoport <rppt@linux.ibm.com> wrote:
>>>
>>> On Mon, Dec 07, 2020 at 09:35:06AM +0000, Marc Zyngier wrote:
>>>> On 2020-12-07 09:09, Ard Biesheuvel wrote:
>>>>> (+ Marc)
>>>>>
>>>>> On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
>>>>>>
>>>>>> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
>>>>>>> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
>>>>>>> do not free the reserved memory for the page map, decrease the section
>>>>>>> size can reduce the waste of reserved memory.
>>>>>>>
>>>>>>> Signed-off-by: Wei Li <liwei213@huawei.com>
>>>>>>> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
>>>>>>> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
>>>>>>> ---
>>>>>>>  arch/arm64/include/asm/sparsemem.h | 2 +-
>>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>
>>>>>>> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
>>>>>>> index 1f43fcc79738..8963bd3def28 100644
>>>>>>> --- a/arch/arm64/include/asm/sparsemem.h
>>>>>>> +++ b/arch/arm64/include/asm/sparsemem.h
>>>>>>> @@ -7,7 +7,7 @@
>>>>>>>
>>>>>>>  #ifdef CONFIG_SPARSEMEM
>>>>>>>  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
>>>>>>> -#define SECTION_SIZE_BITS    30
>>>>>>> +#define SECTION_SIZE_BITS    27
>>>>>>
>>>>>> We chose '30' to avoid running out of bits in the page flags. What
>>>>>> changed?
>>>>>>
>>>>>> With this patch, I can trigger:
>>>>>>
>>>>>> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
>>>>>> SECTION_SIZE
>>>>>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>>>>>>
>>>>>> if I bump up NR_CPUS and NODES_SHIFT.
>>>>>>
>>>>>
>>>>> Does this mean we will run into problems with the GICv3 ITS LPI tables
>>>>> again if we are forced to reduce MAX_ORDER to fit inside
>>>>> SECTION_SIZE_BITS?
>>>>
>>>> Most probably. We are already massively constraint on platforms
>>>> such as TX1, and dividing the max allocatable range by 8 isn't
>>>> going to make it work any better...
>>>
>>> I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is
>>> reduced it should accomodate the existing MAX_ORDER.
>>>
>>> My two pennies.
>>>
>>
>> But include/linux/mmzone.h:1170 has this:
>>
>> #if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>> #endif
>>
>> and Will managed to trigger it after applying this patch.
> 
> Right, because with 64K pages section size of 27 bits is not enough to
> accomodate MAX_ORDER (2^13 pages of 64K).
> 
> Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER
> into account either statically with 
> 
> #ifdef ARM64_4K_PAGES
> #define SECTION_SIZE_BITS <a number>
> #elif ARM64_16K_PAGES
> #define SECTION_SIZE_BITS <a larger number>
> #elif ARM64_64K_PAGES
> #define SECTION_SIZE_BITS <even larger number>
> #else
> #error "and what is the page size?"
> #endif
> 
> or dynamically, like e.g. ia64 does:
> 
> #ifdef CONFIG_FORCE_MAX_ZONEORDER
> #if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS)
> #undef SECTION_SIZE_BITS
> #define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
> #endif

I had proposed the same on the other thread here. But with this the
SECTION_SIZE_BITS becomes 22 in case of 4K page size reducing to an
extent where PMD based vmemmap mapping could not be created. Though
have not looked into much details yet.

Using CONFIG_FORCE_MAX_ZONEORDER seems to the right thing to do. But
if that does not reasonably work for 4K pages, we might have to hard
code it as 27 to have huge page vmemmap mappings.

Re: [PATCH] arm64: mm: decrease the section size to reduce the memory reserved for the page map

[Wei Li]

(+ saberlily + jiapeng)

On 2020/12/7 18:39, Anshuman Khandual wrote:
> 
> 
> On 12/7/20 3:34 PM, Mike Rapoport wrote:
>> On Mon, Dec 07, 2020 at 10:49:26AM +0100, Ard Biesheuvel wrote:
>>> On Mon, 7 Dec 2020 at 10:42, Mike Rapoport <rppt@linux.ibm.com> wrote:
>>>>
>>>> On Mon, Dec 07, 2020 at 09:35:06AM +0000, Marc Zyngier wrote:
>>>>> On 2020-12-07 09:09, Ard Biesheuvel wrote:
>>>>>> (+ Marc)
>>>>>>
>>>>>> On Fri, 4 Dec 2020 at 12:14, Will Deacon <will@kernel.org> wrote:
>>>>>>>
>>>>>>> On Fri, Dec 04, 2020 at 09:44:43AM +0800, Wei Li wrote:
>>>>>>>> For the memory hole, sparse memory model that define SPARSEMEM_VMEMMAP
>>>>>>>> do not free the reserved memory for the page map, decrease the section
>>>>>>>> size can reduce the waste of reserved memory.
>>>>>>>>
>>>>>>>> Signed-off-by: Wei Li <liwei213@huawei.com>
>>>>>>>> Signed-off-by: Baopeng Feng <fengbaopeng2@hisilicon.com>
>>>>>>>> Signed-off-by: Xia Qing <saberlily.xia@hisilicon.com>
>>>>>>>> ---
>>>>>>>>  arch/arm64/include/asm/sparsemem.h | 2 +-
>>>>>>>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>>>>>>>
>>>>>>>> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
>>>>>>>> index 1f43fcc79738..8963bd3def28 100644
>>>>>>>> --- a/arch/arm64/include/asm/sparsemem.h
>>>>>>>> +++ b/arch/arm64/include/asm/sparsemem.h
>>>>>>>> @@ -7,7 +7,7 @@
>>>>>>>>
>>>>>>>>  #ifdef CONFIG_SPARSEMEM
>>>>>>>>  #define MAX_PHYSMEM_BITS     CONFIG_ARM64_PA_BITS
>>>>>>>> -#define SECTION_SIZE_BITS    30
>>>>>>>> +#define SECTION_SIZE_BITS    27
>>>>>>>
>>>>>>> We chose '30' to avoid running out of bits in the page flags. What
>>>>>>> changed?
>>>>>>>
>>>>>>> With this patch, I can trigger:
>>>>>>>
>>>>>>> ./include/linux/mmzone.h:1170:2: error: Allocator MAX_ORDER exceeds
>>>>>>> SECTION_SIZE
>>>>>>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>>>>>>>
>>>>>>> if I bump up NR_CPUS and NODES_SHIFT.
>>>>>>>
>>>>>>
>>>>>> Does this mean we will run into problems with the GICv3 ITS LPI tables
>>>>>> again if we are forced to reduce MAX_ORDER to fit inside
>>>>>> SECTION_SIZE_BITS?
>>>>>
>>>>> Most probably. We are already massively constraint on platforms
>>>>> such as TX1, and dividing the max allocatable range by 8 isn't
>>>>> going to make it work any better...
>>>>
>>>> I don't think MAX_ORDER should shrink. Even if SECTION_SIZE_BITS is
>>>> reduced it should accomodate the existing MAX_ORDER.
>>>>
>>>> My two pennies.
>>>>
>>>
>>> But include/linux/mmzone.h:1170 has this:
>>>
>>> #if (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS
>>> #error Allocator MAX_ORDER exceeds SECTION_SIZE
>>> #endif
>>>
>>> and Will managed to trigger it after applying this patch.
>>
>> Right, because with 64K pages section size of 27 bits is not enough to
>> accomodate MAX_ORDER (2^13 pages of 64K).
>>
>> Which means that definition of SECTION_SIZE_BITS should take MAX_ORDER
>> into account either statically with 
>>
>> #ifdef ARM64_4K_PAGES
>> #define SECTION_SIZE_BITS <a number>
>> #elif ARM64_16K_PAGES
>> #define SECTION_SIZE_BITS <a larger number>
>> #elif ARM64_64K_PAGES
>> #define SECTION_SIZE_BITS <even larger number>
>> #else
>> #error "and what is the page size?"
>> #endif
>>
>> or dynamically, like e.g. ia64 does:
>>
>> #ifdef CONFIG_FORCE_MAX_ZONEORDER
>> #if ((CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS)
>> #undef SECTION_SIZE_BITS
>> #define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
>> #endif
> 
> I had proposed the same on the other thread here. But with this the
> SECTION_SIZE_BITS becomes 22 in case of 4K page size reducing to an
> extent where PMD based vmemmap mapping could not be created. Though
> have not looked into much details yet.
> 
> Using CONFIG_FORCE_MAX_ZONEORDER seems to the right thing to do. But
> if that does not reasonably work for 4K pages, we might have to hard
> code it as 27 to have huge page vmemmap mappings.
> .
>