[PATCH 0/1] arm64: reduce section size for sparsemem

[PATCH 0/1] arm64: reduce section size for sparsemem

[Sudarshan Rajagopalan]

This patch is the follow-up from the discussions in the thread [1].
Reducing the section size has the merit of reducing wastage of reserved memory
for huge memory holes in sparsemem model. Also with smaller section size gives
more grunularity and agility for memory hot(un)plugging.

This patch tends to use the suggestion from David Hildenbrand in thread [1]
to set the least possible SECTION_SIZE_BITS for 4K, 16K and 64K page granule.
That is 27 (128MB) for 4K/16K and 29 (512MB) for 64K page granule.

[1] https://lore.kernel.org/lkml/cover.1609895500.git.sudaraja@codeaurora.org/T/#m8ee60ae69db5e9eb06ca7999c43828d49ccb9626

Sudarshan Rajagopalan (1):
  arm64: reduce section size for sparsemem

 arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

[PATCH 1/1] arm64: reduce section size for sparsemem

[Sudarshan Rajagopalan]

Reducing the section size helps reduce wastage of reserved memory
for huge memory holes in sparsemem model. But having a much smaller
section size bits could break PMD mappings for vmemmap and wouldn't
accomodate the highest order page for certain page size granule configs.
It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal
default value for 4K_PAGES that gives least section size without breaking
PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
that could accomodate MAX_ORDER.

Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
Suggested-by: David Hildenbrand <david@redhat.com>
Cc: Will Deacon <will@kernel.org>
Cc: Anshuman Khandual <anshuman.khandual@arm.com>
Cc: Mike Rapoport <rppt@linux.ibm.com>
Cc: Mark Rutland <mark.rutland@arm.com>
Cc: Suren Baghdasaryan <surenb@google.com>
---
 arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
 1 file changed, 8 insertions(+), 2 deletions(-)

diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
index 1f43fcc79738..ff08ff6b677c 100644
--- a/arch/arm64/include/asm/sparsemem.h
+++ b/arch/arm64/include/asm/sparsemem.h
@@ -7,7 +7,13 @@
 
 #ifdef CONFIG_SPARSEMEM
 #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
-#define SECTION_SIZE_BITS	30
-#endif
+
+#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)
+#define SECTION_SIZE_BITS 27
+#else
+#define SECTION_SIZE_BITS 29
+#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */
+
+#endif /* CONFIG_SPARSEMEM*/
 
 #endif
-- 
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Anshuman Khandual]

+ Catalin

Hello Sudershan,

Could you please change the subject line above as follows for
better classifications and clarity.

arm64/sparsemem: Reduce SECTION_SIZE_BITS

On 1/9/21 4:46 AM, Sudarshan Rajagopalan wrote:
> Reducing the section size helps reduce wastage of reserved memory
> for huge memory holes in sparsemem model. But having a much smaller

There are two distinct benefits of reducing SECTION_SIZE_BITS.

- Improve memory hotplug granularity
- Reduce reserved memory wastage for vmmemmap mappings for sections
  with large memory holes

> section size bits could break PMD mappings for vmemmap and wouldn't
> accomodate the highest order page for certain page size granule configs.

There are constrains in reducing SECTION_SIZE_BIT like

- Should accommodate highest order page for a given config
- Should not break PMD mapping in vmemmap for 4K pages
- Should not consume too many page->flags bits reducing space for other info

Both benefits and constraints should be described in the commit message
for folks to understand the rationale clearly at a later point in time.

> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal

Probably needs some description how we arrived here.

> default value for 4K_PAGES that gives least section size without breaking
> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
> that could accomodate MAX_ORDER.

Did not see this patch earlier and hence ended up writing yet another one.
Here is the draft commit message from that patch, please feel free to use
in part or full. But please do include the benefits, the constraints and
the rationale for arriving at these figures.

    ---------
    memory_block_size_bytes() determines the memory hotplug granularity i.e the
    amount of memory which can be hot added or hot removed from the kernel. The
    generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)
    for memory_block_size_bytes() on platforms like arm64 that does not override.
    
    Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here
    increases memory hotplug granularity, thus improving its agility. A reduced
    section size also reduces memory wastage in vmemmmap mapping for sections
    with large memory holes. A section size bits selection must follow.
    
    (MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS
    
    CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and just following it
    would help achieve the smallest section size.
    
    SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
    
    SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
    SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
    SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
    SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP
    SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP
    
    But there are other problems. Reducing the section size too much would over
    populate /sys/devices/system/memory/ and also consume too many page->flags
    bits in the !vmemmap case. Also section size needs to be multiple of 128MB
    to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.
    
    Given these constraints, lets just reduce the section size to 128MB for 4K
    and 16K base page size configs and to 512MB for 64K base page size config.
    -------------

> 
> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Mike Rapoport <rppt@linux.ibm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
A nit. Please add all relevant mailing lists like LAKML, MM along
with other developers here in the CC list, so that it would never
be missed.

> ---
>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..ff08ff6b677c 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,13 @@
>  
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> -#endif
> +
> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)

Please add a comment, something like

/*
 * Section size must be at least 128MB for 4K base
 * page size config. Otherwise PMD based huge page
 * entries could not be created for vmemmap mappings.
 * 16K follows 4K for simplicity. 
 */

> +#define SECTION_SIZE_BITS 27
> +#else

Please add a comment, something like

/*
 * Section size must be at least 512MB for 64K base
 * page size config. Otherwise it will be less than
 * (MAX_ORDER - 1) and the build process will fail.
 */

> +#define SECTION_SIZE_BITS 29
> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */

This condition here can be flipped (#ifdef CONFIG_ARM64_64K_PAGES) to
be bit cleaner and possibly to reduce the code.

> +
> +#endif /* CONFIG_SPARSEMEM*/
>  
>  #endif
> 

- Anshuman

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Anshuman Khandual]

+ Catalin

Hello Sudershan,

Could you please change the subject line above as follows for
better classifications and clarity.

arm64/sparsemem: Reduce SECTION_SIZE_BITS

On 1/9/21 4:46 AM, Sudarshan Rajagopalan wrote:
> Reducing the section size helps reduce wastage of reserved memory
> for huge memory holes in sparsemem model. But having a much smaller

There are two distinct benefits of reducing SECTION_SIZE_BITS.

- Improve memory hotplug granularity
- Reduce reserved memory wastage for vmmemmap mappings for sections
  with large memory holes

> section size bits could break PMD mappings for vmemmap and wouldn't
> accomodate the highest order page for certain page size granule configs.

There are constrains in reducing SECTION_SIZE_BIT like

- Should accommodate highest order page for a given config
- Should not break PMD mapping in vmemmap for 4K pages
- Should not consume too many page->flags bits reducing space for other info

Both benefits and constraints should be described in the commit message
for folks to understand the rationale clearly at a later point in time.

> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal

Probably needs some description how we arrived here.

> default value for 4K_PAGES that gives least section size without breaking
> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
> that could accomodate MAX_ORDER.

Did not see this patch earlier and hence ended up writing yet another one.
Here is the draft commit message from that patch, please feel free to use
in part or full. But please do include the benefits, the constraints and
the rationale for arriving at these figures.

    ---------
    memory_block_size_bytes() determines the memory hotplug granularity i.e the
    amount of memory which can be hot added or hot removed from the kernel. The
    generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << SECTION_SIZE_BITS)
    for memory_block_size_bytes() on platforms like arm64 that does not override.
    
    Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a reduction here
    increases memory hotplug granularity, thus improving its agility. A reduced
    section size also reduces memory wastage in vmemmmap mapping for sections
    with large memory holes. A section size bits selection must follow.
    
    (MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS
    
    CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and just following it
    would help achieve the smallest section size.
    
    SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
    
    SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
    SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages without THP
    SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with THP
    SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages without THP
    SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with THP
    
    But there are other problems. Reducing the section size too much would over
    populate /sys/devices/system/memory/ and also consume too many page->flags
    bits in the !vmemmap case. Also section size needs to be multiple of 128MB
    to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.
    
    Given these constraints, lets just reduce the section size to 128MB for 4K
    and 16K base page size configs and to 512MB for 64K base page size config.
    -------------

> 
> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Mike Rapoport <rppt@linux.ibm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
A nit. Please add all relevant mailing lists like LAKML, MM along
with other developers here in the CC list, so that it would never
be missed.

> ---
>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..ff08ff6b677c 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,13 @@
>  
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> -#endif
> +
> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)

Please add a comment, something like

/*
 * Section size must be at least 128MB for 4K base
 * page size config. Otherwise PMD based huge page
 * entries could not be created for vmemmap mappings.
 * 16K follows 4K for simplicity. 
 */

> +#define SECTION_SIZE_BITS 27
> +#else

Please add a comment, something like

/*
 * Section size must be at least 512MB for 64K base
 * page size config. Otherwise it will be less than
 * (MAX_ORDER - 1) and the build process will fail.
 */

> +#define SECTION_SIZE_BITS 29
> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */

This condition here can be flipped (#ifdef CONFIG_ARM64_64K_PAGES) to
be bit cleaner and possibly to reduce the code.

> +
> +#endif /* CONFIG_SPARSEMEM*/
>  
>  #endif
> 

- Anshuman

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Will Deacon]

On Fri, Jan 08, 2021 at 03:16:00PM -0800, Sudarshan Rajagopalan wrote:
> Reducing the section size helps reduce wastage of reserved memory
> for huge memory holes in sparsemem model. But having a much smaller
> section size bits could break PMD mappings for vmemmap and wouldn't
> accomodate the highest order page for certain page size granule configs.
> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal
> default value for 4K_PAGES that gives least section size without breaking
> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
> that could accomodate MAX_ORDER.
> 
> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Mike Rapoport <rppt@linux.ibm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> ---
>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..ff08ff6b677c 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,13 @@
>  
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> -#endif
> +
> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)
> +#define SECTION_SIZE_BITS 27
> +#else
> +#define SECTION_SIZE_BITS 29
> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */
> +
> +#endif /* CONFIG_SPARSEMEM*/

Please can you repost this in light of the comments from Anshuman?

Thanks,

Will

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Will Deacon]

On Fri, Jan 08, 2021 at 03:16:00PM -0800, Sudarshan Rajagopalan wrote:
> Reducing the section size helps reduce wastage of reserved memory
> for huge memory holes in sparsemem model. But having a much smaller
> section size bits could break PMD mappings for vmemmap and wouldn't
> accomodate the highest order page for certain page size granule configs.
> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal
> default value for 4K_PAGES that gives least section size without breaking
> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
> that could accomodate MAX_ORDER.
> 
> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
> Suggested-by: David Hildenbrand <david@redhat.com>
> Cc: Will Deacon <will@kernel.org>
> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
> Cc: Mike Rapoport <rppt@linux.ibm.com>
> Cc: Mark Rutland <mark.rutland@arm.com>
> Cc: Suren Baghdasaryan <surenb@google.com>
> ---
>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>  1 file changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h
> index 1f43fcc79738..ff08ff6b677c 100644
> --- a/arch/arm64/include/asm/sparsemem.h
> +++ b/arch/arm64/include/asm/sparsemem.h
> @@ -7,7 +7,13 @@
>  
>  #ifdef CONFIG_SPARSEMEM
>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
> -#define SECTION_SIZE_BITS	30
> -#endif
> +
> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)
> +#define SECTION_SIZE_BITS 27
> +#else
> +#define SECTION_SIZE_BITS 29
> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */
> +
> +#endif /* CONFIG_SPARSEMEM*/

Please can you repost this in light of the comments from Anshuman?

Thanks,

Will

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Sudarshan Rajagopalan]

On 2021-01-20 09:49, Will Deacon wrote:
> On Fri, Jan 08, 2021 at 03:16:00PM -0800, Sudarshan Rajagopalan wrote:
>> Reducing the section size helps reduce wastage of reserved memory
>> for huge memory holes in sparsemem model. But having a much smaller
>> section size bits could break PMD mappings for vmemmap and wouldn't
>> accomodate the highest order page for certain page size granule 
>> configs.
>> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal
>> default value for 4K_PAGES that gives least section size without 
>> breaking
>> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
>> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
>> that could accomodate MAX_ORDER.
>> 
>> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
>> Suggested-by: David Hildenbrand <david@redhat.com>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
>> Cc: Mike Rapoport <rppt@linux.ibm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Suren Baghdasaryan <surenb@google.com>
>> ---
>>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>> 
>> diff --git a/arch/arm64/include/asm/sparsemem.h 
>> b/arch/arm64/include/asm/sparsemem.h
>> index 1f43fcc79738..ff08ff6b677c 100644
>> --- a/arch/arm64/include/asm/sparsemem.h
>> +++ b/arch/arm64/include/asm/sparsemem.h
>> @@ -7,7 +7,13 @@
>> 
>>  #ifdef CONFIG_SPARSEMEM
>>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
>> -#define SECTION_SIZE_BITS	30
>> -#endif
>> +
>> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)
>> +#define SECTION_SIZE_BITS 27
>> +#else
>> +#define SECTION_SIZE_BITS 29
>> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */
>> +
>> +#endif /* CONFIG_SPARSEMEM*/
> 
> Please can you repost this in light of the comments from Anshuman?
> 
> Thanks,
> 
> Will

Sure Will. We were held up with some other critical tasks.. will repost 
the patch by EOD after addressing Anshuman's comments.

-- 
Sudarshan

--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project

Re: [PATCH 1/1] arm64: reduce section size for sparsemem

[Sudarshan Rajagopalan]

On 2021-01-11 03:09, Anshuman Khandual wrote:
> + Catalin
> 
> Hello Sudershan,
> 
> Could you please change the subject line above as follows for
> better classifications and clarity.
> 
> arm64/sparsemem: Reduce SECTION_SIZE_BITS
> 
> On 1/9/21 4:46 AM, Sudarshan Rajagopalan wrote:
>> Reducing the section size helps reduce wastage of reserved memory
>> for huge memory holes in sparsemem model. But having a much smaller
> 
> There are two distinct benefits of reducing SECTION_SIZE_BITS.
> 
> - Improve memory hotplug granularity
> - Reduce reserved memory wastage for vmmemmap mappings for sections
>   with large memory holes
> 
>> section size bits could break PMD mappings for vmemmap and wouldn't
>> accomodate the highest order page for certain page size granule 
>> configs.
> 
> There are constrains in reducing SECTION_SIZE_BIT like
> 
> - Should accommodate highest order page for a given config
> - Should not break PMD mapping in vmemmap for 4K pages
> - Should not consume too many page->flags bits reducing space for other 
> info
> 
> Both benefits and constraints should be described in the commit message
> for folks to understand the rationale clearly at a later point in time.
> 
>> It is determined that SECTION_SIZE_BITS of 27 (128MB) could be ideal
> 
> Probably needs some description how we arrived here.
> 
>> default value for 4K_PAGES that gives least section size without 
>> breaking
>> PMD based vmemmap mappings. For simplicity, 16K_PAGES could follow the
>> same as 4K_PAGES. And the least SECTION_SIZE_BITS for 64K_PAGES is 29
>> that could accomodate MAX_ORDER.
> 
> Did not see this patch earlier and hence ended up writing yet another 
> one.
> Here is the draft commit message from that patch, please feel free to 
> use
> in part or full. But please do include the benefits, the constraints 
> and
> the rationale for arriving at these figures.
> 
>     ---------
>     memory_block_size_bytes() determines the memory hotplug granularity 
> i.e the
>     amount of memory which can be hot added or hot removed from the 
> kernel. The
>     generic value here being MIN_MEMORY_BLOCK_SIZE (1UL << 
> SECTION_SIZE_BITS)
>     for memory_block_size_bytes() on platforms like arm64 that does
> not override.
> 
>     Current SECTION_SIZE_BITS is 30 i.e 1GB which is large and a 
> reduction here
>     increases memory hotplug granularity, thus improving its agility. A 
> reduced
>     section size also reduces memory wastage in vmemmmap mapping for 
> sections
>     with large memory holes. A section size bits selection must follow.
> 
>     (MAX_ORDER - 1 + PAGE_SHIFT) <= SECTION_SIZE_BITS
> 
>     CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 and just 
> following it
>     would help achieve the smallest section size.
> 
>     SECTION_SIZE_BITS = (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT)
> 
>     SECTION_SIZE_BITS = 22 (11 - 1 + 12) i.e 4MB   for 4K pages
>     SECTION_SIZE_BITS = 24 (11 - 1 + 14) i.e 16MB  for 16K pages 
> without THP
>     SECTION_SIZE_BITS = 25 (12 - 1 + 14) i.e 32MB  for 16K pages with 
> THP
>     SECTION_SIZE_BITS = 26 (11 - 1 + 16) i.e 64MB  for 64K pages 
> without THP
>     SECTION_SIZE_BITS = 29 (14 - 1 + 16) i.e 512MB for 64K pages with 
> THP
> 
>     But there are other problems. Reducing the section size too much 
> would over
>     populate /sys/devices/system/memory/ and also consume too many 
> page->flags
>     bits in the !vmemmap case. Also section size needs to be multiple 
> of 128MB
>     to have PMD based vmemmap mapping with CONFIG_ARM64_4K_PAGES.
> 
>     Given these constraints, lets just reduce the section size to 128MB 
> for 4K
>     and 16K base page size configs and to 512MB for 64K base page size 
> config.
>     -------------
> 
>> 
>> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org>
>> Suggested-by: David Hildenbrand <david@redhat.com>
>> Cc: Will Deacon <will@kernel.org>
>> Cc: Anshuman Khandual <anshuman.khandual@arm.com>
>> Cc: Mike Rapoport <rppt@linux.ibm.com>
>> Cc: Mark Rutland <mark.rutland@arm.com>
>> Cc: Suren Baghdasaryan <surenb@google.com>
> A nit. Please add all relevant mailing lists like LAKML, MM along
> with other developers here in the CC list, so that it would never
> be missed.
> 
>> ---
>>  arch/arm64/include/asm/sparsemem.h | 10 ++++++++--
>>  1 file changed, 8 insertions(+), 2 deletions(-)
>> 
>> diff --git a/arch/arm64/include/asm/sparsemem.h 
>> b/arch/arm64/include/asm/sparsemem.h
>> index 1f43fcc79738..ff08ff6b677c 100644
>> --- a/arch/arm64/include/asm/sparsemem.h
>> +++ b/arch/arm64/include/asm/sparsemem.h
>> @@ -7,7 +7,13 @@
>> 
>>  #ifdef CONFIG_SPARSEMEM
>>  #define MAX_PHYSMEM_BITS	CONFIG_ARM64_PA_BITS
>> -#define SECTION_SIZE_BITS	30
>> -#endif
>> +
>> +#if defined(CONFIG_ARM64_4K_PAGES) || defined(CONFIG_ARM64_16K_PAGES)
> 
> Please add a comment, something like
> 
> /*
>  * Section size must be at least 128MB for 4K base
>  * page size config. Otherwise PMD based huge page
>  * entries could not be created for vmemmap mappings.
>  * 16K follows 4K for simplicity.
>  */
> 
>> +#define SECTION_SIZE_BITS 27
>> +#else
> 
> Please add a comment, something like
> 
> /*
>  * Section size must be at least 512MB for 64K base
>  * page size config. Otherwise it will be less than
>  * (MAX_ORDER - 1) and the build process will fail.
>  */
> 
>> +#define SECTION_SIZE_BITS 29
>> +#endif /* CONFIG_ARM64_4K_PAGES || CONFIG_ARM64_16K_PAGES */
> 
> This condition here can be flipped (#ifdef CONFIG_ARM64_64K_PAGES) to
> be bit cleaner and possibly to reduce the code.
> 
>> +
>> +#endif /* CONFIG_SPARSEMEM*/
>> 
>>  #endif
>> 
> 
> - Anshuman

Thanks Anshuman for the comments and suggestions. I have used the 
excerpt you had provided before for the commit message with slight 
modifications. I have pushed the new patch with your comments addressed 
-
https://lore.kernel.org/lkml/43843c5e092bfe3ec4c41e3c8c78a7ee35b69bb0.1611206601.git.sudaraja@codeaurora.org/

Sudarshan

--
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a 
Linux Foundation Collaborative Project