* [PATCH 0/1] arm64: make section size configurable for memory hotplug @ 2021-01-06 1:28 Sudarshan Rajagopalan 2021-01-06 1:28 ` [PATCH 1/1] " Sudarshan Rajagopalan 0 siblings, 1 reply; 11+ messages in thread From: Sudarshan Rajagopalan @ 2021-01-06 1:28 UTC (permalink / raw) To: akpm, linux-mm, linux-kernel; +Cc: Sudarshan Rajagopalan The section size defines the granularity of memory hotplug. This is currently hard coded to 1GB on arm64 linux, which defines that the least size of memblock that can be hotplugged out is 1GB. Some DDR configurations (especially low RAM and dual-rank DDRs) may have section sizes that are less than 1GB (ex. 512MB, 256MB etc.). Having an option to reduce the memblock size to section size or lower gives more granularity of memory hotplug. For example, a system with DDR section size of 512MB and kernel memblock size of 1GB, we would have to remove two segments of DDR sections in order to hotplug out atleast 1 memblock from kernel POV. Section sizes of DDRs vary based on specs (number of ranks, channels, regions etc.) Making this section size configurable helps users to assign based on the DDR being used. The default is set to 1GB which is the current memblock size. Sudarshan Rajagopalan (1): arm64: Make section size configurable for memory hotplug arch/arm64/Kconfig | 11 +++++++++++ arch/arm64/include/asm/sparsemem.h | 4 ++++ 2 files changed, 15 insertions(+) -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project ^ permalink raw reply [flat|nested] 11+ messages in thread
* [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-06 1:28 [PATCH 0/1] arm64: make section size configurable for memory hotplug Sudarshan Rajagopalan @ 2021-01-06 1:28 ` Sudarshan Rajagopalan 2021-01-06 2:20 ` Randy Dunlap 2021-01-06 6:11 ` Anshuman Khandual 0 siblings, 2 replies; 11+ messages in thread From: Sudarshan Rajagopalan @ 2021-01-06 1:28 UTC (permalink / raw) To: akpm, linux-mm, linux-kernel; +Cc: Sudarshan Rajagopalan Currently on arm64, memory section size is hard-coded to 1GB. Make this configurable if memory-hotplug is enabled, to support more finer granularity for hotplug-able memory. Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org> --- arch/arm64/Kconfig | 11 +++++++++++ arch/arm64/include/asm/sparsemem.h | 4 ++++ 2 files changed, 15 insertions(+) diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig index 6d232837cbee..34124eee65da 100644 --- a/arch/arm64/Kconfig +++ b/arch/arm64/Kconfig @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE config SMP def_bool y +config HOTPLUG_SIZE_BITS + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" + depends on SPARSEMEM + depends on MEMORY_HOTPLUG + range 28 30 + default 30 + help + Selects granularity of hotplug memory. Block size for + memory hotplug is represent as a power of 2. + If unsure, stick with default value. + config KERNEL_MODE_NEON def_bool y diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index 1f43fcc79738..3d5310f3aad5 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -7,7 +7,11 @@ #ifdef CONFIG_SPARSEMEM #define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS +#ifndef CONFIG_MEMORY_HOTPLUG #define SECTION_SIZE_BITS 30 +#else +#define SECTION_SIZE_BITS CONFIG_HOTPLUG_SIZE_BITS +#endif #endif #endif -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-06 1:28 ` [PATCH 1/1] " Sudarshan Rajagopalan @ 2021-01-06 2:20 ` Randy Dunlap 2021-01-06 6:11 ` Anshuman Khandual 1 sibling, 0 replies; 11+ messages in thread From: Randy Dunlap @ 2021-01-06 2:20 UTC (permalink / raw) To: Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 1/5/21 5:28 PM, Sudarshan Rajagopalan wrote: > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index 6d232837cbee..34124eee65da 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE > config SMP > def_bool y > > +config HOTPLUG_SIZE_BITS > + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" block size (29 => 512MB, 30 => 1GB)" > + depends on SPARSEMEM > + depends on MEMORY_HOTPLUG > + range 28 30 > + default 30 > + help > + Selects granularity of hotplug memory. Block size for > + memory hotplug is represent as a power of 2. represented > + If unsure, stick with default value. and all of the help text should be indented with 1 tab + 2 spaces. thanks. -- ~Randy ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-06 1:28 ` [PATCH 1/1] " Sudarshan Rajagopalan 2021-01-06 2:20 ` Randy Dunlap @ 2021-01-06 6:11 ` Anshuman Khandual 2021-01-07 12:30 ` David Hildenbrand 2021-01-08 1:01 ` Sudarshan Rajagopalan 1 sibling, 2 replies; 11+ messages in thread From: Anshuman Khandual @ 2021-01-06 6:11 UTC (permalink / raw) To: Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel Hi Sudershan, This patch (and the cover letter) does not copy LAKML even though the entire change here is arm64 specific. Please do copy all applicable mailing lists for a given patch. On 1/6/21 6:58 AM, Sudarshan Rajagopalan wrote: > Currently on arm64, memory section size is hard-coded to 1GB. > Make this configurable if memory-hotplug is enabled, to support > more finer granularity for hotplug-able memory. Section size has always been decided by the platform. It cannot be a configurable option because the user would not know the constraints for memory representation on the platform and besides it also cannot be trusted. > > Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org> > --- > arch/arm64/Kconfig | 11 +++++++++++ > arch/arm64/include/asm/sparsemem.h | 4 ++++ > 2 files changed, 15 insertions(+) > > diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig > index 6d232837cbee..34124eee65da 100644 > --- a/arch/arm64/Kconfig > +++ b/arch/arm64/Kconfig > @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE > config SMP > def_bool y > > +config HOTPLUG_SIZE_BITS > + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" > + depends on SPARSEMEM > + depends on MEMORY_HOTPLUG > + range 28 30 28 would not work for 64K pages. > + default 30 > + help > + Selects granularity of hotplug memory. Block size for > + memory hotplug is represent as a power of 2. > + If unsure, stick with default value. > + > config KERNEL_MODE_NEON > def_bool y > > diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h > index 1f43fcc79738..3d5310f3aad5 100644 > --- a/arch/arm64/include/asm/sparsemem.h > +++ b/arch/arm64/include/asm/sparsemem.h > @@ -7,7 +7,11 @@ > > #ifdef CONFIG_SPARSEMEM > #define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS > +#ifndef CONFIG_MEMORY_HOTPLUG > #define SECTION_SIZE_BITS 30 > +#else > +#define SECTION_SIZE_BITS CONFIG_HOTPLUG_SIZE_BITS > +#endif > #endif > > #endif > There was an inconclusive discussion regarding this last month. https://lore.kernel.org/linux-arm-kernel/20201204014443.43329-1-liwei213@huawei.com/ I have been wondering if this would solve the problem for 4K page size config which requires PMD mapping for the vmemmap mapping while making section size bits dependent on max order. But this has not been tested properly. diff --git a/arch/arm64/include/asm/sparsemem.h b/arch/arm64/include/asm/sparsemem.h index 1f43fcc79738..fe4353cb1dce 100644 --- a/arch/arm64/include/asm/sparsemem.h +++ b/arch/arm64/include/asm/sparsemem.h @@ -7,7 +7,18 @@ #ifdef CONFIG_SPARSEMEM #define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS -#define SECTION_SIZE_BITS 30 -#endif + +#ifdef CONFIG_ARM64_4K_PAGES +#define SECTION_SIZE_BITS 27 +#else +#ifdef CONFIG_FORCE_MAX_ZONEORDER +#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) +#else +#define SECTION_SIZE_BITS 30 +#endif /* CONFIG_FORCE_MAX_ZONEORDER */ + +#endif /* CONFIG_ARM64_4K_PAGES */ + +#endif /* CONFIG_SPARSEMEM*/ #endif ^ permalink raw reply related [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-06 6:11 ` Anshuman Khandual @ 2021-01-07 12:30 ` David Hildenbrand 2021-01-08 7:02 ` Anshuman Khandual 2021-01-08 1:01 ` Sudarshan Rajagopalan 1 sibling, 1 reply; 11+ messages in thread From: David Hildenbrand @ 2021-01-07 12:30 UTC (permalink / raw) To: Anshuman Khandual, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 06.01.21 07:11, Anshuman Khandual wrote: > Hi Sudershan, > > This patch (and the cover letter) does not copy LAKML even though the > entire change here is arm64 specific. Please do copy all applicable > mailing lists for a given patch. > > On 1/6/21 6:58 AM, Sudarshan Rajagopalan wrote: >> Currently on arm64, memory section size is hard-coded to 1GB. >> Make this configurable if memory-hotplug is enabled, to support >> more finer granularity for hotplug-able memory. > > Section size has always been decided by the platform. It cannot be a > configurable option because the user would not know the constraints > for memory representation on the platform and besides it also cannot > be trusted. > >> >> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org> >> --- >> arch/arm64/Kconfig | 11 +++++++++++ >> arch/arm64/include/asm/sparsemem.h | 4 ++++ >> 2 files changed, 15 insertions(+) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index 6d232837cbee..34124eee65da 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE >> config SMP >> def_bool y >> >> +config HOTPLUG_SIZE_BITS >> + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" "Memory hotplug block size" and "HOTPLUG_SIZE_BITS" is confusing. It's the section size. Please use that terminology. (if at all, it would have to be "minimum memory hot(un)plug granularity", but even that is somewhat misleading) "SECTION_SIZE_BITS" But I agree that letting the user configure it is sub-optimal. >> + depends on SPARSEMEM >> + depends on MEMORY_HOTPLUG >> + range 28 30 > > 28 would not work for 64K pages. @Anshuman, what's your feeling about changing this to 128 MB for 4k/16k base pages (as we have on x86-64 right now) and 512 MB for 64k as default for now? (If we worry about the number of section bits in page->flags, we could glue it to vmemmap support where that does not matter) -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-07 12:30 ` David Hildenbrand @ 2021-01-08 7:02 ` Anshuman Khandual 2021-01-08 15:30 ` David Hildenbrand 0 siblings, 1 reply; 11+ messages in thread From: Anshuman Khandual @ 2021-01-08 7:02 UTC (permalink / raw) To: David Hildenbrand, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 1/7/21 6:00 PM, David Hildenbrand wrote: > On 06.01.21 07:11, Anshuman Khandual wrote: >> Hi Sudershan, >> >> This patch (and the cover letter) does not copy LAKML even though the >> entire change here is arm64 specific. Please do copy all applicable >> mailing lists for a given patch. >> >> On 1/6/21 6:58 AM, Sudarshan Rajagopalan wrote: >>> Currently on arm64, memory section size is hard-coded to 1GB. >>> Make this configurable if memory-hotplug is enabled, to support >>> more finer granularity for hotplug-able memory. >> >> Section size has always been decided by the platform. It cannot be a >> configurable option because the user would not know the constraints >> for memory representation on the platform and besides it also cannot >> be trusted. >> >>> >>> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org> >>> --- >>> arch/arm64/Kconfig | 11 +++++++++++ >>> arch/arm64/include/asm/sparsemem.h | 4 ++++ >>> 2 files changed, 15 insertions(+) >>> >>> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >>> index 6d232837cbee..34124eee65da 100644 >>> --- a/arch/arm64/Kconfig >>> +++ b/arch/arm64/Kconfig >>> @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE >>> config SMP >>> def_bool y >>> >>> +config HOTPLUG_SIZE_BITS >>> + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" > > "Memory hotplug block size" and "HOTPLUG_SIZE_BITS" is confusing. It's > the section size. Please use that terminology. > > (if at all, it would have to be "minimum memory hot(un)plug > granularity", but even that is somewhat misleading) > > "SECTION_SIZE_BITS" > > But I agree that letting the user configure it is sub-optimal. > >>> + depends on SPARSEMEM >>> + depends on MEMORY_HOTPLUG >>> + range 28 30 >> >> 28 would not work for 64K pages. > > @Anshuman, what's your feeling about changing this to 128 MB for 4k/16k > base pages (as we have on x86-64 right now) and 512 MB for 64k as > default for now? To summarize, the section size bits for each base page size config should always a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS b. Provide minimum possible section size for a given base page config to have increased agility during memory hotplug operations and reduced vmemmap wastage for sections with holes. c. Allow 4K base page configs to have PMD based vmemmap mappings Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform, the following would always avoid the condition (a) SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) - 22 (11 - 1 + 12) for 4K pages - 24 (11 - 1 + 14) for 16K pages without THP - 25 (12 - 1 + 14) for 16K pages with THP - 26 (11 - 1 + 16) for 64K pages without THP - 29 (14 - 1 + 16) for 64K pages with THP Apart from overriding 4K base page size config to have 27 as section size bits, should not all other values be okay here ? But then wondering what benefit 128MB (27 bits) section size for 16K config would have ? OR the objective here is to match 16K page size config with default x86-64. > > (If we worry about the number of section bits in page->flags, we could > glue it to vmemmap support where that does not matter) Could you please elaborate ? Could smaller section size bits numbers like 22 or 24 create problems in page->flags without changing other parameters like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP i.e 26 bits in section size, fails to boot. As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER, at least for now. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-08 7:02 ` Anshuman Khandual @ 2021-01-08 15:30 ` David Hildenbrand 2021-01-11 4:17 ` Anshuman Khandual 0 siblings, 1 reply; 11+ messages in thread From: David Hildenbrand @ 2021-01-08 15:30 UTC (permalink / raw) To: Anshuman Khandual, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel > To summarize, the section size bits for each base page size config > should always > > a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS Pageblocks must also always fall completely into a section. > > b. Provide minimum possible section size for a given base page config to > have increased agility during memory hotplug operations and reduced > vmemmap wastage for sections with holes. OTOH, making the section size too small (e.g., 16MB) creates way to many memory block devices in /sys/devices/system/memory/, and might consume too many page->flags bits in the !vmemmap case. For bigger setups, we might, similar to x86-64 (e.g., >= 64 GiB), determine the memory_block_size_bytes() at runtime (spanning multiple sections then), once it becomes relevant. > > c. Allow 4K base page configs to have PMD based vmemmap mappings Agreed. > > Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform, > the following would always avoid the condition (a) > > SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) > > - 22 (11 - 1 + 12) for 4K pages > - 24 (11 - 1 + 14) for 16K pages without THP > - 25 (12 - 1 + 14) for 16K pages with THP > - 26 (11 - 1 + 16) for 64K pages without THP > - 29 (14 - 1 + 16) for 64K pages with THP > > Apart from overriding 4K base page size config to have 27 as section size > bits, should not all other values be okay here ? But then wondering what > benefit 128MB (27 bits) section size for 16K config would have ? OR the > objective here is to match 16K page size config with default x86-64. We don't want to have sections that are too small. We don't want to have sections that are too big :) Not sure if we really want to allow setting e.g., a section size of 4 MB. That's just going to hurt. IMHO, something in the range of 64..256 MB is quite a good choice, where possible. > >> >> (If we worry about the number of section bits in page->flags, we could >> glue it to vmemmap support where that does not matter) > > Could you please elaborate ? Could smaller section size bits numbers like > 22 or 24 create problems in page->flags without changing other parameters > like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP Yes, in the !vmemmap case, we have to store the section_nr in there. IIRC, it's less of an issue with section sizes like 128 MB. > i.e 26 bits in section size, fails to boot. 26 bits would mean 64 MB, no? Not sure if that's possible even without THP (MAX_ORDER - 1, pageblock_order ...) on 64k pages. I'd assume 512 MB is the lowest we can go. I'd assume this would crash :) > > As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB > for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER, > at least for now. That's also what I would prefer, keeping it simple. -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-08 15:30 ` David Hildenbrand @ 2021-01-11 4:17 ` Anshuman Khandual 2021-01-11 10:13 ` David Hildenbrand 0 siblings, 1 reply; 11+ messages in thread From: Anshuman Khandual @ 2021-01-11 4:17 UTC (permalink / raw) To: David Hildenbrand, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 1/8/21 9:00 PM, David Hildenbrand wrote: >> To summarize, the section size bits for each base page size config >> should always >> >> a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS > > Pageblocks must also always fall completely into a section. > >> >> b. Provide minimum possible section size for a given base page config to >> have increased agility during memory hotplug operations and reduced >> vmemmap wastage for sections with holes. > > OTOH, making the section size too small (e.g., 16MB) creates way to many > memory block devices in /sys/devices/system/memory/, and might consume > too many page->flags bits in the !vmemmap case. > > For bigger setups, we might, similar to x86-64 (e.g., >= 64 GiB), > determine the memory_block_size_bytes() at runtime (spanning multiple > sections then), once it becomes relevant. > >> >> c. Allow 4K base page configs to have PMD based vmemmap mappings > > Agreed. > >> >> Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform, >> the following would always avoid the condition (a) >> >> SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) >> >> - 22 (11 - 1 + 12) for 4K pages >> - 24 (11 - 1 + 14) for 16K pages without THP >> - 25 (12 - 1 + 14) for 16K pages with THP >> - 26 (11 - 1 + 16) for 64K pages without THP >> - 29 (14 - 1 + 16) for 64K pages with THP >> >> Apart from overriding 4K base page size config to have 27 as section size >> bits, should not all other values be okay here ? But then wondering what >> benefit 128MB (27 bits) section size for 16K config would have ? OR the >> objective here is to match 16K page size config with default x86-64. > > We don't want to have sections that are too small. We don't want to have > sections that are too big :) > > Not sure if we really want to allow setting e.g., a section size of 4 > MB. That's just going to hurt. IMHO, something in the range of 64..256 > MB is quite a good choice, where possible. > >> >>> >>> (If we worry about the number of section bits in page->flags, we could >>> glue it to vmemmap support where that does not matter) >> >> Could you please elaborate ? Could smaller section size bits numbers like >> 22 or 24 create problems in page->flags without changing other parameters >> like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP > > Yes, in the !vmemmap case, we have to store the section_nr in there. > IIRC, it's less of an issue with section sizes like 128 MB. > >> i.e 26 bits in section size, fails to boot. > > 26 bits would mean 64 MB, no? Not sure if that's possible even without > THP (MAX_ORDER - 1, pageblock_order ...) on 64k pages. I'd assume 512 MB > is the lowest we can go. I'd assume this would crash :) > >> >> As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB >> for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER, >> at least for now. > > That's also what I would prefer, keeping it simple. Okay sure, will send a RFC to begin with. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-11 4:17 ` Anshuman Khandual @ 2021-01-11 10:13 ` David Hildenbrand 2021-01-11 10:26 ` Anshuman Khandual 0 siblings, 1 reply; 11+ messages in thread From: David Hildenbrand @ 2021-01-11 10:13 UTC (permalink / raw) To: Anshuman Khandual, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 11.01.21 05:17, Anshuman Khandual wrote: > > > On 1/8/21 9:00 PM, David Hildenbrand wrote: >>> To summarize, the section size bits for each base page size config >>> should always >>> >>> a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS >> >> Pageblocks must also always fall completely into a section. >> >>> >>> b. Provide minimum possible section size for a given base page config to >>> have increased agility during memory hotplug operations and reduced >>> vmemmap wastage for sections with holes. >> >> OTOH, making the section size too small (e.g., 16MB) creates way to many >> memory block devices in /sys/devices/system/memory/, and might consume >> too many page->flags bits in the !vmemmap case. >> >> For bigger setups, we might, similar to x86-64 (e.g., >= 64 GiB), >> determine the memory_block_size_bytes() at runtime (spanning multiple >> sections then), once it becomes relevant. >> >>> >>> c. Allow 4K base page configs to have PMD based vmemmap mappings >> >> Agreed. >> >>> >>> Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform, >>> the following would always avoid the condition (a) >>> >>> SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) >>> >>> - 22 (11 - 1 + 12) for 4K pages >>> - 24 (11 - 1 + 14) for 16K pages without THP >>> - 25 (12 - 1 + 14) for 16K pages with THP >>> - 26 (11 - 1 + 16) for 64K pages without THP >>> - 29 (14 - 1 + 16) for 64K pages with THP >>> >>> Apart from overriding 4K base page size config to have 27 as section size >>> bits, should not all other values be okay here ? But then wondering what >>> benefit 128MB (27 bits) section size for 16K config would have ? OR the >>> objective here is to match 16K page size config with default x86-64. >> >> We don't want to have sections that are too small. We don't want to have >> sections that are too big :) >> >> Not sure if we really want to allow setting e.g., a section size of 4 >> MB. That's just going to hurt. IMHO, something in the range of 64..256 >> MB is quite a good choice, where possible. >> >>> >>>> >>>> (If we worry about the number of section bits in page->flags, we could >>>> glue it to vmemmap support where that does not matter) >>> >>> Could you please elaborate ? Could smaller section size bits numbers like >>> 22 or 24 create problems in page->flags without changing other parameters >>> like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP >> >> Yes, in the !vmemmap case, we have to store the section_nr in there. >> IIRC, it's less of an issue with section sizes like 128 MB. >> >>> i.e 26 bits in section size, fails to boot. >> >> 26 bits would mean 64 MB, no? Not sure if that's possible even without >> THP (MAX_ORDER - 1, pageblock_order ...) on 64k pages. I'd assume 512 MB >> is the lowest we can go. I'd assume this would crash :) >> >>> >>> As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB >>> for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER, >>> at least for now. >> >> That's also what I would prefer, keeping it simple. > > Okay sure, will send a RFC to begin with. > Note that there is https://lkml.kernel.org/r/15cf9a2359197fee0168f820c5c904650d07939e.1610146597.git.sudaraja@codeaurora.org (Sudarshan missed to cc linux-mm) -- Thanks, David / dhildenb ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-11 10:13 ` David Hildenbrand @ 2021-01-11 10:26 ` Anshuman Khandual 0 siblings, 0 replies; 11+ messages in thread From: Anshuman Khandual @ 2021-01-11 10:26 UTC (permalink / raw) To: David Hildenbrand, Sudarshan Rajagopalan, akpm, linux-mm, linux-kernel On 1/11/21 3:43 PM, David Hildenbrand wrote: > On 11.01.21 05:17, Anshuman Khandual wrote: >> >> >> On 1/8/21 9:00 PM, David Hildenbrand wrote: >>>> To summarize, the section size bits for each base page size config >>>> should always >>>> >>>> a. Avoid (MAX_ORDER - 1 + PAGE_SHIFT) > SECTION_SIZE_BITS >>> >>> Pageblocks must also always fall completely into a section. >>> >>>> >>>> b. Provide minimum possible section size for a given base page config to >>>> have increased agility during memory hotplug operations and reduced >>>> vmemmap wastage for sections with holes. >>> >>> OTOH, making the section size too small (e.g., 16MB) creates way to many >>> memory block devices in /sys/devices/system/memory/, and might consume >>> too many page->flags bits in the !vmemmap case. >>> >>> For bigger setups, we might, similar to x86-64 (e.g., >= 64 GiB), >>> determine the memory_block_size_bytes() at runtime (spanning multiple >>> sections then), once it becomes relevant. >>> >>>> >>>> c. Allow 4K base page configs to have PMD based vmemmap mappings >>> >>> Agreed. >>> >>>> >>>> Because CONFIG_FORCE_MAX_ZONEORDER is always defined on arm64 platform, >>>> the following would always avoid the condition (a) >>>> >>>> SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + PAGE_SHIFT) >>>> >>>> - 22 (11 - 1 + 12) for 4K pages >>>> - 24 (11 - 1 + 14) for 16K pages without THP >>>> - 25 (12 - 1 + 14) for 16K pages with THP >>>> - 26 (11 - 1 + 16) for 64K pages without THP >>>> - 29 (14 - 1 + 16) for 64K pages with THP >>>> >>>> Apart from overriding 4K base page size config to have 27 as section size >>>> bits, should not all other values be okay here ? But then wondering what >>>> benefit 128MB (27 bits) section size for 16K config would have ? OR the >>>> objective here is to match 16K page size config with default x86-64. >>> >>> We don't want to have sections that are too small. We don't want to have >>> sections that are too big :) >>> >>> Not sure if we really want to allow setting e.g., a section size of 4 >>> MB. That's just going to hurt. IMHO, something in the range of 64..256 >>> MB is quite a good choice, where possible. >>> >>>> >>>>> >>>>> (If we worry about the number of section bits in page->flags, we could >>>>> glue it to vmemmap support where that does not matter) >>>> >>>> Could you please elaborate ? Could smaller section size bits numbers like >>>> 22 or 24 create problems in page->flags without changing other parameters >>>> like NR_CPUS or NODES_SHIFT ? A quick test with 64K base page without THP >>> >>> Yes, in the !vmemmap case, we have to store the section_nr in there. >>> IIRC, it's less of an issue with section sizes like 128 MB. >>> >>>> i.e 26 bits in section size, fails to boot. >>> >>> 26 bits would mean 64 MB, no? Not sure if that's possible even without >>> THP (MAX_ORDER - 1, pageblock_order ...) on 64k pages. I'd assume 512 MB >>> is the lowest we can go. I'd assume this would crash :) >>> >>>> >>>> As you have suggested, probably constant defaults (128MB for 4K/16K, 512MB >>>> for 64K) might be better than depending on the CONFIG_FORCE_MAX_ZONEORDER, >>>> at least for now. >>> >>> That's also what I would prefer, keeping it simple. >> >> Okay sure, will send a RFC to begin with. >> > > Note that there is > > https://lkml.kernel.org/r/15cf9a2359197fee0168f820c5c904650d07939e.1610146597.git.sudaraja@codeaurora.org > > (Sudarshan missed to cc linux-mm) > Right, some how missed that. Anyways, ended up spending some time testing the change for various configs. ^ permalink raw reply [flat|nested] 11+ messages in thread
* Re: [PATCH 1/1] arm64: make section size configurable for memory hotplug 2021-01-06 6:11 ` Anshuman Khandual 2021-01-07 12:30 ` David Hildenbrand @ 2021-01-08 1:01 ` Sudarshan Rajagopalan 1 sibling, 0 replies; 11+ messages in thread From: Sudarshan Rajagopalan @ 2021-01-08 1:01 UTC (permalink / raw) To: Anshuman Khandual, linux-arm-kernel, Will Deacon Cc: akpm, linux-mm, linux-kernel, Suren Baghdasaryan, Pratik Patel On 2021-01-05 22:11, Anshuman Khandual wrote: Hello Anshuman, thanks for your response. (+ Will) > Hi Sudershan, > > This patch (and the cover letter) does not copy LAKML even though the > entire change here is arm64 specific. Please do copy all applicable > mailing lists for a given patch. I used ./scripts/get_maintainer.pl patch.patch to get the maintainers list. It somehow didn't mention LAKML. I've added the mailing list to this thread. > > On 1/6/21 6:58 AM, Sudarshan Rajagopalan wrote: >> Currently on arm64, memory section size is hard-coded to 1GB. >> Make this configurable if memory-hotplug is enabled, to support >> more finer granularity for hotplug-able memory. > > Section size has always been decided by the platform. It cannot be a > configurable option because the user would not know the constraints > for memory representation on the platform and besides it also cannot > be trusted. > >> >> Signed-off-by: Sudarshan Rajagopalan <sudaraja@codeaurora.org> >> --- >> arch/arm64/Kconfig | 11 +++++++++++ >> arch/arm64/include/asm/sparsemem.h | 4 ++++ >> 2 files changed, 15 insertions(+) >> >> diff --git a/arch/arm64/Kconfig b/arch/arm64/Kconfig >> index 6d232837cbee..34124eee65da 100644 >> --- a/arch/arm64/Kconfig >> +++ b/arch/arm64/Kconfig >> @@ -294,6 +294,17 @@ config ARCH_ENABLE_MEMORY_HOTREMOVE >> config SMP >> def_bool y >> >> +config HOTPLUG_SIZE_BITS >> + int "Memory hotplug block size(29 => 512MB 30 => 1GB)" >> + depends on SPARSEMEM >> + depends on MEMORY_HOTPLUG >> + range 28 30 > > 28 would not work for 64K pages. > >> + default 30 >> + help >> + Selects granularity of hotplug memory. Block size for >> + memory hotplug is represent as a power of 2. >> + If unsure, stick with default value. >> + >> config KERNEL_MODE_NEON >> def_bool y >> >> diff --git a/arch/arm64/include/asm/sparsemem.h >> b/arch/arm64/include/asm/sparsemem.h >> index 1f43fcc79738..3d5310f3aad5 100644 >> --- a/arch/arm64/include/asm/sparsemem.h >> +++ b/arch/arm64/include/asm/sparsemem.h >> @@ -7,7 +7,11 @@ >> >> #ifdef CONFIG_SPARSEMEM >> #define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS >> +#ifndef CONFIG_MEMORY_HOTPLUG >> #define SECTION_SIZE_BITS 30 >> +#else >> +#define SECTION_SIZE_BITS CONFIG_HOTPLUG_SIZE_BITS >> +#endif >> #endif >> >> #endif >> > > There was an inconclusive discussion regarding this last month. > > https://lore.kernel.org/linux-arm-kernel/20201204014443.43329-1-liwei213@huawei.com/ Thanks for pointing out this thread. Looking into all the comments, major concern with reducing the section size seems to be risk of running out of bits in the page flags. And while SECTION_SIZE must be greater or equal to highest order page in the buddy, it must also satisfy cases for 4K page size where it doesn't break PMD mapping for vmemmap - and hence SECTION_SIZE_BITS of 27 could be set for 4K page size that could allow 2MB PMD mappings for each 128M(2^27) block size. While this is the least value that can be set (27 for 4K_PAGE, MAX_ZONEORDER - 1 + PAGE_SHIFT for 16K or 64K_PAGE), are there any concerns with setting higher values (but <= 30bits). It seems like any arbitrary number between this range could be applied that wouldn't break vmemmaps. That's why we were thinking of letting the user configure it since this directly impacts memory hotplug about the granularity or least size that can be hot (up)plugged. The current setting of 1GB for arm64 does poses a lot of challenges in utilizing memory hotplug via a driver, esp. for low RAM targets. I agree its sub-optimal in some sense but wanted to know the maintainers opinion on this. Also, the patch introduced in that thread does seem to help reduce vmemmap memory if there are large holes. So there is some merit in reducing the section size along with memory hotplug leveraging it. > > I have been wondering if this would solve the problem for 4K page size > config which requires PMD mapping for the vmemmap mapping while making > section size bits dependent on max order. But this has not been tested > properly. > > diff --git a/arch/arm64/include/asm/sparsemem.h > b/arch/arm64/include/asm/sparsemem.h > index 1f43fcc79738..fe4353cb1dce 100644 > --- a/arch/arm64/include/asm/sparsemem.h > +++ b/arch/arm64/include/asm/sparsemem.h > @@ -7,7 +7,18 @@ > > #ifdef CONFIG_SPARSEMEM > #define MAX_PHYSMEM_BITS CONFIG_ARM64_PA_BITS > -#define SECTION_SIZE_BITS 30 > -#endif > + > +#ifdef CONFIG_ARM64_4K_PAGES > +#define SECTION_SIZE_BITS 27 > +#else > +#ifdef CONFIG_FORCE_MAX_ZONEORDER > +#define SECTION_SIZE_BITS (CONFIG_FORCE_MAX_ZONEORDER - 1 + > PAGE_SHIFT) > +#else > +#define SECTION_SIZE_BITS 30 > +#endif /* CONFIG_FORCE_MAX_ZONEORDER */ > + > +#endif /* CONFIG_ARM64_4K_PAGES */ > + > +#endif /* CONFIG_SPARSEMEM*/ > > #endif SECTION_SIZE_BITS of 27 for 4K_PAGES should be fine for us. Would you know if there's possibility of this patch above being applied in upstream anytime soon? This is in regards with Generic Kernel Image (GKI) that we are working with Google. If this patch would positively end up in upstream, we could apply it in aosp kernel. Sudarshan -- Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, a Linux Foundation Collaborative Project ^ permalink raw reply [flat|nested] 11+ messages in thread
end of thread, other threads:[~2021-01-11 10:26 UTC | newest] Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-01-06 1:28 [PATCH 0/1] arm64: make section size configurable for memory hotplug Sudarshan Rajagopalan 2021-01-06 1:28 ` [PATCH 1/1] " Sudarshan Rajagopalan 2021-01-06 2:20 ` Randy Dunlap 2021-01-06 6:11 ` Anshuman Khandual 2021-01-07 12:30 ` David Hildenbrand 2021-01-08 7:02 ` Anshuman Khandual 2021-01-08 15:30 ` David Hildenbrand 2021-01-11 4:17 ` Anshuman Khandual 2021-01-11 10:13 ` David Hildenbrand 2021-01-11 10:26 ` Anshuman Khandual 2021-01-08 1:01 ` Sudarshan Rajagopalan
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).