[PATCH] aarch64/mm: speedup memory initialisation

[PATCH] aarch64/mm: speedup memory initialisation

[Hubert Ralf]

On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
all pages in the memory starting at the lowest address until the highest
address is reached. On arm64 this ends up in searching a memmory region
containing for each single page between lowest and highest available
physicall address.

Having a sparse memory system there may be some big holes in the
memory map. For each page in this holes a lookup is done, which is
implemented as a binary search on the available memory blocks.

Adding a memmap_init for aarch64 to do the init only for the available
memory areas reduces the time needed for initialising memory on startup.
On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
execution time is reduced from 378ms to 84ms.

Signed-off-by: Ralf Hubert <ralf.hubert@preh.de>
---
 arch/arm64/include/asm/pgtable.h |  7 +++++++
 arch/arm64/mm/init.c             | 24 ++++++++++++++++++++++++
 2 files changed, 31 insertions(+)

diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
index e09760ece844..8c6eefc08b0b 100644
--- a/arch/arm64/include/asm/pgtable.h
+++ b/arch/arm64/include/asm/pgtable.h
@@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
 	return (lhs == rhs);
 }
 
+#ifdef CONFIG_SPARSEMEM
+/* arch mem_map init routine is needed due to holes in a memmap */
+#   define __HAVE_ARCH_MEMMAP_INIT
+	void memmap_init(unsigned long size, int nid, unsigned long zone,
+			 unsigned long start_pfn);
+#endif /* CONFIG_SPARSEMEM */
+
 /*
  * Huge pte definitions.
  */
diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
index f3c795278def..206b28310872 100644
--- a/arch/arm64/mm/init.c
+++ b/arch/arm64/mm/init.c
@@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn)
 }
 EXPORT_SYMBOL(pfn_valid);
 
+#ifdef CONFIG_SPARSEMEM
+void __meminit
+memmap_init(unsigned long size, int nid, unsigned long zone,
+	    unsigned long start_pfn)
+{
+	struct memblock_region *reg;
+
+	for_each_memblock(memory, reg) {
+		unsigned long start = memblock_region_memory_base_pfn(reg);
+		unsigned long end = memblock_region_memory_end_pfn(reg);
+
+		if (start < start_pfn)
+			start = start_pfn;
+		if (end > start_pfn + size)
+			end = start_pfn + size;
+
+		if (start < end) {
+			memmap_init_zone(end - start, nid, zone, start,
+					 MEMMAP_EARLY, NULL);
+		}
+	}
+}
+#endif /* CONFIG_SPARSEMEM */
+
 static phys_addr_t memory_limit = PHYS_ADDR_MAX;
 
 /*
-- 
2.23.0

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[Robin Murphy]

On 10/09/2019 09:59, Hubert Ralf wrote:
> On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
> all pages in the memory starting at the lowest address until the highest
> address is reached. On arm64 this ends up in searching a memmory region
> containing for each single page between lowest and highest available
> physicall address.
> 
> Having a sparse memory system there may be some big holes in the
> memory map. For each page in this holes a lookup is done, which is
> implemented as a binary search on the available memory blocks.

It sounds like that's one of the many related issues for which the real 
solution is "handle EFI no-map regions in a way which makes pfn_valid() 
not terrible" - if we are prepared to have workarounds for individual 
hot-spots, they should probably carry a big reminder to put things back 
once pfn_valid() is sorted out.

That said, even with the current implementation, the memblock search is 
short-circuited by the valid_section() check, so the impact should 
already be limited to just boundary regions where RAM is weirdly aligned 
and a hole starts or ends mid-section. Of course, regardless of any 
check overhead there might well still be an argument for not walking 
through large avoidable holes one PFN at a time, but the rationale as 
given doesn't seem to quite add up.

Robin.

> Adding a memmap_init for aarch64 to do the init only for the available
> memory areas reduces the time needed for initialising memory on startup.
> On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
> execution time is reduced from 378ms to 84ms.
> 
> Signed-off-by: Ralf Hubert <ralf.hubert@preh.de>
> ---
>   arch/arm64/include/asm/pgtable.h |  7 +++++++
>   arch/arm64/mm/init.c             | 24 ++++++++++++++++++++++++
>   2 files changed, 31 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index e09760ece844..8c6eefc08b0b 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
>   	return (lhs == rhs);
>   }
>   
> +#ifdef CONFIG_SPARSEMEM
> +/* arch mem_map init routine is needed due to holes in a memmap */
> +#   define __HAVE_ARCH_MEMMAP_INIT
> +	void memmap_init(unsigned long size, int nid, unsigned long zone,
> +			 unsigned long start_pfn);
> +#endif /* CONFIG_SPARSEMEM */
> +
>   /*
>    * Huge pte definitions.
>    */
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index f3c795278def..206b28310872 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn)
>   }
>   EXPORT_SYMBOL(pfn_valid);
>   
> +#ifdef CONFIG_SPARSEMEM
> +void __meminit
> +memmap_init(unsigned long size, int nid, unsigned long zone,
> +	    unsigned long start_pfn)
> +{
> +	struct memblock_region *reg;
> +
> +	for_each_memblock(memory, reg) {
> +		unsigned long start = memblock_region_memory_base_pfn(reg);
> +		unsigned long end = memblock_region_memory_end_pfn(reg);
> +
> +		if (start < start_pfn)
> +			start = start_pfn;
> +		if (end > start_pfn + size)
> +			end = start_pfn + size;
> +
> +		if (start < end) {
> +			memmap_init_zone(end - start, nid, zone, start,
> +					 MEMMAP_EARLY, NULL);
> +		}
> +	}
> +}
> +#endif /* CONFIG_SPARSEMEM */
> +
>   static phys_addr_t memory_limit = PHYS_ADDR_MAX;
>   
>   /*
> 

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[James Morse]

Hi Hubert,

(Subject-Nit: The prefix for this part of the kernel is 'arm64: mm:'. 
git log --oneline $file will usually give you enough examples you can 
spot the pattern.)

On 9/10/19 9:59 AM, Hubert Ralf wrote:
> On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
> all pages in the memory starting at the lowest address until the highest
> address is reached. On arm64 this ends up in searching a memmory region
> containing for each single page between lowest and highest available
> physicall address.

> Having a sparse memory system there may be some big holes in the
> memory map. For each page in this holes a lookup is done, which is
> implemented as a binary search on the available memory blocks.
> 
> Adding a memmap_init for aarch64 to do the init only for the available
> memory areas reduces the time needed for initialising memory on startup.
> On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
> execution time is reduced from 378ms to 84ms.

Hmm, there is nothing arm64 specific about this SPARSEMEM behaviour.
Is there any reason this can't be done in core code, where it would 
benefit other architectures too?

(You'd need it to depend on !ARCH_DISCARD_MEMBLOCK as it looks like 
memory-hotplug uses this late).


Thanks,

James

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[Hubert Ralf]

Hi James,
On 09/11/2019, 17:22 +0100 James Morse wrote:
> Hi Hubert,
> 
> (Subject-Nit: The prefix for this part of the kernel is 'arm64: mm:'. 
> git log --oneline $file will usually give you enough examples you can 
> spot the pattern.)
Thanks for the hint.
> 
> On 9/10/19 9:59 AM, Hubert Ralf wrote:
> > On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
> > all pages in the memory starting at the lowest address until the highest
> > address is reached. On arm64 this ends up in searching a memmory region
> > containing for each single page between lowest and highest available
> > physicall address.
> > Having a sparse memory system there may be some big holes in the
> > memory map. For each page in this holes a lookup is done, which is
> > implemented as a binary search on the available memory blocks.
> > 
> > Adding a memmap_init for aarch64 to do the init only for the available
> > memory areas reduces the time needed for initialising memory on startup.
> > On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
> > execution time is reduced from 378ms to 84ms.
> 
> Hmm, there is nothing arm64 specific about this SPARSEMEM behaviour.
> Is there any reason this can't be done in core code, where it would 
> benefit other architectures too?
I'll try to move this to the core code.
> 
> (You'd need it to depend on !ARCH_DISCARD_MEMBLOCK as it looks like 
> memory-hotplug uses this late).
> 
> 
> Thanks,
> 
> James
Thanks,
Ralf
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[Anshuman Khandual]

On 09/10/2019 02:29 PM, Hubert Ralf wrote:
> On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
> all pages in the memory starting at the lowest address until the highest
> address is reached. On arm64 this ends up in searching a memmory region
> containing for each single page between lowest and highest available
> physicall address.
> 
> Having a sparse memory system there may be some big holes in the
> memory map. For each page in this holes a lookup is done, which is
> implemented as a binary search on the available memory blocks.
> 
> Adding a memmap_init for aarch64 to do the init only for the available
> memory areas reduces the time needed for initialising memory on startup.
> On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
> execution time is reduced from 378ms to 84ms.
> 
> Signed-off-by: Ralf Hubert <ralf.hubert@preh.de>
> ---
>  arch/arm64/include/asm/pgtable.h |  7 +++++++
>  arch/arm64/mm/init.c             | 24 ++++++++++++++++++++++++
>  2 files changed, 31 insertions(+)
> 
> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> index e09760ece844..8c6eefc08b0b 100644
> --- a/arch/arm64/include/asm/pgtable.h
> +++ b/arch/arm64/include/asm/pgtable.h
> @@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
>  	return (lhs == rhs);
>  }
>  
> +#ifdef CONFIG_SPARSEMEM
> +/* arch mem_map init routine is needed due to holes in a memmap */
> +#   define __HAVE_ARCH_MEMMAP_INIT

This is not required any more. Its gone with the following commit which
also made generic memmap_init() an weak function currently overridden
only on ia64.

dfb3ccd00a0 ("mm: make memmap_init a proper function")

> +	void memmap_init(unsigned long size, int nid, unsigned long zone,
> +			 unsigned long start_pfn);
> +#endif /* CONFIG_SPARSEMEM */
> +
>  /*
>   * Huge pte definitions.
>   */
> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> index f3c795278def..206b28310872 100644
> --- a/arch/arm64/mm/init.c
> +++ b/arch/arm64/mm/init.c
> @@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn)
>  }
>  EXPORT_SYMBOL(pfn_valid);
>  
> +#ifdef CONFIG_SPARSEMEM
> +void __meminit
> +memmap_init(unsigned long size, int nid, unsigned long zone,
> +	    unsigned long start_pfn)
> +{
> +	struct memblock_region *reg;
> +
> +	for_each_memblock(memory, reg) {
> +		unsigned long start = memblock_region_memory_base_pfn(reg);
> +		unsigned long end = memblock_region_memory_end_pfn(reg);
> +
> +		if (start < start_pfn)
> +			start = start_pfn;
> +		if (end > start_pfn + size)
> +			end = start_pfn + size;
> +
> +		if (start < end) {
> +			memmap_init_zone(end - start, nid, zone, start,
> +					 MEMMAP_EARLY, NULL);
> +		}
> +	}
> +}
> +#endif /* CONFIG_SPARSEMEM */

In generic mmap_init(), the current high cost comes from early_pfn_valid()
check for each pfn in memmap_init_zone() given that early_pfn_valid() is
pfn_valid() when CONFIG_SPARSEMEM which is known to be expensive on arm64.

Though we cannot do anything about pfns which are really present but the
high cost for non present pfns should be eliminated. The following check
in the above for_each_memblock() loop can achieve that.

if (reg->flags & MEMBLOCK_NOMAP)
	continue;

MEMBLOCK_NOMAP universally should not be initialized into a zone and holes
if any should also be universally skipped across platforms. So these changes
can be moved into generic memmap_init() which will benefit other platforms.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[Hubert Ralf]

Am Donnerstag, den 12.09.2019, 13:42 +0530 schrieb Anshuman Khandual:
> 
> On 09/10/2019 02:29 PM, Hubert Ralf wrote:
> > On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
> > all pages in the memory starting at the lowest address until the highest
> > address is reached. On arm64 this ends up in searching a memmory region
> > containing for each single page between lowest and highest available
> > physicall address.
> > 
> > Having a sparse memory system there may be some big holes in the
> > memory map. For each page in this holes a lookup is done, which is
> > implemented as a binary search on the available memory blocks.
> > 
> > Adding a memmap_init for aarch64 to do the init only for the available
> > memory areas reduces the time needed for initialising memory on startup.
> > On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
> > execution time is reduced from 378ms to 84ms.
> > 
> > Signed-off-by: Ralf Hubert <ralf.hubert@preh.de>
> > ---
> >  arch/arm64/include/asm/pgtable.h |  7 +++++++
> >  arch/arm64/mm/init.c             | 24 ++++++++++++++++++++++++
> >  2 files changed, 31 insertions(+)
> > 
> > diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
> > index e09760ece844..8c6eefc08b0b 100644
> > --- a/arch/arm64/include/asm/pgtable.h
> > +++ b/arch/arm64/include/asm/pgtable.h
> > @@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
> >  	return (lhs == rhs);
> >  }
> >  
> > +#ifdef CONFIG_SPARSEMEM
> > +/* arch mem_map init routine is needed due to holes in a memmap */
> > +#   define __HAVE_ARCH_MEMMAP_INIT
> 
> This is not required any more. Its gone with the following commit which
> also made generic memmap_init() an weak function currently overridden
> only on ia64.
> 
> dfb3ccd00a0 ("mm: make memmap_init a proper function")
> 
> > +	void memmap_init(unsigned long size, int nid, unsigned long zone,
> > +			 unsigned long start_pfn);
> > +#endif /* CONFIG_SPARSEMEM */
> > +
> >  /*
> >   * Huge pte definitions.
> >   */
> > diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
> > index f3c795278def..206b28310872 100644
> > --- a/arch/arm64/mm/init.c
> > +++ b/arch/arm64/mm/init.c
> > @@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn)
> >  }
> >  EXPORT_SYMBOL(pfn_valid);
> >  
> > +#ifdef CONFIG_SPARSEMEM
> > +void __meminit
> > +memmap_init(unsigned long size, int nid, unsigned long zone,
> > +	    unsigned long start_pfn)
> > +{
> > +	struct memblock_region *reg;
> > +
> > +	for_each_memblock(memory, reg) {
> > +		unsigned long start = memblock_region_memory_base_pfn(reg);
> > +		unsigned long end = memblock_region_memory_end_pfn(reg);
> > +
> > +		if (start < start_pfn)
> > +			start = start_pfn;
> > +		if (end > start_pfn + size)
> > +			end = start_pfn + size;
> > +
> > +		if (start < end) {
> > +			memmap_init_zone(end - start, nid, zone, start,
> > +					 MEMMAP_EARLY, NULL);
> > +		}
> > +	}
> > +}
> > +#endif /* CONFIG_SPARSEMEM */
> 
> In generic mmap_init(), the current high cost comes from early_pfn_valid()
> check for each pfn in memmap_init_zone() given that early_pfn_valid() is
> pfn_valid() when CONFIG_SPARSEMEM which is known to be expensive on arm64.
> 
> Though we cannot do anything about pfns which are really present but the
> high cost for non present pfns should be eliminated. The following check
> in the above for_each_memblock() loop can achieve that.
> 
> if (reg->flags & MEMBLOCK_NOMAP)
> 	continue;
> 
> MEMBLOCK_NOMAP universally should not be initialized into a zone and holes
> if any should also be universally skipped across platforms. So these changes
> can be moved into generic memmap_init() which will benefit other platforms.
Not sure if I got this. This is a additional short path for memblocks with
the MEMBLOCK_NOMAP flag set, right? For memblocks without these flag the remaining
code still needs to be executed?

In my case I have 4 memblocks with 1.5GB RAM at 0x4000 0000, 0x4 8000 0000,
0x6 0000 0000 and 0x6 8000 0000. None of them has the MEMBLOCK_NOMAP flag set.
_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel

Re: [PATCH] aarch64/mm: speedup memory initialisation

[Anshuman Khandual]

On 09/12/2019 02:10 PM, Hubert Ralf wrote:
> Am Donnerstag, den 12.09.2019, 13:42 +0530 schrieb Anshuman Khandual:
>>
>> On 09/10/2019 02:29 PM, Hubert Ralf wrote:
>>> On ARM64 memmap_init_zone is used during bootmem_init, which iterates over
>>> all pages in the memory starting at the lowest address until the highest
>>> address is reached. On arm64 this ends up in searching a memmory region
>>> containing for each single page between lowest and highest available
>>> physicall address.
>>>
>>> Having a sparse memory system there may be some big holes in the
>>> memory map. For each page in this holes a lookup is done, which is
>>> implemented as a binary search on the available memory blocks.
>>>
>>> Adding a memmap_init for aarch64 to do the init only for the available
>>> memory areas reduces the time needed for initialising memory on startup.
>>> On a Renesas R-CAR M3 based system with a total hole of 20GB bootmem_init
>>> execution time is reduced from 378ms to 84ms.
>>>
>>> Signed-off-by: Ralf Hubert <ralf.hubert@preh.de>
>>> ---
>>>  arch/arm64/include/asm/pgtable.h |  7 +++++++
>>>  arch/arm64/mm/init.c             | 24 ++++++++++++++++++++++++
>>>  2 files changed, 31 insertions(+)
>>>
>>> diff --git a/arch/arm64/include/asm/pgtable.h b/arch/arm64/include/asm/pgtable.h
>>> index e09760ece844..8c6eefc08b0b 100644
>>> --- a/arch/arm64/include/asm/pgtable.h
>>> +++ b/arch/arm64/include/asm/pgtable.h
>>> @@ -298,6 +298,13 @@ static inline int pte_same(pte_t pte_a, pte_t pte_b)
>>>  	return (lhs == rhs);
>>>  }
>>>  
>>> +#ifdef CONFIG_SPARSEMEM
>>> +/* arch mem_map init routine is needed due to holes in a memmap */
>>> +#   define __HAVE_ARCH_MEMMAP_INIT
>>
>> This is not required any more. Its gone with the following commit which
>> also made generic memmap_init() an weak function currently overridden
>> only on ia64.
>>
>> dfb3ccd00a0 ("mm: make memmap_init a proper function")
>>
>>> +	void memmap_init(unsigned long size, int nid, unsigned long zone,
>>> +			 unsigned long start_pfn);
>>> +#endif /* CONFIG_SPARSEMEM */
>>> +
>>>  /*
>>>   * Huge pte definitions.
>>>   */
>>> diff --git a/arch/arm64/mm/init.c b/arch/arm64/mm/init.c
>>> index f3c795278def..206b28310872 100644
>>> --- a/arch/arm64/mm/init.c
>>> +++ b/arch/arm64/mm/init.c
>>> @@ -250,6 +250,30 @@ int pfn_valid(unsigned long pfn)
>>>  }
>>>  EXPORT_SYMBOL(pfn_valid);
>>>  
>>> +#ifdef CONFIG_SPARSEMEM
>>> +void __meminit
>>> +memmap_init(unsigned long size, int nid, unsigned long zone,
>>> +	    unsigned long start_pfn)
>>> +{
>>> +	struct memblock_region *reg;
>>> +
>>> +	for_each_memblock(memory, reg) {
>>> +		unsigned long start = memblock_region_memory_base_pfn(reg);
>>> +		unsigned long end = memblock_region_memory_end_pfn(reg);
>>> +
>>> +		if (start < start_pfn)
>>> +			start = start_pfn;
>>> +		if (end > start_pfn + size)
>>> +			end = start_pfn + size;
>>> +
>>> +		if (start < end) {
>>> +			memmap_init_zone(end - start, nid, zone, start,
>>> +					 MEMMAP_EARLY, NULL);
>>> +		}
>>> +	}
>>> +}
>>> +#endif /* CONFIG_SPARSEMEM */
>>
>> In generic mmap_init(), the current high cost comes from early_pfn_valid()
>> check for each pfn in memmap_init_zone() given that early_pfn_valid() is
>> pfn_valid() when CONFIG_SPARSEMEM which is known to be expensive on arm64.
>>
>> Though we cannot do anything about pfns which are really present but the
>> high cost for non present pfns should be eliminated. The following check
>> in the above for_each_memblock() loop can achieve that.
>>
>> if (reg->flags & MEMBLOCK_NOMAP)
>> 	continue;
>>
>> MEMBLOCK_NOMAP universally should not be initialized into a zone and holes
>> if any should also be universally skipped across platforms. So these changes
>> can be moved into generic memmap_init() which will benefit other platforms.
> Not sure if I got this. This is a additional short path for memblocks with
> the MEMBLOCK_NOMAP flag set, right? For memblocks without these flag the remaining
> code still needs to be executed?

Right, they should be initialized in the zone.

> 
> In my case I have 4 memblocks with 1.5GB RAM at 0x4000 0000, 0x4 8000 0000,
> 0x6 0000 0000 and 0x6 8000 0000. None of them has the MEMBLOCK_NOMAP flag set.

All of them should be added into the zone with memmap_init_zone() because none of
them has the flag.

_______________________________________________
linux-arm-kernel mailing list
linux-arm-kernel@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/linux-arm-kernel