Re: [PATCH 0/13] Parallel struct page initialisation v4

[Waiman Long <waiman.long@hp.com>]

On 05/05/2015 06:45 AM, Mel Gorman wrote:
> On Mon, May 04, 2015 at 02:30:46PM -0700, Andrew Morton wrote:
>>> Before the patch, the boot time from elilo prompt to ssh login was 694s.
>>> After the patch, the boot up time was 346s, a saving of 348s (about 50%).
>> Having to guesstimate the amount of memory which is needed for a
>> successful boot will be painful.  Any number we choose will be wrong
>> 99% of the time.
>>
>> If the kswapd threads have started, all we need to do is to wait: take
>> a little nap in the allocator's page==NULL slowpath.
>>
>> I'm not seeing any reason why we can't start kswapd much earlier -
>> right at the start of do_basic_setup()?
> It doesn't even have to be kswapd, it just should be a thread pinned to
> a done. The difficulty is that dealing with the system hashes means the
> initialisation has to happen before vfs_caches_init_early() when there is
> no scheduler. Those allocations could be delayed further but then there is
> the possibility that the allocations would not be contiguous and they'd
> have to rely on CMA to make the attempt. That potentially alters the
> performance of the large system hashes at run time.
>
> We can scale the amount initialised with memory sizes relatively easy.
> This boots on the same 1TB machine I was testing before but that is
> hardly a surprise.
>
> ---8<---
> mm: meminit: Take into account that large system caches scale linearly with memory
>
> Waiman Long reported a 24TB machine triggered an OOM as parallel memory
> initialisation deferred too much memory for initialisation. The likely
> consumer of this memory was large system hashes that scale with memory
> size. This patch initialises at least 2G per node but scales the amount
> initialised for larger systems.
>
> Signed-off-by: Mel Gorman<mgorman@suse.de>
> ---
>   mm/page_alloc.c | 15 +++++++++++++--
>   1 file changed, 13 insertions(+), 2 deletions(-)
>
> diff --git a/mm/page_alloc.c b/mm/page_alloc.c
> index 598f78d6544c..f7cc6c9fb909 100644
> --- a/mm/page_alloc.c
> +++ b/mm/page_alloc.c
> @@ -266,15 +266,16 @@ static inline bool early_page_nid_uninitialised(unsigned long pfn, int nid)
>    */
>   static inline bool update_defer_init(pg_data_t *pgdat,
>   				unsigned long pfn, unsigned long zone_end,
> +				unsigned long max_initialise,
>   				unsigned long *nr_initialised)
>   {
>   	/* Always populate low zones for address-contrained allocations */
>   	if (zone_end<  pgdat_end_pfn(pgdat))
>   		return true;
>
> -	/* Initialise at least 2G of the highest zone */
> +	/* Initialise at least the requested amount in the highest zone */
>   	(*nr_initialised)++;
> -	if (*nr_initialised>  (2UL<<  (30 - PAGE_SHIFT))&&
> +	if ((*nr_initialised>  max_initialise)&&
>   	(pfn&  (PAGES_PER_SECTION - 1)) == 0) {
>   		pgdat->first_deferred_pfn = pfn;
>   		return false;
> @@ -299,6 +300,7 @@ static inline bool early_page_nid_uninitialised(unsigned long pfn, int nid)
>
>   static inline bool update_defer_init(pg_data_t *pgdat,
>   				unsigned long pfn, unsigned long zone_end,
> +				unsigned long max_initialise,
>   				unsigned long *nr_initialised)
>   {
>   	return true;
> @@ -4457,11 +4459,19 @@ void __meminit memmap_init_zone(unsigned long size, int nid, unsigned long zone,
>   	unsigned long end_pfn = start_pfn + size;
>   	unsigned long pfn;
>   	struct zone *z;
> +	unsigned long max_initialise;
>   	unsigned long nr_initialised = 0;
>
>   	if (highest_memmap_pfn<  end_pfn - 1)
>   		highest_memmap_pfn = end_pfn - 1;
>
> +	/*
> +	 * Initialise at least 2G of a node but also take into account that
> +	 * two large system hashes that can take up an 8th of memory.
> +	 */
> +	max_initialise = min(2UL<<  (30 - PAGE_SHIFT),
> +			(pgdat->node_spanned_pages>>  3));
> +

I think you may be pre-allocating too much memory here. On the 24-TB 
machine, the size of the dentry and inode hash tables were 16G each. So 
the ratio is about is about 32G/24T = 0.13%. I think a shift factor of 
(>> 8) which is about 0.39% should be more than enough. For the 24TB 
machine, that means a preallocated memory of 96+4G which should be even 
more than the 64+4G in the modified kernel that I used. At the same 
time, I think we can also set the minimum to 1G or even 0.5G for better 
performance for systems that have many CPUs, but not as much memory per 
node.

Cheers,
Longman