Re: [PATCH 0/13] Parallel struct page initialisation v4

[Mel Gorman <mgorman@suse.de>]

On Wed, May 06, 2015 at 10:37:28PM -0400, Waiman Long wrote:
> On 05/06/2015 01:58 PM, Waiman Long wrote:
> >On 05/06/2015 06:22 AM, Mel Gorman wrote:
> >>On Wed, May 06, 2015 at 08:12:46AM +0100, Mel Gorman wrote:
> >>>On Tue, May 05, 2015 at 03:25:49PM -0700, Andrew Morton wrote:
> >>>>On Tue, 5 May 2015 23:13:29 +0100 Mel Gorman<mgorman@suse.de>  wrote:
> >>>>
> >>>>>>Alternatively, the page allocator can go off and synchronously
> >>>>>>initialize some pageframes itself.  Keep doing that until the
> >>>>>>allocation attempt succeeds.
> >>>>>>
> >>>>>That was rejected during review of earlier attempts at
> >>>>>this feature on
> >>>>>the grounds that it impacted allocator fast paths.
> >>>>eh?  Changes are only needed on the allocation-attempt-failed path,
> >>>>which is slow-path.
> >>>We'd have to distinguish between falling back to other zones
> >>>because the
> >>>high zone is artifically exhausted and normal ALLOC_BATCH
> >>>exhaustion. We'd
> >>>also have to avoid falling back to remote nodes prematurely.
> >>>While I have
> >>>not tried an implementation, I expected they would need to be
> >>>in the fast
> >>>paths unless I used jump labels to get around it. I'm going to
> >>>try altering
> >>>when we initialise instead so that it happens earlier.
> >>>
> >>Which looks as follows. Waiman, a test on the 24TB machine would be
> >>appreciated again. This patch should be applied instead of "mm: meminit:
> >>Take into account that large system caches scale linearly with memory"
> >>
> >>---8<---
> >>mm: meminit: Finish initialisation of memory before basic setup
> >>
> >>Waiman Long reported that 24TB machines hit OOM during basic setup when
> >>struct page initialisation was deferred. One approach is to
> >>initialise memory
> >>on demand but it interferes with page allocator paths. This
> >>patch creates
> >>dedicated threads to initialise memory before basic setup. It
> >>then blocks
> >>on a rw_semaphore until completion as a wait_queue and counter
> >>is overkill.
> >>This may be slower to boot but it's simplier overall and also
> >>gets rid of a
> >>lot of section mangling which existed so kswapd could do the
> >>initialisation.
> >>
> >>Signed-off-by: Mel Gorman<mgorman@suse.de>
> >>
> >
> >This patch moves the deferred meminit from kswapd to its own
> >kernel threads started after smp_init(). However, the hash table
> >allocation was done earlier than that. It seems like it will still
> >run out of memory in the 24TB machine that I tested on.
> >
> >I will certainly try it out, but I doubt it will solve the problem
> >on its own.
> 
> It turns out that the two new patches did work on the 24-TB
> DragonHawk without the "mm: meminit: Take into account that large
> system caches scale linearly with memory" patch. The bootup time was
> 357s which was just a few seconds slower than the other bootup times
> that I sent you yesterday.
> 

Grand. This is what I expected because the previous failure was not the
hash tables, it was later allocations and the parallel initialisation
was early enough.

> BTW, do you want to change the following log message as kswapd will
> no longer be the one doing deferred meminit?
> 
>     kswapd 0 initialised 396098436 pages in 6024ms
> 

I will.

-- 
Mel Gorman
SUSE Labs