Re: [PATCH v2 0/5] mm: support parallel free of memory

From: Aaron Lu <aaron.lu@intel.com>
To: Dave Hansen <dave.hansen@intel.com>
Cc: Michal Hocko <mhocko@kernel.org>,
	Tim Chen <tim.c.chen@linux.intel.com>,
	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	Tim Chen <tim.c.chen@intel.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Ying Huang <ying.huang@intel.com>
Subject: Re: [PATCH v2 0/5] mm: support parallel free of memory
Date: Wed, 22 Mar 2017 16:02:06 +0800	[thread overview]
Message-ID: <20170322080206.GB2360@aaronlu.sh.intel.com> (raw)
In-Reply-To: <ae4e3597-f664-e5c4-97fb-e07f230d5017@intel.com>

On Tue, Mar 21, 2017 at 07:54:37AM -0700, Dave Hansen wrote:
> On 03/16/2017 02:07 AM, Michal Hocko wrote:
> > On Wed 15-03-17 14:38:34, Tim Chen wrote:
> >> max_active:   time
> >> 1             8.9s   ±0.5%
> >> 2             5.65s  ±5.5%
> >> 4             4.84s  ±0.16%
> >> 8             4.77s  ±0.97%
> >> 16            4.85s  ±0.77%
> >> 32            6.21s  ±0.46%
> > 
> > OK, but this will depend on the HW, right? Also now that I am looking at
> > those numbers more closely. This was about unmapping 320GB area and
> > using 4 times more CPUs you managed to half the run time. Is this really
> > worth it? Sure if those CPUs were idle then this is a clear win but if
> > the system is moderately busy then it doesn't look like a clear win to
> > me.
> 
> This still suffers from zone lock contention.  It scales much better if
> we are freeing memory from more than one zone.  We would expect any
> other generic page allocator scalability improvements to really help
> here, too.
> 
> Aaron, could you make sure to make sure that the memory being freed is
> coming from multiple NUMA nodes?  It might also be interesting to boot

The test machine has 4 nodes and each has 128G memory.
With the test size of 320G, at least 3 nodes are involved.

But since the test is done on an idle system, I *guess* the allocated
memory is physically continuous. Then when they are freed in virtually
continuous order, it's likely that one after another physically continous
1G chunk are sent to the free kworkers. So roughly for the first
128 1G chunks, those workers will all be contending on the same zone.
(well, it shouldn't be 128 kworkers all runnable contending for the same
lock since early launched kworkers will have exited after finishing its
job before some later launched kworkers start).

> with a fake NUMA configuration with a *bunch* of nodes to see what the
> best case looks like when zone lock contention isn't even in play where
> one worker would be working on its own zone.

Good idea, will post results here once I finished the test.

Thanks.