Re: Re: Re: Re: [RFC] autonuma: Support to scan page table asynchronously

From: Mel Gorman <mgorman@suse.de>
To: SeongJae Park <sjpark@amazon.com>
Cc: Mel Gorman <mgorman@techsingularity.net>,
	Peter Zijlstra <peterz@infradead.org>,
	"Huang, Ying" <ying.huang@intel.com>,
	linux-mm@kvack.org, linux-kernel@vger.kernel.org,
	Ingo Molnar <mingo@redhat.com>, Rik van Riel <riel@surriel.com>,
	Daniel Jordan <daniel.m.jordan@oracle.com>,
	Tejun Heo <tj@kernel.org>, Dave Hansen <dave.hansen@intel.com>,
	Tim Chen <tim.c.chen@intel.com>, Aubrey Li <aubrey.li@intel.com>
Subject: Re: Re: Re: Re: [RFC] autonuma: Support to scan page table asynchronously
Date: Fri, 17 Apr 2020 15:46:26 +0100	[thread overview]
Message-ID: <20200417144626.GA3711@suse.de> (raw)
In-Reply-To: <20200417124437.25445-1-sjpark@amazon.com>

On Fri, Apr 17, 2020 at 02:44:37PM +0200, SeongJae Park wrote:
> On Fri, 17 Apr 2020 13:16:29 +0100 Mel Gorman <mgorman@techsingularity.net> wrote:
> 
> > On Fri, Apr 17, 2020 at 12:21:29PM +0200, SeongJae Park wrote:
> > > On Fri, 17 Apr 2020 12:04:17 +0200 Peter Zijlstra <peterz@infradead.org> wrote:
> > > 
> > > > On Fri, Apr 17, 2020 at 09:05:08AM +0200, SeongJae Park wrote:
> > > > > I think the main idea of DAMON[1] might be able to applied here.  Have you
> > > > > considered it?
> > > > > 
> > > > > [1] https://lore.kernel.org/linux-mm/20200406130938.14066-1-sjpark@amazon.com/
> > > > 
> > > > I've ignored that entire thing after you said the information it
> > > > provides was already available through the PMU.
> > > 
> > > Sorry if my answer made you confused.  What I wanted to say was that the
> > > fundamental access checking mechanism that DAMON depends on is PTE Accessed bit
> > > for now, but it could be modified to use PMU or other features instead. 
> > 
> > I would not be inclined to lean towards either approach for NUMA
> > balancing. Fiddling with the accessed bit can have consequences for page
> > aging and residency -- fine for debugging a problem, not to fine for
> > normal usage. I would expect the PMU approach would have high overhead
> > as well as taking over a PMU counter that userspace debugging may expect
> > to be available.
> 
> Good point.  But, isn't it ok to use Accessed bit as long as PG_Idle and
> PG_Young is adjusted properly?  Current DAMON implementation does so, as
> idle_page_tracking also does.
> 

PG_idle and PG_young are page flags that are used by idle tracking. The
accessed bit I'm thinking of is in the PTE bit and it's the PTE that
is changed by page_referenced() during reclaim. So it's not clear how
adjusting the page bits help avoid page aging problems during reclaim.

Maybe your suggestion was to move NUMA balancing to use the PG_idle and
PG_young bits from idle tracking but I'm not sure what that gains us.
This may be because I did not look too closely at DAMON as for debugging
and analysis, the PMU sounded more suitable.

It's not clear to me how looking at the page or pte bit handling of
DAMON helps us reduce the scanning cost of numa balancing. There may
be some benefit in splitting the address space and rescanning sections
that were referenced but it has similar pitfalls to simply tracking the
highest/lowest address that trapped a NUMA hinting fault.

-- 
Mel Gorman
SUSE Labs