Re: Re: [RFC] autonuma: Support to scan page table asynchronously

[SeongJae Park <sjpark@amazon.com>]

On Wed, 15 Apr 2020 16:14:38 +0800 "Huang\, Ying" <ying.huang@intel.com> wrote:

> Mel Gorman <mgorman@techsingularity.net> writes:
> 
> > On Tue, Apr 14, 2020 at 04:19:51PM +0800, Huang Ying wrote:
> >> In current AutoNUMA implementation, the page tables of the processes
> >> are scanned periodically to trigger the NUMA hint page faults.  The
> >> scanning runs in the context of the processes, so will delay the
> >> running of the processes.  In a test with 64 threads pmbench memory
> >> accessing benchmark on a 2-socket server machine with 104 logical CPUs
> >> and 256 GB memory, there are more than 20000 latency outliers that are
> >> > 1 ms in 3600s run time.  These latency outliers are almost all
> >> caused by the AutoNUMA page table scanning.  Because they almost all
> >> disappear after applying this patch to scan the page tables
> >> asynchronously.
> >> 
> >> Because there are idle CPUs in system, the asynchronous running page
> >> table scanning code can run on these idle CPUs instead of the CPUs the
> >> workload is running on.
> >> 
> >> So on system with enough idle CPU time, it's better to scan the page
> >> tables asynchronously to take full advantages of these idle CPU time.
> >> Another scenario which can benefit from this is to scan the page
> >> tables on some service CPUs of the socket, so that the real workload
> >> can run on the isolated CPUs without the latency outliers caused by
> >> the page table scanning.
> >> 
> >> But it's not perfect to scan page tables asynchronously too.  For
> >> example, on system without enough idle CPU time, the CPU time isn't
> >> scheduled fairly because the page table scanning is charged to the
> >> workqueue thread instead of the process/thread it works for.  And
> >> although the page tables are scanned for the target process, it may
> >> run on a CPU that is not in the cpuset of the target process.
> >> 
> >> One possible solution is to let the system administrator to choose the
> >> better behavior for the system via a sysctl knob (implemented in the
> >> patch).  But it's not perfect too.  Because every user space knob adds
> >> maintenance overhead.
> >> 
> >> A better solution may be to back-charge the CPU time to scan the page
> >> tables to the process/thread, and find a way to run the work on the
> >> proper cpuset.  After some googling, I found there's some discussion
> >> about this as in the following thread,
> >> 
> >> https://lkml.org/lkml/2019/6/13/1321
> >> 
> >> So this patch may be not ready to be merged by upstream yet.  It
> >> quantizes the latency outliers caused by the page table scanning in
> >> AutoNUMA.  And it provides a possible way to resolve the issue for
> >> users who cares about it.  And it is a potential customer of the work
> >> related to the cgroup-aware workqueue or other asynchronous execution
> >> mechanisms.
> >> 
> >
> > The caveats you list are the important ones and the reason why it was
> > not done asynchronously. In an earlier implementation all the work was
> > done by a dedicated thread and ultimately abandoned.
> >
> > There is no guarantee there is an idle CPU available and one that is
> > local to the thread that should be doing the scanning. Even if there is,
> > it potentially prevents another task from scheduling on an idle CPU and
> > similarly other workqueue tasks may be delayed waiting on the scanner. The
> > hiding of the cost is also problematic because the CPU cost is hidden
> > and mixed with other unrelated workqueues. It also has the potential
> > to mask bugs. Lets say for example there is a bug whereby a task is
> > scanning excessively, that can be easily missed when the work is done by
> > a workqueue.
> 
> Do you think something like cgroup-aware workqueue is a solution deserve
> to be tried when it's available?  It will not hide the scanning cost,
> because the CPU time will be charged to the original cgroup or task.
> Although the other tasks may be disturbed, cgroup can provide some kind
> of management via cpusets.
> 
> > While it's just an opinion, my preference would be to focus on reducing
> > the cost and amount of scanning done -- particularly for threads. For
> > example, all threads operate on the same address space but there can be
> > significant overlap where all threads are potentially scanning the same
> > areas or regions that the thread has no interest in. One option would be
> > to track the highest and lowest pages accessed and only scan within
> > those regions for example. The tricky part is that library pages may
> > create very wide windows that render the tracking useless but it could
> > at least be investigated. 
> 
> In general, I think it's good to reduce the scanning cost.

I think the main idea of DAMON[1] might be able to applied here.  Have you
considered it?

[1] https://lore.kernel.org/linux-mm/20200406130938.14066-1-sjpark@amazon.com/


Thanks,
SeongJae Park

> 
> Why do you think there will be overlap between the threads of a process?
> If my understanding were correctly, the threads will scan one by one
> instead of simultaneously.  And how to determine whether a vma need to
> be scanned or not?  For example, there may be only a small portion of
> pages been accessed in a vma, but they may be accessed remotely and
> consumes quite some inter-node bandwidth, so need to be migrated.
> 
> Best Regards,
> Huang, Ying
>