Re: RFC: Memory Tiering Kernel Interfaces

From: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
To: Davidlohr Bueso <dave@stgolabs.net>, Wei Xu <weixugc@google.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	Huang Ying <ying.huang@intel.com>,
	Dan Williams <dan.j.williams@intel.com>,
	Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
	Greg Thelen <gthelen@google.com>,
	Jagdish Gediya <jvgediya@linux.ibm.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Alistair Popple <apopple@nvidia.com>,
	Michal Hocko <mhocko@kernel.org>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Brice Goglin <brice.goglin@gmail.com>,
	Feng Tang <feng.tang@intel.com>,
	Jonathan.Cameron@huawei.com
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Date: Mon, 02 May 2022 12:53:11 +0530	[thread overview]
Message-ID: <87ee1cv1nk.fsf@linux.ibm.com> (raw)
In-Reply-To: <20220501175813.tvytoosygtqlh3nn@offworld>

Davidlohr Bueso <dave@stgolabs.net> writes:

> Nice summary, thanks. I don't know who of the interested parties will be
> at lsfmm, but fyi we have a couple of sessions on memory tiering Tuesday
> at 14:00 and 15:00.

Will there be an online option this time? If so, i would like to
participate in this discussion. I have not closely followed LSF/MM
details this year. So not sure how to get the online attend request out.

>
> On Fri, 29 Apr 2022, Wei Xu wrote:
>
>>The current kernel has the basic memory tiering support: Inactive
>>pages on a higher tier NUMA node can be migrated (demoted) to a lower
>>tier NUMA node to make room for new allocations on the higher tier
>>NUMA node.  Frequently accessed pages on a lower tier NUMA node can be
>>migrated (promoted) to a higher tier NUMA node to improve the
>>performance.
>
> Regardless of the promotion algorithm, at some point I see the NUMA hinting
> fault mechanism being in the way of performance. It would be nice if hardware
> began giving us page "heatmaps" instead of having to rely on faulting or
> sampling based ways to identify hot memory.

Power10 hardware can do this. Right now we are looking at integrating
this to MultiGen LRU. We haven't got it to work. One of the challenges is
how to estimate the relative hotness of the page compared to the rest of the
pages in the system. I am looking at the random sampling of the oldest
generation pages (the page list in the shrink page list) and using the hot
and cold page in that random sample to determine the hotness of a
specific page and whether to reclaim it or not.

-aneesh