Re: RFC: Memory Tiering Kernel Interfaces (v2)

From: "Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>
To: Wei Xu <weixugc@google.com>,
	Jonathan Cameron <Jonathan.Cameron@huawei.com>
Cc: Dave Hansen <dave.hansen@intel.com>,
	Alistair Popple <apopple@nvidia.com>,
	Huang Ying <ying.huang@intel.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Greg Thelen <gthelen@google.com>, Yang Shi <shy828301@gmail.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Jagdish Gediya <jvgediya@linux.ibm.com>,
	Michal Hocko <mhocko@kernel.org>,
	Tim C Chen <tim.c.chen@intel.com>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Feng Tang <feng.tang@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Dan Williams <dan.j.williams@intel.com>,
	David Rientjes <rientjes@google.com>,
	Linux MM <linux-mm@kvack.org>,
	Brice Goglin <brice.goglin@gmail.com>,
	Hesham Almatary <hesham.almatary@huawei.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)
Date: Tue, 24 May 2022 18:56:54 +0530	[thread overview]
Message-ID: <8735gzdpsx.fsf@linux.ibm.com> (raw)
In-Reply-To: <CAAPL-u_c+q_uCMJXOtYGg42Fj2gSnD6c8vgYQmi1iVpRwHiQTw@mail.gmail.com>

Wei Xu <weixugc@google.com> writes:

> On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron
> <Jonathan.Cameron@huawei.com> wrote:
>>
>> On Wed, 18 May 2022 00:09:48 -0700
>> Wei Xu <weixugc@google.com> wrote:

...

> Nice :)
>>
>> Initially I thought this was over complicated when compared to just leaving space, but
>> after a chat with Hesham just now you have us both convinced that this is an elegant solution.
>>
>> Few corners probably need fleshing out:
>> *  Use of an allocator for new tiers. Flat number at startup, or new one on write of unique
>>    value to set_memtier perhaps?  Also whether to allow drivers to allocate (I think
>>    we should).
>> *  Multiple tiers with same rank.  My assumption is from demotion path point of view you
>>    fuse them (treat them as if they were a single tier), but keep them expressed
>>    separately in the sysfs interface so that the rank can be changed independently.
>> *  Some guidance on what values make sense for given rank default that might be set by
>>    a driver. If we have multiple GPU vendors, and someone mixes them in a system we
>>    probably don't want the default values they use to result in demotion between them.
>>    This might well be a guidance DOC or appropriate set of #define
>
> All of these are good ideas, though I am afraid that these can make
> tier management too complex for what it's worth.
>
> How about an alternative tier numbering scheme that uses major.minor
> device IDs?  For simplicity, we can just start with 3 major tiers.
> New tiers can be inserted in-between using minor tier IDs.

What drives the creation of a new memory tier here?  Jonathan was
suggesting we could do something similar to writing to set_memtier for
creating a new memory tier.

$ echo "memtier128" > sys/devices/system/node/node1/set_memtier

But I am wondering whether we should implement that now. If we keep
"rank" concept and detach tier index (memtier0 is the memory tier with
index 0) separate from rank, I assume we have enough flexibility for a
future extension that will allow us to create a memory tier from userspace
and assigning it a rank value that helps the device to be placed before or
after DRAM in demotion order. 

ie, For now we will only have memtier0, memtier1, memtier2. We won't add
dynamic creation of memory tiers and the above memory tiers will have
rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2.

-aneesh