Re: RFC: Memory Tiering Kernel Interfaces (v2)

[Jonathan Cameron <Jonathan.Cameron@Huawei.com>]

On Wed, 25 May 2022 17:47:33 +1000
Alistair Popple <apopple@nvidia.com> wrote:

> Wei Xu <weixugc@google.com> writes:
> 
> > On Tue, May 24, 2022 at 6:27 AM Aneesh Kumar K.V
> > <aneesh.kumar@linux.ibm.com> wrote:  
> >>
> >> Wei Xu <weixugc@google.com> writes:
> >>  
> >> > On Wed, May 18, 2022 at 5:00 AM Jonathan Cameron
> >> > <Jonathan.Cameron@huawei.com> wrote:  
> >> >>
> >> >> On Wed, 18 May 2022 00:09:48 -0700
> >> >> Wei Xu <weixugc@google.com> wrote:  
> >>
> >> ...
> >>  
> >> > Nice :)  
> >> >>
> >> >> Initially I thought this was over complicated when compared to just leaving space, but
> >> >> after a chat with Hesham just now you have us both convinced that this is an elegant solution.
> >> >>
> >> >> Few corners probably need fleshing out:
> >> >> *  Use of an allocator for new tiers. Flat number at startup, or new one on write of unique
> >> >>    value to set_memtier perhaps?  Also whether to allow drivers to allocate (I think
> >> >>    we should).
> >> >> *  Multiple tiers with same rank.  My assumption is from demotion path point of view you
> >> >>    fuse them (treat them as if they were a single tier), but keep them expressed
> >> >>    separately in the sysfs interface so that the rank can be changed independently.
> >> >> *  Some guidance on what values make sense for given rank default that might be set by
> >> >>    a driver. If we have multiple GPU vendors, and someone mixes them in a system we
> >> >>    probably don't want the default values they use to result in demotion between them.
> >> >>    This might well be a guidance DOC or appropriate set of #define  
> >> >
> >> > All of these are good ideas, though I am afraid that these can make
> >> > tier management too complex for what it's worth.
> >> >
> >> > How about an alternative tier numbering scheme that uses major.minor
> >> > device IDs?  For simplicity, we can just start with 3 major tiers.
> >> > New tiers can be inserted in-between using minor tier IDs.  
> >>
> >>
> >> What drives the creation of a new memory tier here?  Jonathan was
> >> suggesting we could do something similar to writing to set_memtier for
> >> creating a new memory tier.
> >>
> >> $ echo "memtier128" > sys/devices/system/node/node1/set_memtier
> >>
> >> But I am wondering whether we should implement that now. If we keep
> >> "rank" concept and detach tier index (memtier0 is the memory tier with
> >> index 0) separate from rank, I assume we have enough flexibility for a
> >> future extension that will allow us to create a memory tier from userspace
> >> and assigning it a rank value that helps the device to be placed before or
> >> after DRAM in demotion order.
> >>
> >> ie, For now we will only have memtier0, memtier1, memtier2. We won't add
> >> dynamic creation of memory tiers and the above memory tiers will have
> >> rank value 0, 1, 2 according with demotion order 0 -> 1 -> 2.  
> >
> > Great. So the consensus is to go with the "rank" approach.  The above
> > sounds good to me as a starting point.  
> 
> The rank approach seems good to me too.

Rank is good, but I do slightly worry about accidentally defining ABI
that people care about with the particular numbers used for the initial ranks.

Maybe just x100 on all of them to allow things in between with no change to
this initial set of 3?  So 0, 100, 200

Jonathan

> 
>  - Alistair
> 
> >> -aneesh