Re: RFC: Memory Tiering Kernel Interfaces (v2)

From: Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>
To: Ying Huang <ying.huang@intel.com>, Wei Xu <weixugc@google.com>
Cc: Jonathan Cameron <Jonathan.Cameron@huawei.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Greg Thelen <gthelen@google.com>, Yang Shi <shy828301@gmail.com>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Jagdish Gediya <jvgediya@linux.ibm.com>,
	Michal Hocko <mhocko@kernel.org>,
	Tim C Chen <tim.c.chen@intel.com>,
	Dave Hansen <dave.hansen@intel.com>,
	Alistair Popple <apopple@nvidia.com>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Feng Tang <feng.tang@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Dan Williams <dan.j.williams@intel.com>,
	David Rientjes <rientjes@google.com>,
	Linux MM <linux-mm@kvack.org>,
	Brice Goglin <brice.goglin@gmail.com>,
	Hesham Almatary <hesham.almatary@huawei.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)
Date: Wed, 25 May 2022 15:31:13 +0530	[thread overview]
Message-ID: <594046f8-9ab3-786a-fc48-8a61f1238f52@linux.ibm.com> (raw)
In-Reply-To: <6ce724e5c67d4f7530457897fa08d0a8ba5dd6d0.camel@intel.com>

On 5/25/22 2:33 PM, Ying Huang wrote:
> On Tue, 2022-05-24 at 22:32 -0700, Wei Xu wrote:
>> On Tue, May 24, 2022 at 1:24 AM Ying Huang <ying.huang@intel.com> wrote:
>>>
>>> On Tue, 2022-05-24 at 00:04 -0700, Wei Xu wrote:
>>>> On Thu, May 19, 2022 at 8:06 PM Ying Huang <ying.huang@intel.com> wrote:
>>>>>

...

> 
> OK.  Just to confirm.  Does this mean that we will have fixed device ID,
> for example,
> 
> GPU			memtier255
> DRAM (with CPU)		memtier0
> PMEM			memtier1
> 
> When we add a new memtier, it can be memtier254, or memter2?  The rank
> value will determine the real demotion order.
> 
> I think you may need to send v3 to make sure everyone is at the same
> page.
> 

What we have implemented which we will send as RFC shortly is below.

cd /sys/dekvaneesh@ubuntu-guest:~$ cd /sys/devices/system/
kvaneesh@ubuntu-guest:/sys/devices/system$ pwd
/sys/devices/system
kvaneesh@ubuntu-guest:/sys/devices/system$ ls
clockevents  clocksource  container  cpu  edac  memory  memtier  mpic 
node  power
kvaneesh@ubuntu-guest:/sys/devices/system$ cd memtier/
kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ pwd
/sys/devices/system/memtier
kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ ls
default_rank  max_rank  memtier1  power  uevent
kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat default_rank
1
kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat max_rank
3
kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cd memtier1/
kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ ls
nodelist  power  rank  subsystem  uevent
kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat nodelist
0-3
kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat rank
1
kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cd 
../../node/node1/
kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$ cat memtier
1
kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$
root@ubuntu-guest:/sys/devices/system/node/node1# echo 0 > memtier
root@ubuntu-guest:/sys/devices/system/node/node1# cat memtier
0
root@ubuntu-guest:/sys/devices/system/node/node1# cd ../../memtier/
root@ubuntu-guest:/sys/devices/system/memtier# ls
default_rank  max_rank  memtier0  memtier1  power  uevent
root@ubuntu-guest:/sys/devices/system/memtier# cd memtier0/
root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat nodelist
1
root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat rank
0
root@ubuntu-guest:/sys/devices/system/memtier/memtier0# echo 4 > rank
bash: rank: Permission denied
root@ubuntu-guest:/sys/devices/system/memtier/memtier0#