From: Jonathan Cameron <Jonathan.Cameron@Huawei.com>
To: Wei Xu <weixugc@google.com>
Cc: Aneesh Kumar K V <aneesh.kumar@linux.ibm.com>,
Ying Huang <ying.huang@intel.com>,
Andrew Morton <akpm@linux-foundation.org>,
"Greg Thelen" <gthelen@google.com>,
Yang Shi <shy828301@gmail.com>,
"Linux Kernel Mailing List" <linux-kernel@vger.kernel.org>,
Jagdish Gediya <jvgediya@linux.ibm.com>,
Michal Hocko <mhocko@kernel.org>,
Tim C Chen <tim.c.chen@intel.com>,
Dave Hansen <dave.hansen@intel.com>,
"Alistair Popple" <apopple@nvidia.com>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Feng Tang <feng.tang@intel.com>,
Davidlohr Bueso <dave@stgolabs.net>,
"Dan Williams" <dan.j.williams@intel.com>,
David Rientjes <rientjes@google.com>,
Linux MM <linux-mm@kvack.org>,
Brice Goglin <brice.goglin@gmail.com>,
"Hesham Almatary" <hesham.almatary@huawei.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces (v2)
Date: Thu, 26 May 2022 10:32:11 +0100 [thread overview]
Message-ID: <20220526103211.000001ad@Huawei.com> (raw)
In-Reply-To: <CAAPL-u-3-iOoSK07TE=gUN8a+TsXNNEiFaEArf7+zvTgxDJnRQ@mail.gmail.com>
On Wed, 25 May 2022 10:27:42 -0700
Wei Xu <weixugc@google.com> wrote:
> On Wed, May 25, 2022 at 3:01 AM Aneesh Kumar K V
> <aneesh.kumar@linux.ibm.com> wrote:
> >
> > On 5/25/22 2:33 PM, Ying Huang wrote:
> > > On Tue, 2022-05-24 at 22:32 -0700, Wei Xu wrote:
> > >> On Tue, May 24, 2022 at 1:24 AM Ying Huang <ying.huang@intel.com> wrote:
> > >>>
> > >>> On Tue, 2022-05-24 at 00:04 -0700, Wei Xu wrote:
> > >>>> On Thu, May 19, 2022 at 8:06 PM Ying Huang <ying.huang@intel.com> wrote:
> > >>>>>
> >
> > ...
> >
> > >
> > > OK. Just to confirm. Does this mean that we will have fixed device ID,
> > > for example,
> > >
> > > GPU memtier255
> > > DRAM (with CPU) memtier0
> > > PMEM memtier1
> > >
> > > When we add a new memtier, it can be memtier254, or memter2? The rank
> > > value will determine the real demotion order.
> > >
> > > I think you may need to send v3 to make sure everyone is at the same
> > > page.
> > >
> >
> > What we have implemented which we will send as RFC shortly is below.
> >
> > cd /sys/dekvaneesh@ubuntu-guest:~$ cd /sys/devices/system/
> > kvaneesh@ubuntu-guest:/sys/devices/system$ pwd
> > /sys/devices/system
> > kvaneesh@ubuntu-guest:/sys/devices/system$ ls
> > clockevents clocksource container cpu edac memory memtier mpic
> > node power
> > kvaneesh@ubuntu-guest:/sys/devices/system$ cd memtier/
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ pwd
> > /sys/devices/system/memtier
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ ls
> > default_rank max_rank memtier1 power uevent
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat default_rank
> > 1
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cat max_rank
> > 3
>
> For flexibility, we don't want max_rank to be interpreted as the
> number of memory tiers. Also, we want to leave spaces in rank values
> to allow new memtiers to be inserted when needed. So I'd suggest to
> make max_rank a much larger value (e.g. 255).
>
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier$ cd memtier1/
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ ls
> > nodelist power rank subsystem uevent
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat nodelist
> > 0-3
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cat rank
> > 1
> > kvaneesh@ubuntu-guest:/sys/devices/system/memtier/memtier1$ cd
> > ../../node/node1/
> > kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$ cat memtier
> > 1
> > kvaneesh@ubuntu-guest:/sys/devices/system/node/node1$
> > root@ubuntu-guest:/sys/devices/system/node/node1# echo 0 > memtier
> > root@ubuntu-guest:/sys/devices/system/node/node1# cat memtier
> > 0
> > root@ubuntu-guest:/sys/devices/system/node/node1# cd ../../memtier/
> > root@ubuntu-guest:/sys/devices/system/memtier# ls
> > default_rank max_rank memtier0 memtier1 power uevent
> > root@ubuntu-guest:/sys/devices/system/memtier# cd memtier0/
> > root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat nodelist
> > 1
> > root@ubuntu-guest:/sys/devices/system/memtier/memtier0# cat rank
> > 0
>
> It looks like the example here demonstrates the dynamic creation of
> memtier0. If so, how is the rank of memtier0 determined? If we want
> to support creating new memtiers at runtime, I think an explicit
> interface that specifies both device ID and rank is preferred to avoid
> implicit dependencies between device IDs and ranks.
Why make device ID explicit - it's meaningless I think?
How about a creation interface that is simply writing the rank value
to create a new one? The only race I can see would be to get
two parallel attempts to create a new tier with the same rank.
That seems unlikely to matter unless we support changing rank later.
Two attempts to create the same device ID tier seems more likely to
cause fiddly races.
Jonathan
>
> > root@ubuntu-guest:/sys/devices/system/memtier/memtier0# echo 4 > rank
> > bash: rank: Permission denied
> > root@ubuntu-guest:/sys/devices/system/memtier/memtier0#
next prev parent reply other threads:[~2022-05-26 9:32 UTC|newest]
Thread overview: 47+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-05-12 6:22 RFC: Memory Tiering Kernel Interfaces (v2) Wei Xu
2022-05-12 7:03 ` ying.huang
2022-05-12 7:12 ` Aneesh Kumar K V
2022-05-12 7:18 ` ying.huang
2022-05-12 7:22 ` Wei Xu
2022-05-12 7:36 ` Aneesh Kumar K.V
2022-05-12 8:15 ` Wei Xu
2022-05-12 8:37 ` ying.huang
2022-05-13 2:52 ` ying.huang
2022-05-13 7:00 ` Wei Xu
2022-05-16 1:57 ` ying.huang
2022-05-12 21:12 ` Tim Chen
2022-05-12 21:31 ` Wei Xu
2022-05-12 15:00 ` Jonathan Cameron
2022-05-18 7:09 ` Wei Xu
2022-05-18 12:00 ` Jonathan Cameron
2022-05-24 7:36 ` Wei Xu
2022-05-24 13:26 ` Aneesh Kumar K.V
2022-05-25 5:27 ` Wei Xu
2022-05-25 7:47 ` Alistair Popple
2022-05-25 11:48 ` Jonathan Cameron
2022-05-25 15:32 ` Wei Xu
2022-05-20 3:06 ` Ying Huang
2022-05-24 7:04 ` Wei Xu
2022-05-24 8:24 ` Ying Huang
2022-05-25 5:32 ` Wei Xu
2022-05-25 9:03 ` Ying Huang
2022-05-25 10:01 ` Aneesh Kumar K V
2022-05-25 11:36 ` Mika Penttilä
2022-05-25 15:33 ` Wei Xu
2022-05-25 17:27 ` Wei Xu
2022-05-26 9:32 ` Jonathan Cameron [this message]
2022-05-26 20:30 ` Wei Xu
2022-05-27 9:26 ` Aneesh Kumar K V
2022-05-25 15:36 ` Wei Xu
2022-05-26 1:09 ` Ying Huang
2022-05-26 3:53 ` Wei Xu
2022-05-26 6:54 ` Ying Huang
2022-05-26 7:08 ` Wei Xu
2022-05-26 7:39 ` Ying Huang
2022-05-26 20:55 ` Wei Xu
2022-05-27 9:10 ` Jonathan Cameron
2022-05-30 6:54 ` Ying Huang
2022-05-13 3:25 ` ying.huang
2022-05-13 6:36 ` Wei Xu
2022-05-13 7:04 ` ying.huang
2022-05-13 7:21 ` Wei Xu
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20220526103211.000001ad@Huawei.com \
--to=jonathan.cameron@huawei.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=apopple@nvidia.com \
--cc=baolin.wang@linux.alibaba.com \
--cc=brice.goglin@gmail.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@intel.com \
--cc=dave@stgolabs.net \
--cc=feng.tang@intel.com \
--cc=gthelen@google.com \
--cc=hesham.almatary@huawei.com \
--cc=jvgediya@linux.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=rientjes@google.com \
--cc=shy828301@gmail.com \
--cc=tim.c.chen@intel.com \
--cc=weixugc@google.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).