From: Wei Xu <weixugc@google.com>
To: Tim Chen <tim.c.chen@linux.intel.com>
Cc: Andrew Morton <akpm@linux-foundation.org>,
Dave Hansen <dave.hansen@linux.intel.com>,
Huang Ying <ying.huang@intel.com>,
Dan Williams <dan.j.williams@intel.com>,
Yang Shi <shy828301@gmail.com>, Linux MM <linux-mm@kvack.org>,
Greg Thelen <gthelen@google.com>,
"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
Jagdish Gediya <jvgediya@linux.ibm.com>,
Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
Alistair Popple <apopple@nvidia.com>,
Davidlohr Bueso <dave@stgolabs.net>,
Michal Hocko <mhocko@kernel.org>,
Baolin Wang <baolin.wang@linux.alibaba.com>,
Brice Goglin <brice.goglin@gmail.com>,
Feng Tang <feng.tang@intel.com>,
Jonathan Cameron <Jonathan.Cameron@huawei.com>
Subject: Re: RFC: Memory Tiering Kernel Interfaces
Date: Thu, 5 May 2022 00:02:25 -0700 [thread overview]
Message-ID: <CAAPL-u93P1FmM6+TvKDAKuqbUaTBouswqw7uQhvommMEP1-oVg@mail.gmail.com> (raw)
In-Reply-To: <1b6ab66587cfef8574f38cafdd1796daf2a92346.camel@linux.intel.com>
On Tue, May 3, 2022 at 12:12 PM Tim Chen <tim.c.chen@linux.intel.com> wrote:
>
> On Fri, 2022-04-29 at 19:10 -0700, Wei Xu wrote:
> > The current kernel has the basic memory tiering support: Inactive
> > pages on a higher tier NUMA node can be migrated (demoted) to a lower
> > tier NUMA node to make room for new allocations on the higher tier
> > NUMA node. Frequently accessed pages on a lower tier NUMA node can be
> > migrated (promoted) to a higher tier NUMA node to improve the
> > performance.
> >
> > A tiering relationship between NUMA nodes in the form of demotion path
> > is created during the kernel initialization and updated when a NUMA
> > node is hot-added or hot-removed. The current implementation puts all
> > nodes with CPU into the top tier, and then builds the tiering hierarchy
> > tier-by-tier by establishing the per-node demotion targets based on
> > the distances between nodes.
>
> Thanks for making this proposal. It has many of the elements needed
> for the tiering support.
>
> >
> > The current memory tiering interface needs to be improved to address
> > several important use cases:
> >
> > * The current tiering initialization code always initializes
> > each memory-only NUMA node into a lower tier. But a memory-only
> > NUMA node may have a high performance memory device (e.g. a DRAM
> > device attached via CXL.mem or a DRAM-backed memory-only node on
> > a virtual machine) and should be put into the top tier.
> >
> > * The current tiering hierarchy always puts CPU nodes into the top
> > tier. But on a system with HBM (e.g. GPU memory) devices, these
> > memory-only HBM NUMA nodes should be in the top tier, and DRAM nodes
> > with CPUs are better to be placed into the next lower tier.
> >
> > * Also because the current tiering hierarchy always puts CPU nodes
> > into the top tier, when a CPU is hot-added (or hot-removed) and
> > triggers a memory node from CPU-less into a CPU node (or vice
> > versa), the memory tiering hierarchy gets changed, even though no
> > memory node is added or removed. This can make the tiering
> > hierarchy much less stable.
> >
> > * A higher tier node can only be demoted to selected nodes on the
> > next lower tier, not any other node from the next lower tier. This
> > strict, hard-coded demotion order does not work in all use cases
> > (e.g. some use cases may want to allow cross-socket demotion to
> > another node in the same demotion tier as a fallback when the
> > preferred demotion node is out of space), and has resulted in the
> > feature request for an interface to override the system-wide,
> > per-node demotion order from the userspace.
> >
> > * There are no interfaces for the userspace to learn about the memory
> > tiering hierarchy in order to optimize its memory allocations.
> >
> > I'd like to propose revised memory tiering kernel interfaces based on
> > the discussions in the threads:
> >
> > - https://lore.kernel.org/lkml/20220425201728.5kzm4seu7rep7ndr@offworld/T/
> > - https://lore.kernel.org/linux-mm/20220426114300.00003ad8@Huawei.com/t/
> >
> >
> > Sysfs Interfaces
> > ================
> >
> > * /sys/devices/system/node/memory_tiers
> >
> > Format: node list (one tier per line, in the tier order)
> >
> > When read, list memory nodes by tiers.
> >
> > When written (one tier per line), take the user-provided node-tier
> > assignment as the new tiering hierarchy and rebuild the per-node
> > demotion order. It is allowed to only override the top tiers, in
> > which cases, the kernel will establish the lower tiers automatically.
> >
> >
> > Kernel Representation
> > =====================
> >
> > * nodemask_t node_states[N_TOPTIER_MEMORY]
> >
> > Store all top-tier memory nodes.
> >
> > * nodemask_t memory_tiers[MAX_TIERS]
> >
> > Store memory nodes by tiers.
> >
> > * struct demotion_nodes node_demotion[]
> >
> > where: struct demotion_nodes { nodemask_t preferred; nodemask_t allowed; }
> >
> > For a node N:
> >
> > node_demotion[N].preferred lists all preferred demotion targets;
> >
> > node_demotion[N].allowed lists all allowed demotion targets
> > (initialized to be all the nodes in the same demotion tier).
> >
>
> I assume that the preferred list is auto-configured/initialized based on
> NUMA distances. Not sure why "allowed" list is only to the same demotion
> tier? For example, I think the default should be tier 0 should
> is allowed to demote to tier 1 and tier 2, not just to tier 1. So if we
> fail to demote to tier 1, we can demote to tier 2.
I agree that we can allow demotion to go to all the lower tiers, not
just the immediate next tier. I have mentioned the same idea as well
when replying to Dan's comments.
> Do you also expose the demotion preferred node and allowed
> list via /sys/devices/system/node/memory_tiers, as you have done in the examples?
To keep the memory tier sysfs minimal for now, I didn't propose
exposing the demotion preferred/allowed list in
/sys/devices/system/node/memory_tiers. But now I can see that in the
way that the examples were presented, N_TOPTIER_MEMORY and
node_demotion[] can be thought as part of the memory_tiers output,
which is not the intention.
> > Examples
> > ========
> >
> > * Example 2:
> > Node 0 & 1 are DRAM nodes.
> > Node 2 is a PMEM node and closer to node 0.
> >
> > Node 0 has node 2 as the preferred and only demotion target.
> >
> > Node 1 has no preferred demotion target, but can still demote
> > to node 2.
> >
> > Set mempolicy to prevent cross-socket demotion and memory access,
> > e.g. cpuset.mems=0,2
>
> Do we expect to later allow configuration of the demotion list explicitly?
> Something like:
>
> echo "demotion 0 1 1-3" > /sys/devices/system/node/memory_tiers
>
> to set demotion list for node 0, where preferred demote node is 1,
> allowed demote node list is 1-3.
IMHO, we'd better follow the allocation fallback order for the
demotion node order in each tier and avoid userspace override of
per-node demotion list.
In general, I think we'd better keep the tier assignment of each node
stable. If adding/changing one node can redefine the tiers of other
nodes, it can make tier-based memory accounting very difficult.
Overriding the per-node demotion list can have such undesirable side
effects (if the per-node demotion list is used to redefine tiers).
> Thanks.
>
> Tim
>
>
next prev parent reply other threads:[~2022-05-05 7:02 UTC|newest]
Thread overview: 57+ messages / expand[flat|nested] mbox.gz Atom feed top
2022-04-30 2:10 RFC: Memory Tiering Kernel Interfaces Wei Xu
2022-04-30 3:59 ` Yang Shi
2022-04-30 6:37 ` Wei Xu
2022-05-06 0:01 ` Alistair Popple
2022-05-10 4:32 ` Wei Xu
2022-05-10 5:37 ` Alistair Popple
2022-05-10 11:38 ` Aneesh Kumar K.V
2022-05-11 5:30 ` Wei Xu
2022-05-11 7:34 ` Alistair Popple
2022-05-11 7:49 ` ying.huang
2022-05-11 17:07 ` Wei Xu
2022-05-12 1:42 ` ying.huang
2022-05-12 2:39 ` Wei Xu
2022-05-12 3:13 ` ying.huang
2022-05-12 3:37 ` Wei Xu
2022-05-12 6:24 ` Wei Xu
2022-05-06 18:56 ` Yang Shi
2022-05-09 14:32 ` Hesham Almatary
2022-05-10 3:24 ` Yang Shi
2022-05-10 9:59 ` Hesham Almatary
2022-05-10 12:10 ` Aneesh Kumar K V
2022-05-11 5:42 ` Wei Xu
2022-05-11 7:12 ` Alistair Popple
2022-05-11 9:05 ` Hesham Almatary
2022-05-12 3:02 ` ying.huang
2022-05-12 4:40 ` Aneesh Kumar K V
2022-05-12 4:49 ` Wei Xu
2022-05-10 4:22 ` Wei Xu
2022-05-10 10:01 ` Hesham Almatary
2022-05-10 11:44 ` Aneesh Kumar K.V
2022-05-01 18:35 ` Dan Williams
2022-05-03 6:36 ` Wei Xu
2022-05-06 19:05 ` Yang Shi
2022-05-07 7:56 ` ying.huang
2022-05-01 17:58 ` Davidlohr Bueso
2022-05-02 1:04 ` David Rientjes
2022-05-02 7:23 ` Aneesh Kumar K.V
2022-05-03 2:07 ` Baolin Wang
2022-05-03 6:06 ` Wei Xu
2022-05-03 17:14 ` Alistair Popple
2022-05-03 17:47 ` Dave Hansen
2022-05-03 22:35 ` Alistair Popple
2022-05-03 23:54 ` Dave Hansen
2022-05-04 1:31 ` Wei Xu
2022-05-04 17:02 ` Dave Hansen
2022-05-05 6:35 ` Wei Xu
2022-05-05 14:24 ` Dave Hansen
2022-05-10 4:43 ` Wei Xu
2022-05-02 6:25 ` Aneesh Kumar K.V
2022-05-03 7:02 ` Wei Xu
2022-05-02 15:20 ` Dave Hansen
2022-05-03 7:19 ` Wei Xu
2022-05-03 19:12 ` Tim Chen
2022-05-05 7:02 ` Wei Xu [this message]
2022-05-05 8:57 ` ying.huang
2022-05-05 23:57 ` Alistair Popple
2022-05-06 0:25 ` Alistair Popple
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=CAAPL-u93P1FmM6+TvKDAKuqbUaTBouswqw7uQhvommMEP1-oVg@mail.gmail.com \
--to=weixugc@google.com \
--cc=Jonathan.Cameron@huawei.com \
--cc=akpm@linux-foundation.org \
--cc=aneesh.kumar@linux.ibm.com \
--cc=apopple@nvidia.com \
--cc=baolin.wang@linux.alibaba.com \
--cc=brice.goglin@gmail.com \
--cc=dan.j.williams@intel.com \
--cc=dave.hansen@linux.intel.com \
--cc=dave@stgolabs.net \
--cc=feng.tang@intel.com \
--cc=gthelen@google.com \
--cc=jvgediya@linux.ibm.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=shy828301@gmail.com \
--cc=tim.c.chen@linux.intel.com \
--cc=ying.huang@intel.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).