linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Wei Xu <weixugc@google.com>
To: "Chen, Tim C" <tim.c.chen@intel.com>
Cc: "Huang, Ying" <ying.huang@intel.com>,
	Jagdish Gediya <jvgediya@linux.ibm.com>,
	Yang Shi <shy828301@gmail.com>,
	Dave Hansen <dave.hansen@linux.intel.com>,
	"Williams, Dan J" <dan.j.williams@intel.com>,
	Davidlohr Bueso <dave@stgolabs.net>,
	Linux MM <linux-mm@kvack.org>,
	Linux Kernel Mailing List <linux-kernel@vger.kernel.org>,
	Andrew Morton <akpm@linux-foundation.org>,
	"Aneesh Kumar K.V" <aneesh.kumar@linux.ibm.com>,
	Baolin Wang <baolin.wang@linux.alibaba.com>,
	Greg Thelen <gthelen@google.com>, MichalHocko <mhocko@kernel.org>,
	Brice Goglin <brice.goglin@gmail.com>,
	Tim Chen <tim.c.chen@linux.intel.com>
Subject: Re: [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS
Date: Fri, 29 Apr 2022 19:21:01 -0700	[thread overview]
Message-ID: <CAAPL-u95dyPHYZfHUzYN531AYW+CUBYRYTy5f0rWAhKG9md+Yw@mail.gmail.com> (raw)
In-Reply-To: <DM6PR11MB4107867291AFE0C210D9052ADCFD9@DM6PR11MB4107.namprd11.prod.outlook.com>

On Thu, Apr 28, 2022 at 12:30 PM Chen, Tim C <tim.c.chen@intel.com> wrote:
>
> >
> >On Wed, 2022-04-27 at 09:27 -0700, Wei Xu wrote:
> >> On Wed, Apr 27, 2022 at 12:11 AM ying.huang@intel.com
> >> <ying.huang@intel.com> wrote:
> >> >
> >> > On Mon, 2022-04-25 at 09:56 -0700, Wei Xu wrote:
> >> > > On Sat, Apr 23, 2022 at 8:02 PM ying.huang@intel.com
> >> > > <ying.huang@intel.com> wrote:
> >> > > >
> >> > > > Hi, All,
> >> > > >
> >> > > > On Fri, 2022-04-22 at 16:30 +0530, Jagdish Gediya wrote:
> >> > > >
> >> > > > [snip]
> >> > > >
> >> > > > > I think it is necessary to either have per node demotion
> >> > > > > targets configuration or the user space interface supported by
> >> > > > > this patch series. As we don't have clear consensus on how the
> >> > > > > user interface should look like, we can defer the per node
> >> > > > > demotion target set interface to future until the real need arises.
> >> > > > >
> >> > > > > Current patch series sets N_DEMOTION_TARGET from dax device
> >> > > > > kmem driver, it may be possible that some memory node desired
> >> > > > > as demotion target is not detected in the system from dax-device
> >kmem probe path.
> >> > > > >
> >> > > > > It is also possible that some of the dax-devices are not
> >> > > > > preferred as demotion target e.g. HBM, for such devices, node
> >> > > > > shouldn't be set to N_DEMOTION_TARGETS. In future, Support
> >> > > > > should be added to distinguish such dax-devices and not mark
> >> > > > > them as N_DEMOTION_TARGETS from the kernel, but for now this
> >> > > > > user space interface will be useful to avoid such devices as demotion
> >targets.
> >> > > > >
> >> > > > > We can add read only interface to view per node demotion
> >> > > > > targets from /sys/devices/system/node/nodeX/demotion_targets,
> >> > > > > remove duplicated /sys/kernel/mm/numa/demotion_target
> >> > > > > interface and instead make
> >/sys/devices/system/node/demotion_targets writable.
> >> > > > >
> >> > > > > Huang, Wei, Yang,
> >> > > > > What do you suggest?
> >> > > >
> >> > > > We cannot remove a kernel ABI in practice.  So we need to make
> >> > > > it right at the first time.  Let's try to collect some
> >> > > > information for the kernel ABI definitation.
> >> > > >
> >> > > > The below is just a starting point, please add your requirements.
> >> > > >
> >> > > > 1. Jagdish has some machines with DRAM only NUMA nodes, but they
> >> > > > don't want to use that as the demotion targets.  But I don't
> >> > > > think this is a issue in practice for now, because
> >> > > > demote-in-reclaim is disabled by default.
> >> > > >
> >> > > > 2. For machines with PMEM installed in only 1 of 2 sockets, for
> >> > > > example,
> >> > > >
> >> > > > Node 0 & 2 are cpu + dram nodes and node 1 are slow memory node
> >> > > > near node 0,
> >> > > >
> >> > > > available: 3 nodes (0-2)
> >> > > > node 0 cpus: 0 1
> >> > > > node 0 size: n MB
> >> > > > node 0 free: n MB
> >> > > > node 1 cpus:
> >> > > > node 1 size: n MB
> >> > > > node 1 free: n MB
> >> > > > node 2 cpus: 2 3
> >> > > > node 2 size: n MB
> >> > > > node 2 free: n MB
> >> > > > node distances:
> >> > > > node   0   1   2
> >> > > >   0:  10  40  20
> >> > > >   1:  40  10  80
> >> > > >   2:  20  80  10
> >> > > >
> >> > > > We have 2 choices,
> >> > > >
> >> > > > a)
> >> > > > node    demotion targets
> >> > > > 0       1
> >> > > > 2       1
> >> > > >
> >> > > > b)
> >> > > > node    demotion targets
> >> > > > 0       1
> >> > > > 2       X
> >> > > >
> >> > > > a) is good to take advantage of PMEM.  b) is good to reduce
> >> > > > cross-socket traffic.  Both are OK as defualt configuration.
> >> > > > But some users may prefer the other one.  So we need a user
> >> > > > space ABI to override the default configuration.
> >> > >
> >> > > I think 2(a) should be the system-wide configuration and 2(b) can
> >> > > be achieved with NUMA mempolicy (which needs to be added to
> >demotion).
> >> >
> >> > Unfortunately, some NUMA mempolicy information isn't available at
> >> > demotion time, for example, mempolicy enforced via set_mempolicy()
> >> > is for thread. But I think that cpusets can work for demotion.
> >> >
> >> > > In general, we can view the demotion order in a way similar to
> >> > > allocation fallback order (after all, if we don't demote or
> >> > > demotion lags behind, the allocations will go to these demotion
> >> > > target nodes according to the allocation fallback order anyway).
> >> > > If we initialize the demotion order in that way (i.e. every node
> >> > > can demote to any node in the next tier, and the priority of the
> >> > > target nodes is sorted for each source node), we don't need
> >> > > per-node demotion order override from the userspace.  What we need
> >> > > is to specify what nodes should be in each tier and support NUMA
> >mempolicy in demotion.
> >> >
> >> > This sounds interesting. Tier sounds like a natural and general
> >> > concept for these memory types. It's attracting to use it for user
> >> > space interface too. For example, we may use that for mem_cgroup
> >> > limits of a specific memory type (tier).
> >> >
> >> > And if we take a look at the N_DEMOTION_TARGETS again from the "tier"
> >> > point of view. The nodes are divided to 2 classes via
> >> > N_DEMOTION_TARGETS.
> >> >
> >> > - The nodes without N_DEMOTION_TARGETS are top tier (or tier 0).
> >> >
> >> > - The nodes with N_DEMOTION_TARGETS are non-top tier (or tier 1, 2,
> >> > 3,
> >> > ...)
> >> >
> >>
> >> Yes, this is one of the main reasons why we (Google) want this interface.
> >>
> >> > So, another possibility is to fit N_DEMOTION_TARGETS and its
> >> > overriding into "tier" concept too.  !N_DEMOTION_TARGETS == TIER0.
> >> >
> >> > - All nodes start with TIER0
> >> >
> >> > - TIER0 can be cleared for some nodes via e.g. kmem driver
> >> >
> >> > TIER0 node list can be read or overriden by the user space via the
> >> > following interface,
> >> >
> >> >   /sys/devices/system/node/tier0
> >> >
> >> > In the future, if we want to customize more tiers, we can add tier1,
> >> > tier2, tier3, .....  For now, we can add just tier0.  That is, the
> >> > interface is extensible in the future compared with
> >> > .../node/demote_targets.
> >> >
> >>
> >> This more explicit tier definition interface works, too.
> >>
> >
> >In addition to make tiering definition explicit, more importantly, this makes it
> >much easier to support more than 2 tiers.  For example, for a system with
> >HBM (High Bandwidth Memory), CPU+DRAM, DRAM only, and PMEM, that is,
> >3 tiers, we can put HBM in tier 0, CPU+DRAM and DRAM only in tier 1, and
> >PMEM in tier 2, automatically, or via user space overridding.
> >N_DEMOTION_TARGETS isn't natural to be extended to support this.
>
> Agree with Ying that making the tier explicit is fundamental to the rest of the API.
>
> I think that the tier organization should come before setting the demotion targets,
> not the other way round.
>
> That makes things clear on the demotion direction,  (node in tier X
> demote to tier Y, X<Y).  With that, explicitly specifying the demotion target or
> order is only needed when we truly want that level of control or a demotion
> order.  Otherwise all the higher numbered tiers are valid targets.
> Configuring a tier level for each node is a lot easier than fixing up all
> demotion targets for each and every node.
>
> We can prevent demotion target configuration that goes in the wrong
> direction by looking at the tier level.
>
> Tim
>

I have just posted a RFC on the tier-oriented memory tiering kernel
interface based on the discussions here.  The RFC proposes a sysfs
interface, /sys/devices/system/node/memory_tiers, to display and
override the nodes in each memory tier.  It also proposes that we rely
on the kernel allocation order to select the demotion target node from
the next tier and don't expose a userspace overriding interface for
per-node demotion order.   The RFC drops the approach of CPU nodes as
the top-tier by default, too.

  parent reply	other threads:[~2022-04-30  2:21 UTC|newest]

Thread overview: 67+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2022-04-13  9:22 [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS Jagdish Gediya
2022-04-13  9:22 ` [PATCH v2 1/5] mm: demotion: Set demotion list differently Jagdish Gediya
2022-04-14  7:09   ` ying.huang
2022-04-14  8:48     ` Jagdish Gediya
2022-04-14  8:57       ` ying.huang
2022-04-14  8:55   ` Baolin Wang
2022-04-14  9:02   ` Jonathan Cameron
2022-04-14 10:40     ` Jagdish Gediya
2022-04-21  6:13   ` ying.huang
2022-04-13  9:22 ` [PATCH v2 2/5] mm: demotion: Add new node state N_DEMOTION_TARGETS Jagdish Gediya
2022-04-21  4:33   ` Wei Xu
2022-04-13  9:22 ` [PATCH v2 3/5] mm: demotion: Add support to set targets from userspace Jagdish Gediya
2022-04-21  4:26   ` Wei Xu
2022-04-22  9:13     ` Jagdish Gediya
2022-04-21  5:31   ` Wei Xu
2022-04-13  9:22 ` [PATCH v2 4/5] device-dax/kmem: Set node state as N_DEMOTION_TARGETS Jagdish Gediya
2022-04-13  9:22 ` [PATCH v2 5/5] mm: demotion: Build demotion list based on N_DEMOTION_TARGETS Jagdish Gediya
2022-04-13 21:44 ` [PATCH v2 0/5] mm: demotion: Introduce new node state N_DEMOTION_TARGETS Andrew Morton
2022-04-14 10:16   ` Jagdish Gediya
2022-04-14  7:00 ` ying.huang
2022-04-14 10:19   ` Jagdish Gediya
2022-04-21  3:11   ` Yang Shi
2022-04-21  5:41     ` Wei Xu
2022-04-21  6:24       ` ying.huang
2022-04-21  6:49         ` Wei Xu
2022-04-21  7:08           ` ying.huang
2022-04-21  7:29             ` Wei Xu
2022-04-21  7:45               ` ying.huang
2022-04-21 18:26                 ` Wei Xu
2022-04-22  0:58                   ` ying.huang
2022-04-22  4:46                     ` Wei Xu
2022-04-22  5:40                       ` ying.huang
2022-04-22  6:13                         ` Wei Xu
2022-04-22  6:21                           ` ying.huang
2022-04-22 11:00                             ` Jagdish Gediya
2022-04-22 16:43                               ` Wei Xu
2022-04-22 17:29                                 ` Yang Shi
2022-04-24  3:02                               ` ying.huang
2022-04-25  3:50                                 ` Aneesh Kumar K.V
2022-04-25  6:10                                   ` ying.huang
2022-04-25  8:09                                     ` Aneesh Kumar K V
2022-04-25  8:54                                       ` Aneesh Kumar K V
2022-04-25 20:17                                       ` Davidlohr Bueso
2022-04-26  8:42                                       ` ying.huang
2022-04-26  9:02                                         ` Aneesh Kumar K V
2022-04-26  9:44                                           ` ying.huang
2022-04-27  4:27                                         ` Wei Xu
2022-04-25  7:26                                 ` Jagdish Gediya
2022-04-25 16:56                                 ` Wei Xu
2022-04-27  5:06                                   ` Aneesh Kumar K V
2022-04-27 18:27                                     ` Wei Xu
2022-04-28  0:56                                       ` ying.huang
2022-04-28  4:11                                         ` Wei Xu
2022-04-28 17:14                                           ` Yang Shi
2022-04-29  1:27                                             ` Alistair Popple
2022-04-29  2:21                                               ` ying.huang
2022-04-29  2:58                                                 ` Wei Xu
2022-04-29  3:27                                                   ` ying.huang
2022-04-29  4:45                                                     ` Alistair Popple
2022-04-29 18:53                                                       ` Yang Shi
2022-04-29 18:52                                                   ` Yang Shi
2022-04-27  7:11                                   ` ying.huang
2022-04-27 16:27                                     ` Wei Xu
2022-04-28  8:37                                       ` ying.huang
     [not found]                                         ` <DM6PR11MB4107867291AFE0C210D9052ADCFD9@DM6PR11MB4107.namprd11.prod.outlook.com>
2022-04-30  2:21                                           ` Wei Xu [this message]
2022-04-21 17:56       ` Yang Shi
2022-04-21 23:48         ` ying.huang

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAAPL-u95dyPHYZfHUzYN531AYW+CUBYRYTy5f0rWAhKG9md+Yw@mail.gmail.com \
    --to=weixugc@google.com \
    --cc=akpm@linux-foundation.org \
    --cc=aneesh.kumar@linux.ibm.com \
    --cc=baolin.wang@linux.alibaba.com \
    --cc=brice.goglin@gmail.com \
    --cc=dan.j.williams@intel.com \
    --cc=dave.hansen@linux.intel.com \
    --cc=dave@stgolabs.net \
    --cc=gthelen@google.com \
    --cc=jvgediya@linux.ibm.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=shy828301@gmail.com \
    --cc=tim.c.chen@intel.com \
    --cc=tim.c.chen@linux.intel.com \
    --cc=ying.huang@intel.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).