Re: [PATCH v3] [mm-unstable] mm: Fix memcg reclaim on memory tiered systems

From: Michal Hocko <mhocko@suse.com>
To: Wei Xu <weixugc@google.com>
Cc: Mina Almasry <almasrymina@google.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Roman Gushchin <roman.gushchin@linux.dev>,
	Shakeel Butt <shakeelb@google.com>,
	Muchun Song <songmuchun@bytedance.com>,
	Huang Ying <ying.huang@intel.com>,
	Yang Shi <yang.shi@linux.alibaba.com>,
	Yosry Ahmed <yosryahmed@google.com>,
	fvdl@google.com, linux-mm@kvack.org,
	linux-kernel@vger.kernel.org
Subject: Re: [PATCH v3] [mm-unstable] mm: Fix memcg reclaim on memory tiered systems
Date: Mon, 12 Dec 2022 09:36:49 +0100	[thread overview]
Message-ID: <Y5boIR3LtmuU9gKv@dhcp22.suse.cz> (raw)
In-Reply-To: <CAAPL-u8rgW-JACKUT5ChmGSJiTDABcDRjNzW_QxMjCTk9zO4sg@mail.gmail.com>

On Sat 10-12-22 00:01:28, Wei Xu wrote:
> On Fri, Dec 9, 2022 at 1:16 PM Michal Hocko <mhocko@suse.com> wrote:
> >
> > On Fri 09-12-22 08:41:47, Wei Xu wrote:
> > > On Fri, Dec 9, 2022 at 12:08 AM Michal Hocko <mhocko@suse.com> wrote:
> > > >
> > > > On Thu 08-12-22 16:59:36, Wei Xu wrote:
> > > > [...]
> > > > > > What I really mean is to add demotion nodes to the nodemask along with
> > > > > > the set of nodes you want to reclaim from. To me that sounds like a
> > > > > > more natural interface allowing for all sorts of usecases:
> > > > > > - free up demotion targets (only specify demotion nodes in the mask)
> > > > > > - control where to demote (e.g. select specific demotion target(s))
> > > > > > - do not demote at all (skip demotion nodes from the node mask)
> > > > >
> > > > > For clarification, do you mean to add another argument (e.g.
> > > > > demotion_nodes) in addition to the "nodes" argument?
> > > >
> > > > No, nodes=mask argument should control the domain where the memory
> > > > reclaim should happen. That includes both aging and the reclaim. If the
> > > > mask doesn't contain any lower tier node then no demotion will happen.
> > > > If only a subset of lower tiers are specified then only those could be
> > > > used for the demotion process. Or put it otherwise, the nodemask is not
> > > > only used to filter out zonelists during reclaim it also restricts
> > > > migration targets.
> > > >
> > > > Is this more clear now?
> > >
> > > In that case, how can we request demotion only from toptier nodes
> > > (without counting any reclaimed bytes from other nodes),  which is our
> > > memory tiering use case?
> >
> > I am not sure I follow. Could you be more specific please?
> 
> In our memory tiering use case, we would like to proactively free up
> memory on top-tier nodes by demoting cold pages to lower-tier nodes.
> This is to create enough free top-tier memory for new allocations and
> promotions.  How many pages and how often to demote from top-tier
> nodes can depend on a number of factors (e.g. the amount of free
> top-tier memory, the amount of cold pages, the bandwidth pressure on
> lower-tier, the task tolerance of slower memory on performance) and
> are controlled by the userspace policies.
> 
> Because the purpose of such proactive demotions is to free up top-tier
> memory, not to lower the amount of memory charged to the memcg, we'd
> like that memory.reclaim can demote the specified amount of bytes from
> the given top-tier nodes.  If we have to also provide the lower-tier
> nodes to memory.reclaim to allow demotions, the kernel can reclaim
> from the lower-tier nodes in the same memory.reclaim request. We then
> won't be able to control the amount of bytes to be demoted from
> top-tier nodes.

I am not sure this is something to be handled by the reclaim interface
because now you are creating an ambiguity what the interface should do
and start depend on it. Consider that we will change the reclaim
algorithm in the future and the node you request to demote will simply
reclaim rather than demote. This will break your usecase, right?
-- 
Michal Hocko
SUSE Labs