Re: [RFC] mm: memcg: add priority for soft limit reclaiming

Re: [RFC] mm: memcg: add priority for soft limit reclaiming

[Michal Hocko]

On Tue 24-09-19 15:36:42, Hillf Danton wrote:
> 
> On Mon, 23 Sep 2019 21:28:34 Michal Hocko wrote:
> > 
> > On Mon 23-09-19 21:04:59, Hillf Danton wrote:
> > >
> > > On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
> > > >
> > > > On Thu 19-09-19 21:13:32, Hillf Danton wrote:
> > > > >
> > > > > Currently memory controler is playing increasingly important role in
> > > > > how memory is used and how pages are reclaimed on memory pressure.
> > > > >
> > > > > In daily works memcg is often created for critical tasks and their pre
> > > > > configured memory usage is supposed to be met even on memory pressure.
> > > > > Administrator wants to make it configurable that the pages consumed by
> > > > > memcg-B can be reclaimed by page allocations invoked not by memcg-A but
> > > > > by memcg-C.
> > > >
> > > > I am not really sure I understand the usecase well but this sounds like
> > > > what memory reclaim protection in v2 is aiming at.
> > > >
> > Please describe the usecase.
> > 
> It is for quite a while that task-A has been able to preempt task-B for
> cpu cycles. IOW the physical resource cpu cycles are preemptible.
> 
> Are physical pages are preemptible too in the same manner?
> Nope without priority defined for pages currently (say the link between
> page->nice and task->nice).
> 
> The slrp is added for memcg instead of nice because 1) it is only used
> in the page reclaiming context (in memcg it is soft limit reclaiming),
> and 2) it is difficult to compare reclaimer and reclaimee task->nice
> directly in that context as only info about reclaimer and lru page is
> available.
> 
> Here task->nice is replaced with memcg->slrp in order to do page
> preemption, PP. There is no way for task-A to PP task-B, but the
> group containing task-A can PP the group containing task-B.
> That preemption needs code within 100 lines as you see on top of
> the current memory controller framework.

This is exactly what the reclaim protection in memcg v2 is meant to be
used for. Also soft limit reclaim is absolutely terrible to achieve that
because it is just too gross to result in any smooth experience (just
have a look how it is doing priority 0 scannig!).

I am not going to even go further wrt the implementation because I
belive the priority is even semantically broken wrt hierarchical
behavior.

But really, make sure you look into the existing feature set that memcg
v2 provides already and come back if you find it unsuitable and we can
move from there. Soft limit reclaim is dead and we should let it RIP.
-- 
Michal Hocko
SUSE Labs

Re: [RFC] mm: memcg: add priority for soft limit reclaiming

[Roman Gushchin]

On Tue, Sep 24, 2019 at 03:30:16PM +0200, Michal Hocko wrote:
> On Tue 24-09-19 15:36:42, Hillf Danton wrote:
> > 
> > On Mon, 23 Sep 2019 21:28:34 Michal Hocko wrote:
> > > 
> > > On Mon 23-09-19 21:04:59, Hillf Danton wrote:
> > > >
> > > > On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
> > > > >
> > > > > On Thu 19-09-19 21:13:32, Hillf Danton wrote:
> > > > > >
> > > > > > Currently memory controler is playing increasingly important role in
> > > > > > how memory is used and how pages are reclaimed on memory pressure.
> > > > > >
> > > > > > In daily works memcg is often created for critical tasks and their pre
> > > > > > configured memory usage is supposed to be met even on memory pressure.
> > > > > > Administrator wants to make it configurable that the pages consumed by
> > > > > > memcg-B can be reclaimed by page allocations invoked not by memcg-A but
> > > > > > by memcg-C.
> > > > >
> > > > > I am not really sure I understand the usecase well but this sounds like
> > > > > what memory reclaim protection in v2 is aiming at.
> > > > >
> > > Please describe the usecase.
> > > 
> > It is for quite a while that task-A has been able to preempt task-B for
> > cpu cycles. IOW the physical resource cpu cycles are preemptible.
> > 
> > Are physical pages are preemptible too in the same manner?
> > Nope without priority defined for pages currently (say the link between
> > page->nice and task->nice).
> > 
> > The slrp is added for memcg instead of nice because 1) it is only used
> > in the page reclaiming context (in memcg it is soft limit reclaiming),
> > and 2) it is difficult to compare reclaimer and reclaimee task->nice
> > directly in that context as only info about reclaimer and lru page is
> > available.
> > 
> > Here task->nice is replaced with memcg->slrp in order to do page
> > preemption, PP. There is no way for task-A to PP task-B, but the
> > group containing task-A can PP the group containing task-B.
> > That preemption needs code within 100 lines as you see on top of
> > the current memory controller framework.
> 
> This is exactly what the reclaim protection in memcg v2 is meant to be
> used for. Also soft limit reclaim is absolutely terrible to achieve that
> because it is just too gross to result in any smooth experience (just
> have a look how it is doing priority 0 scannig!).
> 
> I am not going to even go further wrt the implementation because I
> belive the priority is even semantically broken wrt hierarchical
> behavior.
> 
> But really, make sure you look into the existing feature set that memcg
> v2 provides already and come back if you find it unsuitable and we can
> move from there. Soft limit reclaim is dead and we should let it RIP.

Can't agree more here.

Cgroup v2 memory protection mechanisms (memory.low/min) should perfectly
solve the described problem. If not, let's fix them rather than extend soft
reclaim which is already dead.

Thanks!

Re: [RFC] mm: memcg: add priority for soft limit reclaiming

[Michal Hocko]

On Wed 25-09-19 10:35:30, Hillf Danton wrote:
> 
> On Tue, 24 Sep 2019 17:23:35 +0000 from Roman Gushchin
> > 
> > On Tue, Sep 24, 2019 at 03:30:16PM +0200, Michal Hocko wrote:
> > >
> > > But really, make sure you look into the existing feature set that memcg
> > > v2 provides already and come back if you find it unsuitable and we can
> > > move from there. Soft limit reclaim is dead and we should let it RIP.
> > 
> > Can't agree more here.
> > 
> > Cgroup v2 memory protection mechanisms (memory.low/min) should perfectly
> > solve the described problem. If not, let's fix them rather than extend soft
> > reclaim which is already dead.
> > 
> Hehe, IIUC memory.low/min is essentially drawing a line that reclaimers
> would try their best not to cross. Page preemption OTOH is near ten miles
> away from that line though it is now on the shoulder of soft reclaiming.

Dynamic low limit tuning would achieve exactly what you are after - aka
prioritizing some memory consumers over others.
-- 
Michal Hocko
SUSE Labs

Re: [RFC] mm: memcg: add priority for soft limit reclaiming

[Michal Hocko]

On Mon 23-09-19 21:04:59, Hillf Danton wrote:
> 
> On Thu, 19 Sep 2019 21:32:31 +0800 Michal Hocko wrote:
> > 
> > On Thu 19-09-19 21:13:32, Hillf Danton wrote:
> > >
> > > Currently memory controler is playing increasingly important role in
> > > how memory is used and how pages are reclaimed on memory pressure.
> > >
> > > In daily works memcg is often created for critical tasks and their pre
> > > configured memory usage is supposed to be met even on memory pressure.
> > > Administrator wants to make it configurable that the pages consumed by
> > > memcg-B can be reclaimed by page allocations invoked not by memcg-A but
> > > by memcg-C.
> > 
> > I am not really sure I understand the usecase well but this sounds like
> > what memory reclaim protection in v2 is aiming at.
> > 

Please describe the usecase. 

> A tipoint to the v2 stuff please.

Documentation/admin-guide/cgroup-v2.rst
 
> > > That configurability is addressed by adding priority for soft limit
> > > reclaiming to make sure that no pages will be reclaimed from memcg of
> > > higer priortiy in favor of memcg of lower priority.
> > 
> > cgroup v1 interfaces are generally frozen and mostly aimed at backward
> > compatibility. I am especially concerned about adding a new way to
> > control soft limit which is known to be misdesigned and unfixable to
> > behave reasonably.
> >
> An URL to the drafts/works about the new way in your git tree.

Whut?
-- 
Michal Hocko
SUSE Labs

Re: [RFC] mm: memcg: add priority for soft limit reclaiming

[Michal Hocko]

On Thu 19-09-19 21:13:32, Hillf Danton wrote:
> 
> Currently memory controler is playing increasingly important role in
> how memory is used and how pages are reclaimed on memory pressure.
> 
> In daily works memcg is often created for critical tasks and their pre
> configured memory usage is supposed to be met even on memory pressure.
> Administrator wants to make it configurable that the pages consumed by
> memcg-B can be reclaimed by page allocations invoked not by memcg-A but
> by memcg-C.

I am not really sure I understand the usecase well but this sounds like
what memory reclaim protection in v2 is aiming at.
 
> That configurability is addressed by adding priority for soft limit
> reclaiming to make sure that no pages will be reclaimed from memcg of
> higer priortiy in favor of memcg of lower priority.

cgroup v1 interfaces are generally frozen and mostly aimed at backward
compatibility. I am especially concerned about adding a new way to
control soft limit which is known to be misdesigned and unfixable to
behave reasonably.

> Pages are reclaimed with no priority being taken into account by default
> unless user turns it on, and then they are responsible for their smart
> activities almost the same way as they play realtime FIFO/RR games.
> 
> Priority is available only in the direct reclaiming context in order to
> advoid churning in the complex kswapd behavior.
> 
> Cc: Shakeel Butt <shakeelb@google.com>
> Cc: Roman Gushchin <guro@fb.com>
> Cc: Matthew Wilcox <willy@infradead.org>
> Cc: Johannes Weiner <hannes@cmpxchg.org>
> Cc: Michal Hocko <mhocko@suse.cz>
> Signed-off-by: Hillf Danton <hdanton@sina.com>

That being said, you should describe the usecase and explain why v2
interface is not providing what you need. We might think about where to
go from there but extending the soft limit reclaim is almost certainly
not the right way to go.
-- 
Michal Hocko
SUSE Labs