All of lore.kernel.org
 help / color / mirror / Atom feed
From: Michal Hocko <mhocko@kernel.org>
To: Shakeel Butt <shakeelb@google.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>,
	Roman Gushchin <guro@fb.com>, Greg Thelen <gthelen@google.com>,
	Andrew Morton <akpm@linux-foundation.org>,
	Linux MM <linux-mm@kvack.org>, Cgroups <cgroups@vger.kernel.org>,
	LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH] memcg: oom: ignore oom warnings from memory.max
Date: Mon, 4 May 2020 16:11:36 +0200	[thread overview]
Message-ID: <20200504141136.GR22838@dhcp22.suse.cz> (raw)
In-Reply-To: <CALvZod5Ao2PEFPEOckW6URBfxisp9nNpNeon1GuctuHehqk_6Q@mail.gmail.com>

On Mon 04-05-20 06:54:40, Shakeel Butt wrote:
> On Sun, May 3, 2020 at 11:56 PM Michal Hocko <mhocko@kernel.org> wrote:
> >
> > On Thu 30-04-20 11:27:12, Shakeel Butt wrote:
> > > Lowering memory.max can trigger an oom-kill if the reclaim does not
> > > succeed. However if oom-killer does not find a process for killing, it
> > > dumps a lot of warnings.
> >
> > It shouldn't dump much more than the regular OOM report AFAICS. Sure
> > there is "Out of memory and no killable processes..." message printed as
> > well but is that a real problem?
> >
> > > Deleting a memcg does not reclaim memory from it and the memory can
> > > linger till there is a memory pressure. One normal way to proactively
> > > reclaim such memory is to set memory.max to 0 just before deleting the
> > > memcg. However if some of the memcg's memory is pinned by others, this
> > > operation can trigger an oom-kill without any process and thus can log a
> > > lot un-needed warnings. So, ignore all such warnings from memory.max.
> >
> > OK, I can see why you might want to use memory.max for that purpose but
> > I do not really understand why the oom report is a problem here.
> 
> It may not be a problem for an individual or small scale deployment
> but when "sweep before tear down" is the part of the workflow for
> thousands of machines cycling through hundreds of thousands of cgroups
> then we can potentially flood the logs with not useful dumps and may
> hide (or overflow) any useful information in the logs.

If you are doing this in a large scale and the oom report is really a
problem then you shouldn't be resetting hard limit to 0 in the first
place.

> > memory.max can trigger the oom kill and user should be expecting the oom
> > report under that condition. Why is "no eligible task" so special? Is it
> > because you know that there won't be any tasks for your particular case?
> > What about other use cases where memory.max is not used as a "sweep
> > before tear down"?
> 
> What other such use-cases would be? The only use-case I can envision
> of adjusting limits dynamically of a live cgroup are resource
> managers. However for cgroup v2, memory.high is the recommended way to
> limit the usage, so, why would resource managers be changing
> memory.max instead of memory.high? I am not sure. What do you think?

There are different reasons to use the hard limit. Mostly to contain
potential runaways. While high limit might be a sufficient measure to
achieve that as well the hard limit is the last resort. And it clearly
has the oom killer semantic so I am not really sure why you are
comparing the two.

> FB is moving away from limits setting, so, not sure if they have
> thought of these cases.
> 
> BTW for such use-cases, shouldn't we be taking the memcg's oom_lock?

This is a good question. I would have to go and double check the code
but I suspect that this is an omission.
-- 
Michal Hocko
SUSE Labs

  reply	other threads:[~2020-05-04 14:11 UTC|newest]

Thread overview: 78+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-04-30 18:27 [PATCH] memcg: oom: ignore oom warnings from memory.max Shakeel Butt
2020-04-30 18:27 ` Shakeel Butt
2020-04-30 19:06 ` Roman Gushchin
2020-04-30 19:06   ` Roman Gushchin
2020-04-30 19:30   ` Johannes Weiner
2020-04-30 19:30     ` Johannes Weiner
2020-04-30 20:23     ` Roman Gushchin
2020-04-30 20:23       ` Roman Gushchin
2020-04-30 19:31   ` Shakeel Butt
2020-04-30 19:31     ` Shakeel Butt
2020-04-30 19:31     ` Shakeel Butt
2020-04-30 19:29 ` Johannes Weiner
2020-04-30 20:20   ` Shakeel Butt
2020-04-30 20:20     ` Shakeel Butt
2020-04-30 20:20     ` Shakeel Butt
2020-05-04  6:57     ` Michal Hocko
2020-05-04  6:57       ` Michal Hocko
2020-05-04 13:54       ` Shakeel Butt
2020-05-04 13:54         ` Shakeel Butt
2020-05-04 13:54         ` Shakeel Butt
2020-05-01  1:39 ` Yafang Shao
2020-05-01  1:39   ` Yafang Shao
2020-05-01  2:04   ` Shakeel Butt
2020-05-01  2:04     ` Shakeel Butt
2020-05-01  2:12     ` Yafang Shao
2020-05-01  2:12       ` Yafang Shao
2020-05-01  2:12       ` Yafang Shao
2020-05-04  7:03   ` Michal Hocko
2020-05-04  7:03     ` Michal Hocko
2020-05-04  7:26     ` Yafang Shao
2020-05-04  7:26       ` Yafang Shao
2020-05-04  7:26       ` Yafang Shao
2020-05-04  7:35       ` Michal Hocko
2020-05-04  7:40         ` Yafang Shao
2020-05-04  7:40           ` Yafang Shao
2020-05-04  7:40           ` Yafang Shao
2020-05-04  8:03           ` Michal Hocko
2020-05-04  8:03             ` Michal Hocko
2020-05-04  6:56 ` Michal Hocko
2020-05-04  6:56   ` Michal Hocko
2020-05-04 13:54   ` Shakeel Butt
2020-05-04 13:54     ` Shakeel Butt
2020-05-04 14:11     ` Michal Hocko [this message]
2020-05-04 14:53       ` Shakeel Butt
2020-05-04 14:53         ` Shakeel Butt
2020-05-04 14:53         ` Shakeel Butt
2020-05-04 15:00         ` Michal Hocko
2020-05-04 15:35           ` Shakeel Butt
2020-05-04 15:35             ` Shakeel Butt
2020-05-04 15:35             ` Shakeel Butt
2020-05-04 15:39             ` Yafang Shao
2020-05-04 15:39               ` Yafang Shao
2020-05-04 15:39               ` Yafang Shao
2020-05-04 16:06             ` Michal Hocko
2020-05-04 16:06               ` Michal Hocko
2020-05-04 19:23               ` Shakeel Butt
2020-05-04 19:23                 ` Shakeel Butt
2020-05-05  7:13                 ` Michal Hocko
2020-05-05  7:13                   ` Michal Hocko
2020-05-05 15:03                   ` Shakeel Butt
2020-05-05 15:03                     ` Shakeel Butt
2020-05-05 16:57                     ` Johannes Weiner
2020-05-05 16:57                       ` Johannes Weiner
2020-05-05 15:27                 ` Johannes Weiner
2020-05-05 15:27                   ` Johannes Weiner
2020-05-05 15:35                   ` Shakeel Butt
2020-05-05 15:35                     ` Shakeel Butt
2020-05-05 15:35                     ` Shakeel Butt
2020-05-05 15:49                     ` Michal Hocko
2020-05-05 15:49                       ` Michal Hocko
2020-05-05 16:40                     ` Johannes Weiner
2020-05-05 16:40                       ` Johannes Weiner
2020-05-04 14:20     ` Tetsuo Handa
2020-05-04 14:20       ` Tetsuo Handa
2020-05-04 14:57       ` Shakeel Butt
2020-05-04 14:57         ` Shakeel Butt
2020-05-04 15:44         ` Tetsuo Handa
2020-05-04 15:44           ` Tetsuo Handa

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200504141136.GR22838@dhcp22.suse.cz \
    --to=mhocko@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=cgroups@vger.kernel.org \
    --cc=gthelen@google.com \
    --cc=guro@fb.com \
    --cc=hannes@cmpxchg.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=shakeelb@google.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.