linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Paul Furtado <paulfurtado91@gmail.com>
To: Michal Hocko <mhocko@kernel.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	bugzilla-daemon@bugzilla.kernel.org,  linux-mm@kvack.org
Subject: Re: [Bug 207273] New: cgroup with 1.5GB limit and 100MB rss usage OOM-kills processes due to page cache usage after upgrading to kernel 5.4
Date: Wed, 15 Apr 2020 04:34:56 -0400	[thread overview]
Message-ID: <CAKkCftoa3e3cj2jArO5Ekk68_p6igSu+GzqWDrkWVo0WGcuZ4g@mail.gmail.com> (raw)
In-Reply-To: <20200415065059.GV4629@dhcp22.suse.cz>

> You can either try to use cgroup v2 which has much better memcg aware dirty
> throttling implementation so such a large amount of dirty pages doesn't
> accumulate in the first place

I'd love to use cgroup v2, however this is docker + kubernetes so that
would require a lot of changes on our end to make happen, given how
recently container runtimes gained cgroup v2 support.

> I pressume you are using defaults for
> /proc/sys/vm/dirty_{background_}ratio which is a percentage of the
> available memory. I would recommend using their resp. *_bytes
> alternatives and use something like 500M for background and 800M for
> dirty_bytes.

We're using the defaults right now, however, given that this is a
containerized environment, it's problematic to set these values too
low system-wide since the containers all have dedicated volumes with
varying performance (from as low as 100MB/sec to gigabyes). Looking
around, I see that there were patches in the past to set per-cgroup
vm.dirty settings, however it doesn't look like those ever made it
into the kernel unless I'm missing something. In practice, maybe 500M
and 800M wouldn't be so bad though and may improve latency in other
ways. The other problem is that this also sets an upper bound on the
minimum container size for anything that does do IO. That said, I'll
still I'll tune these settings in our infrastructure and see how
things go, but it sounds like something should be done inside the
kernel to help this situation, since it's so easy to trigger, but
looking at the threads that led to the commits you referenced, I can
see that this is complicated.

Thanks,
Paul



On Wed, Apr 15, 2020 at 2:51 AM Michal Hocko <mhocko@kernel.org> wrote:
>
> On Tue 14-04-20 21:25:58, Andrew Morton wrote:
> [...]
> > > Upon upgrading to kernel 5.4, we see constant OOM kills in database containers
> > > that are restoring from backups, with nearly no RSS memory usage. It appears
> > > all the memory is consumed by file_dirty, with applications using minimal
> > > memory. On kernel 4.14.146 and 4.19.75, we do not see this problem, so it
> > > appears to be a new regression.
>
> OK, this is interesting. Because the memcg oom handling has changed in
> 4.19. Older kernels triggered memcg oom only from the page fault path
> while your stack trace is pointing to the write(2) syscall. But if you
> do not see any problem with 4.19 then this is not it.
>
> [...]
>
> > > gfp_mask=0x101c4a(GFP_NOFS|__GFP_HIGHMEM|__GFP_HARDWALL|__GFP_MOVABLE|__GFP_WRITE)
> [...]
> > > memory: usage 1536000kB, limit 1536000kB, failcnt 0
> > > memory+swap: usage 1536000kB, limit 1536000kB, failcnt 490221
> > > kmem: usage 23164kB, limit 9007199254740988kB, failcnt 0
>
> Based on the output I assume you are using cgroup v1
>
> > > Memory cgroup stats for
> > > /kubepods/burstable/pod6900693c-8b2c-4efe-ab52-26e4a6bd9e4c/83216944bb43baf32f0d43ef12c85ebaa2767b3f51846f5fa438bba00b4636d8:
> > > anon 72507392
> > > file 1474740224
> > > kernel_stack 774144
> > > slab 18673664
> > > sock 0
> > > shmem 0
> > > file_mapped 0
> > > file_dirty 1413857280
> > > file_writeback 60555264
>
> This seems to be the crux of the problem. You cannot swap out any memory
> due to memory+swap limit and 95% of the file LRU is dirty. Quite a lot
> of dirty memory to flush. This alone shouldn't be a disaster because
> cgroup v1 does have a hack to throttle the memory reclaim in presence of
> dirty/writeback pages. But note the gfp_mask for the allocation. It
> says GFP_NOFS which means that we cannot apply the throttling and have
> to give up. We used to retry the reclaim even though not much could be
> done with a restricted allocation context and that led to lockups. Then
> we have merged f9c645621a28 ("memcg, oom: don't require __GFP_FS when
> invoking memcg OOM killer") in 5.4 and it has been marked for stable
> trees (4.19+). And this is likely the primary culprit of the issue you
> are seeing.
>
> Now what to do about that. Reverting f9c645621a28 doesn't sound like a
> feasible solution. We could try to put a sleep for restricted
> allocations after memory reclaim fails but we know from the past
> experience this is a bit fishy because a sleep without any feedback on
> the flushing is just not going to work reliably.
>
> Another possibility is to workaround the problem by configuration. You
> can either try to use cgroup v2 which has much better memcg aware dirty
> throttling implementation so such a large amount of dirty pages doesn't
> accumulate in the first place or you can reconfigure global dirty
> limits. I pressume you are using defaults for
> /proc/sys/vm/dirty_{background_}ratio which is a percentage of the
> available memory. I would recommend using their resp. *_bytes
> alternatives and use something like 500M for background and 800M for
> dirty_bytes. That should help in your current situation. The overall IO
> throughput might be smaller so you might need to tune those values a
> bit.
>
> HTH
> --
> Michal Hocko
> SUSE Labs


  reply	other threads:[~2020-04-15  8:35 UTC|newest]

Thread overview: 8+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <bug-207273-27@https.bugzilla.kernel.org/>
2020-04-15  4:25 ` [Bug 207273] New: cgroup with 1.5GB limit and 100MB rss usage OOM-kills processes due to page cache usage after upgrading to kernel 5.4 Andrew Morton
2020-04-15  6:50   ` Michal Hocko
2020-04-15  8:34     ` Paul Furtado [this message]
2020-04-15  9:44       ` Michal Hocko
2021-08-06 20:42         ` Anchal Agarwal
2021-08-11 19:41           ` Anchal Agarwal
2021-08-19  9:35             ` Michal Hocko
2021-08-19  9:41               ` Herrenschmidt, Benjamin

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=CAKkCftoa3e3cj2jArO5Ekk68_p6igSu+GzqWDrkWVo0WGcuZ4g@mail.gmail.com \
    --to=paulfurtado91@gmail.com \
    --cc=akpm@linux-foundation.org \
    --cc=bugzilla-daemon@bugzilla.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).