From: Michal Hocko <mhocko@kernel.org>
To: Petros Angelatos <petrosagg@resin.io>
Cc: linux-mm@kvack.org, cgroups@vger.kernel.org,
Hugh Dickins <hughd@google.com>, Tejun Heo <tj@kernel.org>,
lstoakes@gmail.com
Subject: Re: Memory cgroup invokes OOM killer when there are a lot of dirty pages
Date: Thu, 5 Jul 2018 08:46:37 +0200 [thread overview]
Message-ID: <20180705064637.GB32658@dhcp22.suse.cz> (raw)
In-Reply-To: <CAM1WBj+OQiADXxE2dv0BtS1BG+r_wdE_wTf0-LVq7nMPxgkPPQ@mail.gmail.com>
On Wed 04-07-18 18:45:48, Petros Angelatos wrote:
> > I assume dd just tried to fault a code page in and that failed due to
> > the hard limit and unreclaimable memory. The reason why the memcg v1
> > oom throttling heuristic hasn't kicked in is that there are no pages
> > under writeback. This would match symptoms of the bug fixed by
> > 1c610d5f93c7 ("mm/vmscan: wake up flushers for legacy cgroups too") in
> > 4.16 but there might be more. You should have that fix already so there
> > must be something more in the game. You've said that you are using blkio
> > cgroup, right? What is the configuration? I strongly suspect that none
> > of the writeback has started because of the throttling.
>
> I'm only using a memory cgroup with no blkio restrictions so I'm not
> sure why writeback hasn't started. Another thing I noticed is that
> it's a lot harder to reproduce when the same amount of data is written
> in a single file versus many smaller files. That's why my original
> example code writes 500 files with 1MB of data.
>
> Your mention of writeback gave me the idea to try and do a
> sync_file_range() with SYNC_FILE_RANGE_WRITE after writing each file
> to manually schedule writeback and surprisingly it fixed the problem.
> Is that an indication of a bug in the kernel that doesn't trigger
> writeback in time?
Yeah, it smells so. If you look at 1c610d5f93c7, we've had bug where we
even didn't kick flushers. So it seems they do not start to do a useful
work in time. I would start digging that direction.
> Also, you mentioned that the pagefault is probably due to a code page.
> Would another remedy be to lock the whole executable and dynamic
> libraries in memory with mlock() before starting the IO operations?
That looks like a big hammer to me.
--
Michal Hocko
SUSE Labs
prev parent reply other threads:[~2018-07-05 6:46 UTC|newest]
Thread overview: 4+ messages / expand[flat|nested] mbox.gz Atom feed top
2018-07-03 21:14 Memory cgroup invokes OOM killer when there are a lot of dirty pages Petros Angelatos
2018-07-04 7:50 ` Michal Hocko
2018-07-04 15:45 ` Petros Angelatos
2018-07-05 6:46 ` Michal Hocko [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20180705064637.GB32658@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=hughd@google.com \
--cc=linux-mm@kvack.org \
--cc=lstoakes@gmail.com \
--cc=petrosagg@resin.io \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).