From: Michal Hocko <mhocko@kernel.org>
To: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Cc: Masoud Sharbiani <msharbiani@apple.com>,
Greg KH <gregkh@linuxfoundation.org>,
hannes@cmpxchg.org, vdavydov.dev@gmail.com, linux-mm@kvack.org,
cgroups@vger.kernel.org, linux-kernel@vger.kernel.org
Subject: Re: Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1.
Date: Mon, 5 Aug 2019 10:42:28 +0200 [thread overview]
Message-ID: <20190805084228.GB7597@dhcp22.suse.cz> (raw)
In-Reply-To: <d7efccf4-7f07-10da-077d-a58dafbf627e@I-love.SAKURA.ne.jp>
On Sun 04-08-19 00:51:18, Tetsuo Handa wrote:
> Masoud, will you try this patch?
>
> By the way, is /sys/fs/cgroup/memory/leaker/memory.usage_in_bytes remains non-zero
> despite /sys/fs/cgroup/memory/leaker/tasks became empty due to memcg OOM killer expected?
> Deleting big-data-file.bin after memcg OOM killer reduces some, but still remains
> non-zero.
>
> ----------------------------------------
> >From 2f92c70f390f42185c6e2abb8dda98b1b7d02fa9 Mon Sep 17 00:00:00 2001
> From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Date: Sun, 4 Aug 2019 00:41:30 +0900
> Subject: [PATCH] memcg, oom: don't require __GFP_FS when invoking memcg OOM killer
>
> Masoud Sharbiani noticed that commit 29ef680ae7c21110 ("memcg, oom: move
> out_of_memory back to the charge path") broke memcg OOM called from
> __xfs_filemap_fault() path.
This is very well spotted! I really didn't think of GFP_NOFS although
xfs in the mix could give me some clue.
> It turned out that try_chage() is retrying
> forever without making forward progress because mem_cgroup_oom(GFP_NOFS)
> cannot invoke the OOM killer due to commit 3da88fb3bacfaa33 ("mm, oom:
> move GFP_NOFS check to out_of_memory"). Regarding memcg OOM, we need to
> bypass GFP_NOFS check in order to guarantee forward progress.
This deserves more information about the fix. Why is it OK to trigger
OOM for GFP_NOFS allocations? Doesn't this lead to pre-mature OOM killer
invocation?
You can argue that memcg charges have ignored GFP_NOFS without seeing a
lot of problems. But please document that in the changelog.
It is 3da88fb3bacfaa33 that has introduced this heuristic and I have to
confess I haven't realized the side effect on the memcg side because
OOM was triggered only from the GFP_KERNEL context. So I would point
to 3da88fb3bacfaa33 as introducing the regression albeit silent at the
time.
> Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
> Reported-by: Masoud Sharbiani <msharbiani@apple.com>
> Bisected-by: Masoud Sharbiani <msharbiani@apple.com>
> Fixes: 29ef680ae7c21110 ("memcg, oom: move out_of_memory back to the charge path")
I would say
Fixes: 3da88fb3bacfaa33 # necessary after 29ef680ae7c21110
Other than that I am not really sure about a better fix. Let's see
whether we see some pre-mature memcg OOM reports and think where to get
from there.
With updated changelog
Acked-by: Michal Hocko <mhocko@suse.com>
Thanks!
> ---
> mm/oom_kill.c | 5 +++--
> 1 file changed, 3 insertions(+), 2 deletions(-)
>
> diff --git a/mm/oom_kill.c b/mm/oom_kill.c
> index eda2e2a..26804ab 100644
> --- a/mm/oom_kill.c
> +++ b/mm/oom_kill.c
> @@ -1068,9 +1068,10 @@ bool out_of_memory(struct oom_control *oc)
> * The OOM killer does not compensate for IO-less reclaim.
> * pagefault_out_of_memory lost its gfp context so we have to
> * make sure exclude 0 mask - all other users should have at least
> - * ___GFP_DIRECT_RECLAIM to get here.
> + * ___GFP_DIRECT_RECLAIM to get here. But mem_cgroup_oom() has to
> + * invoke the OOM killer even if it is a GFP_NOFS allocation.
> */
> - if (oc->gfp_mask && !(oc->gfp_mask & __GFP_FS))
> + if (oc->gfp_mask && !(oc->gfp_mask & __GFP_FS) && !is_memcg_oom(oc))
> return true;
>
> /*
> --
> 1.8.3.1
>
--
Michal Hocko
SUSE Labs
next prev parent reply other threads:[~2019-08-05 8:42 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-08-01 18:04 Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1 Masoud Sharbiani
2019-08-01 18:19 ` Greg KH
2019-08-02 1:08 ` Masoud Sharbiani
2019-08-02 8:18 ` Michal Hocko
2019-08-02 7:40 ` Michal Hocko
2019-08-02 14:18 ` Masoud Sharbiani
2019-08-02 14:41 ` Michal Hocko
2019-08-02 18:00 ` Masoud Sharbiani
2019-08-02 19:14 ` Michal Hocko
[not found] ` <A06C5313-B021-4ADA-9897-CE260A9011CC@apple.com>
2019-08-03 2:36 ` Tetsuo Handa
2019-08-03 15:51 ` Tetsuo Handa
2019-08-03 17:41 ` Masoud Sharbiani
2019-08-03 18:24 ` Masoud Sharbiani
2019-08-05 8:42 ` Michal Hocko [this message]
2019-08-05 11:36 ` Tetsuo Handa
2019-08-05 11:44 ` Michal Hocko
2019-08-05 14:00 ` Tetsuo Handa
2019-08-05 14:26 ` Michal Hocko
2019-08-06 10:26 ` Tetsuo Handa
2019-08-06 10:50 ` Michal Hocko
2019-08-06 12:48 ` [PATCH v3] memcg, oom: don't require __GFP_FS when invoking memcg OOM killer Tetsuo Handa
2019-08-05 8:18 ` Possible mem cgroup bug in kernels between 4.18.0 and 5.3-rc1 Michal Hocko
[not found] <20190802121059.13192-1-hdanton@sina.com>
2019-08-02 13:40 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20190805084228.GB7597@dhcp22.suse.cz \
--to=mhocko@kernel.org \
--cc=cgroups@vger.kernel.org \
--cc=gregkh@linuxfoundation.org \
--cc=hannes@cmpxchg.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=msharbiani@apple.com \
--cc=penguin-kernel@I-love.SAKURA.ne.jp \
--cc=vdavydov.dev@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).