From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> To: Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org> Cc: Johannes Weiner <hannes@cmpxchg.org>, linux-mm@kvack.org, David Rientjes <rientjes@google.com>, LKML <linux-kernel@vger.kernel.org>, Jonathan Corbet <corbet@lwn.net> Subject: Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks Date: Wed, 12 Dec 2018 19:23:35 +0900 [thread overview] Message-ID: <38568dbe-f45b-8c55-84e4-d4dcbdbef545@i-love.sakura.ne.jp> (raw) In-Reply-To: <8a71ecd8-e7bc-25de-184f-dfda511ee0d1@i-love.sakura.ne.jp> On 2018/12/07 21:43, Tetsuo Handa wrote: > No response for one month. When can we get to an RCU stall problem syzbot reported? Why not to apply this patch and then think how to address https://lore.kernel.org/lkml/201810100012.w9A0Cjtn047782@www262.sakura.ne.jp/ ? From 0fb58415770a83d6c40d471e1840f8bc4a35ca83 Mon Sep 17 00:00:00 2001 From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Date: Wed, 12 Dec 2018 19:15:51 +0900 Subject: [PATCH] memcg: killed threads should not invoke memcg OOM killer It is possible that a single process group memcg easily swamps the log with no-eligible OOM victim messages after current thread was OOM-killed, due to race between the memcg charge and the OOM reaper [1]. Thread-1 Thread-2 OOM reaper try_charge() mem_cgroup_out_of_memory() mutex_lock(oom_lock) try_charge() mem_cgroup_out_of_memory() mutex_lock(oom_lock) out_of_memory() select_bad_process() oom_kill_process(current) wake_oom_reaper() oom_reap_task() # sets MMF_OOM_SKIP mutex_unlock(oom_lock) out_of_memory() select_bad_process() # no task mutex_unlock(oom_lock) We don't need to invoke the memcg OOM killer if current thread was killed when waiting for oom_lock, for mem_cgroup_oom_synchronize(true) and memory_max_write() can bail out upon SIGKILL, and try_charge() allows already killed/exiting threads to make forward progress. Michal has a plan to use tsk_is_oom_victim() by calling mark_oom_victim() on all thread groups sharing victim's mm. But fatal_signal_pending() in this patch helps regardless of Michal's plan because it will avoid needlessly calling out_of_memory() when current thread is already terminating (e.g. got SIGINT after passing fatal_signal_pending() check in try_charge() and mutex_lock_killable() did not block). [1] https://lkml.kernel.org/r/ea637f9a-5dd0-f927-d26d-d0b4fd8ccb6f@i-love.sakura.ne.jp Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> --- mm/memcontrol.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b860dd4f7..b0d3bf3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1389,8 +1389,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, }; bool ret; - mutex_lock(&oom_lock); - ret = out_of_memory(&oc); + if (mutex_lock_killable(&oom_lock)) + return true; + /* + * A few threads which were not waiting at mutex_lock_killable() can + * fail to bail out. Therefore, check again after holding oom_lock. + */ + ret = fatal_signal_pending(current) || out_of_memory(&oc); mutex_unlock(&oom_lock); return ret; } -- 1.8.3.1
WARNING: multiple messages have this Message-ID (diff)
From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp> To: Michal Hocko <mhocko@kernel.org>, Andrew Morton <akpm@linux-foundation.org> Cc: Johannes Weiner <hannes@cmpxchg.org>, linux-mm@kvack.org, David Rientjes <rientjes@google.com>, LKML <linux-kernel@vger.kernel.org>, Jonathan Corbet <corbet@lwn.net> Subject: Re: [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks Date: Wed, 12 Dec 2018 19:23:35 +0900 [thread overview] Message-ID: <38568dbe-f45b-8c55-84e4-d4dcbdbef545@i-love.sakura.ne.jp> (raw) In-Reply-To: <8a71ecd8-e7bc-25de-184f-dfda511ee0d1@i-love.sakura.ne.jp> On 2018/12/07 21:43, Tetsuo Handa wrote: > No response for one month. When can we get to an RCU stall problem syzbot reported? Why not to apply this patch and then think how to address https://lore.kernel.org/lkml/201810100012.w9A0Cjtn047782@www262.sakura.ne.jp/ ? >From 0fb58415770a83d6c40d471e1840f8bc4a35ca83 Mon Sep 17 00:00:00 2001 From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> Date: Wed, 12 Dec 2018 19:15:51 +0900 Subject: [PATCH] memcg: killed threads should not invoke memcg OOM killer It is possible that a single process group memcg easily swamps the log with no-eligible OOM victim messages after current thread was OOM-killed, due to race between the memcg charge and the OOM reaper [1]. Thread-1 Thread-2 OOM reaper try_charge() mem_cgroup_out_of_memory() mutex_lock(oom_lock) try_charge() mem_cgroup_out_of_memory() mutex_lock(oom_lock) out_of_memory() select_bad_process() oom_kill_process(current) wake_oom_reaper() oom_reap_task() # sets MMF_OOM_SKIP mutex_unlock(oom_lock) out_of_memory() select_bad_process() # no task mutex_unlock(oom_lock) We don't need to invoke the memcg OOM killer if current thread was killed when waiting for oom_lock, for mem_cgroup_oom_synchronize(true) and memory_max_write() can bail out upon SIGKILL, and try_charge() allows already killed/exiting threads to make forward progress. Michal has a plan to use tsk_is_oom_victim() by calling mark_oom_victim() on all thread groups sharing victim's mm. But fatal_signal_pending() in this patch helps regardless of Michal's plan because it will avoid needlessly calling out_of_memory() when current thread is already terminating (e.g. got SIGINT after passing fatal_signal_pending() check in try_charge() and mutex_lock_killable() did not block). [1] https://lkml.kernel.org/r/ea637f9a-5dd0-f927-d26d-d0b4fd8ccb6f@i-love.sakura.ne.jp Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp> --- mm/memcontrol.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/mm/memcontrol.c b/mm/memcontrol.c index b860dd4f7..b0d3bf3 100644 --- a/mm/memcontrol.c +++ b/mm/memcontrol.c @@ -1389,8 +1389,13 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask, }; bool ret; - mutex_lock(&oom_lock); - ret = out_of_memory(&oc); + if (mutex_lock_killable(&oom_lock)) + return true; + /* + * A few threads which were not waiting at mutex_lock_killable() can + * fail to bail out. Therefore, check again after holding oom_lock. + */ + ret = fatal_signal_pending(current) || out_of_memory(&oc); mutex_unlock(&oom_lock); return ret; } -- 1.8.3.1
next prev parent reply other threads:[~2018-12-12 10:24 UTC|newest] Thread overview: 36+ messages / expand[flat|nested] mbox.gz Atom feed top 2018-10-22 7:13 [RFC PATCH 0/2] oom, memcg: do not report racy no-eligible OOM Michal Hocko 2018-10-22 7:13 ` Michal Hocko 2018-10-22 7:13 ` [RFC PATCH 1/2] mm, oom: marks all killed tasks as oom victims Michal Hocko 2018-10-22 7:13 ` Michal Hocko 2018-10-22 7:58 ` Tetsuo Handa 2018-10-22 8:48 ` Michal Hocko 2018-10-22 9:42 ` Tetsuo Handa 2018-10-22 10:43 ` Michal Hocko 2018-10-22 10:56 ` Tetsuo Handa 2018-10-22 11:12 ` Michal Hocko 2018-10-22 11:16 ` [RFC PATCH v2 " Michal Hocko 2018-10-22 11:16 ` Michal Hocko 2018-10-22 7:13 ` [RFC PATCH 2/2] memcg: do not report racy no-eligible OOM tasks Michal Hocko 2018-10-22 7:13 ` Michal Hocko 2018-10-22 11:45 ` Tetsuo Handa 2018-10-22 12:03 ` Michal Hocko 2018-10-22 13:20 ` Tetsuo Handa 2018-10-22 13:43 ` Michal Hocko 2018-10-22 15:12 ` Tetsuo Handa 2018-10-23 1:01 ` Tetsuo Handa 2018-10-23 11:42 ` Michal Hocko 2018-10-23 12:10 ` Michal Hocko 2018-10-23 12:33 ` Tetsuo Handa 2018-10-23 12:48 ` Michal Hocko 2018-10-26 14:25 ` Johannes Weiner 2018-10-26 19:25 ` Michal Hocko 2018-10-26 19:33 ` Michal Hocko 2018-10-27 1:10 ` Tetsuo Handa 2018-11-06 9:44 ` Tetsuo Handa 2018-11-06 9:44 ` Tetsuo Handa 2018-11-06 12:42 ` Michal Hocko 2018-11-07 9:45 ` Tetsuo Handa 2018-11-07 10:08 ` Michal Hocko 2018-12-07 12:43 ` Tetsuo Handa 2018-12-12 10:23 ` Tetsuo Handa [this message] 2018-12-12 10:23 ` Tetsuo Handa
Reply instructions: You may reply publicly to this message via plain-text email using any one of the following methods: * Save the following mbox file, import it into your mail client, and reply-to-all from there: mbox Avoid top-posting and favor interleaved quoting: https://en.wikipedia.org/wiki/Posting_style#Interleaved_style * Reply using the --to, --cc, and --in-reply-to switches of git-send-email(1): git send-email \ --in-reply-to=38568dbe-f45b-8c55-84e4-d4dcbdbef545@i-love.sakura.ne.jp \ --to=penguin-kernel@i-love.sakura.ne.jp \ --cc=akpm@linux-foundation.org \ --cc=corbet@lwn.net \ --cc=hannes@cmpxchg.org \ --cc=linux-kernel@vger.kernel.org \ --cc=linux-mm@kvack.org \ --cc=mhocko@kernel.org \ --cc=rientjes@google.com \ /path/to/YOUR_REPLY https://kernel.org/pub/software/scm/git/docs/git-send-email.html * If your mail client supports setting the In-Reply-To header via mailto: links, try the mailto: linkBe sure your reply has a Subject: header at the top and a blank line before the message body.
This is an external index of several public inboxes, see mirroring instructions on how to clone and mirror all data and code used by this external index.