From: David Rientjes <rientjes@google.com>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Michal Hocko <mhocko@kernel.org>,
Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>,
Vlastimil Babka <vbabka@suse.cz>,
Robert Kolchmeyer <rkolchmeyer@google.com>,
linux-kernel@vger.kernel.org, linux-mm@kvack.org
Subject: [patch v3] mm, oom: prevent soft lockup on memcg oom for UP systems
Date: Wed, 18 Mar 2020 15:03:52 -0700 (PDT) [thread overview]
Message-ID: <alpine.DEB.2.21.2003181458100.70237@chino.kir.corp.google.com> (raw)
In-Reply-To: <alpine.DEB.2.21.2003181437270.70237@chino.kir.corp.google.com>
When a process is oom killed as a result of memcg limits and the victim
is waiting to exit, nothing ends up actually yielding the processor back
to the victim on UP systems with preemption disabled. Instead, the
charging process simply loops in memcg reclaim and eventually soft
lockups.
For example, on an UP system with a memcg limited to 100MB, if three
processes each charge 40MB of heap with swap disabled, one of the charging
processes can loop endlessly trying to charge memory which starves the oom
victim.
Memory cgroup out of memory: Killed process 808 (repro) total-vm:41944kB,
anon-rss:35344kB, file-rss:504kB, shmem-rss:0kB, UID:0 pgtables:108kB
oom_score_adj:0
watchdog: BUG: soft lockup - CPU#0 stuck for 23s! [repro:806]
CPU: 0 PID: 806 Comm: repro Not tainted 5.6.0-rc5+ #136
RIP: 0010:shrink_lruvec+0x4e9/0xa40
...
Call Trace:
shrink_node+0x40d/0x7d0
do_try_to_free_pages+0x13f/0x470
try_to_free_mem_cgroup_pages+0x16d/0x230
try_charge+0x247/0xac0
mem_cgroup_try_charge+0x10a/0x220
mem_cgroup_try_charge_delay+0x1e/0x40
handle_mm_fault+0xdf2/0x15f0
do_user_addr_fault+0x21f/0x420
page_fault+0x2f/0x40
Make sure that once the oom killer has been called that we forcibly yield
if current is not the chosen victim regardless of priority to allow for
memory freeing. The same situation can theoretically occur in the page
allocator, so do this after dropping oom_lock there as well.
We used to have a short sleep after oom killing, but commit 9bfe5ded054b
("mm, oom: remove sleep from under oom_lock") removed it because sleeping
inside the oom_lock is dangerous. This patch restores the sleep outside of
the lock.
Suggested-by: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
Tested-by: Robert Kolchmeyer <rkolchmeyer@google.com>
Cc: stable@vger.kernel.org
Signed-off-by: David Rientjes <rientjes@google.com>
---
mm/memcontrol.c | 6 ++++++
mm/page_alloc.c | 6 ++++++
2 files changed, 12 insertions(+)
diff --git a/mm/memcontrol.c b/mm/memcontrol.c
--- a/mm/memcontrol.c
+++ b/mm/memcontrol.c
@@ -1576,6 +1576,12 @@ static bool mem_cgroup_out_of_memory(struct mem_cgroup *memcg, gfp_t gfp_mask,
*/
ret = should_force_charge() || out_of_memory(&oc);
mutex_unlock(&oom_lock);
+ /*
+ * Give a killed process a good chance to exit before trying to
+ * charge memory again.
+ */
+ if (ret)
+ schedule_timeout_killable(1);
return ret;
}
diff --git a/mm/page_alloc.c b/mm/page_alloc.c
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3861,6 +3861,12 @@ __alloc_pages_may_oom(gfp_t gfp_mask, unsigned int order,
}
out:
mutex_unlock(&oom_lock);
+ /*
+ * Give a killed process a good chance to exit before trying to
+ * allocate memory again.
+ */
+ if (*did_some_progress)
+ schedule_timeout_killable(1);
return page;
}
next prev parent reply other threads:[~2020-03-18 22:03 UTC|newest]
Thread overview: 37+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-03-10 21:39 [patch] mm, oom: prevent soft lockup on memcg oom for UP systems David Rientjes
2020-03-10 22:05 ` Tetsuo Handa
2020-03-10 22:55 ` David Rientjes
2020-03-11 9:34 ` Tetsuo Handa
2020-03-11 19:38 ` David Rientjes
2020-03-11 22:04 ` Tetsuo Handa
2020-03-11 22:14 ` David Rientjes
2020-03-12 0:12 ` Tetsuo Handa
2020-03-12 18:07 ` David Rientjes
2020-03-12 22:32 ` Andrew Morton
2020-03-16 9:31 ` Michal Hocko
2020-03-16 10:04 ` Tetsuo Handa
2020-03-16 10:14 ` Michal Hocko
2020-03-13 0:15 ` Tetsuo Handa
2020-03-13 22:01 ` David Rientjes
2020-03-13 23:15 ` Tetsuo Handa
2020-03-13 23:32 ` Tetsuo Handa
2020-03-16 23:59 ` David Rientjes
2020-03-17 3:18 ` Tetsuo Handa
2020-03-17 4:09 ` David Rientjes
2020-03-18 0:55 ` [patch v2] " David Rientjes
2020-03-18 9:42 ` Michal Hocko
2020-03-18 21:40 ` David Rientjes
2020-03-18 22:03 ` David Rientjes [this message]
2020-03-19 7:09 ` [patch v3] " Michal Hocko
2020-03-12 4:23 ` [patch] " Tetsuo Handa
2020-03-10 22:10 ` Michal Hocko
2020-03-10 23:02 ` David Rientjes
2020-03-11 8:27 ` Michal Hocko
2020-03-11 19:45 ` David Rientjes
2020-03-12 8:32 ` Michal Hocko
2020-03-12 18:20 ` David Rientjes
2020-03-12 20:16 ` Michal Hocko
2020-03-16 9:32 ` Michal Hocko
2020-03-11 0:18 ` Andrew Morton
2020-03-11 0:34 ` David Rientjes
2020-03-11 8:36 ` Michal Hocko
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=alpine.DEB.2.21.2003181458100.70237@chino.kir.corp.google.com \
--to=rientjes@google.com \
--cc=akpm@linux-foundation.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mhocko@kernel.org \
--cc=penguin-kernel@i-love.sakura.ne.jp \
--cc=rkolchmeyer@google.com \
--cc=vbabka@suse.cz \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).