All of lore.kernel.org
 help / color / mirror / Atom feed
From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
To: Michal Hocko <mhocko@kernel.org>, hannes@cmpxchg.org
Cc: linux-mm@kvack.org, syzkaller-bugs@googlegroups.com,
	Michal Hocko <mhocko@suse.com>,
	guro@fb.com, kirill.shutemov@linux.intel.com,
	linux-kernel@vger.kernel.org, rientjes@google.com,
	yang.s@alibaba-inc.com
Subject: Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks
Date: Fri, 12 Oct 2018 19:47:03 +0900	[thread overview]
Message-ID: <7bb967ec-81ae-2c07-7e58-9ad23167bb66@i-love.sakura.ne.jp> (raw)
In-Reply-To: <201810110637.w9B6b9eH044188@www262.sakura.ne.jp>

On 2018/10/11 15:37, Tetsuo Handa wrote:
> Michal Hocko wrote:
>> Once we are here, make sure that the reason to trigger the OOM is
>> printed without ratelimiting because this is really valuable to
>> debug what happened.
> 
> Here is my version.
> 

Hmm, per mem_cgroup flag would be better than per task_struct flag.

From 5e80805e4988f23372a3b352fd4185e6bad53f58 Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Fri, 12 Oct 2018 16:24:49 +0900
Subject: [PATCH v2] mm: memcontrol: Don't flood OOM messages with no eligible task.

syzbot is hitting RCU stall at shmem_fault() [1].
This is because memcg-OOM events with no eligible task (current thread
is marked as OOM-unkillable) continued calling dump_header() from
out_of_memory() enabled by commit 3100dab2aa09dc6e ("mm: memcontrol:
print proper OOM header when no eligible victim left.").

Let's make sure that next dump_header() waits for at least 60 seconds from
previous "Out of memory and no killable processes..." message, if requested
memcg already reported it. This change allows threads in requested memcg to
complete memory allocation requests for doing recovery operation, and also
allows us to know whether threads in requested memcg is doing recovery
operation.

[1] https://syzkaller.appspot.com/bug?id=4ae3fff7fcf4c33a47c1192d2d62d2e03efffa64

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reported-by: syzbot <syzbot+77e6b28a7a7106ad0def@syzkaller.appspotmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
---
 include/linux/memcontrol.h |  3 +++
 mm/oom_kill.c              | 23 +++++++++++++++++++++++
 2 files changed, 26 insertions(+)

diff --git a/include/linux/memcontrol.h b/include/linux/memcontrol.h
index 652f602..8da2340 100644
--- a/include/linux/memcontrol.h
+++ b/include/linux/memcontrol.h
@@ -230,6 +230,9 @@ struct mem_cgroup {
 	 */
 	bool oom_group;
 
+	/* Previous OOM-kill request failed due to no eligible task? */
+	bool oom_no_eligible_warned;
+
 	/* protected by memcg_oom_lock */
 	bool		oom_lock;
 	int		under_oom;
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index f10aa53..52a8319 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -1106,6 +1106,21 @@ bool out_of_memory(struct oom_control *oc)
 	select_bad_process(oc);
 	/* Found nothing?!?! */
 	if (!oc->chosen) {
+#ifdef CONFIG_MEMCG
+		/*
+		 * Don't flood with dump_header() if already reported, in case
+		 * current->signal->oom_score_adj == OOM_SCORE_ADJ_MIN. Maybe
+		 * this variable should be per "struct mem_cgroup". But since
+		 * we can't prove that multiple concurrent memcg OOM without
+		 * eligible task won't cause flooding, choose global variable
+		 * for safety.
+		 */
+		static u64 last_warned;
+
+		if (is_memcg_oom(oc) && oc->memcg->oom_no_eligible_warned &&
+		    time_before64(get_jiffies_64(), last_warned + 60 * HZ))
+			return false;
+#endif
 		dump_header(oc, NULL);
 		pr_warn("Out of memory and no killable processes...\n");
 		/*
@@ -1115,6 +1130,14 @@ bool out_of_memory(struct oom_control *oc)
 		 */
 		if (!is_sysrq_oom(oc) && !is_memcg_oom(oc))
 			panic("System is deadlocked on memory\n");
+#ifdef CONFIG_MEMCG
+		if (is_memcg_oom(oc)) {
+			oc->memcg->oom_no_eligible_warned = true;
+			last_warned = get_jiffies_64();
+		}
+	} else if (is_memcg_oom(oc)) {
+		oc->memcg->oom_no_eligible_warned = false;
+#endif
 	}
 	if (oc->chosen && oc->chosen != (void *)-1UL)
 		oom_kill_process(oc, !is_memcg_oom(oc) ? "Out of memory" :
-- 
1.8.3.1

WARNING: multiple messages have this Message-ID
From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
To: Michal Hocko <mhocko@kernel.org>, hannes@cmpxchg.org
Cc: linux-mm@kvack.org, syzkaller-bugs@googlegroups.com,
	Michal Hocko <mhocko@suse.com>,
	guro@fb.com, kirill.shutemov@linux.intel.com,
	linux-kernel@vger.kernel.org, rientjes@google.com,
	yang.s@alibaba-inc.com
Subject: Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks
Date: Fri, 12 Oct 2018 19:47:03 +0900	[thread overview]
Message-ID: <7bb967ec-81ae-2c07-7e58-9ad23167bb66@i-love.sakura.ne.jp> (raw)
In-Reply-To: <201810110637.w9B6b9eH044188@www262.sakura.ne.jp>

On 2018/10/11 15:37, Tetsuo Handa wrote:
> Michal Hocko wrote:
>> Once we are here, make sure that the reason to trigger the OOM is
>> printed without ratelimiting because this is really valuable to
>> debug what happened.
> 
> Here is my version.
> 

Hmm, per mem_cgroup flag would be better than per task_struct flag.

  reply	other threads:[~2018-10-12 10:47 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-10  0:08 INFO: rcu detected stall in shmem_fault syzbot
2018-10-10  0:12 ` Tetsuo Handa
2018-10-10  4:11   ` David Rientjes
2018-10-10  7:55     ` Dmitry Vyukov
2018-10-10  9:13       ` Michal Hocko
2018-10-10  9:33         ` Dmitry Vyukov
2018-10-10  9:02     ` Michal Hocko
2018-10-10  8:59   ` Michal Hocko
2018-10-10 10:43     ` Tetsuo Handa
2018-10-10 11:35       ` Michal Hocko
2018-10-10 11:48         ` Sergey Senozhatsky
2018-10-10 12:25           ` Michal Hocko
2018-10-10 12:29             ` Dmitry Vyukov
2018-10-10 12:36               ` Dmitry Vyukov
2018-10-10 13:10                 ` Tetsuo Handa
2018-10-10 13:17                   ` Dmitry Vyukov
2018-10-11  1:17                   ` Sergey Senozhatsky
2018-10-10 15:17               ` Sergey Senozhatsky
2018-10-10 14:19         ` Tetsuo Handa
2018-10-10 15:11 ` [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks Michal Hocko
2018-10-10 15:11   ` Michal Hocko
2018-10-11  6:37   ` Tetsuo Handa
2018-10-11  6:37     ` Tetsuo Handa
2018-10-12 10:47     ` Tetsuo Handa [this message]
2018-10-12 10:47       ` Tetsuo Handa
2018-10-12 11:20   ` Johannes Weiner
2018-10-12 12:08     ` Michal Hocko
2018-10-12 12:10       ` Tetsuo Handa
2018-10-12 12:41         ` Johannes Weiner
2018-10-12 12:58           ` Tetsuo Handa
2018-10-13 11:09             ` Tetsuo Handa
2018-10-13 11:22               ` Johannes Weiner
2018-10-13 11:28                 ` Tetsuo Handa
2018-10-15  8:19                   ` Michal Hocko
2018-10-15 10:57                     ` Tetsuo Handa
2018-10-15 11:24                       ` Michal Hocko
2018-10-15 12:47                         ` Tetsuo Handa
2018-10-15 13:35                           ` Michal Hocko
2018-10-16  0:55                             ` Tetsuo Handa
2018-10-16  9:20                               ` Michal Hocko
2018-10-16 11:05                                 ` Tetsuo Handa
2018-10-16 11:17                                   ` Michal Hocko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=7bb967ec-81ae-2c07-7e58-9ad23167bb66@i-love.sakura.ne.jp \
    --to=penguin-kernel@i-love.sakura.ne.jp \
    --cc=guro@fb.com \
    --cc=hannes@cmpxchg.org \
    --cc=kirill.shutemov@linux.intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=mhocko@suse.com \
    --cc=rientjes@google.com \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=yang.s@alibaba-inc.com \
    --subject='Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.