All of lore.kernel.org
 help / color / mirror / Atom feed
From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-mm@kvack.org, syzkaller-bugs@googlegroups.com,
	Michal Hocko <mhocko@suse.com>,
	guro@fb.com, hannes@cmpxchg.org, kirill.shutemov@linux.intel.com,
	linux-kernel@vger.kernel.org, rientjes@google.com,
	yang.s@alibaba-inc.com
Subject: Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks
Date: Thu, 11 Oct 2018 15:37:09 +0900	[thread overview]
Message-ID: <201810110637.w9B6b9eH044188@www262.sakura.ne.jp> (raw)
In-Reply-To: <20181010151135.25766-1-mhocko@kernel.org>

Michal Hocko wrote:
> Once we are here, make sure that the reason to trigger the OOM is
> printed without ratelimiting because this is really valuable to
> debug what happened.

Here is my version.

From 0c9ab34fd01837d4c85794042ecb9e922c9eed5a Mon Sep 17 00:00:00 2001
From: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Date: Thu, 11 Oct 2018 15:27:42 +0900
Subject: [PATCH] mm: memcontrol: Don't flood OOM messages with no eligible task.

syzbot is hitting RCU stall at shmem_fault() [1].
This is because memcg-OOM events with no eligible task (current thread
is marked as OOM-unkillable) continued calling dump_header() from
out_of_memory() enabled by commit 3100dab2aa09dc6e ("mm: memcontrol:
print proper OOM header when no eligible victim left.").

Let's make sure that next dump_header() waits for at least 60 seconds from
previous "Out of memory and no killable processes..." message, if current
thread already reported it. This change allows current thread to complete
memory allocation requests for doing recovery operation, and also allows
us to know whether the current thread is doing recovery operation.

[1] https://syzkaller.appspot.com/bug?id=4ae3fff7fcf4c33a47c1192d2d62d2e03efffa64

Signed-off-by: Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>
Reported-by: syzbot <syzbot+77e6b28a7a7106ad0def@syzkaller.appspotmail.com>
Cc: Johannes Weiner <hannes@cmpxchg.org>
Cc: Michal Hocko <mhocko@suse.com>
---
 include/linux/sched.h |  1 +
 mm/oom_kill.c         | 23 +++++++++++++++++++++++
 2 files changed, 24 insertions(+)

diff --git a/include/linux/sched.h b/include/linux/sched.h
index 977cb57..701cf3c 100644
--- a/include/linux/sched.h
+++ b/include/linux/sched.h
@@ -723,6 +723,7 @@ struct task_struct {
 #endif
 #ifdef CONFIG_MEMCG
 	unsigned			in_user_fault:1;
+	unsigned			memcg_oom_no_eligible_warned:1;
 #ifdef CONFIG_MEMCG_KMEM
 	unsigned			memcg_kmem_skip_account:1;
 #endif
diff --git a/mm/oom_kill.c b/mm/oom_kill.c
index f10aa53..f954d99 100644
--- a/mm/oom_kill.c
+++ b/mm/oom_kill.c
@@ -1106,6 +1106,21 @@ bool out_of_memory(struct oom_control *oc)
 	select_bad_process(oc);
 	/* Found nothing?!?! */
 	if (!oc->chosen) {
+#ifdef CONFIG_MEMCG
+		/*
+		 * Don't flood with dump_header() if already reported, in case
+		 * current->signal->oom_score_adj == OOM_SCORE_ADJ_MIN. Maybe
+		 * this variable should be per "struct mem_cgroup". But since
+		 * we can't prove that multiple concurrent memcg OOM without
+		 * eligible task won't cause flooding, choose global variable
+		 * for safety.
+		 */
+		static u64 last_warned;
+
+		if (is_memcg_oom(oc) && current->memcg_oom_no_eligible_warned
+		    && time_before64(get_jiffies_64(), last_warned + 60 * HZ))
+			return false;
+#endif
 		dump_header(oc, NULL);
 		pr_warn("Out of memory and no killable processes...\n");
 		/*
@@ -1115,6 +1130,14 @@ bool out_of_memory(struct oom_control *oc)
 		 */
 		if (!is_sysrq_oom(oc) && !is_memcg_oom(oc))
 			panic("System is deadlocked on memory\n");
+#ifdef CONFIG_MEMCG
+		if (is_memcg_oom(oc)) {
+			last_warned = get_jiffies_64();
+			current->memcg_oom_no_eligible_warned = 1;
+		}
+	} else if (is_memcg_oom(oc)) {
+		current->memcg_oom_no_eligible_warned = 0;
+#endif
 	}
 	if (oc->chosen && oc->chosen != (void *)-1UL)
 		oom_kill_process(oc, !is_memcg_oom(oc) ? "Out of memory" :
-- 
1.8.3.1


WARNING: multiple messages have this Message-ID
From: Tetsuo Handa <penguin-kernel@i-love.sakura.ne.jp>
To: Michal Hocko <mhocko@kernel.org>
Cc: linux-mm@kvack.org, syzkaller-bugs@googlegroups.com,
	Michal Hocko <mhocko@suse.com>,
	guro@fb.com, hannes@cmpxchg.org, kirill.shutemov@linux.intel.com,
	linux-kernel@vger.kernel.org, rientjes@google.com,
	yang.s@alibaba-inc.com
Subject: Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks
Date: Thu, 11 Oct 2018 15:37:09 +0900	[thread overview]
Message-ID: <201810110637.w9B6b9eH044188@www262.sakura.ne.jp> (raw)
In-Reply-To: <20181010151135.25766-1-mhocko@kernel.org>

Michal Hocko wrote:
> Once we are here, make sure that the reason to trigger the OOM is
> printed without ratelimiting because this is really valuable to
> debug what happened.

Here is my version.

  reply	other threads:[~2018-10-11  6:37 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2018-10-10  0:08 INFO: rcu detected stall in shmem_fault syzbot
2018-10-10  0:12 ` Tetsuo Handa
2018-10-10  4:11   ` David Rientjes
2018-10-10  7:55     ` Dmitry Vyukov
2018-10-10  9:13       ` Michal Hocko
2018-10-10  9:33         ` Dmitry Vyukov
2018-10-10  9:02     ` Michal Hocko
2018-10-10  8:59   ` Michal Hocko
2018-10-10 10:43     ` Tetsuo Handa
2018-10-10 11:35       ` Michal Hocko
2018-10-10 11:48         ` Sergey Senozhatsky
2018-10-10 12:25           ` Michal Hocko
2018-10-10 12:29             ` Dmitry Vyukov
2018-10-10 12:36               ` Dmitry Vyukov
2018-10-10 13:10                 ` Tetsuo Handa
2018-10-10 13:17                   ` Dmitry Vyukov
2018-10-11  1:17                   ` Sergey Senozhatsky
2018-10-10 15:17               ` Sergey Senozhatsky
2018-10-10 14:19         ` Tetsuo Handa
2018-10-10 15:11 ` [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks Michal Hocko
2018-10-10 15:11   ` Michal Hocko
2018-10-11  6:37   ` Tetsuo Handa [this message]
2018-10-11  6:37     ` Tetsuo Handa
2018-10-12 10:47     ` Tetsuo Handa
2018-10-12 10:47       ` Tetsuo Handa
2018-10-12 11:20   ` Johannes Weiner
2018-10-12 12:08     ` Michal Hocko
2018-10-12 12:10       ` Tetsuo Handa
2018-10-12 12:41         ` Johannes Weiner
2018-10-12 12:58           ` Tetsuo Handa
2018-10-13 11:09             ` Tetsuo Handa
2018-10-13 11:22               ` Johannes Weiner
2018-10-13 11:28                 ` Tetsuo Handa
2018-10-15  8:19                   ` Michal Hocko
2018-10-15 10:57                     ` Tetsuo Handa
2018-10-15 11:24                       ` Michal Hocko
2018-10-15 12:47                         ` Tetsuo Handa
2018-10-15 13:35                           ` Michal Hocko
2018-10-16  0:55                             ` Tetsuo Handa
2018-10-16  9:20                               ` Michal Hocko
2018-10-16 11:05                                 ` Tetsuo Handa
2018-10-16 11:17                                   ` Michal Hocko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=201810110637.w9B6b9eH044188@www262.sakura.ne.jp \
    --to=penguin-kernel@i-love.sakura.ne.jp \
    --cc=guro@fb.com \
    --cc=hannes@cmpxchg.org \
    --cc=kirill.shutemov@linux.intel.com \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@kernel.org \
    --cc=mhocko@suse.com \
    --cc=rientjes@google.com \
    --cc=syzkaller-bugs@googlegroups.com \
    --cc=yang.s@alibaba-inc.com \
    --subject='Re: [RFC PATCH] memcg, oom: throttle dump_header for memcg ooms without eligible tasks' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.