All of lore.kernel.org
 help / color / mirror / Atom feed
From: Michal Hocko <mhocko@kernel.org>
To: <linux-mm@kvack.org>
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Mel Gorman <mgorman@suse.de>,
	Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
	LKML <linux-kernel@vger.kernel.org>,
	Michal Hocko <mhocko@suse.com>
Subject: [PATCH] mm: warn about allocations which stall for too long
Date: Fri, 23 Sep 2016 10:15:55 +0200	[thread overview]
Message-ID: <20160923081555.14645-1-mhocko@kernel.org> (raw)

From: Michal Hocko <mhocko@suse.com>

Currently we do warn only about allocation failures but small
allocations are basically nofail and they might loop in the page
allocator for a long time.  Especially when the reclaim cannot make
any progress - e.g. GFP_NOFS cannot invoke the oom killer and rely on
a different context to make a forward progress in case there is a lot
memory used by filesystems.

Give us at least a clue when something like this happens and warn about
allocations which take more than 10s. Print the basic allocation context
information along with the cumulative time spent in the allocation as
well as the allocation stack. Repeat the warning after every 10 seconds so
that we know that the problem is permanent rather than ephemeral.

Signed-off-by: Michal Hocko <mhocko@suse.com>
---

Hi,
I am sending this as an RFC because I am not really sure what is the reasonable
timeout when to warn. I went with 10s because that should be close to "for ever"
from the user perspective. But maybe a shorter would be helpful as well?
I didn't go with a tunable because I would rather not add a new one.

Thoughts? Ideas?

 mm/page_alloc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5155485057cb..d5faab8aa94d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3485,6 +3485,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	enum compact_result compact_result;
 	int compaction_retries = 0;
 	int no_progress_loops = 0;
+	unsigned long alloc_start = jiffies;
+	unsigned int stall_timeout = 10 * HZ;
 
 	/*
 	 * In the slowpath, we sanity check order to avoid ever trying to
@@ -3659,6 +3661,15 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	else
 		no_progress_loops++;
 
+	/* Make sure we know about allocations which stall for too long */
+	if (!(gfp_mask & __GFP_NOWARN) && time_after(jiffies, alloc_start + stall_timeout)) {
+		pr_warn("%s: page alloction stalls for %ums: order:%u mode:%#x(%pGg)\n",
+				current->comm, jiffies_to_msecs(jiffies-alloc_start),
+				order, gfp_mask, &gfp_mask);
+		stall_timeout += 10 * HZ;
+		dump_stack();
+	}
+
 	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
 				 did_some_progress > 0, no_progress_loops))
 		goto retry;
-- 
2.9.3

WARNING: multiple messages have this Message-ID
From: Michal Hocko <mhocko@kernel.org>
To: linux-mm@kvack.org
Cc: Andrew Morton <akpm@linux-foundation.org>,
	Johannes Weiner <hannes@cmpxchg.org>,
	Mel Gorman <mgorman@suse.de>,
	Tetsuo Handa <penguin-kernel@I-love.SAKURA.ne.jp>,
	LKML <linux-kernel@vger.kernel.org>,
	Michal Hocko <mhocko@suse.com>
Subject: [PATCH] mm: warn about allocations which stall for too long
Date: Fri, 23 Sep 2016 10:15:55 +0200	[thread overview]
Message-ID: <20160923081555.14645-1-mhocko@kernel.org> (raw)

From: Michal Hocko <mhocko@suse.com>

Currently we do warn only about allocation failures but small
allocations are basically nofail and they might loop in the page
allocator for a long time.  Especially when the reclaim cannot make
any progress - e.g. GFP_NOFS cannot invoke the oom killer and rely on
a different context to make a forward progress in case there is a lot
memory used by filesystems.

Give us at least a clue when something like this happens and warn about
allocations which take more than 10s. Print the basic allocation context
information along with the cumulative time spent in the allocation as
well as the allocation stack. Repeat the warning after every 10 seconds so
that we know that the problem is permanent rather than ephemeral.

Signed-off-by: Michal Hocko <mhocko@suse.com>
---

Hi,
I am sending this as an RFC because I am not really sure what is the reasonable
timeout when to warn. I went with 10s because that should be close to "for ever"
from the user perspective. But maybe a shorter would be helpful as well?
I didn't go with a tunable because I would rather not add a new one.

Thoughts? Ideas?

 mm/page_alloc.c | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/mm/page_alloc.c b/mm/page_alloc.c
index 5155485057cb..d5faab8aa94d 100644
--- a/mm/page_alloc.c
+++ b/mm/page_alloc.c
@@ -3485,6 +3485,8 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	enum compact_result compact_result;
 	int compaction_retries = 0;
 	int no_progress_loops = 0;
+	unsigned long alloc_start = jiffies;
+	unsigned int stall_timeout = 10 * HZ;
 
 	/*
 	 * In the slowpath, we sanity check order to avoid ever trying to
@@ -3659,6 +3661,15 @@ __alloc_pages_slowpath(gfp_t gfp_mask, unsigned int order,
 	else
 		no_progress_loops++;
 
+	/* Make sure we know about allocations which stall for too long */
+	if (!(gfp_mask & __GFP_NOWARN) && time_after(jiffies, alloc_start + stall_timeout)) {
+		pr_warn("%s: page alloction stalls for %ums: order:%u mode:%#x(%pGg)\n",
+				current->comm, jiffies_to_msecs(jiffies-alloc_start),
+				order, gfp_mask, &gfp_mask);
+		stall_timeout += 10 * HZ;
+		dump_stack();
+	}
+
 	if (should_reclaim_retry(gfp_mask, order, ac, alloc_flags,
 				 did_some_progress > 0, no_progress_loops))
 		goto retry;
-- 
2.9.3

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

             reply	other threads:[~2016-09-23  8:16 UTC|newest]

Thread overview: 42+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2016-09-23  8:15 Michal Hocko [this message]
2016-09-23  8:15 ` Michal Hocko
2016-09-23  8:29 ` Hillf Danton
2016-09-23  8:29   ` Hillf Danton
2016-09-23  8:32   ` Michal Hocko
2016-09-23  8:32     ` Michal Hocko
2016-09-23  8:44     ` Hillf Danton
2016-09-23  8:44       ` Hillf Danton
2016-09-23  9:15       ` Michal Hocko
2016-09-23  9:15         ` Michal Hocko
2016-09-23 14:36 ` Tetsuo Handa
2016-09-23 14:36   ` Tetsuo Handa
2016-09-23 15:02   ` Michal Hocko
2016-09-23 15:02     ` Michal Hocko
2016-09-24  3:00     ` Tetsuo Handa
2016-09-24  3:00       ` Tetsuo Handa
2016-09-26  8:17       ` Michal Hocko
2016-09-26  8:17         ` Michal Hocko
2016-09-27 12:57         ` Tetsuo Handa
2016-09-27 12:57           ` Tetsuo Handa
2016-09-29  8:48           ` Michal Hocko
2016-09-29  8:48             ` Michal Hocko
2016-09-23 17:34 ` Dave Hansen
2016-09-23 17:34   ` Dave Hansen
2016-09-24 13:19   ` Balbir Singh
2016-09-24 13:19     ` Balbir Singh
2016-09-26  8:13     ` Michal Hocko
2016-09-26  8:13       ` Michal Hocko
2016-09-26  8:12   ` Michal Hocko
2016-09-26  8:12     ` Michal Hocko
2016-09-29  8:44 ` [PATCH 0/2] " Michal Hocko
2016-09-29  8:44   ` Michal Hocko
2016-09-29  8:44   ` [PATCH 1/2] mm: consolidate warn_alloc_failed users Michal Hocko
2016-09-29  8:44     ` Michal Hocko
2016-09-29  9:23     ` Vlastimil Babka
2016-09-29  9:23       ` Vlastimil Babka
2016-09-29  8:44   ` [PATCH 2/2] mm: warn about allocations which stall for too long Michal Hocko
2016-09-29  8:44     ` Michal Hocko
2016-09-29  9:02     ` Tetsuo Handa
2016-09-29  9:02       ` Tetsuo Handa
2016-09-29  9:10       ` Michal Hocko
2016-09-29  9:10         ` Michal Hocko

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20160923081555.14645-1-mhocko@kernel.org \
    --to=mhocko@kernel.org \
    --cc=akpm@linux-foundation.org \
    --cc=hannes@cmpxchg.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mgorman@suse.de \
    --cc=mhocko@suse.com \
    --cc=penguin-kernel@I-love.SAKURA.ne.jp \
    --subject='Re: [PATCH] mm: warn about allocations which stall for too long' \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.