linux-kernel.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
From: Mel Gorman <mgorman@techsingularity.net>
To: Andrew Morton <akpm@linux-foundation.org>
Cc: Michal Hocko <mhocko@suse.com>, Vlastimil Babka <vbabka@suse.cz>,
	Alexey Avramov <hakavlad@inbox.lv>,
	Rik van Riel <riel@surriel.com>, Mike Galbraith <efault@gmx.de>,
	Darrick Wong <djwong@kernel.org>,
	regressions@lists.linux.dev,
	Linux-fsdevel <linux-fsdevel@vger.kernel.org>,
	Linux-MM <linux-mm@kvack.org>,
	LKML <linux-kernel@vger.kernel.org>,
	Mel Gorman <mgorman@techsingularity.net>
Subject: [PATCH 1/1] mm: vmscan: Reduce throttling due to a failure to make progress
Date: Thu, 25 Nov 2021 15:18:53 +0000	[thread overview]
Message-ID: <20211125151853.8540-1-mgorman@techsingularity.net> (raw)

Mike Galbraith, Alexey Avramov and Darrick Wong all reported similar
problems due to reclaim throttling for excessive lengths of time.
In Alexey's case, a memory hog that should go OOM quickly stalls for
several minutes before stalling. In Mike and Darrick's cases, a small
memcg environment stalled excessively even though the system had enough
memory overall.

Commit 69392a403f49 ("mm/vmscan: throttle reclaim when no progress is being
made") introduced the problem although commit a19594ca4a8b ("mm/vmscan:
increase the timeout if page reclaim is not making progress") made it
worse. Systems at or near an OOM state that cannot be recovered must
reach OOM quickly and memcg should kill tasks if a memcg is near OOM.

To address this, only stall for the first zone in the zonelist, reduce
the timeout to 1 tick for VMSCAN_THROTTLE_NOPROGRESS and only stall if
the scan control nr_reclaimed is 0 and kswapd is still active.  If kswapd
has stopped reclaiming due to excessive failures, do not stall at all so
that OOM triggers relatively quickly.

Alexey's test case was the most straight forward

	for i in {1..3}; do tail /dev/zero; done

On vanilla 5.16-rc1, this test stalled and was reset after 10 minutes.
After the patch, the test gets killed after roughly 15 seconds which is
the same length of time taken in 5.15.

Link: https://lore.kernel.org/r/99e779783d6c7fce96448a3402061b9dc1b3b602.camel@gmx.de
Link: https://lore.kernel.org/r/20211124011954.7cab9bb4@mail.inbox.lv
Link: https://lore.kernel.org/r/20211022144651.19914-1-mgorman@techsingularity.net

Fixes: 69392a403f49 ("mm/vmscan: throttle reclaim when no progress is being made")
Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
Tested-by: Darrick J. Wong <djwong@kernel.org>
---
 mm/vmscan.c | 21 ++++++++++++++++++---
 1 file changed, 18 insertions(+), 3 deletions(-)

diff --git a/mm/vmscan.c b/mm/vmscan.c
index fb9584641ac7..176ddd28df21 100644
--- a/mm/vmscan.c
+++ b/mm/vmscan.c
@@ -1057,7 +1057,17 @@ void reclaim_throttle(pg_data_t *pgdat, enum vmscan_throttle_state reason)
 
 		break;
 	case VMSCAN_THROTTLE_NOPROGRESS:
-		timeout = HZ/2;
+		timeout = 1;
+
+		/*
+		 * If kswapd is disabled, reschedule if necessary but do not
+		 * throttle as the system is likely near OOM.
+		 */
+		if (pgdat->kswapd_failures >= MAX_RECLAIM_RETRIES) {
+			cond_resched();
+			return;
+		}
+
 		break;
 	case VMSCAN_THROTTLE_ISOLATED:
 		timeout = HZ/50;
@@ -3395,7 +3405,7 @@ static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
 		return;
 
 	/* Throttle if making no progress at high prioities. */
-	if (sc->priority < DEF_PRIORITY - 2)
+	if (sc->priority < DEF_PRIORITY - 2 && !sc->nr_reclaimed)
 		reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS);
 }
 
@@ -3415,6 +3425,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
 	unsigned long nr_soft_scanned;
 	gfp_t orig_mask;
 	pg_data_t *last_pgdat = NULL;
+	pg_data_t *first_pgdat = NULL;
 
 	/*
 	 * If the number of buffer_heads in the machine exceeds the maximum
@@ -3478,14 +3489,18 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
 			/* need some check for avoid more shrink_zone() */
 		}
 
+		if (!first_pgdat)
+			first_pgdat = zone->zone_pgdat;
+
 		/* See comment about same check for global reclaim above */
 		if (zone->zone_pgdat == last_pgdat)
 			continue;
 		last_pgdat = zone->zone_pgdat;
 		shrink_node(zone->zone_pgdat, sc);
-		consider_reclaim_throttle(zone->zone_pgdat, sc);
 	}
 
+	consider_reclaim_throttle(first_pgdat, sc);
+
 	/*
 	 * Restore to original mask to avoid the impact on the caller if we
 	 * promoted it to __GFP_HIGHMEM.
-- 
2.31.1


             reply	other threads:[~2021-11-25 15:19 UTC|newest]

Thread overview: 38+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2021-11-25 15:18 Mel Gorman [this message]
2021-11-26 10:14 ` [PATCH 1/1] mm: vmscan: Reduce throttling due to a failure to make progress Vlastimil Babka
2021-11-26 10:26 ` Mike Galbraith
2021-11-26 16:12 ` Alexey Avramov
2021-11-26 16:52   ` Mel Gorman
2021-11-27 19:26     ` Alexey Avramov
2021-11-28 10:00       ` Mike Galbraith
2021-11-28 11:39         ` Mike Galbraith
2021-11-28 12:35           ` Mike Galbraith
2021-11-28 18:38             ` Mike Galbraith
2021-11-28 11:58         ` Alexey Avramov
2021-11-29  8:26         ` Mel Gorman
2021-11-29 15:01   ` Mel Gorman
2021-11-30 10:14     ` Mike Galbraith
2021-11-30 11:22       ` Mel Gorman
2021-11-30 12:00         ` Mike Galbraith
2021-11-30 12:51           ` Mike Galbraith
2021-11-30 13:09             ` Mel Gorman
2021-12-01  4:32               ` Mike Galbraith
2021-11-30 16:03     ` Alexey Avramov
2021-11-30 17:27       ` Mel Gorman
2021-11-30 17:59         ` Mike Galbraith
2021-12-01 13:01           ` Mel Gorman
2021-12-01 13:52             ` Mike Galbraith
2021-12-01 15:06               ` Mel Gorman
2021-11-30 18:38         ` Alexey Avramov
2021-12-01 14:00           ` Mel Gorman
2021-12-01 17:29             ` Darrick J. Wong
2021-12-02  9:43               ` Mel Gorman
2021-12-02 16:09                 ` Darrick J. Wong
2021-12-02  3:11             ` Mike Galbraith
2021-12-02 10:13               ` Mel Gorman
2021-12-02 10:51                 ` Mike Galbraith
2021-12-02 11:42             ` Alexey Avramov
2021-12-02 12:14               ` Alexey Avramov
2021-12-02 12:15               ` Mel Gorman
2021-12-02 12:22                 ` Alexey Avramov
2021-12-02 13:18 Mel Gorman

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20211125151853.8540-1-mgorman@techsingularity.net \
    --to=mgorman@techsingularity.net \
    --cc=akpm@linux-foundation.org \
    --cc=djwong@kernel.org \
    --cc=efault@gmx.de \
    --cc=hakavlad@inbox.lv \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-kernel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=mhocko@suse.com \
    --cc=regressions@lists.linux.dev \
    --cc=riel@surriel.com \
    --cc=vbabka@suse.cz \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).