From: "Darrick J. Wong" <djwong@kernel.org>
To: Mel Gorman <mgorman@techsingularity.net>
Cc: Andrew Morton <akpm@linux-foundation.org>,
NeilBrown <neilb@suse.de>, Theodore Ts'o <tytso@mit.edu>,
Andreas Dilger <adilger.kernel@dilger.ca>,
Matthew Wilcox <willy@infradead.org>,
Michal Hocko <mhocko@suse.com>,
Dave Chinner <david@fromorbit.com>,
Rik van Riel <riel@surriel.com>, Vlastimil Babka <vbabka@suse.cz>,
Johannes Weiner <hannes@cmpxchg.org>,
Jonathan Corbet <corbet@lwn.net>, Linux-MM <linux-mm@kvack.org>,
Linux-fsdevel <linux-fsdevel@vger.kernel.org>,
LKML <linux-kernel@vger.kernel.org>
Subject: Re: [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made
Date: Tue, 23 Nov 2021 17:49:14 -0800 [thread overview]
Message-ID: <20211124014914.GB265983@magnolia> (raw)
In-Reply-To: <20211124011912.GA265983@magnolia>
On Tue, Nov 23, 2021 at 05:19:12PM -0800, Darrick J. Wong wrote:
> On Fri, Oct 22, 2021 at 03:46:46PM +0100, Mel Gorman wrote:
> > Memcg reclaim throttles on congestion if no reclaim progress is made.
> > This makes little sense, it might be due to writeback or a host of
> > other factors.
> >
> > For !memcg reclaim, it's messy. Direct reclaim primarily is throttled
> > in the page allocator if it is failing to make progress. Kswapd
> > throttles if too many pages are under writeback and marked for
> > immediate reclaim.
> >
> > This patch explicitly throttles if reclaim is failing to make progress.
>
> Hi Mel,
>
> Ever since Christoph broke swapfiles, I've been carrying around a little
> fstest in my dev tree[1] that tries to exercise paging things in and out
> of a swapfile. Sadly I've been trapped in about three dozen customer
> escalations for over a month, which means I haven't been able to do much
> upstream in weeks. Like submit this test upstream. :(
>
> Now that I've finally gotten around to trying out a 5.16-rc2 build, I
> notice that the runtime of this test has gone from ~5s to 2 hours.
> Among other things that it does, the test sets up a cgroup with a memory
> controller limiting the memory usage to 25MB, then runs a program that
> tries to dirty 50MB of memory. There's 2GB of memory in the VM, so
> we're not running reclaim globally, but the cgroup gets throttled very
> severely.
>
> AFAICT the system is mostly idle, but it's difficult to tell because ps
> and top also get stuck waiting for this cgroup for whatever reason. My
> uninformed spculation is that usemem_and_swapoff takes a page fault
> while dirtying the 50MB memory buffer, prepares to pull a page in from
> swap, tries to evict another page to stay under the memcg limit, but
> that decides that it's making no progress and calls
> reclaim_throttle(..., VMSCAN_THROTTLE_NOPROGRESS).
>
> The sleep is uninterruptible, so I can't even kill -9 fstests to shut it
> down. Eventually we either finish the test or (for the mlock part) the
> OOM killer actually kills the process, but this takes a very long time.
>
> Any thoughts? For now I can just hack around this by skipping
> reclaim_throttle if cgroup_reclaim() == true, but that's probably not
> the correct fix. :)
Update: after adding timing information to usemem_and_swapoff, it looks
like dirtying the 50MB buffer takes ~22s (up from 0.06s on 5.15). The
mlock call stalls for ~280s until the OOM killer kills it (up from
nearly instantaneous on 5.15), and the swapon/swapoff variant takes
20 minutes to hours depending on the run.
--D
> --D
>
> [1] https://git.kernel.org/pub/scm/linux/kernel/git/djwong/xfstests-dev.git/commit/?h=test-swapfile-io&id=0d0ad843cea366d0ab0a7d8d984e5cd1deba5b43
>
> >
> > [vbabka@suse.cz: Remove redundant code]
> > Signed-off-by: Mel Gorman <mgorman@techsingularity.net>
> > Acked-by: Vlastimil Babka <vbabka@suse.cz>
> > ---
> > include/linux/mmzone.h | 1 +
> > include/trace/events/vmscan.h | 4 +++-
> > mm/memcontrol.c | 10 +---------
> > mm/vmscan.c | 28 ++++++++++++++++++++++++++++
> > 4 files changed, 33 insertions(+), 10 deletions(-)
> >
> > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h
> > index 9ccd8d95291b..00e305cfb3ec 100644
> > --- a/include/linux/mmzone.h
> > +++ b/include/linux/mmzone.h
> > @@ -276,6 +276,7 @@ enum lru_list {
> > enum vmscan_throttle_state {
> > VMSCAN_THROTTLE_WRITEBACK,
> > VMSCAN_THROTTLE_ISOLATED,
> > + VMSCAN_THROTTLE_NOPROGRESS,
> > NR_VMSCAN_THROTTLE,
> > };
> >
> > diff --git a/include/trace/events/vmscan.h b/include/trace/events/vmscan.h
> > index d4905bd9e9c4..f25a6149d3ba 100644
> > --- a/include/trace/events/vmscan.h
> > +++ b/include/trace/events/vmscan.h
> > @@ -29,11 +29,13 @@
> >
> > #define _VMSCAN_THROTTLE_WRITEBACK (1 << VMSCAN_THROTTLE_WRITEBACK)
> > #define _VMSCAN_THROTTLE_ISOLATED (1 << VMSCAN_THROTTLE_ISOLATED)
> > +#define _VMSCAN_THROTTLE_NOPROGRESS (1 << VMSCAN_THROTTLE_NOPROGRESS)
> >
> > #define show_throttle_flags(flags) \
> > (flags) ? __print_flags(flags, "|", \
> > {_VMSCAN_THROTTLE_WRITEBACK, "VMSCAN_THROTTLE_WRITEBACK"}, \
> > - {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"} \
> > + {_VMSCAN_THROTTLE_ISOLATED, "VMSCAN_THROTTLE_ISOLATED"}, \
> > + {_VMSCAN_THROTTLE_NOPROGRESS, "VMSCAN_THROTTLE_NOPROGRESS"} \
> > ) : "VMSCAN_THROTTLE_NONE"
> >
> >
> > diff --git a/mm/memcontrol.c b/mm/memcontrol.c
> > index 6da5020a8656..8b33152c9b85 100644
> > --- a/mm/memcontrol.c
> > +++ b/mm/memcontrol.c
> > @@ -3465,19 +3465,11 @@ static int mem_cgroup_force_empty(struct mem_cgroup *memcg)
> >
> > /* try to free all pages in this cgroup */
> > while (nr_retries && page_counter_read(&memcg->memory)) {
> > - int progress;
> > -
> > if (signal_pending(current))
> > return -EINTR;
> >
> > - progress = try_to_free_mem_cgroup_pages(memcg, 1,
> > - GFP_KERNEL, true);
> > - if (!progress) {
> > + if (!try_to_free_mem_cgroup_pages(memcg, 1, GFP_KERNEL, true))
> > nr_retries--;
> > - /* maybe some writeback is necessary */
> > - congestion_wait(BLK_RW_ASYNC, HZ/10);
> > - }
> > -
> > }
> >
> > return 0;
> > diff --git a/mm/vmscan.c b/mm/vmscan.c
> > index 1e54e636b927..0450f6867d61 100644
> > --- a/mm/vmscan.c
> > +++ b/mm/vmscan.c
> > @@ -3323,6 +3323,33 @@ static inline bool compaction_ready(struct zone *zone, struct scan_control *sc)
> > return zone_watermark_ok_safe(zone, 0, watermark, sc->reclaim_idx);
> > }
> >
> > +static void consider_reclaim_throttle(pg_data_t *pgdat, struct scan_control *sc)
> > +{
> > + /* If reclaim is making progress, wake any throttled tasks. */
> > + if (sc->nr_reclaimed) {
> > + wait_queue_head_t *wqh;
> > +
> > + wqh = &pgdat->reclaim_wait[VMSCAN_THROTTLE_NOPROGRESS];
> > + if (waitqueue_active(wqh))
> > + wake_up(wqh);
> > +
> > + return;
> > + }
> > +
> > + /*
> > + * Do not throttle kswapd on NOPROGRESS as it will throttle on
> > + * VMSCAN_THROTTLE_WRITEBACK if there are too many pages under
> > + * writeback and marked for immediate reclaim at the tail of
> > + * the LRU.
> > + */
> > + if (current_is_kswapd())
> > + return;
> > +
> > + /* Throttle if making no progress at high prioities. */
> > + if (sc->priority < DEF_PRIORITY - 2)
> > + reclaim_throttle(pgdat, VMSCAN_THROTTLE_NOPROGRESS, HZ/10);
> > +}
> > +
> > /*
> > * This is the direct reclaim path, for page-allocating processes. We only
> > * try to reclaim pages from zones which will satisfy the caller's allocation
> > @@ -3407,6 +3434,7 @@ static void shrink_zones(struct zonelist *zonelist, struct scan_control *sc)
> > continue;
> > last_pgdat = zone->zone_pgdat;
> > shrink_node(zone->zone_pgdat, sc);
> > + consider_reclaim_throttle(zone->zone_pgdat, sc);
> > }
> >
> > /*
> > --
> > 2.31.1
> >
next prev parent reply other threads:[~2021-11-24 1:49 UTC|newest]
Thread overview: 23+ messages / expand[flat|nested] mbox.gz Atom feed top
2021-10-22 14:46 [PATCH v5 0/8] Remove dependency on congestion_wait in mm/ Mel Gorman
2021-10-22 14:46 ` [PATCH 1/8] mm/vmscan: Throttle reclaim until some writeback completes if congested Mel Gorman
2021-10-22 14:46 ` [PATCH 2/8] mm/vmscan: Throttle reclaim and compaction when too may pages are isolated Mel Gorman
2021-10-22 14:46 ` [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made Mel Gorman
2021-11-24 1:19 ` Darrick J. Wong
2021-11-24 1:49 ` Darrick J. Wong [this message]
2021-11-24 14:35 ` Mel Gorman
2021-11-24 18:02 ` Darrick J. Wong
2021-11-24 10:32 ` Mel Gorman
2021-11-24 10:43 ` Vlastimil Babka
2021-11-24 10:53 ` Mel Gorman
2021-11-24 17:24 ` Mike Galbraith
2021-10-22 14:46 ` [PATCH 4/8] mm/writeback: Throttle based on page writeback instead of congestion Mel Gorman
2021-10-22 14:46 ` [PATCH 5/8] mm/page_alloc: Remove the throttling logic from the page allocator Mel Gorman
2021-10-25 10:07 ` Vlastimil Babka
2021-10-22 14:46 ` [PATCH 6/8] mm/vmscan: Centralise timeout values for reclaim_throttle Mel Gorman
2021-10-22 14:46 ` [PATCH 7/8] mm/vmscan: Increase the timeout if page reclaim is not making progress Mel Gorman
2021-10-22 14:46 ` [PATCH 8/8] mm/vmscan: Delay waking of tasks throttled on NOPROGRESS Mel Gorman
-- strict thread matches above, loose matches on Subject: below --
2021-10-19 9:01 [PATCH v4 0/8] Remove dependency on congestion_wait in mm/ Mel Gorman
2021-10-19 9:01 ` [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made Mel Gorman
2021-10-08 13:53 [PATCH v3 0/8] Remove dependency on congestion_wait in mm/ Mel Gorman
2021-10-08 13:53 ` [PATCH 3/8] mm/vmscan: Throttle reclaim when no progress is being made Mel Gorman
2021-10-14 12:31 ` Vlastimil Babka
2021-10-14 13:03 ` Mel Gorman
2021-10-14 15:45 ` Vlastimil Babka
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20211124014914.GB265983@magnolia \
--to=djwong@kernel.org \
--cc=adilger.kernel@dilger.ca \
--cc=akpm@linux-foundation.org \
--cc=corbet@lwn.net \
--cc=david@fromorbit.com \
--cc=hannes@cmpxchg.org \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=mgorman@techsingularity.net \
--cc=mhocko@suse.com \
--cc=neilb@suse.de \
--cc=riel@surriel.com \
--cc=tytso@mit.edu \
--cc=vbabka@suse.cz \
--cc=willy@infradead.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).