From: Josef Bacik <josef@toxicpanda.com>
To: Dave Chinner <david@fromorbit.com>
Cc: Josef Bacik <josef@toxicpanda.com>,
linux-xfs@vger.kernel.org, linux-mm@kvack.org,
linux-fsdevel@vger.kernel.org
Subject: Re: [PATCH V2 00/26] mm, xfs: non-blocking inode reclaim
Date: Fri, 11 Oct 2019 20:19:21 -0400 [thread overview]
Message-ID: <20191012001919.lknks3k2at5xpxwf@macbook-pro-91.dhcp.thefacebook.com> (raw)
In-Reply-To: <20191011234842.GQ16973@dread.disaster.area>
On Sat, Oct 12, 2019 at 10:48:42AM +1100, Dave Chinner wrote:
> On Fri, Oct 11, 2019 at 03:03:08PM -0400, Josef Bacik wrote:
> > On Wed, Oct 09, 2019 at 02:20:58PM +1100, Dave Chinner wrote:
> > > Hi folks,
> > >
> > > This is the second version of the RFC I originally posted here:
> > >
> > > https://lore.kernel.org/linux-xfs/20190801021752.4986-1-david@fromorbit.com/
> > >
> > > The original description of the patchset is below, the issues and
> > > approach to solving them has not changed. THere is some
> > > restructuring of the patch set - the first few patches are all the
> > > XFS fixes that can be merged regardless of the rest of the patchset,
> > > but the non-blocking reclaim is somewhat dependent of them for
> > > correct behaviour. The second set of patches are the shrinker
> > > infrastructure changes needed for the shrinkers to feed back
> > > reclaim progress to the main reclaim instructure and act on the
> > > feedback. The last set of patches are the XFS changes needed to
> > > convert inode reclaim over to a non-blocking, IO-less algorithm.
> >
> > I looked through the MM patches and other than the congestion thing they look
> > reasonable. I think I can probably use this stuff to drop the use of the btree
> > inode. However I'm wondering if it would be a good idea to add an explicit
> > backoff thing for heavy metadata dirty'ing operations. Btrfs generates a lot
> > more dirty metadata than most, partly why my attempt to deal with this was tied
> > to using balance dirty pages since it already has all of the backoff logic.
>
> That's an orthorgonal problem, I think. We still need the IO-less
> reclaim in XFS regardless of how we throttle build up of dirty
> metadata...
>
> > Perhaps an explict balance_dirty_metadata() that we put after all
> > metadata operations so we have a good way to throttle dirtiers
> > when we aren't able to keep up? Just a thought, based on my
> > previous experiences trying to tackle this issue for btrfs, what
> > you've done already may be enough to address these concerns.
>
> The biggest issue is that different filesystems need different
> mechanisms for throttling dirty metadata build-up. In ext4/XFS, the
> amount of dirty metadata is bound by the log size, but that can
> still be massively more metadata than the disk subsystem can handle
> in a finite time.
>
> IOWs, for XFS, the way to throttle dirty metadata buildup is to
> limit the amount of log space we allow the filesystem to use when we
> are able to throttle incoming transaction reservations. Nothing in
> the VFS/mm subsystem can see any of this inside XFS, so I'm not
> really sure how generic we could make a metadata dirtying throttle
> implementation....
>
Ok, I just read the mm patches and made assumptions about what you were trying
to accomplish. I suppose I should probably dig my stuff back out. Thanks,
Josef
next prev parent reply other threads:[~2019-10-12 0:19 UTC|newest]
Thread overview: 87+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-10-09 3:20 [PATCH V2 00/26] mm, xfs: non-blocking inode reclaim Dave Chinner
2019-10-09 3:20 ` [PATCH 01/26] xfs: Lower CIL flush limit for large logs Dave Chinner
2019-10-11 12:39 ` Brian Foster
2019-10-30 17:08 ` Darrick J. Wong
2019-10-09 3:21 ` [PATCH 02/26] xfs: Throttle commits on delayed background CIL push Dave Chinner
2019-10-11 12:38 ` Brian Foster
2019-10-09 3:21 ` [PATCH 03/26] xfs: don't allow log IO to be throttled Dave Chinner
2019-10-11 9:35 ` Christoph Hellwig
2019-10-11 12:39 ` Brian Foster
2019-10-30 17:14 ` Darrick J. Wong
2019-10-09 3:21 ` [PATCH 04/26] xfs: Improve metadata buffer reclaim accountability Dave Chinner
2019-10-11 12:39 ` Brian Foster
2019-10-11 12:57 ` Christoph Hellwig
2019-10-11 23:14 ` Dave Chinner
2019-10-11 23:13 ` Dave Chinner
2019-10-12 12:05 ` Brian Foster
2019-10-13 3:14 ` Dave Chinner
2019-10-14 13:05 ` Brian Foster
2019-10-30 17:25 ` Darrick J. Wong
2019-10-30 21:43 ` Dave Chinner
2019-10-31 3:06 ` Darrick J. Wong
2019-10-31 20:50 ` Dave Chinner
2019-10-31 21:05 ` Darrick J. Wong
2019-10-31 21:22 ` Christoph Hellwig
2019-11-03 21:26 ` Dave Chinner
2019-11-04 23:08 ` Darrick J. Wong
2019-10-09 3:21 ` [PATCH 05/26] xfs: correctly acount for reclaimable slabs Dave Chinner
2019-10-11 12:39 ` Brian Foster
2019-10-30 17:16 ` Darrick J. Wong
2019-10-09 3:21 ` [PATCH 06/26] xfs: synchronous AIL pushing Dave Chinner
2019-10-11 9:42 ` Christoph Hellwig
2019-10-11 12:40 ` Brian Foster
2019-10-11 23:15 ` Dave Chinner
2019-10-09 3:21 ` [PATCH 07/26] xfs: tail updates only need to occur when LSN changes Dave Chinner
2019-10-11 9:50 ` Christoph Hellwig
2019-10-11 12:40 ` Brian Foster
2019-10-09 3:21 ` [PATCH 08/26] mm: directed shrinker work deferral Dave Chinner
2019-10-14 8:46 ` Christoph Hellwig
2019-10-14 13:06 ` Brian Foster
2019-10-18 7:59 ` Dave Chinner
2019-10-09 3:21 ` [PATCH 09/26] shrinkers: use defer_work for GFP_NOFS sensitive shrinkers Dave Chinner
2019-10-09 3:21 ` [PATCH 10/26] mm: factor shrinker work calculations Dave Chinner
2019-10-09 3:21 ` [PATCH 11/26] shrinker: defer work only to kswapd Dave Chinner
2019-10-09 3:21 ` [PATCH 12/26] shrinker: clean up variable types and tracepoints Dave Chinner
2019-10-09 3:21 ` [PATCH 13/26] mm: reclaim_state records pages reclaimed, not slabs Dave Chinner
2019-10-09 3:21 ` [PATCH 14/26] mm: back off direct reclaim on excessive shrinker deferral Dave Chinner
2019-10-11 16:21 ` Matthew Wilcox
2019-10-11 23:20 ` Dave Chinner
2019-10-09 3:21 ` [PATCH 15/26] mm: kswapd backoff for shrinkers Dave Chinner
2019-10-09 3:21 ` [PATCH 16/26] xfs: synchronous AIL pushing Dave Chinner
2019-10-11 10:18 ` Christoph Hellwig
2019-10-11 15:29 ` Brian Foster
2019-10-11 23:27 ` Dave Chinner
2019-10-12 12:08 ` Brian Foster
2019-10-09 3:21 ` [PATCH 17/26] xfs: don't block kswapd in inode reclaim Dave Chinner
2019-10-11 15:29 ` Brian Foster
2019-10-09 3:21 ` [PATCH 18/26] xfs: reduce kswapd blocking on inode locking Dave Chinner
2019-10-11 10:29 ` Christoph Hellwig
2019-10-09 3:21 ` [PATCH 19/26] xfs: kill background reclaim work Dave Chinner
2019-10-11 10:31 ` Christoph Hellwig
2019-10-09 3:21 ` [PATCH 20/26] xfs: use AIL pushing for inode reclaim IO Dave Chinner
2019-10-11 17:38 ` Brian Foster
2019-10-09 3:21 ` [PATCH 21/26] xfs: remove mode from xfs_reclaim_inodes() Dave Chinner
2019-10-11 10:39 ` Christoph Hellwig
2019-10-14 13:07 ` Brian Foster
2019-10-09 3:21 ` [PATCH 22/26] xfs: track reclaimable inodes using a LRU list Dave Chinner
2019-10-11 10:42 ` Christoph Hellwig
2019-10-14 13:07 ` Brian Foster
2019-10-09 3:21 ` [PATCH 23/26] xfs: reclaim inodes from the LRU Dave Chinner
2019-10-11 10:56 ` Christoph Hellwig
2019-10-30 23:25 ` Dave Chinner
2019-10-09 3:21 ` [PATCH 24/26] xfs: remove unusued old inode reclaim code Dave Chinner
2019-10-09 3:21 ` [PATCH 25/26] xfs: rework unreferenced inode lookups Dave Chinner
2019-10-11 12:55 ` Christoph Hellwig
2019-10-11 13:39 ` Peter Zijlstra
2019-10-11 23:38 ` Dave Chinner
2019-10-14 13:07 ` Brian Foster
2019-10-17 1:24 ` Dave Chinner
2019-10-17 7:57 ` Brian Foster
2019-10-18 20:29 ` Dave Chinner
2019-10-09 3:21 ` [PATCH 26/26] xfs: use xfs_ail_push_all_sync in xfs_reclaim_inodes Dave Chinner
2019-10-11 9:55 ` Christoph Hellwig
2019-10-09 7:06 ` [PATCH V2 00/26] mm, xfs: non-blocking inode reclaim Christoph Hellwig
2019-10-11 19:03 ` Josef Bacik
2019-10-11 23:48 ` Dave Chinner
2019-10-12 0:19 ` Josef Bacik [this message]
2019-10-12 0:48 ` Dave Chinner
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20191012001919.lknks3k2at5xpxwf@macbook-pro-91.dhcp.thefacebook.com \
--to=josef@toxicpanda.com \
--cc=david@fromorbit.com \
--cc=linux-fsdevel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=linux-xfs@vger.kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).