Re: Massive I/O usage from btrfs-cleaner after upgrading to 5.16

[Filipe Manana <fdmanana@kernel.org>]

On Thu, Jan 20, 2022 at 12:02 PM François-Xavier Thomas
<fx.thomas@gmail.com> wrote:
>
> > What if on top of those patches, you also add this one:
> > https://pastebin.com/raw/EbEfk1tF
>
> That's exactly patch 2 in my stack of patches in fact, is that the correct link?

It was the correct link, but I forgot that I had already given it to
you (there's another thread from another
user that reported defrag/autodefrag issues in 5.16 as well).

Ok, so new patches to try and the new stack of patches should be:

1) https://patchwork.kernel.org/project/linux-btrfs/patch/bcbfce0ff7e21bbfed2484b1457e560edf78020d.1642436805.git.fdmanana@suse.com/

2) https://patchwork.kernel.org/project/linux-btrfs/patch/20220118071904.29991-1-wqu@suse.com/

3) https://patchwork.kernel.org/project/linux-btrfs/patch/20220118115352.52126-1-wqu@suse.com/

4) https://patchwork.kernel.org/project/linux-btrfs/patch/5cb3ce140c84b0283be685bae8a5d75d5d19af08.1642688018.git.fdmanana@suse.com/

5) https://patchwork.kernel.org/project/linux-btrfs/patch/3fe2f747e0a9319064d59d051dc3f993fc41b172.1642698605.git.fdmanana@suse.com/

6) https://patchwork.kernel.org/project/linux-btrfs/patch/20aad8ccf0fbdecddd49216f25fa772754f77978.1642700395.git.fdmanana@suse.com/

Hope that helps.
Thanks.


>
> On Thu, Jan 20, 2022 at 12:45 PM Filipe Manana <fdmanana@kernel.org> wrote:
> >
> > On Thu, Jan 20, 2022 at 11:37 AM François-Xavier Thomas
> > <fx.thomas@gmail.com> wrote:
> > >
> > > Hi Felipe,
> > >
> > > > So, try with these two more patches on top of that:
> > >
> > > Thanks, I did just that, see graph with annotations:
> > > https://i.imgur.com/pu66nz0.png
> > >
> > > No visible improvement, average baseline I/O (for roughly similar
> > > workloads, the server I'm testing it on is not very busy I/O-wise) is
> > > still 3-4x higher in 5.16 than in 5.15 with autodefrag enabled.
> >
> > What if on top of those patches, you also add this one:
> >
> > https://pastebin.com/raw/EbEfk1tF
> >
> > Can you see if it helps?
> >
> > >
> > > The good news is that patch 2 did fix a large part of the issues 5.16.0 had.
> > > I also checked that disabling autodefrag immediately brings I/O rate
> > > back to how it was in 5.15.
> >
> > At least that!
> > Thanks.
> >
> > >
> > > >> Some people reported that 5.16.1 improved the situation for them, so
> > > > I don't see how that's possible, nothing was added to 5.16.1 that
> > > > involves defrag.
> > > > Might just be a coincidence.
> > >
> > > Yes, I found no evidence that official 5.16.1 is any better than the
> > > rest on my side.
> > >
> > > François-Xavier
> > >
> > > On Wed, Jan 19, 2022 at 11:14 AM Filipe Manana <fdmanana@kernel.org> wrote:
> > > >
> > > > On Wed, Jan 19, 2022 at 9:44 AM François-Xavier Thomas
> > > > <fx.thomas@gmail.com> wrote:
> > > > >
> > > > > Hi,
> > > > >
> > > > > More details on graph[0]:
> > > > > - First patch (1-byte file) on 5.16.0 did not have a significant impact.
> > > > > - Both patches on 5.16.0 did reduce a large part of the I/O but still
> > > > > have a high baseline I/O compared to 5.15
> > > >
> > > > So, try with these two more patches on top of that:
> > > >
> > > > https://patchwork.kernel.org/project/linux-btrfs/patch/20220118071904.29991-1-wqu@suse.com/
> > > >
> > > > https://patchwork.kernel.org/project/linux-btrfs/patch/20220118115352.52126-1-wqu@suse.com/
> > > >
> > > > >
> > > > > Some people reported that 5.16.1 improved the situation for them, so
> > > >
> > > > I don't see how that's possible, nothing was added to 5.16.1 that
> > > > involves defrag.
> > > > Might just be a coincidence.
> > > >
> > > > Thanks.
> > > >
> > > > > I'm testing that. It's too early to tell but for now the baseline I/O
> > > > > still seems to be high compared to 5.15. Will update with more results
> > > > > tomorrow.
> > > > >
> > > > > François-Xavier
> > > > >
> > > > > [0] https://i.imgur.com/agzAKGc.png
> > > > >
> > > > > On Mon, Jan 17, 2022 at 10:37 PM François-Xavier Thomas
> > > > > <fx.thomas@gmail.com> wrote:
> > > > > >
> > > > > > Hi Filipe,
> > > > > >
> > > > > > Thank you so much for the hints!
> > > > > >
> > > > > > I compiled 5.16 with the 1-byte file patch and have been running it
> > > > > > for a couple of hours now. I/O seems to have been gradually increasing
> > > > > > compared to 5.15, but I will wait for tomorrow to have a clearer view
> > > > > > on the graphs, then I'll try the both patches.
> > > > > >
> > > > > > François-Xavier
> > > > > >
> > > > > > On Mon, Jan 17, 2022 at 5:59 PM Filipe Manana <fdmanana@kernel.org> wrote:
> > > > > > >
> > > > > > > On Mon, Jan 17, 2022 at 12:02:08PM +0000, Filipe Manana wrote:
> > > > > > > > On Mon, Jan 17, 2022 at 11:06:42AM +0100, François-Xavier Thomas wrote:
> > > > > > > > > Hello all,
> > > > > > > > >
> > > > > > > > > Just in case someone is having the same issue: Btrfs (in the
> > > > > > > > > btrfs-cleaner process) is taking a large amount of disk IO after
> > > > > > > > > upgrading to 5.16 on one of my volumes, and multiple other people seem
> > > > > > > > > to be having the same issue, see discussion in [0].
> > > > > > > > >
> > > > > > > > > [1] is a close-up screenshot of disk I/O history (blue line is write
> > > > > > > > > ops, going from a baseline of some 10 ops/s to around 1k ops/s). I
> > > > > > > > > downgraded from 5.16 to 5.15 in the middle, which immediately restored
> > > > > > > > > previous performance.
> > > > > > > > >
> > > > > > > > > Common options between affected people are: ssd, autodefrag. No error
> > > > > > > > > in the logs, and no other issue aside from performance (the volume
> > > > > > > > > works just fine for accessing data).
> > > > > > > > >
> > > > > > > > > One person reports that SMART stats show a massive amount of blocks
> > > > > > > > > being written; unfortunately I do not have historical data for that so
> > > > > > > > > I cannot confirm, but this sounds likely given what I see on what
> > > > > > > > > should be a relatively new SSD.
> > > > > > > > >
> > > > > > > > > Any idea of what it could be related to?
> > > > > > > >
> > > > > > > > There was a big refactor of the defrag code that landed in 5.16.
> > > > > > > >
> > > > > > > > On a quick glance, when using autodefrag it seems we now can end up in an
> > > > > > > > infinite loop by marking the same range for degrag (IO) over and over.
> > > > > > > >
> > > > > > > > Can you try the following patch? (also at https://pastebin.com/raw/QR27Jv6n)
> > > > > > >
> > > > > > > Actually try this one instead:
> > > > > > >
> > > > > > > https://pastebin.com/raw/EbEfk1tF
> > > > > > >
> > > > > > > Also, there's a bug with defrag running into an (almost) infinite loop when
> > > > > > > attempting to defrag a 1 byte file. Someone ran into this and I've just sent
> > > > > > > a fix for it:
> > > > > > >
> > > > > > > https://patchwork.kernel.org/project/linux-btrfs/patch/bcbfce0ff7e21bbfed2484b1457e560edf78020d.1642436805.git.fdmanana@suse.com/
> > > > > > >
> > > > > > > Maybe that is what you are running into when using autodefrag.
> > > > > > > Firt try that fix for the 1 byte file case, and if after that you still run
> > > > > > > into problems, then try with the other patch above as well (both patches
> > > > > > > applied).
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > >
> > > > > > > > diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
> > > > > > > > index a5bd6926f7ff..0a9f6125a566 100644
> > > > > > > > --- a/fs/btrfs/ioctl.c
> > > > > > > > +++ b/fs/btrfs/ioctl.c
> > > > > > > > @@ -1213,6 +1213,13 @@ static int defrag_collect_targets(struct btrfs_inode *inode,
> > > > > > > >                 if (em->generation < newer_than)
> > > > > > > >                         goto next;
> > > > > > > >
> > > > > > > > +               /*
> > > > > > > > +                * Skip extents already under IO, otherwise we can end up in an
> > > > > > > > +                * infinite loop when using auto defrag.
> > > > > > > > +                */
> > > > > > > > +               if (em->generation == (u64)-1)
> > > > > > > > +                       goto next;
> > > > > > > > +
> > > > > > > >                 /*
> > > > > > > >                  * For do_compress case, we want to compress all valid file
> > > > > > > >                  * extents, thus no @extent_thresh or mergeable check.
> > > > > > > >
> > > > > > > >
> > > > > > > > >
> > > > > > > > > François-Xavier
> > > > > > > > >
> > > > > > > > > [0] https://www.reddit.com/r/btrfs/comments/s4nrzb/massive_performance_degradation_after_upgrading/
> > > > > > > > > [1] https://imgur.com/oYhYat1