Re: Massive I/O usage from btrfs-cleaner after upgrading to 5.16

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/1/20 20:02, François-Xavier Thomas wrote:
>> What if on top of those patches, you also add this one:
>> https://pastebin.com/raw/EbEfk1tF
>
> That's exactly patch 2 in my stack of patches in fact, is that the correct link?

Mind to share the full stack of patches or diffs?

I'd say for the known autodefrag, the following seems to solve the
problem for at least one reporter:

- btrfs: fix too long loop when defragging a 1 byte file

https://patchwork.kernel.org/project/linux-btrfs/patch/bcbfce0ff7e21bbfed2484b1457e560edf78020d.1642436805.git.fdmanana@suse.com/

- [v2] btrfs: defrag: fix the wrong number of defragged sectors

https://patchwork.kernel.org/project/linux-btrfs/patch/20220118071904.29991-1-wqu@suse.com/

- btrfs: defrag: properly update range->start for autodefrag

https://patchwork.kernel.org/project/linux-btrfs/patch/20220118115352.52126-1-wqu@suse.com/

Thanks,
Qu

>
> On Thu, Jan 20, 2022 at 12:45 PM Filipe Manana <fdmanana@kernel.org> wrote:
>>
>> On Thu, Jan 20, 2022 at 11:37 AM François-Xavier Thomas
>> <fx.thomas@gmail.com> wrote:
>>>
>>> Hi Felipe,
>>>
>>>> So, try with these two more patches on top of that:
>>>
>>> Thanks, I did just that, see graph with annotations:
>>> https://i.imgur.com/pu66nz0.png
>>>
>>> No visible improvement, average baseline I/O (for roughly similar
>>> workloads, the server I'm testing it on is not very busy I/O-wise) is
>>> still 3-4x higher in 5.16 than in 5.15 with autodefrag enabled.
>>
>> What if on top of those patches, you also add this one:
>>
>> https://pastebin.com/raw/EbEfk1tF
>>
>> Can you see if it helps?
>>
>>>
>>> The good news is that patch 2 did fix a large part of the issues 5.16.0 had.
>>> I also checked that disabling autodefrag immediately brings I/O rate
>>> back to how it was in 5.15.
>>
>> At least that!
>> Thanks.
>>
>>>
>>>>> Some people reported that 5.16.1 improved the situation for them, so
>>>> I don't see how that's possible, nothing was added to 5.16.1 that
>>>> involves defrag.
>>>> Might just be a coincidence.
>>>
>>> Yes, I found no evidence that official 5.16.1 is any better than the
>>> rest on my side.
>>>
>>> François-Xavier
>>>
>>> On Wed, Jan 19, 2022 at 11:14 AM Filipe Manana <fdmanana@kernel.org> wrote:
>>>>
>>>> On Wed, Jan 19, 2022 at 9:44 AM François-Xavier Thomas
>>>> <fx.thomas@gmail.com> wrote:
>>>>>
>>>>> Hi,
>>>>>
>>>>> More details on graph[0]:
>>>>> - First patch (1-byte file) on 5.16.0 did not have a significant impact.
>>>>> - Both patches on 5.16.0 did reduce a large part of the I/O but still
>>>>> have a high baseline I/O compared to 5.15
>>>>
>>>> So, try with these two more patches on top of that:
>>>>
>>>> https://patchwork.kernel.org/project/linux-btrfs/patch/20220118071904.29991-1-wqu@suse.com/
>>>>
>>>> https://patchwork.kernel.org/project/linux-btrfs/patch/20220118115352.52126-1-wqu@suse.com/
>>>>
>>>>>
>>>>> Some people reported that 5.16.1 improved the situation for them, so
>>>>
>>>> I don't see how that's possible, nothing was added to 5.16.1 that
>>>> involves defrag.
>>>> Might just be a coincidence.
>>>>
>>>> Thanks.
>>>>
>>>>> I'm testing that. It's too early to tell but for now the baseline I/O
>>>>> still seems to be high compared to 5.15. Will update with more results
>>>>> tomorrow.
>>>>>
>>>>> François-Xavier
>>>>>
>>>>> [0] https://i.imgur.com/agzAKGc.png
>>>>>
>>>>> On Mon, Jan 17, 2022 at 10:37 PM François-Xavier Thomas
>>>>> <fx.thomas@gmail.com> wrote:
>>>>>>
>>>>>> Hi Filipe,
>>>>>>
>>>>>> Thank you so much for the hints!
>>>>>>
>>>>>> I compiled 5.16 with the 1-byte file patch and have been running it
>>>>>> for a couple of hours now. I/O seems to have been gradually increasing
>>>>>> compared to 5.15, but I will wait for tomorrow to have a clearer view
>>>>>> on the graphs, then I'll try the both patches.
>>>>>>
>>>>>> François-Xavier
>>>>>>
>>>>>> On Mon, Jan 17, 2022 at 5:59 PM Filipe Manana <fdmanana@kernel.org> wrote:
>>>>>>>
>>>>>>> On Mon, Jan 17, 2022 at 12:02:08PM +0000, Filipe Manana wrote:
>>>>>>>> On Mon, Jan 17, 2022 at 11:06:42AM +0100, François-Xavier Thomas wrote:
>>>>>>>>> Hello all,
>>>>>>>>>
>>>>>>>>> Just in case someone is having the same issue: Btrfs (in the
>>>>>>>>> btrfs-cleaner process) is taking a large amount of disk IO after
>>>>>>>>> upgrading to 5.16 on one of my volumes, and multiple other people seem
>>>>>>>>> to be having the same issue, see discussion in [0].
>>>>>>>>>
>>>>>>>>> [1] is a close-up screenshot of disk I/O history (blue line is write
>>>>>>>>> ops, going from a baseline of some 10 ops/s to around 1k ops/s). I
>>>>>>>>> downgraded from 5.16 to 5.15 in the middle, which immediately restored
>>>>>>>>> previous performance.
>>>>>>>>>
>>>>>>>>> Common options between affected people are: ssd, autodefrag. No error
>>>>>>>>> in the logs, and no other issue aside from performance (the volume
>>>>>>>>> works just fine for accessing data).
>>>>>>>>>
>>>>>>>>> One person reports that SMART stats show a massive amount of blocks
>>>>>>>>> being written; unfortunately I do not have historical data for that so
>>>>>>>>> I cannot confirm, but this sounds likely given what I see on what
>>>>>>>>> should be a relatively new SSD.
>>>>>>>>>
>>>>>>>>> Any idea of what it could be related to?
>>>>>>>>
>>>>>>>> There was a big refactor of the defrag code that landed in 5.16.
>>>>>>>>
>>>>>>>> On a quick glance, when using autodefrag it seems we now can end up in an
>>>>>>>> infinite loop by marking the same range for degrag (IO) over and over.
>>>>>>>>
>>>>>>>> Can you try the following patch? (also at https://pastebin.com/raw/QR27Jv6n)
>>>>>>>
>>>>>>> Actually try this one instead:
>>>>>>>
>>>>>>> https://pastebin.com/raw/EbEfk1tF
>>>>>>>
>>>>>>> Also, there's a bug with defrag running into an (almost) infinite loop when
>>>>>>> attempting to defrag a 1 byte file. Someone ran into this and I've just sent
>>>>>>> a fix for it:
>>>>>>>
>>>>>>> https://patchwork.kernel.org/project/linux-btrfs/patch/bcbfce0ff7e21bbfed2484b1457e560edf78020d.1642436805.git.fdmanana@suse.com/
>>>>>>>
>>>>>>> Maybe that is what you are running into when using autodefrag.
>>>>>>> Firt try that fix for the 1 byte file case, and if after that you still run
>>>>>>> into problems, then try with the other patch above as well (both patches
>>>>>>> applied).
>>>>>>>
>>>>>>> Thanks.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>>
>>>>>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>>>>>> index a5bd6926f7ff..0a9f6125a566 100644
>>>>>>>> --- a/fs/btrfs/ioctl.c
>>>>>>>> +++ b/fs/btrfs/ioctl.c
>>>>>>>> @@ -1213,6 +1213,13 @@ static int defrag_collect_targets(struct btrfs_inode *inode,
>>>>>>>>                  if (em->generation < newer_than)
>>>>>>>>                          goto next;
>>>>>>>>
>>>>>>>> +               /*
>>>>>>>> +                * Skip extents already under IO, otherwise we can end up in an
>>>>>>>> +                * infinite loop when using auto defrag.
>>>>>>>> +                */
>>>>>>>> +               if (em->generation == (u64)-1)
>>>>>>>> +                       goto next;
>>>>>>>> +
>>>>>>>>                  /*
>>>>>>>>                   * For do_compress case, we want to compress all valid file
>>>>>>>>                   * extents, thus no @extent_thresh or mergeable check.
>>>>>>>>
>>>>>>>>
>>>>>>>>>
>>>>>>>>> François-Xavier
>>>>>>>>>
>>>>>>>>> [0] https://www.reddit.com/r/btrfs/comments/s4nrzb/massive_performance_degradation_after_upgrading/
>>>>>>>>> [1] https://imgur.com/oYhYat1