Re: Massive I/O usage from btrfs-cleaner after upgrading to 5.16

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/1/22 03:39, François-Xavier Thomas wrote:
> Thanks, will add that to the list and test. FYI the 6 patches didn't
> seem to have much additional effect today compared to my previous
> stack of 4.

Good and bad news.

Good news is, I got my way to reproduce (at least part of) the problem.

With fsstress, a way to trigger autodefrag at will, and io accounting
for data/metadata read/write, it's clear the newer kernel is indeed
causing more IO.

v5.15 (or just revert the defrag code) causes around 8.7% of total data
IO for autodefrag.

While v5.16, even with the 6 patches, causes 18% of total data IO for
autodefrag.


Then bad news.

I have seen cases where v5.15 doesn't defrag ranges which is completely
sane to defrag.

Something like this:

         item 59 key (287 EXTENT_DATA 118784) itemoff 6211 itemsize 53
                 generation 85 type 1 (regular)
                 extent data disk byte 339296256 nr 8192
                 extent data offset 0 nr 8192 ram 8192
                 extent compression 0 (none)
         item 60 key (287 EXTENT_DATA 126976) itemoff 6158 itemsize 53
                 generation 85 type 1 (regular)
                 extent data disk byte 300445696 nr 4096
                 extent data offset 0 nr 4096 ram 4096
                 extent compression 0 (none)
         item 61 key (287 EXTENT_DATA 131072) itemoff 6105 itemsize 53
                 generation 85 type 1 (regular)
                 extent data disk byte 339304448 nr 4096
                 extent data offset 0 nr 4096 ram 4096
                 extent compression 0 (none)
         item 62 key (287 EXTENT_DATA 135168) itemoff 6052 itemsize 53
                 generation 85 type 1 (regular)
                 extent data disk byte 301170688 nr 4096
                 extent data offset 0 nr 4096 ram 4096
                 extent compression 0 (none)
         item 63 key (287 EXTENT_DATA 139264) itemoff 5999 itemsize 53
                 generation 85 type 1 (regular)
                 extent data disk byte 339308544 nr 106496
                 extent data offset 0 nr 106496 ram 106496
                 extent compression 0 (none)

This 124K range is definitely sane to defrag (and the newer_than
parameter is only 35, all extents are a good fit).

But older kernel by some reason (still under investigation) doesn't
choose to defrag at all, while newer kernel is pretty happy to defrag.

Although there are cases newer kernel is doing too small defrag which
doesn't make sense, with such cases fixed, it still results 15% of total
IO for autodefrag.

I'm afraid there may be some bugs or questionable behaviors in the old
defrag code that is not defragging all good candidates.

So even with more fixes, we may just end up with more IO for autodefrag,
purely because old code is not defragging as hard.

Thanks,
Qu
>
> On Fri, Jan 21, 2022 at 11:49 AM Filipe Manana <fdmanana@kernel.org> wrote:
>>
>> On Thu, Jan 20, 2022 at 6:21 PM François-Xavier Thomas
>> <fx.thomas@gmail.com> wrote:
>>>
>>>> Ok, so new patches to try
>>>
>>> Nice, thanks, I'll let you know how that goes tomorrow!
>>
>> You can also get more one on top of those 6:
>>
>> https://pastebin.com/raw/p87HX6AF
>>
>> Thanks.