Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo <quwenruo.btrfs@gmx.com>]

On 2022/2/5 03:34, Benjamin Xiao wrote:
> There's definitely still an issue even with the patches. I ended up
> disabling autodefrag again and rebooting my computer after btrfs-cleaner
> wrote about 300-ish GB to my SSD.
>
> The patch does help things out a bit compared to before where it was a
> constant non-stop stream of IO, but 300GB worth of extra writes for 33GB
> of actual data doesn't seem normal.

Have you tried v5.15.x with that diff?

Thanks,
Qu

>
> Ben
>
> On Fri, Feb 4 2022 at 09:36:22 AM -0800, Benjamin Xiao
> <ben.r.xiao@gmail.com> wrote:
>> Okay, I just tested right now with my custom 5.16.5 kernel with your 3
>> patches applied. Redownloading the same game, I noticed that there was
>> significantly less IO load during the download, which is great. It
>> looked kinda bursty instead, with periods of no load, and then periods
>> of load ranging in the 150-200MB/s range.
>>
>> After the download, I am no longer getting that constant 90-150MB/s
>> disk write from btrfs-cleaner, but I am seeing periodic bursts of it
>> every 10 seconds or so. These bursts last for around 3 seconds and
>> load is anywhere from 50-300MB/s.
>>
>> Is this normal? Does autodefrag only defrag newly written data, or
>> does it sometimes go back and run defrag on data written previously? I
>> am gonna let it run for a bit to see if it eventually subsides.
>>
>> Ben
>>
>> On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo
>> <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> On 2022/2/4 12:32, Benjamin Xiao wrote:
>>>> Okay, I just compiled a custom Arch kernel with your patches applied.
>>>> Will test soon. Besides enabling autodefrag and redownloading a game
>>>> from Steam, what other sorts of tests should I do?
>>>
>>> As long as your workload can trigger the problem reliably, nothing
>>> \x7felse.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Ben
>>>>
>>>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> Even after the defrag patches that landed in 5.16.5, I
>>>>>>> am still \x7f\x7f\x7f\x7f\x7fseeing
>>>>>>> lots of cpu usage and disk writes to my SSD when
>>>>>>> autodefrag is \x7f\x7f\x7f\x7f\x7fenabled.
>>>>>>> I kinda expected slightly more IO during writes
>>>>>>> compared to 5.15, \x7f\x7f\x7f\x7f\x7fbut
>>>>>>> what I am actually seeing is massive amounts of
>>>>>>> btrfs-cleaner i/o \x7f\x7f\x7f\x7f\x7feven
>>>>>>> when no programs are actively writing to the disk.
>>>>>>>
>>>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>>>> partition. In my case, I was downloading Strange
>>>>>>> Brigade, which \x7f\x7f\x7f\x7f\x7fis a
>>>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>>>> download, iostat will start reporting disk writes
>>>>>>> around 300 \x7f\x7f\x7f\x7f\x7fMB/s, even
>>>>>>> though Steam itself reports disk usage of 40-45MB/s.
>>>>>>> After the \x7f\x7f\x7f\x7f\x7fdownload
>>>>>>> finishes and nothing else is being written to disk, I
>>>>>>> still see \x7f\x7f\x7f\x7f\x7faround
>>>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I
>>>>>>> can see \x7f\x7f\x7f\x7f\x7fbtrfs
>>>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>>>
>>>>>>> I left it running for a while to see if it was just some
>>>>>>> \x7f\x7f\x7f\x7f\x7fmaintenance
>>>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>>>> reboot, but it actually prevented me from properly
>>>>>>> rebooting. \x7f\x7f\x7f\x7f\x7fAfter
>>>>>>> systemd timed out, my system finally shutdown.
>>>>>>>
>>>>>>> Here are my mount options:
>>>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Compression, I guess that's the reason.
>>>>>>
>>>>>>  From the very beginning, btrfs defrag doesn't handle
>>>>>> compressed \x7f\x7f\x7f\x7fextent
>>>>>> well.
>>>>>>
>>>>>> Even if a compressed extent is already at its maximum
>>>>>> capacity, \x7f\x7f\x7f\x7fbtrfs
>>>>>> will still try to defrag it.
>>>>>>
>>>>>> I believe the behavior is masked by other problems in older
>>>>>> \x7f\x7f\x7f\x7fkernels thus
>>>>>> not that obvious.
>>>>>>
>>>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>>>>
>>>>> And if possible, please try this diff on v5.15.x, and see if
>>>>> v5.15 \x7f\x7f\x7fis
>>>>> really doing less IO than v5.16.x.
>>>>>
>>>>> The diff will solve a problem in the old code, where autodefrag is
>>>>> almost not working.
>>>>>
>>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>>> index cc61813213d8..f6f2468d4883 100644
>>>>> --- a/fs/btrfs/ioctl.c
>>>>> +++ b/fs/btrfs/ioctl.c
>>>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode
>>>>> *inode, \x7f\x7f\x7fstruct
>>>>> file *file,
>>>>>                         continue;
>>>>>                 }
>>>>>
>>>>> -               if (!newer_than) {
>>>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>>>> -                                  PAGE_SHIFT) - i;
>>>>> -                       cluster = min(cluster, max_cluster);
>>>>> -               } else {
>>>>> -                       cluster = max_cluster;
>>>>> -               }
>>>>> +               cluster = (PAGE_ALIGN(defrag_end) >>
>>>>> PAGE_SHIFT) - \x7f\x7f\x7fi;
>>>>> +               cluster = min(cluster, max_cluster);
>>>>>
>>>>>                 if (i + cluster > ra_index) {
>>>>>                         ra_index = max(i, ra_index);
>>>>>
>>>>>>
>>>>>> There are patches to address the compression related
>>>>>> problem, but \x7f\x7f\x7f\x7fnot
>>>>>> yet merged:
>>>>>>
>>>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>>>
>>>>>> Mind to test them to see if that's the case?
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>> I've disabled autodefrag again for now to save my SSD,
>>>>>>> but just \x7f\x7f\x7f\x7f\x7fwanted
>>>>>>> to say that there is still an issue. Have the defrag
>>>>>>> issues been \x7f\x7f\x7f\x7f\x7ffully
>>>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>>>> Phoronix say? XD
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
>
>