Still seeing high autodefrag IO with kernel 5.16.5

Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao]

Hello all,

Even after the defrag patches that landed in 5.16.5, I am still seeing 
lots of cpu usage and disk writes to my SSD when autodefrag is enabled. 
I kinda expected slightly more IO during writes compared to 5.15, but 
what I am actually seeing is massive amounts of btrfs-cleaner i/o even 
when no programs are actively writing to the disk.

I can reproduce it quite reliably on my 2TB Btrfs Steam library 
partition. In my case, I was downloading Strange Brigade, which is a 
roughly 25GB download and 33.65GB on disk. Somewhere during the 
download, iostat will start reporting disk writes around 300 MB/s, even 
though Steam itself reports disk usage of 40-45MB/s. After the download 
finishes and nothing else is being written to disk, I still see around 
90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs 
cleaner and other btrfs processes writing a lot of data.

I left it running for a while to see if it was just some maintenance 
tasks that Btrfs needed to do, but it just kept going. I tried to 
reboot, but it actually prevented me from properly rebooting. After 
systemd timed out, my system finally shutdown.

Here are my mount options:
rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/

I've disabled autodefrag again for now to save my SSD, but just wanted 
to say that there is still an issue. Have the defrag issues been fully 
fixed or are there more patches incoming despite what Reddit and 
Phoronix say? XD

Thanks!
Ben

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/4 04:05, Benjamin Xiao wrote:
> Hello all,
>
> Even after the defrag patches that landed in 5.16.5, I am still seeing
> lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
> I kinda expected slightly more IO during writes compared to 5.15, but
> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
> when no programs are actively writing to the disk.
>
> I can reproduce it quite reliably on my 2TB Btrfs Steam library
> partition. In my case, I was downloading Strange Brigade, which is a
> roughly 25GB download and 33.65GB on disk. Somewhere during the
> download, iostat will start reporting disk writes around 300 MB/s, even
> though Steam itself reports disk usage of 40-45MB/s. After the download
> finishes and nothing else is being written to disk, I still see around
> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
> cleaner and other btrfs processes writing a lot of data.
>
> I left it running for a while to see if it was just some maintenance
> tasks that Btrfs needed to do, but it just kept going. I tried to
> reboot, but it actually prevented me from properly rebooting. After
> systemd timed out, my system finally shutdown.
>
> Here are my mount options:
> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/

Compression, I guess that's the reason.

 From the very beginning, btrfs defrag doesn't handle compressed extent
well.

Even if a compressed extent is already at its maximum capacity, btrfs
will still try to defrag it.

I believe the behavior is masked by other problems in older kernels thus
not that obvious.

But after rework of defrag in v5.16, this behavior is more exposed.

There are patches to address the compression related problem, but not
yet merged:

https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387

Mind to test them to see if that's the case?

Thanks,
Qu
>
>
> I've disabled autodefrag again for now to save my SSD, but just wanted
> to say that there is still an issue. Have the defrag issues been fully
> fixed or are there more patches incoming despite what Reddit and
> Phoronix say? XD
>
> Thanks!
> Ben
>
>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/4 09:17, Qu Wenruo wrote:
>
>
> On 2022/2/4 04:05, Benjamin Xiao wrote:
>> Hello all,
>>
>> Even after the defrag patches that landed in 5.16.5, I am still seeing
>> lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
>> I kinda expected slightly more IO during writes compared to 5.15, but
>> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
>> when no programs are actively writing to the disk.
>>
>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>> partition. In my case, I was downloading Strange Brigade, which is a
>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>> download, iostat will start reporting disk writes around 300 MB/s, even
>> though Steam itself reports disk usage of 40-45MB/s. After the download
>> finishes and nothing else is being written to disk, I still see around
>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>> cleaner and other btrfs processes writing a lot of data.
>>
>> I left it running for a while to see if it was just some maintenance
>> tasks that Btrfs needed to do, but it just kept going. I tried to
>> reboot, but it actually prevented me from properly rebooting. After
>> systemd timed out, my system finally shutdown.
>>
>> Here are my mount options:
>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>
>
> Compression, I guess that's the reason.
>
>  From the very beginning, btrfs defrag doesn't handle compressed extent
> well.
>
> Even if a compressed extent is already at its maximum capacity, btrfs
> will still try to defrag it.
>
> I believe the behavior is masked by other problems in older kernels thus
> not that obvious.
>
> But after rework of defrag in v5.16, this behavior is more exposed.

And if possible, please try this diff on v5.15.x, and see if v5.15 is
really doing less IO than v5.16.x.

The diff will solve a problem in the old code, where autodefrag is
almost not working.

diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index cc61813213d8..f6f2468d4883 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
file *file,
                         continue;
                 }

-               if (!newer_than) {
-                       cluster = (PAGE_ALIGN(defrag_end) >>
-                                  PAGE_SHIFT) - i;
-                       cluster = min(cluster, max_cluster);
-               } else {
-                       cluster = max_cluster;
-               }
+               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
+               cluster = min(cluster, max_cluster);

                 if (i + cluster > ra_index) {
                         ra_index = max(i, ra_index);

>
> There are patches to address the compression related problem, but not
> yet merged:
>
> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>
> Mind to test them to see if that's the case?
>
> Thanks,
> Qu
>>
>>
>> I've disabled autodefrag again for now to save my SSD, but just wanted
>> to say that there is still an issue. Have the defrag issues been fully
>> fixed or are there more patches incoming despite what Reddit and
>> Phoronix say? XD
>>
>> Thanks!
>> Ben
>>
>>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao]

Okay, I just compiled a custom Arch kernel with your patches applied. 
Will test soon. Besides enabling autodefrag and redownloading a game 
from Steam, what other sorts of tests should I do?

Ben

On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo 
<quwenruo.btrfs@gmx.com> wrote:
> 
> 
> On 2022/2/4 09:17, Qu Wenruo wrote:
>> 
>> 
>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>> Hello all,
>>> 
>>> Even after the defrag patches that landed in 5.16.5, I am still 
>>> seeing
>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>> enabled.
>>> I kinda expected slightly more IO during writes compared to 5.15, 
>>> but
>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o 
>>> even
>>> when no programs are actively writing to the disk.
>>> 
>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>> partition. In my case, I was downloading Strange Brigade, which is a
>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>> download, iostat will start reporting disk writes around 300 MB/s, 
>>> even
>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>> download
>>> finishes and nothing else is being written to disk, I still see 
>>> around
>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>> cleaner and other btrfs processes writing a lot of data.
>>> 
>>> I left it running for a while to see if it was just some maintenance
>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>> reboot, but it actually prevented me from properly rebooting. After
>>> systemd timed out, my system finally shutdown.
>>> 
>>> Here are my mount options:
>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>> 
>> 
>> Compression, I guess that's the reason.
>> 
>>  From the very beginning, btrfs defrag doesn't handle compressed 
>> extent
>> well.
>> 
>> Even if a compressed extent is already at its maximum capacity, btrfs
>> will still try to defrag it.
>> 
>> I believe the behavior is masked by other problems in older kernels 
>> thus
>> not that obvious.
>> 
>> But after rework of defrag in v5.16, this behavior is more exposed.
> 
> And if possible, please try this diff on v5.15.x, and see if v5.15 is
> really doing less IO than v5.16.x.
> 
> The diff will solve a problem in the old code, where autodefrag is
> almost not working.
> 
> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
> index cc61813213d8..f6f2468d4883 100644
> --- a/fs/btrfs/ioctl.c
> +++ b/fs/btrfs/ioctl.c
> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, 
> struct
> file *file,
>                         continue;
>                 }
> 
> -               if (!newer_than) {
> -                       cluster = (PAGE_ALIGN(defrag_end) >>
> -                                  PAGE_SHIFT) - i;
> -                       cluster = min(cluster, max_cluster);
> -               } else {
> -                       cluster = max_cluster;
> -               }
> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
> +               cluster = min(cluster, max_cluster);
> 
>                 if (i + cluster > ra_index) {
>                         ra_index = max(i, ra_index);
> 
>> 
>> There are patches to address the compression related problem, but not
>> yet merged:
>> 
>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>> 
>> Mind to test them to see if that's the case?
>> 
>> Thanks,
>> Qu
>>> 
>>> 
>>> I've disabled autodefrag again for now to save my SSD, but just 
>>> wanted
>>> to say that there is still an issue. Have the defrag issues been 
>>> fully
>>> fixed or are there more patches incoming despite what Reddit and
>>> Phoronix say? XD
>>> 
>>> Thanks!
>>> Ben
>>> 
>>> 

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/4 12:32, Benjamin Xiao wrote:
> Okay, I just compiled a custom Arch kernel with your patches applied.
> Will test soon. Besides enabling autodefrag and redownloading a game
> from Steam, what other sorts of tests should I do?

As long as your workload can trigger the problem reliably, nothing else.

Thanks,
Qu

>
> Ben
>
> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
> <quwenruo.btrfs@gmx.com> wrote:
>>
>>
>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>> Hello all,
>>>>
>>>> Even after the defrag patches that landed in 5.16.5, I am still seeing
>>>> lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
>>>> I kinda expected slightly more IO during writes compared to 5.15, but
>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
>>>> when no programs are actively writing to the disk.
>>>>
>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>> partition. In my case, I was downloading Strange Brigade, which is a
>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>> download, iostat will start reporting disk writes around 300 MB/s, even
>>>> though Steam itself reports disk usage of 40-45MB/s. After the download
>>>> finishes and nothing else is being written to disk, I still see around
>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>>> cleaner and other btrfs processes writing a lot of data.
>>>>
>>>> I left it running for a while to see if it was just some maintenance
>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>> reboot, but it actually prevented me from properly rebooting. After
>>>> systemd timed out, my system finally shutdown.
>>>>
>>>> Here are my mount options:
>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>
>>>>
>>>
>>> Compression, I guess that's the reason.
>>>
>>>  From the very beginning, btrfs defrag doesn't handle compressed extent
>>> well.
>>>
>>> Even if a compressed extent is already at its maximum capacity, btrfs
>>> will still try to defrag it.
>>>
>>> I believe the behavior is masked by other problems in older kernels thus
>>> not that obvious.
>>>
>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>
>> And if possible, please try this diff on v5.15.x, and see if v5.15 is
>> really doing less IO than v5.16.x.
>>
>> The diff will solve a problem in the old code, where autodefrag is
>> almost not working.
>>
>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>> index cc61813213d8..f6f2468d4883 100644
>> --- a/fs/btrfs/ioctl.c
>> +++ b/fs/btrfs/ioctl.c
>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
>> file *file,
>>                         continue;
>>                 }
>>
>> -               if (!newer_than) {
>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>> -                                  PAGE_SHIFT) - i;
>> -                       cluster = min(cluster, max_cluster);
>> -               } else {
>> -                       cluster = max_cluster;
>> -               }
>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
>> +               cluster = min(cluster, max_cluster);
>>
>>                 if (i + cluster > ra_index) {
>>                         ra_index = max(i, ra_index);
>>
>>>
>>> There are patches to address the compression related problem, but not
>>> yet merged:
>>>
>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>
>>> Mind to test them to see if that's the case?
>>>
>>> Thanks,
>>> Qu
>>>>
>>>>
>>>> I've disabled autodefrag again for now to save my SSD, but just wanted
>>>> to say that there is still an issue. Have the defrag issues been fully
>>>> fixed or are there more patches incoming despite what Reddit and
>>>> Phoronix say? XD
>>>>
>>>> Thanks!
>>>> Ben
>>>>
>>>>
>
>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao]

Okay, I just tested right now with my custom 5.16.5 kernel with your 3 
patches applied. Redownloading the same game, I noticed that there was 
significantly less IO load during the download, which is great. It 
looked kinda bursty instead, with periods of no load, and then periods 
of load ranging in the 150-200MB/s range.

After the download, I am no longer getting that constant 90-150MB/s 
disk write from btrfs-cleaner, but I am seeing periodic bursts of it 
every 10 seconds or so. These bursts last for around 3 seconds and load 
is anywhere from 50-300MB/s.

Is this normal? Does autodefrag only defrag newly written data, or does 
it sometimes go back and run defrag on data written previously? I am 
gonna let it run for a bit to see if it eventually subsides.

Ben

On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo 
<quwenruo.btrfs@gmx.com> wrote:
> 
> 
> On 2022/2/4 12:32, Benjamin Xiao wrote:
>> Okay, I just compiled a custom Arch kernel with your patches applied.
>> Will test soon. Besides enabling autodefrag and redownloading a game
>> from Steam, what other sorts of tests should I do?
> 
> As long as your workload can trigger the problem reliably, nothing 
> else.
> 
> Thanks,
> Qu
> 
>> 
>> Ben
>> 
>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>> <quwenruo.btrfs@gmx.com> wrote:
>>> 
>>> 
>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>> 
>>>> 
>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>> Hello all,
>>>>> 
>>>>> Even after the defrag patches that landed in 5.16.5, I am still 
>>>>> seeing
>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>>>> enabled.
>>>>> I kinda expected slightly more IO during writes compared to 5.15, 
>>>>> but
>>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o 
>>>>> even
>>>>> when no programs are actively writing to the disk.
>>>>> 
>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>> partition. In my case, I was downloading Strange Brigade, which 
>>>>> is a
>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>> download, iostat will start reporting disk writes around 300 
>>>>> MB/s, even
>>>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>>>> download
>>>>> finishes and nothing else is being written to disk, I still see 
>>>>> around
>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see 
>>>>> btrfs
>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>> 
>>>>> I left it running for a while to see if it was just some 
>>>>> maintenance
>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>> reboot, but it actually prevented me from properly rebooting. 
>>>>> After
>>>>> systemd timed out, my system finally shutdown.
>>>>> 
>>>>> Here are my mount options:
>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>> 
>>>>> 
>>>> 
>>>> Compression, I guess that's the reason.
>>>> 
>>>>  From the very beginning, btrfs defrag doesn't handle compressed 
>>>> extent
>>>> well.
>>>> 
>>>> Even if a compressed extent is already at its maximum capacity, 
>>>> btrfs
>>>> will still try to defrag it.
>>>> 
>>>> I believe the behavior is masked by other problems in older 
>>>> kernels thus
>>>> not that obvious.
>>>> 
>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>> 
>>> And if possible, please try this diff on v5.15.x, and see if v5.15 
>>> is
>>> really doing less IO than v5.16.x.
>>> 
>>> The diff will solve a problem in the old code, where autodefrag is
>>> almost not working.
>>> 
>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>> index cc61813213d8..f6f2468d4883 100644
>>> --- a/fs/btrfs/ioctl.c
>>> +++ b/fs/btrfs/ioctl.c
>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, 
>>> struct
>>> file *file,
>>>                         continue;
>>>                 }
>>> 
>>> -               if (!newer_than) {
>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>> -                                  PAGE_SHIFT) - i;
>>> -                       cluster = min(cluster, max_cluster);
>>> -               } else {
>>> -                       cluster = max_cluster;
>>> -               }
>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - 
>>> i;
>>> +               cluster = min(cluster, max_cluster);
>>> 
>>>                 if (i + cluster > ra_index) {
>>>                         ra_index = max(i, ra_index);
>>> 
>>>> 
>>>> There are patches to address the compression related problem, but 
>>>> not
>>>> yet merged:
>>>> 
>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>> 
>>>> Mind to test them to see if that's the case?
>>>> 
>>>> Thanks,
>>>> Qu
>>>>> 
>>>>> 
>>>>> I've disabled autodefrag again for now to save my SSD, but just 
>>>>> wanted
>>>>> to say that there is still an issue. Have the defrag issues been 
>>>>> fully
>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>> Phoronix say? XD
>>>>> 
>>>>> Thanks!
>>>>> Ben
>>>>> 
>>>>> 
>> 
>> 

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Benjamin Xiao]

There's definitely still an issue even with the patches. I ended up 
disabling autodefrag again and rebooting my computer after 
btrfs-cleaner wrote about 300-ish GB to my SSD.

The patch does help things out a bit compared to before where it was a 
constant non-stop stream of IO, but 300GB worth of extra writes for 
33GB of actual data doesn't seem normal.

Ben

On Fri, Feb 4 2022 at 09:36:22 AM -0800, Benjamin Xiao 
<ben.r.xiao@gmail.com> wrote:
> Okay, I just tested right now with my custom 5.16.5 kernel with your 
> 3 patches applied. Redownloading the same game, I noticed that there 
> was significantly less IO load during the download, which is great. 
> It looked kinda bursty instead, with periods of no load, and then 
> periods of load ranging in the 150-200MB/s range.
> 
> After the download, I am no longer getting that constant 90-150MB/s 
> disk write from btrfs-cleaner, but I am seeing periodic bursts of it 
> every 10 seconds or so. These bursts last for around 3 seconds and 
> load is anywhere from 50-300MB/s.
> 
> Is this normal? Does autodefrag only defrag newly written data, or 
> does it sometimes go back and run defrag on data written previously? 
> I am gonna let it run for a bit to see if it eventually subsides.
> 
> Ben
> 
> On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo 
> <quwenruo.btrfs@gmx.com> wrote:
>> 
>> 
>> On 2022/2/4 12:32, Benjamin Xiao wrote:
>>> Okay, I just compiled a custom Arch kernel with your patches 
>>> applied.
>>> Will test soon. Besides enabling autodefrag and redownloading a game
>>> from Steam, what other sorts of tests should I do?
>> 
>> As long as your workload can trigger the problem reliably, nothing 
>> \x7felse.
>> 
>> Thanks,
>> Qu
>> 
>>> 
>>> Ben
>>> 
>>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>>> <quwenruo.btrfs@gmx.com> wrote:
>>>> 
>>>> 
>>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>> 
>>>>> 
>>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>>> Hello all,
>>>>>> 
>>>>>> Even after the defrag patches that landed in 5.16.5, I am still 
>>>>>> \x7f\x7f\x7f\x7f\x7fseeing
>>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>>>>> \x7f\x7f\x7f\x7f\x7fenabled.
>>>>>> I kinda expected slightly more IO during writes compared to 
>>>>>> 5.15, \x7f\x7f\x7f\x7f\x7fbut
>>>>>> what I am actually seeing is massive amounts of btrfs-cleaner 
>>>>>> i/o \x7f\x7f\x7f\x7f\x7feven
>>>>>> when no programs are actively writing to the disk.
>>>>>> 
>>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>>> partition. In my case, I was downloading Strange Brigade, which 
>>>>>> \x7f\x7f\x7f\x7f\x7fis a
>>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>>> download, iostat will start reporting disk writes around 300 
>>>>>> \x7f\x7f\x7f\x7f\x7fMB/s, even
>>>>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>>>>> \x7f\x7f\x7f\x7f\x7fdownload
>>>>>> finishes and nothing else is being written to disk, I still see 
>>>>>> \x7f\x7f\x7f\x7f\x7faround
>>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see 
>>>>>> \x7f\x7f\x7f\x7f\x7fbtrfs
>>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>> 
>>>>>> I left it running for a while to see if it was just some 
>>>>>> \x7f\x7f\x7f\x7f\x7fmaintenance
>>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>>> reboot, but it actually prevented me from properly rebooting. 
>>>>>> \x7f\x7f\x7f\x7f\x7fAfter
>>>>>> systemd timed out, my system finally shutdown.
>>>>>> 
>>>>>> Here are my mount options:
>>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>> 
>>>>>> 
>>>>> 
>>>>> Compression, I guess that's the reason.
>>>>> 
>>>>>  From the very beginning, btrfs defrag doesn't handle compressed 
>>>>> \x7f\x7f\x7f\x7fextent
>>>>> well.
>>>>> 
>>>>> Even if a compressed extent is already at its maximum capacity, 
>>>>> \x7f\x7f\x7f\x7fbtrfs
>>>>> will still try to defrag it.
>>>>> 
>>>>> I believe the behavior is masked by other problems in older 
>>>>> \x7f\x7f\x7f\x7fkernels thus
>>>>> not that obvious.
>>>>> 
>>>>> But after rework of defrag in v5.16, this behavior is more 
>>>>> exposed.
>>>> 
>>>> And if possible, please try this diff on v5.15.x, and see if v5.15 
>>>> \x7f\x7f\x7fis
>>>> really doing less IO than v5.16.x.
>>>> 
>>>> The diff will solve a problem in the old code, where autodefrag is
>>>> almost not working.
>>>> 
>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>> index cc61813213d8..f6f2468d4883 100644
>>>> --- a/fs/btrfs/ioctl.c
>>>> +++ b/fs/btrfs/ioctl.c
>>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, 
>>>> \x7f\x7f\x7fstruct
>>>> file *file,
>>>>                         continue;
>>>>                 }
>>>> 
>>>> -               if (!newer_than) {
>>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>>> -                                  PAGE_SHIFT) - i;
>>>> -                       cluster = min(cluster, max_cluster);
>>>> -               } else {
>>>> -                       cluster = max_cluster;
>>>> -               }
>>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - 
>>>> \x7f\x7f\x7fi;
>>>> +               cluster = min(cluster, max_cluster);
>>>> 
>>>>                 if (i + cluster > ra_index) {
>>>>                         ra_index = max(i, ra_index);
>>>> 
>>>>> 
>>>>> There are patches to address the compression related problem, but 
>>>>> \x7f\x7f\x7f\x7fnot
>>>>> yet merged:
>>>>> 
>>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>> 
>>>>> Mind to test them to see if that's the case?
>>>>> 
>>>>> Thanks,
>>>>> Qu
>>>>>> 
>>>>>> 
>>>>>> I've disabled autodefrag again for now to save my SSD, but just 
>>>>>> \x7f\x7f\x7f\x7f\x7fwanted
>>>>>> to say that there is still an issue. Have the defrag issues been 
>>>>>> \x7f\x7f\x7f\x7f\x7ffully
>>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>>> Phoronix say? XD
>>>>>> 
>>>>>> Thanks!
>>>>>> Ben
>>>>>> 
>>>>>> 
>>> 
>>> 
> 
> 

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/5 03:34, Benjamin Xiao wrote:
> There's definitely still an issue even with the patches. I ended up
> disabling autodefrag again and rebooting my computer after btrfs-cleaner
> wrote about 300-ish GB to my SSD.
>
> The patch does help things out a bit compared to before where it was a
> constant non-stop stream of IO, but 300GB worth of extra writes for 33GB
> of actual data doesn't seem normal.

Have you tried v5.15.x with that diff?

Thanks,
Qu

>
> Ben
>
> On Fri, Feb 4 2022 at 09:36:22 AM -0800, Benjamin Xiao
> <ben.r.xiao@gmail.com> wrote:
>> Okay, I just tested right now with my custom 5.16.5 kernel with your 3
>> patches applied. Redownloading the same game, I noticed that there was
>> significantly less IO load during the download, which is great. It
>> looked kinda bursty instead, with periods of no load, and then periods
>> of load ranging in the 150-200MB/s range.
>>
>> After the download, I am no longer getting that constant 90-150MB/s
>> disk write from btrfs-cleaner, but I am seeing periodic bursts of it
>> every 10 seconds or so. These bursts last for around 3 seconds and
>> load is anywhere from 50-300MB/s.
>>
>> Is this normal? Does autodefrag only defrag newly written data, or
>> does it sometimes go back and run defrag on data written previously? I
>> am gonna let it run for a bit to see if it eventually subsides.
>>
>> Ben
>>
>> On Fri, Feb 4 2022 at 02:20:44 PM +0800, Qu Wenruo
>> <quwenruo.btrfs@gmx.com> wrote:
>>>
>>>
>>> On 2022/2/4 12:32, Benjamin Xiao wrote:
>>>> Okay, I just compiled a custom Arch kernel with your patches applied.
>>>> Will test soon. Besides enabling autodefrag and redownloading a game
>>>> from Steam, what other sorts of tests should I do?
>>>
>>> As long as your workload can trigger the problem reliably, nothing
>>> \x7felse.
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> Ben
>>>>
>>>> On Fri, Feb 4 2022 at 09:54:19 AM +0800, Qu Wenruo
>>>> <quwenruo.btrfs@gmx.com> wrote:
>>>>>
>>>>>
>>>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> Even after the defrag patches that landed in 5.16.5, I
>>>>>>> am still \x7f\x7f\x7f\x7f\x7fseeing
>>>>>>> lots of cpu usage and disk writes to my SSD when
>>>>>>> autodefrag is \x7f\x7f\x7f\x7f\x7fenabled.
>>>>>>> I kinda expected slightly more IO during writes
>>>>>>> compared to 5.15, \x7f\x7f\x7f\x7f\x7fbut
>>>>>>> what I am actually seeing is massive amounts of
>>>>>>> btrfs-cleaner i/o \x7f\x7f\x7f\x7f\x7feven
>>>>>>> when no programs are actively writing to the disk.
>>>>>>>
>>>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>>>> partition. In my case, I was downloading Strange
>>>>>>> Brigade, which \x7f\x7f\x7f\x7f\x7fis a
>>>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>>>> download, iostat will start reporting disk writes
>>>>>>> around 300 \x7f\x7f\x7f\x7f\x7fMB/s, even
>>>>>>> though Steam itself reports disk usage of 40-45MB/s.
>>>>>>> After the \x7f\x7f\x7f\x7f\x7fdownload
>>>>>>> finishes and nothing else is being written to disk, I
>>>>>>> still see \x7f\x7f\x7f\x7f\x7faround
>>>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I
>>>>>>> can see \x7f\x7f\x7f\x7f\x7fbtrfs
>>>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>>>
>>>>>>> I left it running for a while to see if it was just some
>>>>>>> \x7f\x7f\x7f\x7f\x7fmaintenance
>>>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>>>> reboot, but it actually prevented me from properly
>>>>>>> rebooting. \x7f\x7f\x7f\x7f\x7fAfter
>>>>>>> systemd timed out, my system finally shutdown.
>>>>>>>
>>>>>>> Here are my mount options:
>>>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Compression, I guess that's the reason.
>>>>>>
>>>>>>  From the very beginning, btrfs defrag doesn't handle
>>>>>> compressed \x7f\x7f\x7f\x7fextent
>>>>>> well.
>>>>>>
>>>>>> Even if a compressed extent is already at its maximum
>>>>>> capacity, \x7f\x7f\x7f\x7fbtrfs
>>>>>> will still try to defrag it.
>>>>>>
>>>>>> I believe the behavior is masked by other problems in older
>>>>>> \x7f\x7f\x7f\x7fkernels thus
>>>>>> not that obvious.
>>>>>>
>>>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>>>>
>>>>> And if possible, please try this diff on v5.15.x, and see if
>>>>> v5.15 \x7f\x7f\x7fis
>>>>> really doing less IO than v5.16.x.
>>>>>
>>>>> The diff will solve a problem in the old code, where autodefrag is
>>>>> almost not working.
>>>>>
>>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>>> index cc61813213d8..f6f2468d4883 100644
>>>>> --- a/fs/btrfs/ioctl.c
>>>>> +++ b/fs/btrfs/ioctl.c
>>>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode
>>>>> *inode, \x7f\x7f\x7fstruct
>>>>> file *file,
>>>>>                         continue;
>>>>>                 }
>>>>>
>>>>> -               if (!newer_than) {
>>>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>>>> -                                  PAGE_SHIFT) - i;
>>>>> -                       cluster = min(cluster, max_cluster);
>>>>> -               } else {
>>>>> -                       cluster = max_cluster;
>>>>> -               }
>>>>> +               cluster = (PAGE_ALIGN(defrag_end) >>
>>>>> PAGE_SHIFT) - \x7f\x7f\x7fi;
>>>>> +               cluster = min(cluster, max_cluster);
>>>>>
>>>>>                 if (i + cluster > ra_index) {
>>>>>                         ra_index = max(i, ra_index);
>>>>>
>>>>>>
>>>>>> There are patches to address the compression related
>>>>>> problem, but \x7f\x7f\x7f\x7fnot
>>>>>> yet merged:
>>>>>>
>>>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>>>
>>>>>> Mind to test them to see if that's the case?
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>> I've disabled autodefrag again for now to save my SSD,
>>>>>>> but just \x7f\x7f\x7f\x7f\x7fwanted
>>>>>>> to say that there is still an issue. Have the defrag
>>>>>>> issues been \x7f\x7f\x7f\x7f\x7ffully
>>>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>>>> Phoronix say? XD
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Ben
>>>>>>>
>>>>>>>
>>>>
>>>>
>>
>>
>
>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/6 15:51, Rylee Randall wrote:
> I am experiencing the same issue on 5.16.7, near the end of a large
> steam game download btrfs-cleaner sits at the top of iotop, and shut
> downs take about ten minutes because of various btrfs hangs.
>
> I compiled 5.15.21 with the mentioned patch and tried to recreate the
> issue and so far have been unable to. I seem to get far faster an dmore
> consistent write speeds from steam, and rather than btrfs-cleaner being
> the main source of io usage it is steam. btrfs-cleaner is far down the
> list along with various other btrfs- tasks.

Thanks for the report, this indeed looks like the bug in v5.15 that it
doesn't defrag a lot of extents is not the root cause.

Mind to re-check with the following branch?

https://github.com/adam900710/linux/tree/autodefrag_fixes

It has one extra patch to emulate the older behavior of not using
btrfs_get_em(), which can cause quite some problem for autodefrag.

Thanks,
Qu

>
> On 4/2/22 12:54, Qu Wenruo wrote:
>>
>>
>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>> Hello all,
>>>>
>>>> Even after the defrag patches that landed in 5.16.5, I am still seeing
>>>> lots of cpu usage and disk writes to my SSD when autodefrag is enabled.
>>>> I kinda expected slightly more IO during writes compared to 5.15, but
>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
>>>> when no programs are actively writing to the disk.
>>>>
>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>> partition. In my case, I was downloading Strange Brigade, which is a
>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>> download, iostat will start reporting disk writes around 300 MB/s, even
>>>> though Steam itself reports disk usage of 40-45MB/s. After the download
>>>> finishes and nothing else is being written to disk, I still see around
>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>>> cleaner and other btrfs processes writing a lot of data.
>>>>
>>>> I left it running for a while to see if it was just some maintenance
>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>> reboot, but it actually prevented me from properly rebooting. After
>>>> systemd timed out, my system finally shutdown.
>>>>
>>>> Here are my mount options:
>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>
>>>>
>>>
>>> Compression, I guess that's the reason.
>>>
>>>  From the very beginning, btrfs defrag doesn't handle compressed extent
>>> well.
>>>
>>> Even if a compressed extent is already at its maximum capacity, btrfs
>>> will still try to defrag it.
>>>
>>> I believe the behavior is masked by other problems in older kernels thus
>>> not that obvious.
>>>
>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>
>> And if possible, please try this diff on v5.15.x, and see if v5.15 is
>> really doing less IO than v5.16.x.
>>
>> The diff will solve a problem in the old code, where autodefrag is
>> almost not working.
>>
>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>> index cc61813213d8..f6f2468d4883 100644
>> --- a/fs/btrfs/ioctl.c
>> +++ b/fs/btrfs/ioctl.c
>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
>> file *file,
>>                         continue;
>>                 }
>>
>> -               if (!newer_than) {
>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>> -                                  PAGE_SHIFT) - i;
>> -                       cluster = min(cluster, max_cluster);
>> -               } else {
>> -                       cluster = max_cluster;
>> -               }
>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
>> +               cluster = min(cluster, max_cluster);
>>
>>                 if (i + cluster > ra_index) {
>>                         ra_index = max(i, ra_index);
>>
>>>
>>> There are patches to address the compression related problem, but not
>>> yet merged:
>>>
>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>
>>> Mind to test them to see if that's the case?
>>>
>>> Thanks,
>>> Qu
>>>>
>>>>
>>>> I've disabled autodefrag again for now to save my SSD, but just wanted
>>>> to say that there is still an issue. Have the defrag issues been fully
>>>> fixed or are there more patches incoming despite what Reddit and
>>>> Phoronix say? XD
>>>>
>>>> Thanks!
>>>> Ben
>>>>
>>>>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Jean-Denis Girard]

Hi list,

I'm also experiencing high IO with autodefrag on 5.16.6 patched with:
  - https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
  - https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
  - btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.

After about 24 hours uptime, btrfs-cleaner is most of the time at 90 % 
CPU usage

%CPU  %MEM     TIME+ COMMAND
77,6   0,0  29:03.25 btrfs-cleaner
19,1   0,0   3:40.22 kworker/u24:8-btrfs-endio-write
19,1   0,0   1:50.46 kworker/u24:27-btrfs-endio-write
18,4   0,0   4:14.82 kworker/u24:7-btrfs-endio-write
17,4   0,0   4:08.08 kworker/u24:10-events_unbound
17,4   0,0   4:23.41 kworker/u24:9-btrfs-delalloc
15,8   0,0   3:41.21 kworker/u24:3-btrfs-endio-write
15,1   0,0   2:12.61 kworker/u24:26-blkcg_punt_bio
14,8   0,0   3:40.13 kworker/u24:18-btrfs-delalloc
12,8   0,0   3:55.70 kworker/u24:20-btrfs-delalloc
12,5   0,0   4:01.79 kworker/u24:1-btrfs-delalloc
12,2   0,0   4:04.63 kworker/u24:22-btrfs-endio-write
11,5   0,0   1:11.90 kworker/u24:11-btrfs-endio-write
11,2   0,0   1:25.96 kworker/u24:0-blkcg_punt_bio
10,2   0,0   4:24.17 kworker/u24:24+events_unbound
...

Load average is 5,56, 6,88, 6,71 while just writing that mail.

And iostat shows:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
            1,5%    0,0%   26,8%    6,3%    0,0%   65,4%

       tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn 
   kB_dscd Device
  33893,50         0,0k       172,2M        57,8M       0,0k       1,7G 
    577,7M nvme0n1
    169,40         0,0k         1,2M         0,0k       0,0k      12,5M 
      0,0k sda
    178,90         0,0k         1,3M         0,0k       0,0k      12,5M 
      0,0k sdb

I hope that helps.


Thanks,
-- 
Jean-Denis Girard

SysNux                   Systèmes   Linux   en   Polynésie  française
https://www.sysnux.pf/   Tél: +689 40.50.10.40 / GSM: +689 87.797.527


Le 05/02/2022 à 23:26, Qu Wenruo a écrit :
> 
> 
> On 2022/2/6 15:51, Rylee Randall wrote:
>> I am experiencing the same issue on 5.16.7, near the end of a large
>> steam game download btrfs-cleaner sits at the top of iotop, and shut
>> downs take about ten minutes because of various btrfs hangs.
>>
>> I compiled 5.15.21 with the mentioned patch and tried to recreate the
>> issue and so far have been unable to. I seem to get far faster an dmore
>> consistent write speeds from steam, and rather than btrfs-cleaner being
>> the main source of io usage it is steam. btrfs-cleaner is far down the
>> list along with various other btrfs- tasks.
> 
> Thanks for the report, this indeed looks like the bug in v5.15 that it
> doesn't defrag a lot of extents is not the root cause.
> 
> Mind to re-check with the following branch?
> 
> https://github.com/adam900710/linux/tree/autodefrag_fixes
> 
> It has one extra patch to emulate the older behavior of not using
> btrfs_get_em(), which can cause quite some problem for autodefrag.
> 
> Thanks,
> Qu
> 
>>
>> On 4/2/22 12:54, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>> Hello all,
>>>>>
>>>>> Even after the defrag patches that landed in 5.16.5, I am still seeing
>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is 
>>>>> enabled.
>>>>> I kinda expected slightly more IO during writes compared to 5.15, but
>>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
>>>>> when no programs are actively writing to the disk.
>>>>>
>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>> partition. In my case, I was downloading Strange Brigade, which is a
>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>> download, iostat will start reporting disk writes around 300 MB/s, 
>>>>> even
>>>>> though Steam itself reports disk usage of 40-45MB/s. After the 
>>>>> download
>>>>> finishes and nothing else is being written to disk, I still see around
>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>
>>>>> I left it running for a while to see if it was just some maintenance
>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>> reboot, but it actually prevented me from properly rebooting. After
>>>>> systemd timed out, my system finally shutdown.
>>>>>
>>>>> Here are my mount options:
>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/ 
>>>>>
>>>>>
>>>>>
>>>>
>>>> Compression, I guess that's the reason.
>>>>
>>>>  From the very beginning, btrfs defrag doesn't handle compressed extent
>>>> well.
>>>>
>>>> Even if a compressed extent is already at its maximum capacity, btrfs
>>>> will still try to defrag it.
>>>>
>>>> I believe the behavior is masked by other problems in older kernels 
>>>> thus
>>>> not that obvious.
>>>>
>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>>
>>> And if possible, please try this diff on v5.15.x, and see if v5.15 is
>>> really doing less IO than v5.16.x.
>>>
>>> The diff will solve a problem in the old code, where autodefrag is
>>> almost not working.
>>>
>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>> index cc61813213d8..f6f2468d4883 100644
>>> --- a/fs/btrfs/ioctl.c
>>> +++ b/fs/btrfs/ioctl.c
>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
>>> file *file,
>>>                         continue;
>>>                 }
>>>
>>> -               if (!newer_than) {
>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>> -                                  PAGE_SHIFT) - i;
>>> -                       cluster = min(cluster, max_cluster);
>>> -               } else {
>>> -                       cluster = max_cluster;
>>> -               }
>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
>>> +               cluster = min(cluster, max_cluster);
>>>
>>>                 if (i + cluster > ra_index) {
>>>                         ra_index = max(i, ra_index);
>>>
>>>>
>>>> There are patches to address the compression related problem, but not
>>>> yet merged:
>>>>
>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>
>>>> Mind to test them to see if that's the case?
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>>
>>>>> I've disabled autodefrag again for now to save my SSD, but just wanted
>>>>> to say that there is still an issue. Have the defrag issues been fully
>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>> Phoronix say? XD
>>>>>
>>>>> Thanks!
>>>>> Ben
>>>>>
>>>>>
> 

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 4663 bytes --]



On 2022/2/7 01:43, Jean-Denis Girard wrote:
> Hi list,
>
> I'm also experiencing high IO with autodefrag on 5.16.6 patched
> with: -
> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
>
>
- https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
> - btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.

OK, btrfs_search_forward() doesn't help in this case I guess.

>
> After about 24 hours uptime, btrfs-cleaner is most of the time at 90
> % CPU usage
>
> %CPU  %MEM     TIME+ COMMAND 77,6   0,0  29:03.25 btrfs-cleaner 19,1
> 0,0   3:40.22 kworker/u24:8-btrfs-endio-write 19,1   0,0   1:50.46
> kworker/u24:27-btrfs-endio-write 18,4   0,0   4:14.82
> kworker/u24:7-btrfs-endio-write 17,4   0,0   4:08.08
> kworker/u24:10-events_unbound 17,4   0,0   4:23.41
> kworker/u24:9-btrfs-delalloc 15,8   0,0   3:41.21
> kworker/u24:3-btrfs-endio-write 15,1   0,0   2:12.61
> kworker/u24:26-blkcg_punt_bio 14,8   0,0   3:40.13
> kworker/u24:18-btrfs-delalloc 12,8   0,0   3:55.70
> kworker/u24:20-btrfs-delalloc 12,5   0,0   4:01.79
> kworker/u24:1-btrfs-delalloc 12,2   0,0   4:04.63
> kworker/u24:22-btrfs-endio-write 11,5   0,0   1:11.90
> kworker/u24:11-btrfs-endio-write 11,2   0,0   1:25.96
> kworker/u24:0-blkcg_punt_bio 10,2   0,0   4:24.17
> kworker/u24:24+events_unbound ...
>
> Load average is 5,56, 6,88, 6,71 while just writing that mail.
>
> And iostat shows:
>
> avg-cpu:  %user   %nice %system %iowait  %steal   %idle 1,5%    0,0%
> 26,8%    6,3%    0,0%   65,4%
>
> tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn
> kB_dscd Device 33893,50         0,0k       172,2M        57,8M
> 0,0k       1,7G 577,7M nvme0n1 169,40         0,0k         1,2M
> 0,0k       0,0k      12,5M 0,0k sda 178,90         0,0k         1,3M
> 0,0k       0,0k      12,5M 0,0k sdb
>
> I hope that helps.

And what are the mount options?

 From the CPU usage, it looks like there are some infinite loop, but I
can't find out why by a quick glance.

And since there is no extra read, it looks like it's doing defrag again
and again on the same inode and range.


Mind to apply attached `debug.diff`?

Then you still need to provide `/sys/kernel/debug/tracing/trace`, which
can be very large and full of added debug info.
(This means this can slow down your system)

The example of my local test would result something like this in above
trace file:

# tracer: nop
#
# entries-in-buffer/entries-written: 11/11   #P:16
#
#                                _-----=> irqs-off/BH-disabled
#                               / _----=> need-resched
#                              | / _---=> hardirq/softirq
#                              || / _--=> preempt-depth
#                              ||| / _-=> migrate-disable
#                              |||| /     delay
#           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
#              | |         |   |||||     |         |
    btrfs-cleaner-1770    [014] .....   961.534329: btrfs_defrag_file:
entry r/i=5/257 start=0 len=131072 newer=7 thres=262144
    btrfs-cleaner-1770    [014] .....   961.534333:
defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0
len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534335:
defrag_collect_targets: add target r/i=5/257 start=16384 len=16384
em=16384 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534336:
defrag_collect_targets: add target r/i=5/257 start=32768 len=16384
em=32768 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534337:
defrag_collect_targets: add target r/i=5/257 start=49152 len=16384
em=49152 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534564:
defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0
len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534567:
defrag_collect_targets: add target r/i=5/257 start=16384 len=16384
em=16384 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534568:
defrag_collect_targets: add target r/i=5/257 start=32768 len=16384
em=32768 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534569:
defrag_collect_targets: add target r/i=5/257 start=49152 len=16384
em=49152 len=16384 gen=7 newer=7
    btrfs-cleaner-1770    [014] .....   961.534580: btrfs_defrag_file:
defrag target r/i=5/257 start=0 len=65536
    btrfs-cleaner-1770    [014] .....   961.534681:
btrfs_run_defrag_inodes: defrag finish r/i=5/257 ret=0 defragged=16
last_scanned=131072 last_off=0 cycled=0

Thanks,
Qu


>
>
> Thanks,

[-- Attachment #2: debug.diff --]
[-- Type: text/x-patch, Size: 2457 bytes --]

diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index f5de8ab9787e..be2e20d163ef 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -304,6 +304,11 @@ static int __btrfs_run_defrag_inode(struct btrfs_fs_info *fs_info,
 	sb_start_write(fs_info->sb);
 	ret = btrfs_defrag_file(inode, NULL, &ctrl);
 	sb_end_write(fs_info->sb);
+	trace_printk("defrag finish r/i=%lld/%llu ret=%d defragged=%llu last_scanned=%llu last_off=%llu cycled=%d\n",
+		     BTRFS_I(inode)->root->root_key.objectid,
+		     btrfs_ino(BTRFS_I(inode)), ret, ctrl.sectors_defragged,
+		     ctrl.last_scanned,
+		     defrag->last_offset, defrag->cycled);
 	/*
 	 * if we filled the whole defrag batch, there
 	 * must be more work to do.  Queue this defrag
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 133e3e2e2e79..a0501c88922b 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -1420,6 +1420,11 @@ static int defrag_collect_targets(struct btrfs_inode *inode,
 add:
 		last_is_target = true;
 		range_len = min(extent_map_end(em), start + len) - cur;
+
+		trace_printk("add target r/i=%lld/%llu start=%llu len=%llu em=%llu len=%llu gen=%lld newer=%llu\n",
+			inode->root->root_key.objectid,
+			btrfs_ino(inode), cur, range_len, em->start, em->len,
+			em->generation, ctrl->newer_than);
 		/*
 		 * This one is a good target, check if it can be merged into
 		 * last range of the target list.
@@ -1520,6 +1525,9 @@ static int defrag_one_locked_target(struct btrfs_inode *inode,
 			 EXTENT_DEFRAG, 0, 0, cached_state);
 	set_extent_defrag(&inode->io_tree, start, start + len - 1, cached_state);
 
+	trace_printk("defrag target r/i=%lld/%llu start=%llu len=%llu\n",
+			inode->root->root_key.objectid,
+			btrfs_ino(inode), start, len);
 	/* Update the page status */
 	for (i = start_index - first_index; i <= last_index - first_index; i++) {
 		ClearPageChecked(pages[i]);
@@ -1786,6 +1794,12 @@ int btrfs_defrag_file(struct inode *inode, struct file_ra_state *ra,
 	if (start_index < inode->i_mapping->writeback_index)
 		inode->i_mapping->writeback_index = start_index;
 
+	trace_printk("entry r/i=%lld/%llu start=%llu len=%llu newer=%llu thres=%u\n",
+		BTRFS_I(inode)->root->root_key.objectid,
+		btrfs_ino(BTRFS_I(inode)),
+		ctrl->last_scanned, last_byte + 1 - ctrl->last_scanned,
+		ctrl->newer_than, ctrl->extent_thresh);
+
 	while (ctrl->last_scanned < last_byte) {
 		const unsigned long prev_sectors_defragged = ctrl->sectors_defragged;
 		u64 cluster_end;

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Jean-Denis Girard]

[-- Attachment #1: Type: text/plain, Size: 895 bytes --]

Le 06/02/2022 à 15:16, Qu Wenruo a écrit :
> And what are the mount options?

Here are the mount options:

/dev/nvme0n1p2 on / type btrfs 
(rw,noatime,nodiratime,seclabel,compress-force=zstd:3,ssd,discard=async,space_cache,autodefrag,skip_balance,subvolid=5,subvol=/)
/dev/sdb on /home/Partage type btrfs 
(rw,noatime,nodiratime,seclabel,compress-force=zstd:3,space_cache=v2,autodefrag,skip_balance,subvolid=1061,subvol=/Partage)
/dev/sdb on /home/Photos type btrfs 
(rw,noatime,nodiratime,seclabel,compress-force=zstd:3,space_cache=v2,autodefrag,skip_balance,subvolid=1062,subvol=/Photos)
  ...

I have 13 subvolumes on /dev/sdb.

> Mind to apply attached `debug.diff`?

I can't try that immediately, but will do.


Thanks,
-- 
Jean-Denis Girard

SysNux                   Systèmes   Linux   en   Polynésie  française
https://www.sysnux.pf/   Tél: +689 40.50.10.40 / GSM: +689 87.797.527

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

[-- Attachment #1: Type: text/plain, Size: 5520 bytes --]



On 2022/2/6 17:26, Qu Wenruo wrote:
>
>
> On 2022/2/6 15:51, Rylee Randall wrote:
>> I am experiencing the same issue on 5.16.7, near the end of a large
>> steam game download btrfs-cleaner sits at the top of iotop, and shut
>> downs take about ten minutes because of various btrfs hangs.
>>
>> I compiled 5.15.21 with the mentioned patch and tried to recreate the
>> issue and so far have been unable to. I seem to get far faster an dmore
>> consistent write speeds from steam, and rather than btrfs-cleaner being
>> the main source of io usage it is steam. btrfs-cleaner is far down the
>> list along with various other btrfs- tasks.
>
> Thanks for the report, this indeed looks like the bug in v5.15 that it
> doesn't defrag a lot of extents is not the root cause.
>
> Mind to re-check with the following branch?
>
> https://github.com/adam900710/linux/tree/autodefrag_fixes
>
> It has one extra patch to emulate the older behavior of not using
> btrfs_get_em(), which can cause quite some problem for autodefrag.


And if anyone is still experiencing the problem even with that branch,
there is a diff to do the ultimate debug (thus can slow down the system).

After applying this diff upon that branch, every auto defrag try will
have several lines added to `/sys/kernel/debug/tracing/trace`.
(Which can be pretty large).

And that trace file would greatly help us to locate which extent is
defragged again and again.

Thanks,
Qu
>
> Thanks,
> Qu
>
>>
>> On 4/2/22 12:54, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>
>>>>
>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>> Hello all,
>>>>>
>>>>> Even after the defrag patches that landed in 5.16.5, I am still seeing
>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is
>>>>> enabled.
>>>>> I kinda expected slightly more IO during writes compared to 5.15, but
>>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o even
>>>>> when no programs are actively writing to the disk.
>>>>>
>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>> partition. In my case, I was downloading Strange Brigade, which is a
>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>> download, iostat will start reporting disk writes around 300 MB/s,
>>>>> even
>>>>> though Steam itself reports disk usage of 40-45MB/s. After the
>>>>> download
>>>>> finishes and nothing else is being written to disk, I still see around
>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>
>>>>> I left it running for a while to see if it was just some maintenance
>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>> reboot, but it actually prevented me from properly rebooting. After
>>>>> systemd timed out, my system finally shutdown.
>>>>>
>>>>> Here are my mount options:
>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>
>>>>>
>>>>>
>>>>
>>>> Compression, I guess that's the reason.
>>>>
>>>>  From the very beginning, btrfs defrag doesn't handle compressed extent
>>>> well.
>>>>
>>>> Even if a compressed extent is already at its maximum capacity, btrfs
>>>> will still try to defrag it.
>>>>
>>>> I believe the behavior is masked by other problems in older kernels
>>>> thus
>>>> not that obvious.
>>>>
>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>>
>>> And if possible, please try this diff on v5.15.x, and see if v5.15 is
>>> really doing less IO than v5.16.x.
>>>
>>> The diff will solve a problem in the old code, where autodefrag is
>>> almost not working.
>>>
>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>> index cc61813213d8..f6f2468d4883 100644
>>> --- a/fs/btrfs/ioctl.c
>>> +++ b/fs/btrfs/ioctl.c
>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode, struct
>>> file *file,
>>>                         continue;
>>>                 }
>>>
>>> -               if (!newer_than) {
>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>> -                                  PAGE_SHIFT) - i;
>>> -                       cluster = min(cluster, max_cluster);
>>> -               } else {
>>> -                       cluster = max_cluster;
>>> -               }
>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
>>> +               cluster = min(cluster, max_cluster);
>>>
>>>                 if (i + cluster > ra_index) {
>>>                         ra_index = max(i, ra_index);
>>>
>>>>
>>>> There are patches to address the compression related problem, but not
>>>> yet merged:
>>>>
>>>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387
>>>>
>>>> Mind to test them to see if that's the case?
>>>>
>>>> Thanks,
>>>> Qu
>>>>>
>>>>>
>>>>> I've disabled autodefrag again for now to save my SSD, but just wanted
>>>>> to say that there is still an issue. Have the defrag issues been fully
>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>> Phoronix say? XD
>>>>>
>>>>> Thanks!
>>>>> Ben
>>>>>
>>>>>

[-- Attachment #2: debug.diff --]
[-- Type: text/x-patch, Size: 2457 bytes --]

diff --git a/fs/btrfs/file.c b/fs/btrfs/file.c
index f5de8ab9787e..be2e20d163ef 100644
--- a/fs/btrfs/file.c
+++ b/fs/btrfs/file.c
@@ -304,6 +304,11 @@ static int __btrfs_run_defrag_inode(struct btrfs_fs_info *fs_info,
 	sb_start_write(fs_info->sb);
 	ret = btrfs_defrag_file(inode, NULL, &ctrl);
 	sb_end_write(fs_info->sb);
+	trace_printk("defrag finish r/i=%lld/%llu ret=%d defragged=%llu last_scanned=%llu last_off=%llu cycled=%d\n",
+		     BTRFS_I(inode)->root->root_key.objectid,
+		     btrfs_ino(BTRFS_I(inode)), ret, ctrl.sectors_defragged,
+		     ctrl.last_scanned,
+		     defrag->last_offset, defrag->cycled);
 	/*
 	 * if we filled the whole defrag batch, there
 	 * must be more work to do.  Queue this defrag
diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
index 133e3e2e2e79..a0501c88922b 100644
--- a/fs/btrfs/ioctl.c
+++ b/fs/btrfs/ioctl.c
@@ -1420,6 +1420,11 @@ static int defrag_collect_targets(struct btrfs_inode *inode,
 add:
 		last_is_target = true;
 		range_len = min(extent_map_end(em), start + len) - cur;
+
+		trace_printk("add target r/i=%lld/%llu start=%llu len=%llu em=%llu len=%llu gen=%lld newer=%llu\n",
+			inode->root->root_key.objectid,
+			btrfs_ino(inode), cur, range_len, em->start, em->len,
+			em->generation, ctrl->newer_than);
 		/*
 		 * This one is a good target, check if it can be merged into
 		 * last range of the target list.
@@ -1520,6 +1525,9 @@ static int defrag_one_locked_target(struct btrfs_inode *inode,
 			 EXTENT_DEFRAG, 0, 0, cached_state);
 	set_extent_defrag(&inode->io_tree, start, start + len - 1, cached_state);
 
+	trace_printk("defrag target r/i=%lld/%llu start=%llu len=%llu\n",
+			inode->root->root_key.objectid,
+			btrfs_ino(inode), start, len);
 	/* Update the page status */
 	for (i = start_index - first_index; i <= last_index - first_index; i++) {
 		ClearPageChecked(pages[i]);
@@ -1786,6 +1794,12 @@ int btrfs_defrag_file(struct inode *inode, struct file_ra_state *ra,
 	if (start_index < inode->i_mapping->writeback_index)
 		inode->i_mapping->writeback_index = start_index;
 
+	trace_printk("entry r/i=%lld/%llu start=%llu len=%llu newer=%llu thres=%u\n",
+		BTRFS_I(inode)->root->root_key.objectid,
+		btrfs_ino(BTRFS_I(inode)),
+		ctrl->last_scanned, last_byte + 1 - ctrl->last_scanned,
+		ctrl->newer_than, ctrl->extent_thresh);
+
 	while (ctrl->last_scanned < last_byte) {
 		const unsigned long prev_sectors_defragged = ctrl->sectors_defragged;
 		u64 cluster_end;

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/7 12:29, Rylee Randall wrote:
> I built a kernel from that branch and tried everything I could think of
> to try to reproduce the issue, but no matter what I tried btrfs-cleaner
> would only show up on iotop briefly every now and then, io did not hang
> and disk performance was as expected.
>
>
> I think the branch resolves the issue, or at least prevents me from
> being able to reproduce.

Just in case, you can still use that diff, it will only populate ftrace
event log buffer, which is much faster than dmesg log buffer.

And since you haven't seen huge IO/CPU usage, the extra debug message
shouldn't cause any observable performance drop.

And in case you observed huge CPU usage from btrfs-cleaner under
whatever load, you can direct grab the trace file, and no need to
rebuild a kernel module (and may fail to reproduce in next try).

Thanks,
Qu

>
> On 7/2/22 14:05, Qu Wenruo wrote:
>>
>>
>> On 2022/2/6 17:26, Qu Wenruo wrote:
>>>
>>>
>>> On 2022/2/6 15:51, Rylee Randall wrote:
>>>> I am experiencing the same issue on 5.16.7, near the end of a large
>>>> steam game download btrfs-cleaner sits at the top of iotop, and shut
>>>> downs take about ten minutes because of various btrfs hangs.
>>>>
>>>> I compiled 5.15.21 with the mentioned patch and tried to recreate the
>>>> issue and so far have been unable to. I seem to get far faster an dmore
>>>> consistent write speeds from steam, and rather than btrfs-cleaner being
>>>> the main source of io usage it is steam. btrfs-cleaner is far down the
>>>> list along with various other btrfs- tasks.
>>>
>>> Thanks for the report, this indeed looks like the bug in v5.15 that it
>>> doesn't defrag a lot of extents is not the root cause.
>>>
>>> Mind to re-check with the following branch?
>>>
>>> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fadam900710%2Flinux%2Ftree%2Fautodefrag_fixes&data=04%7C01%7C%7C4ca72e2e2774416246b208d9e9e6c2d8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637797999607143470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=LQyHFTmNG%2FKIQzPT6gt%2BZugspRGTHISPFtwi7zRdqhQ%3D&reserved=0
>>>
>>>
>>> It has one extra patch to emulate the older behavior of not using
>>> btrfs_get_em(), which can cause quite some problem for autodefrag.
>>
>>
>> And if anyone is still experiencing the problem even with that branch,
>> there is a diff to do the ultimate debug (thus can slow down the system).
>>
>> After applying this diff upon that branch, every auto defrag try will
>> have several lines added to `/sys/kernel/debug/tracing/trace`.
>> (Which can be pretty large).
>>
>> And that trace file would greatly help us to locate which extent is
>> defragged again and again.
>>
>> Thanks,
>> Qu
>>>
>>> Thanks,
>>> Qu
>>>
>>>>
>>>> On 4/2/22 12:54, Qu Wenruo wrote:
>>>>>
>>>>>
>>>>> On 2022/2/4 09:17, Qu Wenruo wrote:
>>>>>>
>>>>>>
>>>>>> On 2022/2/4 04:05, Benjamin Xiao wrote:
>>>>>>> Hello all,
>>>>>>>
>>>>>>> Even after the defrag patches that landed in 5.16.5, I am still
>>>>>>> seeing
>>>>>>> lots of cpu usage and disk writes to my SSD when autodefrag is
>>>>>>> enabled.
>>>>>>> I kinda expected slightly more IO during writes compared to 5.15,
>>>>>>> but
>>>>>>> what I am actually seeing is massive amounts of btrfs-cleaner i/o
>>>>>>> even
>>>>>>> when no programs are actively writing to the disk.
>>>>>>>
>>>>>>> I can reproduce it quite reliably on my 2TB Btrfs Steam library
>>>>>>> partition. In my case, I was downloading Strange Brigade, which is a
>>>>>>> roughly 25GB download and 33.65GB on disk. Somewhere during the
>>>>>>> download, iostat will start reporting disk writes around 300 MB/s,
>>>>>>> even
>>>>>>> though Steam itself reports disk usage of 40-45MB/s. After the
>>>>>>> download
>>>>>>> finishes and nothing else is being written to disk, I still see
>>>>>>> around
>>>>>>> 90-150MB/s worth of disk writes. Checking in iotop, I can see btrfs
>>>>>>> cleaner and other btrfs processes writing a lot of data.
>>>>>>>
>>>>>>> I left it running for a while to see if it was just some maintenance
>>>>>>> tasks that Btrfs needed to do, but it just kept going. I tried to
>>>>>>> reboot, but it actually prevented me from properly rebooting. After
>>>>>>> systemd timed out, my system finally shutdown.
>>>>>>>
>>>>>>> Here are my mount options:
>>>>>>> rw,relatime,compress-force=zstd:2,ssd,autodefrag,space_cache=v2,subvolid=5,subvol=/
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> Compression, I guess that's the reason.
>>>>>>
>>>>>>  From the very beginning, btrfs defrag doesn't handle compressed
>>>>>> extent
>>>>>> well.
>>>>>>
>>>>>> Even if a compressed extent is already at its maximum capacity, btrfs
>>>>>> will still try to defrag it.
>>>>>>
>>>>>> I believe the behavior is masked by other problems in older kernels
>>>>>> thus
>>>>>> not that obvious.
>>>>>>
>>>>>> But after rework of defrag in v5.16, this behavior is more exposed.
>>>>>
>>>>> And if possible, please try this diff on v5.15.x, and see if v5.15 is
>>>>> really doing less IO than v5.16.x.
>>>>>
>>>>> The diff will solve a problem in the old code, where autodefrag is
>>>>> almost not working.
>>>>>
>>>>> diff --git a/fs/btrfs/ioctl.c b/fs/btrfs/ioctl.c
>>>>> index cc61813213d8..f6f2468d4883 100644
>>>>> --- a/fs/btrfs/ioctl.c
>>>>> +++ b/fs/btrfs/ioctl.c
>>>>> @@ -1524,13 +1524,8 @@ int btrfs_defrag_file(struct inode *inode,
>>>>> struct
>>>>> file *file,
>>>>>                         continue;
>>>>>                 }
>>>>>
>>>>> -               if (!newer_than) {
>>>>> -                       cluster = (PAGE_ALIGN(defrag_end) >>
>>>>> -                                  PAGE_SHIFT) - i;
>>>>> -                       cluster = min(cluster, max_cluster);
>>>>> -               } else {
>>>>> -                       cluster = max_cluster;
>>>>> -               }
>>>>> +               cluster = (PAGE_ALIGN(defrag_end) >> PAGE_SHIFT) - i;
>>>>> +               cluster = min(cluster, max_cluster);
>>>>>
>>>>>                 if (i + cluster > ra_index) {
>>>>>                         ra_index = max(i, ra_index);
>>>>>
>>>>>>
>>>>>> There are patches to address the compression related problem, but not
>>>>>> yet merged:
>>>>>>
>>>>>> https://apac01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpatchwork.kernel.org%2Fproject%2Flinux-btrfs%2Flist%2F%3Fseries%3D609387&data=04%7C01%7C%7C4ca72e2e2774416246b208d9e9e6c2d8%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637797999607143470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FfyqNM8d%2BTXZXq3jE4d5M1OdO9hPeTIk5Ly6yaWWeuM%3D&reserved=0
>>>>>>
>>>>>>
>>>>>> Mind to test them to see if that's the case?
>>>>>>
>>>>>> Thanks,
>>>>>> Qu
>>>>>>>
>>>>>>>
>>>>>>> I've disabled autodefrag again for now to save my SSD, but just
>>>>>>> wanted
>>>>>>> to say that there is still an issue. Have the defrag issues been
>>>>>>> fully
>>>>>>> fixed or are there more patches incoming despite what Reddit and
>>>>>>> Phoronix say? XD
>>>>>>>
>>>>>>> Thanks!
>>>>>>> Ben
>>>>>>>
>>>>>>>

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Jean-Denis Girard]

[-- Attachment #1: Type: text/plain, Size: 5839 bytes --]

Hi Qu,

Sorry for late reply, and I still have not applied the last diff you sent.

But I wanted to report that without rebooting (uptime is 3 days 7 hours) 
or doing anything special, my system now behaves much better. 
btrfs-cleaner sometimes appears in top, but not for long period, and CPU 
usage is low. iostat seems reasonnable too.

Not sure what to conclude...


Thanks,
-- 
Jean-Denis Girard

SysNux                   Systèmes   Linux   en   Polynésie  française
https://www.sysnux.pf/   Tél: +689 40.50.10.40 / GSM: +689 87.797.527


Le 06/02/2022 à 15:16, Qu Wenruo a écrit :
> 
> 
> On 2022/2/7 01:43, Jean-Denis Girard wrote:
>> Hi list,
>>
>> I'm also experiencing high IO with autodefrag on 5.16.6 patched
>> with: -
>> https://patchwork.kernel.org/project/linux-btrfs/list/?series=609387,
>>
>>
> - https://patchwork.kernel.org/project/linux-btrfs/list/?series=611509,
>> - btrfs-defrag-bring-back-the-old-file-extent-search-behavior.diff.
> 
> OK, btrfs_search_forward() doesn't help in this case I guess.
> 
>>
>> After about 24 hours uptime, btrfs-cleaner is most of the time at 90
>> % CPU usage
>>
>> %CPU  %MEM     TIME+ COMMAND 77,6   0,0  29:03.25 btrfs-cleaner 19,1
>> 0,0   3:40.22 kworker/u24:8-btrfs-endio-write 19,1   0,0   1:50.46
>> kworker/u24:27-btrfs-endio-write 18,4   0,0   4:14.82
>> kworker/u24:7-btrfs-endio-write 17,4   0,0   4:08.08
>> kworker/u24:10-events_unbound 17,4   0,0   4:23.41
>> kworker/u24:9-btrfs-delalloc 15,8   0,0   3:41.21
>> kworker/u24:3-btrfs-endio-write 15,1   0,0   2:12.61
>> kworker/u24:26-blkcg_punt_bio 14,8   0,0   3:40.13
>> kworker/u24:18-btrfs-delalloc 12,8   0,0   3:55.70
>> kworker/u24:20-btrfs-delalloc 12,5   0,0   4:01.79
>> kworker/u24:1-btrfs-delalloc 12,2   0,0   4:04.63
>> kworker/u24:22-btrfs-endio-write 11,5   0,0   1:11.90
>> kworker/u24:11-btrfs-endio-write 11,2   0,0   1:25.96
>> kworker/u24:0-blkcg_punt_bio 10,2   0,0   4:24.17
>> kworker/u24:24+events_unbound ...
>>
>> Load average is 5,56, 6,88, 6,71 while just writing that mail.
>>
>> And iostat shows:
>>
  hours uptime>> avg-cpu:  %user   %nice %system %iowait  %st days eal   
%idle 1,5%    0,0%
>> 26,8%    6,3%    0,0%   65,4%
>>
>> tps    kB_read/s    kB_wrtn/s    kB_dscd/s    kB_read    kB_wrtn
>> kB_dscd Device 33893,50         0,0k       172,2M        57,8M
>> 0,0k       1,7G 577,7M nvme0n1 169,40         0,0k         1,2M
>> 0,0k       0,0k      12,5M 0,0k sda 178,90         0,0k         1,3M
>> 0,0k       0,0k      12,5M 0,0k sdb
>>
>> I hope that helps.
> 
> And what are the mount options?
> 
>  From the CPU usage, it looks like there are some infinite loop, but I
> can't find out why by a quick glance.
> 
> And since there is no extra read, it looks like it's doing defrag again
> and again on the same inode and range.
> 
> 
> Mind to apply attached `debug.diff`?
> 
> Then you still need to provide `/sys/kernel/debug/tracing/trace`, which
> can be very large and full of added debug info.
> (This means this can slow down your system)
> 
> The example of my local test would result something like this in above
> trace file:
> 
> # tracer: nop
> #
> # entries-in-buffer/entries-written: 11/11   #P:16
> #
> #                                _-----=> irqs-off/BH-disabled
> #                               / _----=> need-resched
> #                              | / _---=> hardirq/softirq
> #                              || / _--=> preempt-depth
> #                              ||| / _-=> migrate-disable
> #                              |||| /     delay
> #           TASK-PID     CPU#  |||||  TIMESTAMP  FUNCTION
> #              | |         |   |||||     |         |
>     btrfs-cleaner-1770    [014] .....   961.534329: btrfs_defrag_file:
> entry r/i=5/257 start=0 len=131072 newer=7 thres=262144
>     btrfs-cleaner-1770    [014] .....   961.534333:
> defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0
> len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534335:
> defrag_collect_targets: add target r/i=5/257 start=16384 len=16384
> em=16384 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534336:
> defrag_collect_targets: add target r/i=5/257 start=32768 len=16384
> em=32768 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534337:
> defrag_collect_targets: add target r/i=5/257 start=49152 len=16384
> em=49152 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534564:
> defrag_collect_targets: add target r/i=5/257 start=0 len=16384 em=0
> len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534567:
> defrag_collect_targets: add target r/i=5/257 start=16384 len=16384
> em=16384 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534568:
> defrag_collect_targets: add target r/i=5/257 start=32768 len=16384
> em=32768 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534569:
> defrag_collect_targets: add target r/i=5/257 start=49152 len=16384
> em=49152 len=16384 gen=7 newer=7
>     btrfs-cleaner-1770    [014] .....   961.534580: btrfs_defrag_file:
> defrag target r/i=5/257 start=0 len=65536
>     btrfs-cleaner-1770    [014] .....   961.534681:
> btrfs_run_defrag_inodes: defrag finish r/i=5/257 ret=0 defragged=16
> last_scanned=131072 last_off=0 cycled=0
> 
> Thanks,
> Qu
> 
> 
>>
>>
>> Thanks,

[-- Attachment #2: Signature cryptographique S/MIME --]
[-- Type: application/pkcs7-signature, Size: 4003 bytes --]

Re: Still seeing high autodefrag IO with kernel 5.16.5

[Qu Wenruo]

On 2022/2/9 09:56, Jean-Denis Girard wrote:
> Hi Qu,
>
> Sorry for late reply, and I still have not applied the last diff you sent.
>
> But I wanted to report that without rebooting (uptime is 3 days 7 hours)
> or doing anything special, my system now behaves much better.
> btrfs-cleaner sometimes appears in top, but not for long period, and CPU
> usage is low. iostat seems reasonnable too.

Well, at least it's some good news.

BTW I have updated that branch on github, feel free to try (better with
the extra diff).

Just in case you have hit such situation again.

Meanwhile I'll try to rework autodefrag behavior to be more accurate.

Right now, autodefrag will try to scan the whole file if there is any
small writ to it.
And there is some questionable check on whether autodefrag should rescan
the whole file.

I want to change it to only scan ranges adjacent to the small write, so
we can avoid possible problem related to the whole file scan behavior.

Thanks,
Qu

>
> Not sure what to conclude...
>
>
> Thanks,