[BUG] 500-2000% performance regression w/ 5.10

[BUG] 500-2000% performance regression w/ 5.10

[René Rebe]

Hey there,

as a long time btrfs user I noticed some some things became very slow
w/ Linux kernel 5.10. I found a very simple test case, namely extracting
a huge tarball like:

  tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst

Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%

To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
a similar, albeit not as shocking regression from 5.2 seconds
to ~34 seconds or∫~650%.

Somehow testing that in a VM did over virtio did not produce
as different results, although it was already 35 seconds slow
with 5.9.

# first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
  btrfs: use btrfs_start_delalloc_roots in shrink_delalloc

Now just this single commit does obviously not revert cleanly,
and I did not have the time today to look into the rather more
complex code today.

I hope this helps improve this for the next release, maybe you
want to test on bare metal, too.

Greetings,
	René	https://youtu.be/NhUMdvLyKJc

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin, https://exactcode.com
 https://exactscan.com | https://ocrkit.com | https://t2sde.org | https://rene.rebe.de

Re: [BUG] 500-2000% performance regression w/ 5.10

[Qu Wenruo]

On 2020/12/22 上午3:45, René Rebe wrote:
> Hey there,
>
> as a long time btrfs user I noticed some some things became very slow
> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
> a huge tarball like:
>
>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>
> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>
> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
> a similar, albeit not as shocking regression from 5.2 seconds
> to ~34 seconds or∫~650%.
>
> Somehow testing that in a VM did over virtio did not produce
> as different results, although it was already 35 seconds slow
> with 5.9.
>
> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc

This means metadata space is not enough and we go shrink_delalloc() to
free some metadata space.

My concern is, why we need to go shrink_delalloc() in the first place.

Normally either the fs has enough unallocated space (thus we can
over-commit) or has enough unused metadata space.

We only need to shrink delalloc if we have no unallocated space, and not
enough space for the over-estimated metadata reserve.


Would you please try to provide the `btrfs fi usage` output of your test
drive?
My initial guess is, this is related to fs usage/layout.

Thanks,
Qu
>
> Now just this single commit does obviously not revert cleanly,
> and I did not have the time today to look into the rather more
> complex code today.
>
> I hope this helps improve this for the next release, maybe you
> want to test on bare metal, too.
>
> Greetings,
> 	René	https://youtu.be/NhUMdvLyKJc
>

Re: [BUG] 500-2000% performance regression w/ 5.10

[Rene Rebe]

Dear Qu,

On 12/22/20 9:28 AM, Qu Wenruo wrote:
> On 2020/12/22 上午3:45, René Rebe wrote:
>> Hey there,
>>
>> as a long time btrfs user I noticed some some things became very slow
>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>> a huge tarball like:
>>
>>    tar xf 
>> /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>
>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>
>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>> a similar, albeit not as shocking regression from 5.2 seconds
>> to ~34 seconds or∫~650%.
>>
>> Somehow testing that in a VM did over virtio did not produce
>> as different results, although it was already 35 seconds slow
>> with 5.9.
>>
>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>
> This means metadata space is not enough and we go shrink_delalloc() to
> free some metadata space.
>
> My concern is, why we need to go shrink_delalloc() in the first place.
>
> Normally either the fs has enough unallocated space (thus we can
> over-commit) or has enough unused metadata space.
>
> We only need to shrink delalloc if we have no unallocated space, and not
> enough space for the over-estimated metadata reserve.
>
>
> Would you please try to provide the `btrfs fi usage` output of your test
> drive?
> My initial guess is, this is related to fs usage/layout.


Thank you for looking into this and your reply.

That was my initial thoguht, too, and thus I already had run the test on 
a brand new, previously unused 1TB SSD with a fresh mkfs.btrfs with 
similar results to the long used drive:

# btrfs fi usage /mnt/
Overall:
     Device size:                 931.51GiB
     Device allocated:              4.01GiB
     Device unallocated:          927.50GiB
     Device missing:                  0.00B
     Used:                        640.00KiB
     Free (estimated):            930.50GiB      (min: 930.50GiB)
     Data ratio:                       1.00
     Metadata ratio:                   1.00
     Global reserve:                3.25MiB      (used: 0.00B)

Data,single: Size:3.00GiB, Used:512.00KiB
    /dev/nvme0n1    3.00GiB

Metadata,single: Size:1.01GiB, Used:112.00KiB
    /dev/nvme0n1    1.01GiB

System,single: Size:4.00MiB, Used:16.00KiB
    /dev/nvme0n1    4.00MiB

Unallocated:
    /dev/nvme0n1  927.50GiB


# mkfs.btrfs /dev/nvme0n1 -f

# mount /dev/nvme0n1 /mnt

# cat /t2/download/mirror/f/firefox-84.0.source.tar.zst > /dev/null
# time tar -x -C /mnt -f /t2/download/mirror/f/firefox-84.0.source.tar.zst

I hope this helps,

   René

> Thanks,
> Qu
>>
>> Now just this single commit does obviously not revert cleanly,
>> and I did not have the time today to look into the rather more
>> complex code today.
>>
>> I hope this helps improve this for the next release, maybe you
>> want to test on bare metal, too.
>>
>> Greetings,
>>     René    https://youtu.be/NhUMdvLyKJc
>>

Re: [BUG] 500-2000% performance regression w/ 5.10

[Josef Bacik]

On 12/21/20 2:45 PM, René Rebe wrote:
> Hey there,
> 
> as a long time btrfs user I noticed some some things became very slow
> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
> a huge tarball like:
> 
>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
> 
> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
> 
> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
> a similar, albeit not as shocking regression from 5.2 seconds
> to ~34 seconds or∫~650%.
> 
> Somehow testing that in a VM did over virtio did not produce
> as different results, although it was already 35 seconds slow
> with 5.9.
> 
> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
> 

Eesh that's not great, I'll work it out in the morning, thanks,

Josef

Re: [BUG] 500-2000% performance regression w/ 5.10

[Josef Bacik]

Hello,

Could you give this a try?  I'm not able to reproduce the problem, but I'm
testing inside of a VM.  I'm in the middle of Christmas stuff, but I'll get
ahold of a giant machine at work tomorrow and see if I can reproduce there.
Meanwhile can you give this a shot?  I have a sneaking suspicion why it happens
on your baremetal and not in VM's, and this will be a partial enough of a revert
of the patch you bisected to validate what I'm thinking.  THanks,

Josef


Test this to see if it fixes the problem.

Signed-off-by: Josef Bacik <josef@toxicpanda.com>
---
 fs/btrfs/space-info.c | 25 ++++++++++++++++++++++++-
 1 file changed, 24 insertions(+), 1 deletion(-)

diff --git a/fs/btrfs/space-info.c b/fs/btrfs/space-info.c
index 67e55c5479b8..7b2867e915c6 100644
--- a/fs/btrfs/space-info.c
+++ b/fs/btrfs/space-info.c
@@ -480,6 +480,28 @@ static inline u64 calc_reclaim_items_nr(struct btrfs_fs_info *fs_info,
 
 #define EXTENT_SIZE_PER_ITEM	SZ_256K
 
+static void btrfs_writeback_inodes_sb_nr(struct btrfs_fs_info *fs_info,
+					 unsigned long nr_pages, u64 nr_items)
+{
+	struct super_block *sb = fs_info->sb;
+
+	if (down_read_trylock(&sb->s_umount)) {
+		writeback_inodes_sb_nr(sb, nr_pages, WB_REASON_FS_FREE_SPACE);
+		up_read(&sb->s_umount);
+	} else {
+		/*
+		 * We needn't worry the filesystem going from r/w to r/o though
+		 * we don't acquire ->s_umount mutex, because the filesystem
+		 * should guarantee the delalloc inodes list be empty after
+		 * the filesystem is readonly(all dirty pages are written to
+		 * the disk).
+		 */
+		btrfs_start_delalloc_roots(fs_info, nr_items, true);
+		if (!current->journal_info)
+			btrfs_wait_ordered_roots(fs_info, nr_items, 0, (u64)-1);
+       }
+}
+
 /*
  * shrink metadata reservation for delalloc
  */
@@ -532,7 +554,8 @@ static void shrink_delalloc(struct btrfs_fs_info *fs_info,
 
 	loops = 0;
 	while ((delalloc_bytes || dio_bytes) && loops < 3) {
-		btrfs_start_delalloc_roots(fs_info, items, true);
+		unsigned long nr_pages = min(delalloc_bytes, to_reclaim) >> PAGE_SHIFT;
+		btrfs_writeback_inodes_sb_nr(fs_info, nr_pages, items);
 
 		loops++;
 		if (wait_ordered && !trans) {
-- 
2.26.2

Re: [BUG] 500-2000% performance regression w/ 5.10

[Holger Hoffstätte]

On 2020-12-23 20:41, Josef Bacik wrote:
> Could you give this a try?  I'm not able to reproduce the problem, but I'm

Since I wanted to rule out NVME/block layer/scheduler etc. I tried and
could reproduce it immediately, see below. Didn't notice it earlier since
most of btrfs is read-mostly.. :(

> testing inside of a VM.  I'm in the middle of Christmas stuff, but I'll get
> ahold of a giant machine at work tomorrow and see if I can reproduce there.
> Meanwhile can you give this a shot?  I have a sneaking suspicion why it happens
> on your baremetal and not in VM's, and this will be a partial enough of a revert
> of the patch you bisected to validate what I'm thinking.  THanks,

The patch doesn't apply to 5.10.x since btrfs_start_delalloc_roots() does not
have the trailing true/false argument yet. I removed it, which seemed to have
worked. :}

Results using -dsingle/-msingle/space tree, all on tmpfs:

Unpatched:

kernel tree, ~1.1G:
$time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.37s user 3.26s system 6% cpu 52.144 total

-> slow as hell since it's thousands of small files. Writeback runs at ~5-10 MB/s.

large file:
$fallocate -l 2G /tmp/largefile
$time (cp -a /tmp/largefile /tmp/butter && sync -f /tmp/butter)
( cp -a /tmp/largefile /tmp/butter && sync -f /tmp/butter; )  0.00s user 0.91s system 75% cpu 1.215 total

-> OK-ish since it's just one big file.

With your patch & the 'true' arg to btrfs_start_delalloc_roots() removed:

kernel tree:
$time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.28s user 2.44s system 60% cpu 4.475 total

rewrite:
$time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.28s user 2.87s system 93% cpu 3.357 total

Clearly better.

Hope this helps :)

Holger

Re: [BUG] 500-2000% performance regression w/ 5.10

[Josef Bacik]

On 12/23/20 3:31 PM, Holger Hoffstätte wrote:
> On 2020-12-23 20:41, Josef Bacik wrote:
>> Could you give this a try?  I'm not able to reproduce the problem, but 
>> I'm
> 
> Since I wanted to rule out NVME/block layer/scheduler etc. I tried and
> could reproduce it immediately, see below. Didn't notice it earlier since
> most of btrfs is read-mostly.. :(
> 
>> testing inside of a VM.  I'm in the middle of Christmas stuff, but 
>> I'll get
>> ahold of a giant machine at work tomorrow and see if I can reproduce 
>> there.
>> Meanwhile can you give this a shot?  I have a sneaking suspicion why 
>> it happens
>> on your baremetal and not in VM's, and this will be a partial enough 
>> of a revert
>> of the patch you bisected to validate what I'm thinking.  THanks,
> 
> The patch doesn't apply to 5.10.x since btrfs_start_delalloc_roots() 
> does not
> have the trailing true/false argument yet. I removed it, which seemed to 
> have
> worked. :}
> 
> Results using -dsingle/-msingle/space tree, all on tmpfs:
> 
> Unpatched:
> 
> kernel tree, ~1.1G:
> $time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
> ( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.37s 
> user 3.26s system 6% cpu 52.144 total
> 
> -> slow as hell since it's thousands of small files. Writeback runs at 
> ~5-10 MB/s.
> 
> large file:
> $fallocate -l 2G /tmp/largefile
> $time (cp -a /tmp/largefile /tmp/butter && sync -f /tmp/butter)
> ( cp -a /tmp/largefile /tmp/butter && sync -f /tmp/butter; )  0.00s user 
> 0.91s system 75% cpu 1.215 total
> 
> -> OK-ish since it's just one big file.
> 
> With your patch & the 'true' arg to btrfs_start_delalloc_roots() removed:
> 
> kernel tree:
> $time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
> ( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.28s 
> user 2.44s system 60% cpu 4.475 total
> 
> rewrite:
> $time (cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter)
> ( cp -a /tmp/linux-5.10.3 /tmp/butter && sync -f /tmp/butter; )  0.28s 
> user 2.87s system 93% cpu 3.357 total
> 
> Clearly better.
> 
> Hope this helps :)
> 

Oops I built on top of our devel branch and not 5.10, my bad.  This does 
help and validates my theory.  I'll work out a proper fix soon for 
everybody to test, probably tomorrow so I don't risk being murdered by 
my wife.  Thanks,

Josef

Re: [BUG] 500-2000% performance regression w/ 5.10

[Josef Bacik]

On 12/21/20 2:45 PM, René Rebe wrote:
> Hey there,
> 
> as a long time btrfs user I noticed some some things became very slow
> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
> a huge tarball like:
> 
>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
> 
> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
> 
> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
> a similar, albeit not as shocking regression from 5.2 seconds
> to ~34 seconds or∫~650%.
> 
> Somehow testing that in a VM did over virtio did not produce
> as different results, although it was already 35 seconds slow
> with 5.9.
> 
> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
> 
> Now just this single commit does obviously not revert cleanly,
> and I did not have the time today to look into the rather more
> complex code today.
> 
> I hope this helps improve this for the next release, maybe you
> want to test on bare metal, too.
> 

Alright to close the loop with this, this slipped through the cracks 
because I was doing a lot of performance related work, and specifically 
had been testing with these patches on top of everything

https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/

These patches bring the performance up to around 40% higher than 
baseline.  In the meantime we'll probably push this partial revert into 
5.10 stable so performance isn't sucking in the meantime.  Thanks,

Josef

Re: [BUG] 500-2000% performance regression w/ 5.10

[René Rebe]

Hi Josef,

> On 24. Dec 2020, at 19:09, Josef Bacik <josef@toxicpanda.com> wrote:
> 
> On 12/21/20 2:45 PM, René Rebe wrote:
>> Hey there,
>> as a long time btrfs user I noticed some some things became very slow
>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>> a huge tarball like:
>>   tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>> a similar, albeit not as shocking regression from 5.2 seconds
>> to ~34 seconds or∫~650%.
>> Somehow testing that in a VM did over virtio did not produce
>> as different results, although it was already 35 seconds slow
>> with 5.9.
>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>   btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>> Now just this single commit does obviously not revert cleanly,
>> and I did not have the time today to look into the rather more
>> complex code today.
>> I hope this helps improve this for the next release, maybe you
>> want to test on bare metal, too.
> 
> Alright to close the loop with this, this slipped through the cracks because I was doing a lot of performance related work, and specifically had been testing with these patches on top of everything
> 
> https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/
> 
> These patches bring the performance up to around 40% higher than baseline

I indeed tested the linux-btrfs for-5.11 and found the performance some 50% better. I would hope that can be brought back to 5.9 levels sometime soon ;-)

> .  In the meantime we'll probably push this partial revert into 5.10 stable so performance isn't sucking in the meantime.  Thanks,

That certainly makes sense for the LTS kernel series.

Thanks for looking into this,
Merry Christmas,
	René Rebe

-- 
 ExactCODE GmbH, Lietzenburger Str. 42, DE-10789 Berlin, https://exactcode.com
 https://exactscan.com | https://ocrkit.com | https://t2sde.org | https://rene.rebe.de

Re: [BUG] 500-2000% performance regression w/ 5.10

[Nikolay Borisov]

On 24.12.20 г. 20:09 ч., Josef Bacik wrote:
> On 12/21/20 2:45 PM, René Rebe wrote:
>> Hey there,
>>
>> as a long time btrfs user I noticed some some things became very slow
>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>> a huge tarball like:
>>
>>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>
>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>
>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>> a similar, albeit not as shocking regression from 5.2 seconds
>> to ~34 seconds or∫~650%.
>>
>> Somehow testing that in a VM did over virtio did not produce
>> as different results, although it was already 35 seconds slow
>> with 5.9.
>>
>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>>
>> Now just this single commit does obviously not revert cleanly,
>> and I did not have the time today to look into the rather more
>> complex code today.
>>
>> I hope this helps improve this for the next release, maybe you
>> want to test on bare metal, too.
>>
> 
> Alright to close the loop with this, this slipped through the cracks
> because I was doing a lot of performance related work, and specifically
> had been testing with these patches on top of everything
> 
> https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/
> 
> 
> These patches bring the performance up to around 40% higher than
> baseline.  In the meantime we'll probably push this partial revert into
> 5.10 stable so performance isn't sucking in the meantime.  Thanks,

For the sake of clarity it's important to note that baseline in this
case is kernel 4.19 (since the original regressions was spotted in 5.0).
There has already been a couple of open interpretations of what this
means e.g 40% better than 5.10, suggesting performance is worse.

> 
> Josef
> 

Re: [BUG] 500-2000% performance regression w/ 5.10

[Nikolay Borisov]

On 24.12.20 г. 23:11 ч., René Rebe wrote:
> Hi Josef,
> 
>> On 24. Dec 2020, at 19:09, Josef Bacik <josef@toxicpanda.com> wrote:
>>
>> On 12/21/20 2:45 PM, René Rebe wrote:
>>> Hey there,
>>> as a long time btrfs user I noticed some some things became very slow
>>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>>> a huge tarball like:
>>>   tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>>> a similar, albeit not as shocking regression from 5.2 seconds
>>> to ~34 seconds or∫~650%.
>>> Somehow testing that in a VM did over virtio did not produce
>>> as different results, although it was already 35 seconds slow
>>> with 5.9.
>>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>>   btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>>> Now just this single commit does obviously not revert cleanly,
>>> and I did not have the time today to look into the rather more
>>> complex code today.
>>> I hope this helps improve this for the next release, maybe you
>>> want to test on bare metal, too.
>>
>> Alright to close the loop with this, this slipped through the cracks because I was doing a lot of performance related work, and specifically had been testing with these patches on top of everything
>>
>> https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/
>>
>> These patches bring the performance up to around 40% higher than baseline
> 
> I indeed tested the linux-btrfs for-5.11 and found the performance some 50% better. I would hope that can be brought back to 5.9 levels sometime soon ;-)

Do you mean 50% better as compared to 5.9?
> 
>> .  In the meantime we'll probably push this partial revert into 5.10 stable so performance isn't sucking in the meantime.  Thanks,
> 
> That certainly makes sense for the LTS kernel series.
> 
> Thanks for looking into this,
> Merry Christmas,
> 	René Rebe
> 

Re: [BUG] 500-2000% performance regression w/ 5.10

[Rene Rebe]

Hey there,

On 12/26/20 1:24 PM, Nikolay Borisov wrote:
>
> On 24.12.20 г. 23:11 ч., René Rebe wrote:
>> Hi Josef,
>>
>>> On 24. Dec 2020, at 19:09, Josef Bacik <josef@toxicpanda.com> wrote:
>>>
>>> On 12/21/20 2:45 PM, René Rebe wrote:
>>>> Hey there,
>>>> as a long time btrfs user I noticed some some things became very slow
>>>> w/ Linux kernel 5.10. I found a very simple test case, namely extracting
>>>> a huge tarball like:
>>>>    tar xf /usr/src/t2-clean/download/mirror/f/firefox-84.0.source.tar.zst
>>>> Why my external, USB3 road-warrior SSD on a Ryzen 5950x this
>>>> went from ~15 seconds w/ 5.9 to nearly 5 minutes in 5.10, or 2000%
>>>> To rule out USB, I also tested a brand new PCIe 4.0 SSD, with
>>>> a similar, albeit not as shocking regression from 5.2 seconds
>>>> to ~34 seconds or∫~650%.
>>>> Somehow testing that in a VM did over virtio did not produce
>>>> as different results, although it was already 35 seconds slow
>>>> with 5.9.
>>>> # first bad commit: [38d715f494f2f1dddbf3d0c6e50aefff49519232]
>>>>    btrfs: use btrfs_start_delalloc_roots in shrink_delalloc
>>>> Now just this single commit does obviously not revert cleanly,
>>>> and I did not have the time today to look into the rather more
>>>> complex code today.
>>>> I hope this helps improve this for the next release, maybe you
>>>> want to test on bare metal, too.
>>> Alright to close the loop with this, this slipped through the cracks because I was doing a lot of performance related work, and specifically had been testing with these patches on top of everything
>>>
>>> https://lore.kernel.org/linux-btrfs/cover.1602249928.git.josef@toxicpanda.com/
>>>
>>> These patches bring the performance up to around 40% higher than baseline
>> I indeed tested the linux-btrfs for-5.11 and found the performance some 50% better. I would hope that can be brought back to 5.9 levels sometime soon ;-)
> Do you mean 50% better as compared to 5.9?


Sorry for any confusion, I meant 50% better than the bisected regression 
as found with 5.10, not yet as good as 5.9 has been before.


     René

>>> .  In the meantime we'll probably push this partial revert into 5.10 stable so performance isn't sucking in the meantime.  Thanks,
>> That certainly makes sense for the LTS kernel series.
>>
>> Thanks for looking into this,
>> Merry Christmas,
>> 	René Rebe
>>

Re: [BUG] 500-2000% performance regression w/ 5.10

[Chris Murphy]

The problem is worse on SSD than on HDD. It actually makes the SSD
*slower* than an HDD, on 5.10. For this workload

HDD
5.9.16-200.fc33.x86_64
mq-deadline kyber [bfq] none

$ time tar -xf /tmp/firefox-85.0b4.source.tar.xz && time sync

real    1m27.299s
user    0m27.294s
sys    0m14.134s

real    0m8.890s
user    0m0.001s
sys    0m0.344s

----------------
HDD
5.10.4-200.fc33.x86_64
mq-deadline kyber [bfq] none

$ time tar -xf /tmp/firefox-85.0b4.source.tar.xz && time sync

real    2m14.936s
user    0m54.396s
sys    0m47.082s

real    0m7.726s
user    0m0.001s
sys    0m0.382s

====================

SSD, compress=zstd:1
5.9.16-200.fc33.x86_64
[mq-deadline] kyber bfq none

$ time tar -xf /tmp/firefox-85.0b4.source.tar.xz && time sync

real    0m41.947s
user    0m29.359s
sys    0m18.088s

real    0m2.042s
user    0m0.000s
sys    0m0.065s
----------------
SSD, compress=zstd:1
5.10.4-200.fc33.x86_64
[mq-deadline] kyber bfq none

$ time tar -xf /tmp/firefox-85.0b4.source.tar.xz && time sync

real    2m59.581s
user    1m4.097s
sys    0m56.323s

real    0m1.492s
user    0m0.000s
sys    0m0.077s