[PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Qu Wenruo]

There is a long existing bug in btrfs defrag code that it will always
try to defrag compressed extents, even they are already at max capacity.

This will not reduce the number of extents, but only waste IO/CPU.

The kernel fix is titled:

  btrfs: defrag: don't defrag extents which is already at its max capacity

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/257.out |  2 ++
 2 files changed, 81 insertions(+)
 create mode 100755 tests/btrfs/257
 create mode 100644 tests/btrfs/257.out

diff --git a/tests/btrfs/257 b/tests/btrfs/257
new file mode 100755
index 00000000..326687dc
--- /dev/null
+++ b/tests/btrfs/257
@@ -0,0 +1,79 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 257
+#
+# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
+# at their max capacity.
+#
+. ./common/preamble
+_begin_fstest auto quick defrag
+
+# Import common functions.
+. ./common/filter
+. ./common/btrfs
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_require_scratch
+
+# Needs 4K sectorsize, as larger sectorsize can change the file layout.
+_require_btrfs_support_sectorsize 4096
+
+get_extent_disk_sector()
+{
+	local file=$1
+	local offset=$2
+
+	$XFS_IO_PROG -c "fiemap $offset" "$file" | _filter_xfs_io_fiemap |\
+		head -n1 | $AWK_PROG '{print $3}'
+}
+
+_scratch_mkfs >> $seqres.full
+
+# Need datacow to show which range is defragged, and we're testing
+# autodefrag with compression
+_scratch_mount -o datacow,autodefrag,compress
+
+# Btrfs uses 128K as compressed extent max size, so this would result
+# exactly two extents, which are all at their max size
+$XFS_IO_PROG -f -c "pwrite -S 0xee 0 128k" -c sync \
+		-c "pwrite -S 0xff 128k 128k" -c sync \
+		$SCRATCH_MNT/foobar >> $seqres.full
+
+old_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
+old_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
+old_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
+
+echo "=== File extent layout before defrag ===" >> $seqres.full
+$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
+
+$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/foobar" >> $seqres.full
+
+new_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
+new_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
+new_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
+
+echo "=== File extent layout before defrag ===" >> $seqres.full
+$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
+
+if [ $new_csum != $old_csum ]; then
+	echo "file content changed"
+fi
+
+if [ $new_extent1 != $old_extent1 ]; then
+	echo "the first extent get defragged"
+fi
+
+if [ $new_extent2 != $old_extent2 ]; then
+	echo "the second extent get defragged"
+fi
+
+echo "Silence is golden"
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/257.out b/tests/btrfs/257.out
new file mode 100644
index 00000000..cc3693f3
--- /dev/null
+++ b/tests/btrfs/257.out
@@ -0,0 +1,2 @@
+QA output created by 257
+Silence is golden
-- 
2.34.1

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Filipe Manana]

On Thu, Jan 27, 2022 at 01:53:06PM +0800, Qu Wenruo wrote:
> There is a long existing bug in btrfs defrag code that it will always
> try to defrag compressed extents, even they are already at max capacity.
> 
> This will not reduce the number of extents, but only waste IO/CPU.
> 
> The kernel fix is titled:
> 
>   btrfs: defrag: don't defrag extents which is already at its max capacity
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/257.out |  2 ++
>  2 files changed, 81 insertions(+)
>  create mode 100755 tests/btrfs/257
>  create mode 100644 tests/btrfs/257.out
> 
> diff --git a/tests/btrfs/257 b/tests/btrfs/257
> new file mode 100755
> index 00000000..326687dc
> --- /dev/null
> +++ b/tests/btrfs/257
> @@ -0,0 +1,79 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test 257
> +#
> +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
> +# at their max capacity.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick defrag

Missing the 'compress' group.

> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/btrfs
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_require_scratch
> +
> +# Needs 4K sectorsize, as larger sectorsize can change the file layout.
> +_require_btrfs_support_sectorsize 4096

Hum?
I don't understand why that's needed for this test.
The maximum size of a compressed extent is the same for all sector sizes.

> +
> +get_extent_disk_sector()
> +{
> +	local file=$1
> +	local offset=$2
> +
> +	$XFS_IO_PROG -c "fiemap $offset" "$file" | _filter_xfs_io_fiemap |\
> +		head -n1 | $AWK_PROG '{print $3}'
> +}

This is copy pasted from the previous test:

https://lore.kernel.org/linux-btrfs/20220127054543.28964-1-wqu@suse.com/T/#u

Could go somewhere into common/*, if there isn't already anything providing
the same functionality.

> +
> +_scratch_mkfs >> $seqres.full
> +
> +# Need datacow to show which range is defragged, and we're testing
> +# autodefrag with compression
> +_scratch_mount -o datacow,autodefrag,compress

The autodefrag is not needed. We are triggering a manual defrag below, and
that's all that's needed to trigger the issue.

> +
> +# Btrfs uses 128K as compressed extent max size, so this would result
> +# exactly two extents, which are all at their max size
> +$XFS_IO_PROG -f -c "pwrite -S 0xee 0 128k" -c sync \
> +		-c "pwrite -S 0xff 128k 128k" -c sync \
> +		$SCRATCH_MNT/foobar >> $seqres.full

We don't need to do a sync after every write. If you write 256K at once,
it will result in 2 128K extents anyway. The comment and the way we are
calling xfs_io gives the wrong idea that user space can influence the max
extent size.

A more interesting test would be, say, to write 2M or 4M at once for
example, which will result in many 128K extents. It would also make the
test more robust in case the default defrag threshold changes one day
for some reason (e.g. btrfs-progs might decide to start calling the
ioctl with a higher threshold one day). And then just check that the
output of fiemap is the same before and after the defrag attempt, so
it's not even necessary to manually compare the sectors of each
extent and use get_extent_disk_sector().

Thanks.

> +
> +old_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
> +old_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
> +old_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
> +
> +echo "=== File extent layout before defrag ===" >> $seqres.full
> +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
> +
> +$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/foobar" >> $seqres.full
> +
> +new_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
> +new_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
> +new_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
> +
> +echo "=== File extent layout before defrag ===" >> $seqres.full
> +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
> +
> +if [ $new_csum != $old_csum ]; then
> +	echo "file content changed"
> +fi
> +
> +if [ $new_extent1 != $old_extent1 ]; then
> +	echo "the first extent get defragged"
> +fi
> +
> +if [ $new_extent2 != $old_extent2 ]; then
> +	echo "the second extent get defragged"
> +fi
> +
> +echo "Silence is golden"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/257.out b/tests/btrfs/257.out
> new file mode 100644
> index 00000000..cc3693f3
> --- /dev/null
> +++ b/tests/btrfs/257.out
> @@ -0,0 +1,2 @@
> +QA output created by 257
> +Silence is golden
> -- 
> 2.34.1
> 

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Filipe Manana]

On Thu, Jan 27, 2022 at 11:37:46AM +0000, Filipe Manana wrote:
> On Thu, Jan 27, 2022 at 01:53:06PM +0800, Qu Wenruo wrote:
> > There is a long existing bug in btrfs defrag code that it will always
> > try to defrag compressed extents, even they are already at max capacity.
> > 
> > This will not reduce the number of extents, but only waste IO/CPU.
> > 
> > The kernel fix is titled:
> > 
> >   btrfs: defrag: don't defrag extents which is already at its max capacity
> > 
> > Signed-off-by: Qu Wenruo <wqu@suse.com>
> > ---
> >  tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
> >  tests/btrfs/257.out |  2 ++
> >  2 files changed, 81 insertions(+)
> >  create mode 100755 tests/btrfs/257
> >  create mode 100644 tests/btrfs/257.out
> > 
> > diff --git a/tests/btrfs/257 b/tests/btrfs/257
> > new file mode 100755
> > index 00000000..326687dc
> > --- /dev/null
> > +++ b/tests/btrfs/257
> > @@ -0,0 +1,79 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
> > +#
> > +# FS QA Test 257
> > +#
> > +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
> > +# at their max capacity.
> > +#
> > +. ./common/preamble
> > +_begin_fstest auto quick defrag
> 
> Missing the 'compress' group.
> 
> > +
> > +# Import common functions.
> > +. ./common/filter
> > +. ./common/btrfs
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs btrfs
> > +_require_scratch
> > +
> > +# Needs 4K sectorsize, as larger sectorsize can change the file layout.
> > +_require_btrfs_support_sectorsize 4096
> 
> Hum?
> I don't understand why that's needed for this test.
> The maximum size of a compressed extent is the same for all sector sizes.
> 
> > +
> > +get_extent_disk_sector()
> > +{
> > +	local file=$1
> > +	local offset=$2
> > +
> > +	$XFS_IO_PROG -c "fiemap $offset" "$file" | _filter_xfs_io_fiemap |\
> > +		head -n1 | $AWK_PROG '{print $3}'
> > +}
> 
> This is copy pasted from the previous test:
> 
> https://lore.kernel.org/linux-btrfs/20220127054543.28964-1-wqu@suse.com/T/#u
> 
> Could go somewhere into common/*, if there isn't already anything providing
> the same functionality.
> 
> > +
> > +_scratch_mkfs >> $seqres.full
> > +
> > +# Need datacow to show which range is defragged, and we're testing
> > +# autodefrag with compression
> > +_scratch_mount -o datacow,autodefrag,compress
> 
> The autodefrag is not needed. We are triggering a manual defrag below, and
> that's all that's needed to trigger the issue.

-o datacow is superfluous here, as compression forces COW.

Also, this got to be the longest commit subject I had ever seen :)

  >>> len("btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size")
  115

Something shorter like "btrfs: test defrag with compressed extents" would be
perfectly fine, further details can be left in the changelog and comments in
the test case.

> 
> > +
> > +# Btrfs uses 128K as compressed extent max size, so this would result
> > +# exactly two extents, which are all at their max size
> > +$XFS_IO_PROG -f -c "pwrite -S 0xee 0 128k" -c sync \
> > +		-c "pwrite -S 0xff 128k 128k" -c sync \
> > +		$SCRATCH_MNT/foobar >> $seqres.full
> 
> We don't need to do a sync after every write. If you write 256K at once,
> it will result in 2 128K extents anyway. The comment and the way we are
> calling xfs_io gives the wrong idea that user space can influence the max
> extent size.
> 
> A more interesting test would be, say, to write 2M or 4M at once for
> example, which will result in many 128K extents. It would also make the
> test more robust in case the default defrag threshold changes one day
> for some reason (e.g. btrfs-progs might decide to start calling the
> ioctl with a higher threshold one day). And then just check that the
> output of fiemap is the same before and after the defrag attempt, so
> it's not even necessary to manually compare the sectors of each
> extent and use get_extent_disk_sector().
> 
> Thanks.
> 
> > +
> > +old_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
> > +old_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
> > +old_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
> > +
> > +echo "=== File extent layout before defrag ===" >> $seqres.full
> > +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
> > +
> > +$BTRFS_UTIL_PROG filesystem defrag "$SCRATCH_MNT/foobar" >> $seqres.full
> > +
> > +new_csum=$(_md5_checksum $SCRATCH_MNT/foobar)
> > +new_extent1=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 0)
> > +new_extent2=$(get_extent_disk_sector "$SCRATCH_MNT/foobar" 128k)
> > +
> > +echo "=== File extent layout before defrag ===" >> $seqres.full
> > +$XFS_IO_PROG -c "fiemap -v" "$SCRATCH_MNT/foobar" >> $seqres.full
> > +
> > +if [ $new_csum != $old_csum ]; then
> > +	echo "file content changed"
> > +fi
> > +
> > +if [ $new_extent1 != $old_extent1 ]; then
> > +	echo "the first extent get defragged"
> > +fi
> > +
> > +if [ $new_extent2 != $old_extent2 ]; then
> > +	echo "the second extent get defragged"
> > +fi
> > +
> > +echo "Silence is golden"
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/btrfs/257.out b/tests/btrfs/257.out
> > new file mode 100644
> > index 00000000..cc3693f3
> > --- /dev/null
> > +++ b/tests/btrfs/257.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 257
> > +Silence is golden
> > -- 
> > 2.34.1
> > 

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Ritesh Harjani]

On 22/01/27 01:53PM, Qu Wenruo wrote:
> There is a long existing bug in btrfs defrag code that it will always
> try to defrag compressed extents, even they are already at max capacity.
>
> This will not reduce the number of extents, but only waste IO/CPU.
>
> The kernel fix is titled:
>
>   btrfs: defrag: don't defrag extents which is already at its max capacity
>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
>  tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/257.out |  2 ++
>  2 files changed, 81 insertions(+)
>  create mode 100755 tests/btrfs/257
>  create mode 100644 tests/btrfs/257.out
>
> diff --git a/tests/btrfs/257 b/tests/btrfs/257
> new file mode 100755
> index 00000000..326687dc
> --- /dev/null
> +++ b/tests/btrfs/257
> @@ -0,0 +1,79 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test 257
> +#
> +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
> +# at their max capacity.

Haven't really looked into this fstest. But it is a good practice to add the
commit id and the title here for others to easily refer kernel commit.

-ritesh

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Qu Wenruo]

On 2022/1/27 23:38, Ritesh Harjani wrote:
> On 22/01/27 01:53PM, Qu Wenruo wrote:
>> There is a long existing bug in btrfs defrag code that it will always
>> try to defrag compressed extents, even they are already at max capacity.
>>
>> This will not reduce the number of extents, but only waste IO/CPU.
>>
>> The kernel fix is titled:
>>
>>    btrfs: defrag: don't defrag extents which is already at its max capacity
>>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>>   tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
>>   tests/btrfs/257.out |  2 ++
>>   2 files changed, 81 insertions(+)
>>   create mode 100755 tests/btrfs/257
>>   create mode 100644 tests/btrfs/257.out
>>
>> diff --git a/tests/btrfs/257 b/tests/btrfs/257
>> new file mode 100755
>> index 00000000..326687dc
>> --- /dev/null
>> +++ b/tests/btrfs/257
>> @@ -0,0 +1,79 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
>> +#
>> +# FS QA Test 257
>> +#
>> +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
>> +# at their max capacity.
>
> Haven't really looked into this fstest. But it is a good practice to add the
> commit id and the title here for others to easily refer kernel commit.

Isn't that already in the commit message?

Thanks,
Qu
>
> -ritesh
>

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Ritesh Harjani]

On 22/01/28 06:20AM, Qu Wenruo wrote:
>
>
> On 2022/1/27 23:38, Ritesh Harjani wrote:
> > On 22/01/27 01:53PM, Qu Wenruo wrote:
> > > There is a long existing bug in btrfs defrag code that it will always
> > > try to defrag compressed extents, even they are already at max capacity.
> > >
> > > This will not reduce the number of extents, but only waste IO/CPU.
> > >
> > > The kernel fix is titled:
> > >
> > >    btrfs: defrag: don't defrag extents which is already at its max capacity
> > >
> > > Signed-off-by: Qu Wenruo <wqu@suse.com>
> > > ---
> > >   tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
> > >   tests/btrfs/257.out |  2 ++
> > >   2 files changed, 81 insertions(+)
> > >   create mode 100755 tests/btrfs/257
> > >   create mode 100644 tests/btrfs/257.out
> > >
> > > diff --git a/tests/btrfs/257 b/tests/btrfs/257
> > > new file mode 100755
> > > index 00000000..326687dc
> > > --- /dev/null
> > > +++ b/tests/btrfs/257
> > > @@ -0,0 +1,79 @@
> > > +#! /bin/bash
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
> > > +#
> > > +# FS QA Test 257
> > > +#
> > > +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
> > > +# at their max capacity.
> >
> > Haven't really looked into this fstest. But it is a good practice to add the
> > commit id and the title here for others to easily refer kernel commit.
>
> Isn't that already in the commit message?

Yes, that's true. And thanks for adding that.
I generally found mentioning commit-id and commit-title
in the description section of the test too to be lot more helpful.

For e.g. tests/btrfs/232

# FS QA Test 232
#
# Test that performing io and exhausting qgroup limit won't deadlock. This
# exercises issues fixed by the following kernel commits:
#
# 4f6a49de64fd ("btrfs: unlock extents in btrfs_zero_range in case of quota
# reservation errors")
# 4d14c5cde5c2 ("btrfs: don't flush from btrfs_delayed_inode_reserve_metadata")

Though I don't think it is mandatory, but as I said, it is generally helpful
for anyone to refer to commit directly / title directly from here if it has
a commit-id (might be it's just me :))

Thanks!
-ritesh

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Qu Wenruo]

On 2022/1/28 10:56, Ritesh Harjani wrote:
> On 22/01/28 06:20AM, Qu Wenruo wrote:
>>
>>
>> On 2022/1/27 23:38, Ritesh Harjani wrote:
>>> On 22/01/27 01:53PM, Qu Wenruo wrote:
>>>> There is a long existing bug in btrfs defrag code that it will always
>>>> try to defrag compressed extents, even they are already at max capacity.
>>>>
>>>> This will not reduce the number of extents, but only waste IO/CPU.
>>>>
>>>> The kernel fix is titled:
>>>>
>>>>     btrfs: defrag: don't defrag extents which is already at its max capacity
>>>>
>>>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>>>> ---
>>>>    tests/btrfs/257     | 79 +++++++++++++++++++++++++++++++++++++++++++++
>>>>    tests/btrfs/257.out |  2 ++
>>>>    2 files changed, 81 insertions(+)
>>>>    create mode 100755 tests/btrfs/257
>>>>    create mode 100644 tests/btrfs/257.out
>>>>
>>>> diff --git a/tests/btrfs/257 b/tests/btrfs/257
>>>> new file mode 100755
>>>> index 00000000..326687dc
>>>> --- /dev/null
>>>> +++ b/tests/btrfs/257
>>>> @@ -0,0 +1,79 @@
>>>> +#! /bin/bash
>>>> +# SPDX-License-Identifier: GPL-2.0
>>>> +# Copyright (C) 2022 SUSE Linux Products GmbH. All Rights Reserved.
>>>> +#
>>>> +# FS QA Test 257
>>>> +#
>>>> +# Make sure btrfs defrag ioctl won't defrag compressed extents which are already
>>>> +# at their max capacity.
>>>
>>> Haven't really looked into this fstest. But it is a good practice to add the
>>> commit id and the title here for others to easily refer kernel commit.
>>
>> Isn't that already in the commit message?
>
> Yes, that's true. And thanks for adding that.
> I generally found mentioning commit-id and commit-title
> in the description section of the test too to be lot more helpful.

This is in fact discussed before, I used to include the fixes in the
test description, but later move them into the commit message.

In the long run, the test should and would all pass, thus there is
really no need to bother mentioning it.

For the guys who really need to bother the test failure, aka QA testers
or some developers in the future causing some regression, they will
check the full commit messages anyway.

And the fixes tag has its own problems, like at the time of fstests
merging, the fixes may not yet being merged into mainline, or the title
may change.

Thus mentioning something volatile in the test description can be a
little confusing, and hiding it into the commit message may be preferred.

Thanks,
Qu

>
> For e.g. tests/btrfs/232
>
> # FS QA Test 232
> #
> # Test that performing io and exhausting qgroup limit won't deadlock. This
> # exercises issues fixed by the following kernel commits:
> #
> # 4f6a49de64fd ("btrfs: unlock extents in btrfs_zero_range in case of quota
> # reservation errors")
> # 4d14c5cde5c2 ("btrfs: don't flush from btrfs_delayed_inode_reserve_metadata")
>
> Though I don't think it is mandatory, but as I said, it is generally helpful
> for anyone to refer to commit directly / title directly from here if it has
> a commit-id (might be it's just me :))
>
> Thanks!
> -ritesh
>

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[David Sterba]

On Thu, Jan 27, 2022 at 01:53:06PM +0800, Qu Wenruo wrote:
> There is a long existing bug in btrfs defrag code that it will always
> try to defrag compressed extents, even they are already at max capacity.

As commended under the patch, this not considered a bug, because the
defrag ioctl is expected to reshuffle the extents, with or without
compression and improving the compression ratio if asked to recompress
with hither level. What is not perfect is the kernel side that could try
harder to merge extents into bigger contiguous chunks, but as long as
the compression is involved it's not possible to decide if the extents
should be skipped or not.

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Qu Wenruo]

On 2022/2/1 23:14, David Sterba wrote:
> On Thu, Jan 27, 2022 at 01:53:06PM +0800, Qu Wenruo wrote:
>> There is a long existing bug in btrfs defrag code that it will always
>> try to defrag compressed extents, even they are already at max capacity.
>
> As commended under the patch, this not considered a bug, because the
> defrag ioctl is expected to reshuffle the extents, with or without
> compression and improving the compression ratio if asked to recompress
> with hither level. What is not perfect is the kernel side that could try
> harder to merge extents into bigger contiguous chunks, but as long as
> the compression is involved it's not possible to decide if the extents
> should be skipped or not.

What I can do is to add extra test to make sure if "btrfs fi defrag -c"
always defrag the file no matter whatever.

To me, these two factors don't conflict with each other at all.

Thanks,
Qu

Re: [PATCH] btrfs: add test case to verify that btrfs won't waste IO/CPU to defrag compressed extents already at their max size

[Qu Wenruo]

On 2022/2/2 08:07, Qu Wenruo wrote:
>
>
> On 2022/2/1 23:14, David Sterba wrote:
>> On Thu, Jan 27, 2022 at 01:53:06PM +0800, Qu Wenruo wrote:
>>> There is a long existing bug in btrfs defrag code that it will always
>>> try to defrag compressed extents, even they are already at max capacity.
>>
>> As commended under the patch, this not considered a bug, because the
>> defrag ioctl is expected to reshuffle the extents, with or without
>> compression and improving the compression ratio if asked to recompress
>> with hither level. What is not perfect is the kernel side that could try
>> harder to merge extents into bigger contiguous chunks, but as long as
>> the compression is involved it's not possible to decide if the extents
>> should be skipped or not.
>
> What I can do is to add extra test to make sure if "btrfs fi defrag -c"
> always defrag the file no matter whatever.
>
> To me, these two factors don't conflict with each other at all.

Here comes the new test to make sure "btrfs fi defrag -c" can do what it
should do:
https://patchwork.kernel.org/project/linux-btrfs/patch/20220202083158.68262-1-wqu@suse.com/

And of course, current (with the max capacity check) kernel can pass
both tests without problem, proving those two aspects are not in
conflict at all.

Thanks,
Qu

>
> Thanks,
> Qu