* [PATCH 1/4] fstests: add missing checks of fallocate feature
2021-04-23 11:26 [PATCH 0/4] fstests: first few support patches for zoned btrfs Johannes Thumshirn
@ 2021-04-23 11:26 ` Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 2/4] btrfs: require discard functionality from scratch device Johannes Thumshirn
` (2 subsequent siblings)
3 siblings, 0 replies; 8+ messages in thread
From: Johannes Thumshirn @ 2021-04-23 11:26 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Naohiro Aota
From: Naohiro Aota <naohiro.aota@wdc.com>
Many test cases use xfs_io -c 'falloc' but forgot to add
_require_xfs_io_command "falloc". This will fail the test case if we run
the test case on a file system without fallcoate support e.g. F2FS.
While we believe that normal fallocate(mode = 0) is always supported on
Linux, it is not true. Fallocate is disabled in several implementations of
zoned block support for file systems because the pre-allocated region will
break the sequential writing rule.
Currently, several test cases unconditionally call fallocate(). Let's add
_require_xfs_io_command "falloc" to properly check the feature is supported
by a testing file system.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
---
tests/btrfs/013 | 1 +
tests/btrfs/016 | 1 +
tests/btrfs/025 | 1 +
tests/btrfs/034 | 1 +
tests/btrfs/037 | 1 +
tests/btrfs/046 | 1 +
tests/btrfs/107 | 1 +
tests/ext4/001 | 1 +
tests/f2fs/001 | 1 +
tests/generic/456 | 1 +
tests/xfs/042 | 1 +
tests/xfs/114 | 1 +
tests/xfs/118 | 1 +
tests/xfs/331 | 1 +
tests/xfs/341 | 1 +
tests/xfs/342 | 1 +
tests/xfs/423 | 1 +
17 files changed, 17 insertions(+)
diff --git a/tests/btrfs/013 b/tests/btrfs/013
index 9252c82a2076..5e03ed4a4b4b 100755
--- a/tests/btrfs/013
+++ b/tests/btrfs/013
@@ -33,6 +33,7 @@ trap "_cleanup ; exit \$status" 0 1 2 3 15
# real QA test starts here
_supported_fs btrfs
_require_scratch
+_require_xfs_io_command "falloc"
rm -f $seqres.full
diff --git a/tests/btrfs/016 b/tests/btrfs/016
index 8fd237cbdb64..015ec17f93d6 100755
--- a/tests/btrfs/016
+++ b/tests/btrfs/016
@@ -35,6 +35,7 @@ _supported_fs btrfs
_require_test
_require_scratch
_require_fssum
+_require_xfs_io_command "falloc"
_scratch_mkfs > /dev/null 2>&1
diff --git a/tests/btrfs/025 b/tests/btrfs/025
index 42cd7cefe825..5c8140552bfb 100755
--- a/tests/btrfs/025
+++ b/tests/btrfs/025
@@ -31,6 +31,7 @@ _cleanup()
# real QA test starts here
_supported_fs btrfs
_require_scratch
+_require_xfs_io_command "falloc"
rm -f $seqres.full
diff --git a/tests/btrfs/034 b/tests/btrfs/034
index bc7a4aae3886..07c84c347d3b 100755
--- a/tests/btrfs/034
+++ b/tests/btrfs/034
@@ -28,6 +28,7 @@ _cleanup()
# real QA test starts here
_supported_fs btrfs
_require_scratch
+_require_xfs_io_command "falloc"
rm -f $seqres.full
diff --git a/tests/btrfs/037 b/tests/btrfs/037
index 1cfaf5be58c8..9ef199a93413 100755
--- a/tests/btrfs/037
+++ b/tests/btrfs/037
@@ -35,6 +35,7 @@ _cleanup()
# real QA test starts here
_supported_fs btrfs
_require_scratch
+_require_xfs_io_command "falloc"
rm -f $seqres.full
diff --git a/tests/btrfs/046 b/tests/btrfs/046
index 882db8eacc4e..a1d82e1cdd54 100755
--- a/tests/btrfs/046
+++ b/tests/btrfs/046
@@ -37,6 +37,7 @@ _cleanup()
_supported_fs btrfs
_require_test
_require_scratch
+_require_xfs_io_command "falloc"
_require_fssum
rm -f $seqres.full
diff --git a/tests/btrfs/107 b/tests/btrfs/107
index e57c9dead499..80db5ab9822d 100755
--- a/tests/btrfs/107
+++ b/tests/btrfs/107
@@ -34,6 +34,7 @@ rm -f $seqres.full
_supported_fs btrfs
_require_scratch
+_require_xfs_io_command "falloc"
# Use 64K file size to match any sectorsize
# And with a unaligned tailing range to ensure it will be at least 2 pages
diff --git a/tests/ext4/001 b/tests/ext4/001
index bbb74f1ea5bc..9650303d15b5 100755
--- a/tests/ext4/001
+++ b/tests/ext4/001
@@ -29,6 +29,7 @@ trap "_cleanup ; exit \$status" 0 1 2 3 15
# real QA test starts here
_supported_fs ext4
+_require_xfs_io_command "falloc"
_require_xfs_io_command "fzero"
_require_test
diff --git a/tests/f2fs/001 b/tests/f2fs/001
index 98bd2682d60f..0753a09b5576 100755
--- a/tests/f2fs/001
+++ b/tests/f2fs/001
@@ -36,6 +36,7 @@ _cleanup()
_supported_fs f2fs
_require_scratch
+_require_xfs_io_command "falloc"
testfile=$SCRATCH_MNT/testfile
dummyfile=$SCRATCH_MNT/dummyfile
diff --git a/tests/generic/456 b/tests/generic/456
index 2f9df5e5edc4..65667d449dd3 100755
--- a/tests/generic/456
+++ b/tests/generic/456
@@ -38,6 +38,7 @@ trap "_cleanup; exit \$status" 0 1 2 3 15
# real QA test starts here
_supported_fs generic
_require_scratch
+_require_xfs_io_command "falloc"
_require_dm_target flakey
_require_xfs_io_command "falloc" "-k"
_require_xfs_io_command "fzero"
diff --git a/tests/xfs/042 b/tests/xfs/042
index b55d642c5170..fcd5181cf590 100755
--- a/tests/xfs/042
+++ b/tests/xfs/042
@@ -31,6 +31,7 @@ trap "_cleanup ; exit \$status" 0 1 2 3 15
# real QA test starts here
_supported_fs xfs
+_require_xfs_io_command "falloc"
_require_scratch
diff --git a/tests/xfs/114 b/tests/xfs/114
index b936452461c6..3f5575a61dfb 100755
--- a/tests/xfs/114
+++ b/tests/xfs/114
@@ -32,6 +32,7 @@ _cleanup()
_supported_fs xfs
_require_test_program "punch-alternating"
_require_xfs_scratch_rmapbt
+_require_xfs_io_command "falloc"
_require_xfs_io_command "fcollapse"
_require_xfs_io_command "finsert"
diff --git a/tests/xfs/118 b/tests/xfs/118
index 5e23617b39dd..9a431821aa62 100755
--- a/tests/xfs/118
+++ b/tests/xfs/118
@@ -41,6 +41,7 @@ _supported_fs xfs
_require_scratch
_require_command "$XFS_FSR_PROG" "xfs_fsr"
+_require_xfs_io_command "falloc"
# 50M
_scratch_mkfs_sized $((50 * 1024 * 1024)) >> $seqres.full 2>&1
diff --git a/tests/xfs/331 b/tests/xfs/331
index 4ea54e2a534b..8e92b2e36a8d 100755
--- a/tests/xfs/331
+++ b/tests/xfs/331
@@ -33,6 +33,7 @@ _require_xfs_scratch_rmapbt
_require_scratch_reflink
_require_xfs_io_command "falloc"
_require_test_program "punch-alternating"
+_require_xfs_io_command "falloc"
rm -f "$seqres.full"
diff --git a/tests/xfs/341 b/tests/xfs/341
index e1fbe588d9eb..8bf05087e1ba 100755
--- a/tests/xfs/341
+++ b/tests/xfs/341
@@ -31,6 +31,7 @@ _require_realtime
_require_xfs_scratch_rmapbt
_require_test_program "punch-alternating"
_disable_dmesg_check
+_require_xfs_io_command "falloc"
rm -f "$seqres.full"
diff --git a/tests/xfs/342 b/tests/xfs/342
index 2be5f7698f01..4db222d65fb2 100755
--- a/tests/xfs/342
+++ b/tests/xfs/342
@@ -30,6 +30,7 @@ _supported_fs xfs
_require_realtime
_require_xfs_scratch_rmapbt
_require_test_program "punch-alternating"
+_require_xfs_io_command "falloc"
rm -f "$seqres.full"
diff --git a/tests/xfs/423 b/tests/xfs/423
index 8d51a9a60585..183c9cf5eded 100755
--- a/tests/xfs/423
+++ b/tests/xfs/423
@@ -35,6 +35,7 @@ _cleanup()
_supported_fs xfs
_require_test_program "punch-alternating"
_require_xfs_io_command "scrub"
+_require_xfs_io_command "falloc"
_require_scratch
echo "Format and populate"
--
2.30.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 2/4] btrfs: require discard functionality from scratch device
2021-04-23 11:26 [PATCH 0/4] fstests: first few support patches for zoned btrfs Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 1/4] fstests: add missing checks of fallocate feature Johannes Thumshirn
@ 2021-04-23 11:26 ` Johannes Thumshirn
2021-04-25 8:26 ` Eryu Guan
2021-04-23 11:26 ` [PATCH 3/4] common/rc: introduce zone check commands Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 4/4] btrfs: add test for zone auto reclaim Johannes Thumshirn
3 siblings, 1 reply; 8+ messages in thread
From: Johannes Thumshirn @ 2021-04-23 11:26 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Naohiro Aota, Johannes Thumshirn
Some test cases for btrfs require the scratch device to support discard.
Check if the scratch device does support discard before trying to execute
the test.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
common/rc | 8 ++++++++
tests/btrfs/116 | 1 +
tests/btrfs/156 | 1 +
3 files changed, 10 insertions(+)
diff --git a/common/rc b/common/rc
index 11ff7635700b..76a7265e23ba 100644
--- a/common/rc
+++ b/common/rc
@@ -3587,6 +3587,14 @@ _require_batched_discard()
$FSTRIM_PROG $1 > /dev/null 2>&1 || _notrun "FITRIM not supported on $1"
}
+_require_scratch_discard()
+{
+ local sdev="$(_short_dev $SCRATCH_DEV)"
+ local discard=$(cat /sys/block/$sdev/queue/discard_granularity)
+
+ [ $discard -gt 0 ] || _notrun "discard not supported"
+}
+
_require_dumpe2fs()
{
if [ -z "$DUMPE2FS_PROG" ]; then
diff --git a/tests/btrfs/116 b/tests/btrfs/116
index 3ed097eccf03..f4db439caef8 100755
--- a/tests/btrfs/116
+++ b/tests/btrfs/116
@@ -29,6 +29,7 @@ _cleanup()
# real QA test starts here
_supported_fs btrfs
_require_scratch
+_require_scratch_discard
rm -f $seqres.full
diff --git a/tests/btrfs/156 b/tests/btrfs/156
index 89c80e7161e2..56206d99c801 100755
--- a/tests/btrfs/156
+++ b/tests/btrfs/156
@@ -42,6 +42,7 @@ rm -f $seqres.full
_supported_fs btrfs
_require_scratch
_require_fstrim
+_require_scratch_discard
# 1024fs size
fs_size=$((1024 * 1024 * 1024))
--
2.30.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 2/4] btrfs: require discard functionality from scratch device
2021-04-23 11:26 ` [PATCH 2/4] btrfs: require discard functionality from scratch device Johannes Thumshirn
@ 2021-04-25 8:26 ` Eryu Guan
0 siblings, 0 replies; 8+ messages in thread
From: Eryu Guan @ 2021-04-25 8:26 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: fstests, linux-btrfs, Naohiro Aota
On Fri, Apr 23, 2021 at 08:26:32PM +0900, Johannes Thumshirn wrote:
> Some test cases for btrfs require the scratch device to support discard.
> Check if the scratch device does support discard before trying to execute
> the test.
>
> Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
> ---
> common/rc | 8 ++++++++
> tests/btrfs/116 | 1 +
> tests/btrfs/156 | 1 +
> 3 files changed, 10 insertions(+)
>
> diff --git a/common/rc b/common/rc
> index 11ff7635700b..76a7265e23ba 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -3587,6 +3587,14 @@ _require_batched_discard()
> $FSTRIM_PROG $1 > /dev/null 2>&1 || _notrun "FITRIM not supported on $1"
> }
>
> +_require_scratch_discard()
> +{
> + local sdev="$(_short_dev $SCRATCH_DEV)"
> + local discard=$(cat /sys/block/$sdev/queue/discard_granularity)
> +
> + [ $discard -gt 0 ] || _notrun "discard not supported"
> +}
> +
> _require_dumpe2fs()
> {
> if [ -z "$DUMPE2FS_PROG" ]; then
> diff --git a/tests/btrfs/116 b/tests/btrfs/116
> index 3ed097eccf03..f4db439caef8 100755
> --- a/tests/btrfs/116
> +++ b/tests/btrfs/116
> @@ -29,6 +29,7 @@ _cleanup()
> # real QA test starts here
> _supported_fs btrfs
> _require_scratch
> +_require_scratch_discard
>
> rm -f $seqres.full
>
> diff --git a/tests/btrfs/156 b/tests/btrfs/156
> index 89c80e7161e2..56206d99c801 100755
> --- a/tests/btrfs/156
> +++ b/tests/btrfs/156
> @@ -42,6 +42,7 @@ rm -f $seqres.full
> _supported_fs btrfs
> _require_scratch
> _require_fstrim
> +_require_scratch_discard
These two tests already have _require_batched_discard, which will make
sure discard is supported by the device, by actually doing fstrim on
$SCRATCH_MNT. I think that should do the work?
Thanks,
Eryu
>
> # 1024fs size
> fs_size=$((1024 * 1024 * 1024))
> --
> 2.30.0
^ permalink raw reply [flat|nested] 8+ messages in thread
* [PATCH 3/4] common/rc: introduce zone check commands
2021-04-23 11:26 [PATCH 0/4] fstests: first few support patches for zoned btrfs Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 1/4] fstests: add missing checks of fallocate feature Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 2/4] btrfs: require discard functionality from scratch device Johannes Thumshirn
@ 2021-04-23 11:26 ` Johannes Thumshirn
2021-04-23 11:26 ` [PATCH 4/4] btrfs: add test for zone auto reclaim Johannes Thumshirn
3 siblings, 0 replies; 8+ messages in thread
From: Johannes Thumshirn @ 2021-04-23 11:26 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Naohiro Aota
From: Naohiro Aota <naohiro.aota@wdc.com>
Introduce some zone related helper functions: _zone_type(),
_require_zoned_device(), and _require_non_zoned_device(). They all take a
device path as an argument.
_zone_type() return the zone type of the device according to the value
returned from "/sys/block/<disk>/queue/zoned". See
Documentation/ABI/testing/sysfs-block for a detail.
_require_zoned_device() checks if the device is zoned. If not, it skips the
current test. _require_non_zoned_device() does the opposite.
Signed-off-by: Naohiro Aota <naohiro.aota@wdc.com>
---
common/rc | 44 ++++++++++++++++++++++++++++++++++++++++++++
1 file changed, 44 insertions(+)
diff --git a/common/rc b/common/rc
index 76a7265e23ba..18436e33dcf3 100644
--- a/common/rc
+++ b/common/rc
@@ -1931,6 +1931,50 @@ _require_dm_target()
fi
}
+_zone_type()
+{
+ local target=$1
+ if [ -z $target ]; then
+ echo "Usage: _zone_type <device>"
+ exit 1
+ fi
+ local sdev=`_short_dev $target`
+
+ if [ -e /sys/block/${sdev}/queue/zoned ]; then
+ cat /sys/block/${sdev}/queue/zoned
+ else
+ echo none
+ fi
+}
+
+_require_zoned_device()
+{
+ local target=$1
+ if [ -z $target ]; then
+ echo "Usage: _require_zoned_device <device>"
+ exit 1
+ fi
+
+ local type=`_zone_type ${target}`
+ if [ "${type}" = "none" ]; then
+ _notrun "this test require zoned block device"
+ fi
+}
+
+_require_non_zoned_device()
+{
+ local target=$1
+ if [ -z $target ]; then
+ echo "Usage: _require_non_zoned_device <device>"
+ exit 1
+ fi
+
+ local type=`_zone_type ${target}`
+ if [ "${type}" != "none" ]; then
+ _notrun "this test require non-zoned block device"
+ fi
+}
+
# this test requires the ext4 kernel support crc feature on scratch device
#
_require_scratch_ext4_crc()
--
2.30.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* [PATCH 4/4] btrfs: add test for zone auto reclaim
2021-04-23 11:26 [PATCH 0/4] fstests: first few support patches for zoned btrfs Johannes Thumshirn
` (2 preceding siblings ...)
2021-04-23 11:26 ` [PATCH 3/4] common/rc: introduce zone check commands Johannes Thumshirn
@ 2021-04-23 11:26 ` Johannes Thumshirn
2021-04-25 8:33 ` Eryu Guan
3 siblings, 1 reply; 8+ messages in thread
From: Johannes Thumshirn @ 2021-04-23 11:26 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Naohiro Aota, Johannes Thumshirn
Add a test for the patch titled "btrfs: zoned: automatically reclaim
zones".
This test creates a two file on a newly created FS in a way that when we
delete the first one, an auto reclaim process will be triggered by the FS.
After the reclaim process, it verifies that the data was moved to another
zone and old zone was successfully reset.
Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
---
common/config | 1 +
tests/btrfs/236 | 102 ++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/236.out | 2 +
tests/btrfs/group | 1 +
4 files changed, 106 insertions(+)
create mode 100755 tests/btrfs/236
create mode 100644 tests/btrfs/236.out
diff --git a/common/config b/common/config
index a47e462c7792..1a26934985dd 100644
--- a/common/config
+++ b/common/config
@@ -226,6 +226,7 @@ export FSVERITY_PROG="$(type -P fsverity)"
export OPENSSL_PROG="$(type -P openssl)"
export ACCTON_PROG="$(type -P accton)"
export E2IMAGE_PROG="$(type -P e2image)"
+export BLKZONE_PROG="$(type -P blkzone)"
# use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
# newer systems have udevadm command but older systems like RHEL5 don't.
diff --git a/tests/btrfs/236 b/tests/btrfs/236
new file mode 100755
index 000000000000..18964699f68e
--- /dev/null
+++ b/tests/btrfs/236
@@ -0,0 +1,102 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Western Digital Corporation. All Rights Reserved.
+#
+# FS QA Test 236
+#
+# Test that zone autoreclaim works as expected, that is: if the dirty
+# threashold is exceeded the data gets relocated to new block group and the
+# old block group gets deleted. On block group deletion, the underlying device
+# zone also needs to be reset.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_require_scratch
+_require_btrfs_command inspect-internal dump-tree
+_require_command "$BLKZONE_PROG" blkzone
+_require_zoned_device "$SCRATCH_DEV"
+
+get_data_bg()
+{
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t CHUNK $SCRATCH_DEV |\
+ grep -A 1 "CHUNK_ITEM" | grep -B 1 "type DATA" |\
+ grep -Eo "CHUNK_ITEM [[:digit:]]+" | cut -d ' ' -f 2
+}
+
+zonesize=$(cat /sys/block/$(_short_dev $SCRATCH_DEV)/queue/chunk_sectors)
+zonesize=$((zonesize << 9))
+
+_scratch_mkfs >/dev/null 2>&1
+_scratch_mount -o commit=5
+
+uuid=$(findmnt -n -o UUID "$SCRATCH_MNT")
+reclaim_threshold=75
+echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
+fill_percent=$((reclaim_threshold + 2))
+rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
+fill_size=$((zonesize * fill_percent / 100))
+rest=$((zonesize * rest_percent / 100))
+
+# step 1, fill FS over $fillsize
+$XFS_IO_PROG -fdc "pwrite -W 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
+$XFS_IO_PROG -fdc "pwrite -W 0 $rest" $SCRATCH_MNT/$seq.test2 >> $seqres.full
+sleep 5
+
+zones_before=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
+echo "Before reclaim: $zones_before zones open" >> $seqres.full
+old_data_zone=$(get_data_bg)
+old_data_zone=$((old_data_zone >> 9))
+printf "Old data zone 0x%x\n" $old_data_zone >> $seqres.full
+
+# step 2, delete the 1st $fill_size sized file to trigger reclaim
+rm $SCRATCH_MNT/$seq.test1
+$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
+sleep 10
+
+# check that we don't have more zones open than before
+zones_after=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
+echo "After reclaim: $zones_after zones open" >> $seqres.full
+
+# Check that old data zone was reset
+old_wptr=$($BLKZONE_PROG report -o $old_data_zone -c 1 $SCRATCH_DEV |\
+ grep -Eo "wptr 0x[[:xdigit:]]+" | cut -d ' ' -f 2)
+if [ "$old_wptr" != "0x000000" ]; then
+ _fail "Old wptr still at $old_wptr"
+fi
+
+new_data_zone=$(get_data_bg)
+new_data_zone=$((new_data_zone >> 9))
+printf "New data zone 0x%x\n" $new_data_zone >> $seqres.full
+
+# Check that data was really relocated to a different zone
+if [ $old_data_zone -eq $new_data_zone ]; then
+ _fail "New zone same as old zone"
+fi
+
+# success, all done
+echo "Silence is golden"
+status=0
+exit
diff --git a/tests/btrfs/236.out b/tests/btrfs/236.out
new file mode 100644
index 000000000000..b6b6e0cad9a7
--- /dev/null
+++ b/tests/btrfs/236.out
@@ -0,0 +1,2 @@
+QA output created by 236
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 331dd432fac3..62c9c761e974 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -238,3 +238,4 @@
233 auto quick subvolume
234 auto quick compress rw
235 auto quick send
+236 auto quick balance
--
2.30.0
^ permalink raw reply related [flat|nested] 8+ messages in thread* Re: [PATCH 4/4] btrfs: add test for zone auto reclaim
2021-04-23 11:26 ` [PATCH 4/4] btrfs: add test for zone auto reclaim Johannes Thumshirn
@ 2021-04-25 8:33 ` Eryu Guan
2021-04-26 6:08 ` Johannes Thumshirn
0 siblings, 1 reply; 8+ messages in thread
From: Eryu Guan @ 2021-04-25 8:33 UTC (permalink / raw)
To: Johannes Thumshirn; +Cc: fstests, linux-btrfs, Naohiro Aota
On Fri, Apr 23, 2021 at 08:26:34PM +0900, Johannes Thumshirn wrote:
> Add a test for the patch titled "btrfs: zoned: automatically reclaim
> zones".
>
> This test creates a two file on a newly created FS in a way that when we
> delete the first one, an auto reclaim process will be triggered by the FS.
>
> After the reclaim process, it verifies that the data was moved to another
> zone and old zone was successfully reset.
>
> Signed-off-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Test looks fine to me from fstests' perspective, some minor issues
inline. But I'm not familiar with zoned btrfs, I'd like btrfs folks help
review the patch as well.
> ---
> common/config | 1 +
> tests/btrfs/236 | 102 ++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/236.out | 2 +
> tests/btrfs/group | 1 +
> 4 files changed, 106 insertions(+)
> create mode 100755 tests/btrfs/236
> create mode 100644 tests/btrfs/236.out
>
> diff --git a/common/config b/common/config
> index a47e462c7792..1a26934985dd 100644
> --- a/common/config
> +++ b/common/config
> @@ -226,6 +226,7 @@ export FSVERITY_PROG="$(type -P fsverity)"
> export OPENSSL_PROG="$(type -P openssl)"
> export ACCTON_PROG="$(type -P accton)"
> export E2IMAGE_PROG="$(type -P e2image)"
> +export BLKZONE_PROG="$(type -P blkzone)"
>
> # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
> # newer systems have udevadm command but older systems like RHEL5 don't.
> diff --git a/tests/btrfs/236 b/tests/btrfs/236
> new file mode 100755
> index 000000000000..18964699f68e
> --- /dev/null
> +++ b/tests/btrfs/236
> @@ -0,0 +1,102 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Western Digital Corporation. All Rights Reserved.
> +#
> +# FS QA Test 236
> +#
> +# Test that zone autoreclaim works as expected, that is: if the dirty
> +# threashold is exceeded the data gets relocated to new block group and the
> +# old block group gets deleted. On block group deletion, the underlying device
> +# zone also needs to be reset.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +_supported_fs btrfs
> +_require_scratch
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$BLKZONE_PROG" blkzone
> +_require_zoned_device "$SCRATCH_DEV"
> +
> +get_data_bg()
> +{
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t CHUNK $SCRATCH_DEV |\
> + grep -A 1 "CHUNK_ITEM" | grep -B 1 "type DATA" |\
> + grep -Eo "CHUNK_ITEM [[:digit:]]+" | cut -d ' ' -f 2
> +}
> +
> +zonesize=$(cat /sys/block/$(_short_dev $SCRATCH_DEV)/queue/chunk_sectors)
> +zonesize=$((zonesize << 9))
> +
> +_scratch_mkfs >/dev/null 2>&1
> +_scratch_mount -o commit=5
Any reason to use "-o commit=5" mount option? A comment would be good.
> +
> +uuid=$(findmnt -n -o UUID "$SCRATCH_MNT")
> +reclaim_threshold=75
> +echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
> +fill_percent=$((reclaim_threshold + 2))
> +rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
> +fill_size=$((zonesize * fill_percent / 100))
> +rest=$((zonesize * rest_percent / 100))
> +
> +# step 1, fill FS over $fillsize
> +$XFS_IO_PROG -fdc "pwrite -W 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
> +$XFS_IO_PROG -fdc "pwrite -W 0 $rest" $SCRATCH_MNT/$seq.test2 >> $seqres.full
> +sleep 5
> +
> +zones_before=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
> +echo "Before reclaim: $zones_before zones open" >> $seqres.full
> +old_data_zone=$(get_data_bg)
> +old_data_zone=$((old_data_zone >> 9))
> +printf "Old data zone 0x%x\n" $old_data_zone >> $seqres.full
> +
> +# step 2, delete the 1st $fill_size sized file to trigger reclaim
> +rm $SCRATCH_MNT/$seq.test1
> +$BTRFS_UTIL_PROG filesystem sync $SCRATCH_MNT
> +sleep 10
Why are above two sleeps needed? They make test time longer. Remove them
if they're not required, otherwise adding some comments would be good.
> +
> +# check that we don't have more zones open than before
> +zones_after=$($BLKZONE_PROG report $SCRATCH_DEV | grep -v -e em -e nw | wc -l)
> +echo "After reclaim: $zones_after zones open" >> $seqres.full
> +
> +# Check that old data zone was reset
> +old_wptr=$($BLKZONE_PROG report -o $old_data_zone -c 1 $SCRATCH_DEV |\
> + grep -Eo "wptr 0x[[:xdigit:]]+" | cut -d ' ' -f 2)
> +if [ "$old_wptr" != "0x000000" ]; then
> + _fail "Old wptr still at $old_wptr"
> +fi
> +
> +new_data_zone=$(get_data_bg)
> +new_data_zone=$((new_data_zone >> 9))
> +printf "New data zone 0x%x\n" $new_data_zone >> $seqres.full
> +
> +# Check that data was really relocated to a different zone
> +if [ $old_data_zone -eq $new_data_zone ]; then
> + _fail "New zone same as old zone"
> +fi
> +
> +# success, all done
> +echo "Silence is golden"
> +status=0
> +exit
> diff --git a/tests/btrfs/236.out b/tests/btrfs/236.out
> new file mode 100644
> index 000000000000..b6b6e0cad9a7
> --- /dev/null
> +++ b/tests/btrfs/236.out
> @@ -0,0 +1,2 @@
> +QA output created by 236
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 331dd432fac3..62c9c761e974 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -238,3 +238,4 @@
> 233 auto quick subvolume
> 234 auto quick compress rw
> 235 auto quick send
> +236 auto quick balance
I don't see a balance operation, does it fit into balance group?
Thanks,
Eryu
P.S. I've applied the first patch in this patchset.
> --
> 2.30.0
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: [PATCH 4/4] btrfs: add test for zone auto reclaim
2021-04-25 8:33 ` Eryu Guan
@ 2021-04-26 6:08 ` Johannes Thumshirn
0 siblings, 0 replies; 8+ messages in thread
From: Johannes Thumshirn @ 2021-04-26 6:08 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Naohiro Aota
On 25/04/2021 10:33, Eryu Guan wrote:
[...]
>> +_scratch_mkfs >/dev/null 2>&1
>> +_scratch_mount -o commit=5
>
> Any reason to use "-o commit=5" mount option? A comment would be good.
>
This is to increase the transaction frequency.
[...]
>
> Why are above two sleeps needed? They make test time longer. Remove them
> if they're not required, otherwise adding some comments would be good.
We need the sleeps to ensure, that the data really has reached the disks,
i.e. a transaction has happened. Otherwise we won't see a) that the zone
states have changed on disk and b) zoned btrfs won't transfer deleted
bytes to zone unusable bytes, which trigger the reclaim. I'll add comments
for both.
[...]
>> +236 auto quick balance
>
> I don't see a balance operation, does it fit into balance group?
Autoreclaim internally triggers a balance of one block group, so I thought
it would be a good fit.
Hope I could clarify some things.
> Thanks,
> Eryu
>
> P.S. I've applied the first patch in this patchset.
Thanks :)
^ permalink raw reply [flat|nested] 8+ messages in thread