[PATCH] btrfs: add a test case to verify scrub speed throttle works

[PATCH] btrfs: add a test case to verify scrub speed throttle works

[Qu Wenruo]

We introduced scrub speed throttle in commit eb3b50536642 ("btrfs: scrub:
per-device bandwidth control"),  but it is not that well documented
(e.g. what's the unit of the sysfs interface), nor tested by any test
case.

This patch will add a test case for this functionality.

The test case itself is pretty straightforward:

- Fill the fs with 2G file as scrub workload
- Scrub without any throttle to grab the initial speed
- Set the throttle to half of the initial speed
- Scrub again and check the speed against the throttle

The test case has an assumption that we can exclusively use all the
performance of the underlying disk.
But for cloud environment it's not ensured 100%, thus the test case is
not included in auto group to avoid false alerts.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
Changelog:
v2:
- Instead of a hardcoded speed, run scrub to grab the performance and
  set the throttle to half of the original speed
  This reduced the test runtime from 60s to 30s on a SATA SSD.

- Use "btrfs scrub status" to grab raw scrub speed
  The output of "btrfs scrub start -B" can not be switched to raw mode,
  which makes later parsing harder.
---
 tests/btrfs/282     | 92 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/282.out |  3 ++
 2 files changed, 95 insertions(+)
 create mode 100755 tests/btrfs/282
 create mode 100644 tests/btrfs/282.out

diff --git a/tests/btrfs/282 b/tests/btrfs/282
new file mode 100755
index 00000000..78b56528
--- /dev/null
+++ b/tests/btrfs/282
@@ -0,0 +1,92 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 282
+#
+# Make sure scrub speed limitation works as expected.
+#
+. ./common/preamble
+_begin_fstest scrub
+
+# Override the default cleanup function.
+# _cleanup()
+# {
+# 	cd /
+# 	rm -r -f $tmp.*
+# }
+
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_wants_kernel_commit eb3b50536642 \
+	"btrfs: scrub: per-device bandwidth control"
+
+# We want at least 5G for the scratch device.
+_require_scratch_size $(( 5 * 1024 * 1024))
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+uuid=$(findmnt -n -o UUID $SCRATCH_MNT)
+
+devinfo_dir="/sys/fs/btrfs/${uuid}/devinfo/1"
+
+# Check if we have the sysfs interface first.
+if [ ! -f "${devinfo_dir}/scrub_speed_max" ]; then
+	_notrun "No sysfs interface for scrub speed throttle"
+fi
+
+# Create a 2G file for later scrub workload.
+# The 2G size is chosen to fit even DUP on a 5G disk.
+$XFS_IO_PROG -f -c "pwrite -i /dev/urandom 0 2G" $SCRATCH_MNT/file | _filter_xfs_io
+
+# Writeback above data, as scrub only verify the committed data.
+sync
+
+# The first scrub, mostly to grab the speed of the scrub.
+$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >> $seqres.full
+
+# We grab the rate from "scrub status" which supports raw bytes reporting
+#
+# The output looks like this:
+# UUID:             62eaabc5-93e8-445f-b8a7-6f027934aea7
+# Scrub started:    Thu Jan  5 14:59:12 2023
+# Status:           finished
+# Duration:         0:00:02
+# Total to scrub:   1076166656
+# Rate:             538083328/s
+# Error summary:    no errors found
+#
+# What we care is that Rate line.
+init_speed=$($BTRFS_UTIL_PROG scrub status --raw $SCRATCH_MNT | grep "Rate:" |\
+	     $AWK_PROG '{print $2}' | cut -f1 -d\/)
+
+# This can happen for older progs
+if [ -z "$init_speed" ]; then
+	_notrun "btrfs-progs doesn't support scrub rate reporting"
+fi
+
+# Cycle mount to drop any possible cache.
+_scratch_cycle_mount
+
+target_speed=$(( $init_speed / 2 ))
+echo "$target_speed" > "${devinfo_dir}/scrub_speed_max"
+
+# The second scrub, to check the throttled speed.
+$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >> $seqres.full
+speed=$($BTRFS_UTIL_PROG scrub status --raw $SCRATCH_MNT | grep "Rate:" |\
+	     $AWK_PROG '{print $2}' | cut -f1 -d\/)
+
+# We gave a +- 10% tolerance for the throttle
+if [ "$speed" -gt "$(( $target_speed * 11 / 10 ))" -o \
+     "$speed" -lt "$(( $target_speed * 9 / 10))" ]; then
+	echo "scrub speed $speed Bytes/s is not properly throttled, target is $target_speed Bytes/s"
+fi
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/282.out b/tests/btrfs/282.out
new file mode 100644
index 00000000..8d53e7eb
--- /dev/null
+++ b/tests/btrfs/282.out
@@ -0,0 +1,3 @@
+QA output created by 282
+wrote 2147483648/2147483648 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.39.0

Re: [PATCH] btrfs: add a test case to verify scrub speed throttle works

[Anand Jain]

On 1/5/23 15:18, Qu Wenruo wrote:
> We introduced scrub speed throttle in commit eb3b50536642 ("btrfs: scrub:
> per-device bandwidth control"),  but it is not that well documented
> (e.g. what's the unit of the sysfs interface), nor tested by any test
> case.
> 
> This patch will add a test case for this functionality.
> 
> The test case itself is pretty straightforward:
> 
> - Fill the fs with 2G file as scrub workload
> - Scrub without any throttle to grab the initial speed
> - Set the throttle to half of the initial speed
> - Scrub again and check the speed against the throttle
> 
> The test case has an assumption that we can exclusively use all the
> performance of the underlying disk.
> But for cloud environment it's not ensured 100%, thus the test case is
> not included in auto group to avoid false alerts.
> 
> Signed-off-by: Qu Wenruo <wqu@suse.com>

LGTM.

Reviewed-by: Anand Jain <anand.jain@oralce.com>

-

> ---
> Changelog:
> v2:
> - Instead of a hardcoded speed, run scrub to grab the performance and
>    set the throttle to half of the original speed
>    This reduced the test runtime from 60s to 30s on a SATA SSD.
> 
> - Use "btrfs scrub status" to grab raw scrub speed
>    The output of "btrfs scrub start -B" can not be switched to raw mode,
>    which makes later parsing harder.
> ---
>   tests/btrfs/282     | 92 +++++++++++++++++++++++++++++++++++++++++++++
>   tests/btrfs/282.out |  3 ++
>   2 files changed, 95 insertions(+)
>   create mode 100755 tests/btrfs/282
>   create mode 100644 tests/btrfs/282.out
> 
> diff --git a/tests/btrfs/282 b/tests/btrfs/282
> new file mode 100755
> index 00000000..78b56528
> --- /dev/null
> +++ b/tests/btrfs/282
> @@ -0,0 +1,92 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# FS QA Test 282
> +#
> +# Make sure scrub speed limitation works as expected.
> +#
> +. ./common/preamble
> +_begin_fstest scrub
> +
> +# Override the default cleanup function.
> +# _cleanup()
> +# {
> +# 	cd /
> +# 	rm -r -f $tmp.*
> +# }
> +
> +. ./common/filter
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_wants_kernel_commit eb3b50536642 \
> +	"btrfs: scrub: per-device bandwidth control"
> +
> +# We want at least 5G for the scratch device.
> +_require_scratch_size $(( 5 * 1024 * 1024))
> +
> +_scratch_mkfs >> $seqres.full 2>&1
> +_scratch_mount
> +
> +uuid=$(findmnt -n -o UUID $SCRATCH_MNT)
> +
> +devinfo_dir="/sys/fs/btrfs/${uuid}/devinfo/1"
> +
> +# Check if we have the sysfs interface first.
> +if [ ! -f "${devinfo_dir}/scrub_speed_max" ]; then
> +	_notrun "No sysfs interface for scrub speed throttle"
> +fi
> +
> +# Create a 2G file for later scrub workload.
> +# The 2G size is chosen to fit even DUP on a 5G disk.
> +$XFS_IO_PROG -f -c "pwrite -i /dev/urandom 0 2G" $SCRATCH_MNT/file | _filter_xfs_io
> +
> +# Writeback above data, as scrub only verify the committed data.
> +sync
> +
> +# The first scrub, mostly to grab the speed of the scrub.
> +$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >> $seqres.full
> +
> +# We grab the rate from "scrub status" which supports raw bytes reporting
> +#
> +# The output looks like this:
> +# UUID:             62eaabc5-93e8-445f-b8a7-6f027934aea7
> +# Scrub started:    Thu Jan  5 14:59:12 2023
> +# Status:           finished
> +# Duration:         0:00:02
> +# Total to scrub:   1076166656
> +# Rate:             538083328/s
> +# Error summary:    no errors found
> +#
> +# What we care is that Rate line.
> +init_speed=$($BTRFS_UTIL_PROG scrub status --raw $SCRATCH_MNT | grep "Rate:" |\
> +	     $AWK_PROG '{print $2}' | cut -f1 -d\/)
> +
> +# This can happen for older progs
> +if [ -z "$init_speed" ]; then
> +	_notrun "btrfs-progs doesn't support scrub rate reporting"
> +fi
> +
> +# Cycle mount to drop any possible cache.
> +_scratch_cycle_mount
> +
> +target_speed=$(( $init_speed / 2 ))
> +echo "$target_speed" > "${devinfo_dir}/scrub_speed_max"
> +
> +# The second scrub, to check the throttled speed.
> +$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT >> $seqres.full
> +speed=$($BTRFS_UTIL_PROG scrub status --raw $SCRATCH_MNT | grep "Rate:" |\
> +	     $AWK_PROG '{print $2}' | cut -f1 -d\/)
> +
> +# We gave a +- 10% tolerance for the throttle
> +if [ "$speed" -gt "$(( $target_speed * 11 / 10 ))" -o \
> +     "$speed" -lt "$(( $target_speed * 9 / 10))" ]; then
> +	echo "scrub speed $speed Bytes/s is not properly throttled, target is $target_speed Bytes/s"
> +fi
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/282.out b/tests/btrfs/282.out
> new file mode 100644
> index 00000000..8d53e7eb
> --- /dev/null
> +++ b/tests/btrfs/282.out
> @@ -0,0 +1,3 @@
> +QA output created by 282
> +wrote 2147483648/2147483648 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)

Re: [PATCH] btrfs: add a test case to verify scrub speed throttle works

[Qu Wenruo]

On 2023/1/5 06:17, Anand Jain wrote:
> On 1/4/23 16:21, Qu Wenruo wrote:
>> We introduced scrub speed throttle in commit eb3b50536642 ("btrfs: scrub:
>> per-device bandwidth control"),  but it is not that well documented
>> (e.g. what's the unit of the sysfs interface), nor tested by any test
>> case.
>>
>> This patch will add a test case for this functionality.
>>
>> The test case itself is pretty straightforward:
>>
>> - Fill the fs with 2G file as scrub workload
>> - Set scrub speed limit to 50MiB/s
>> - Scrub and compare the reported rate against above 50MiB/s throttle
>>
>> However the test case has an assumption that the underlying disk must be
>> faster than our 50MiB/s, which should be pretty common in
>> baremetal/exclusive VMs.
>> But for cloud environment it's not ensured 100%, thus the test case is
>> not included in auto group to avoid false alerts.
> 
> 
> Why not scrub the data twice: first to determine the speed, and then to 
> check if it can be throttled to ~50%?

Sounds great.

Thanks,
Qu
> 
> -Anand
> 
> 

Re: [PATCH] btrfs: add a test case to verify scrub speed throttle works

[Anand Jain]

On 1/4/23 16:21, Qu Wenruo wrote:
> We introduced scrub speed throttle in commit eb3b50536642 ("btrfs: scrub:
> per-device bandwidth control"),  but it is not that well documented
> (e.g. what's the unit of the sysfs interface), nor tested by any test
> case.
> 
> This patch will add a test case for this functionality.
> 
> The test case itself is pretty straightforward:
> 
> - Fill the fs with 2G file as scrub workload
> - Set scrub speed limit to 50MiB/s
> - Scrub and compare the reported rate against above 50MiB/s throttle
> 
> However the test case has an assumption that the underlying disk must be
> faster than our 50MiB/s, which should be pretty common in
> baremetal/exclusive VMs.
> But for cloud environment it's not ensured 100%, thus the test case is
> not included in auto group to avoid false alerts.


Why not scrub the data twice: first to determine the speed, and then to 
check if it can be throttled to ~50%?

-Anand

[PATCH] btrfs: add a test case to verify scrub speed throttle works

[Qu Wenruo]

We introduced scrub speed throttle in commit eb3b50536642 ("btrfs: scrub:
per-device bandwidth control"),  but it is not that well documented
(e.g. what's the unit of the sysfs interface), nor tested by any test
case.

This patch will add a test case for this functionality.

The test case itself is pretty straightforward:

- Fill the fs with 2G file as scrub workload
- Set scrub speed limit to 50MiB/s
- Scrub and compare the reported rate against above 50MiB/s throttle

However the test case has an assumption that the underlying disk must be
faster than our 50MiB/s, which should be pretty common in
baremetal/exclusive VMs.
But for cloud environment it's not ensured 100%, thus the test case is
not included in auto group to avoid false alerts.

Signed-off-by: Qu Wenruo <wqu@suse.com>
---
 tests/btrfs/282     | 83 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/282.out |  3 ++
 2 files changed, 86 insertions(+)
 create mode 100755 tests/btrfs/282
 create mode 100644 tests/btrfs/282.out

diff --git a/tests/btrfs/282 b/tests/btrfs/282
new file mode 100755
index 00000000..9a6677ec
--- /dev/null
+++ b/tests/btrfs/282
@@ -0,0 +1,83 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (C) 2023 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# FS QA Test 282
+#
+# Make sure scrub speed limitation works as expected.
+#
+. ./common/preamble
+_begin_fstest scrub
+
+# Override the default cleanup function.
+# _cleanup()
+# {
+# 	cd /
+# 	rm -r -f $tmp.*
+# }
+
+. ./common/filter
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_wants_kernel_commit eb3b50536642 \
+	"btrfs: scrub: per-device bandwidth control"
+
+# We want at least 5G for the scratch device.
+_require_scratch_size $(( 5 * 1024 * 1024))
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+uuid=$(findmnt -n -o UUID $SCRATCH_MNT)
+
+devinfo_dir="/sys/fs/btrfs/${uuid}/devinfo/1"
+
+# Check if we have the sysfs interface first.
+if [ ! -f "${devinfo_dir}/scrub_speed_max" ]; then
+	_notrun "No sysfs interface for scrub speed throttle"
+fi
+
+# Create a 2G file for later scrub workload.
+# The 2G size is chosen to fit even DUP on a 5G disk.
+$XFS_IO_PROG -f -c "pwrite -i /dev/urandom 0 2G" $SCRATCH_MNT/file | _filter_xfs_io
+
+# Writeback above data, as scrub only verify the committed data.
+sync
+
+throttle_mb=50
+# Those are floor and ceiling for us to compare the result, give it a
+# generous +- 10% tolerance.
+throttle_mb_ceiling=55
+throttle_mb_floor=45
+
+# Set the speed limit to 50MiB/s, which should be slower than almost all
+# modern HDD.
+# This would take around 40 sec to scrub above data for SINGLE, double for DUP.
+# With extra time spent on writing the data.
+echo $(($throttle_mb * 1024 * 1024)) > "${devinfo_dir}/scrub_speed_max"
+
+$BTRFS_UTIL_PROG scrub start -B $SCRATCH_MNT > $tmp.scrub_result
+cat $tmp.scrub_result >> $seqres.full
+
+# The output looks like this:
+# Scrub done for 42d25bc2-e8b7-432e-9850-f3314aefffc6
+# Scrub started:    Wed Jan  4 13:12:00 2023
+# Status:           finished
+# Duration:         0:00:30
+# Total to scrub:   7.52GiB
+# Rate:             205.22MiB/s
+# Error summary:    no errors found
+#
+# What we care is that Rate line, and only the int part.
+speed=$(grep "Rate:" $tmp.scrub_result | $AWK_PROG '{print $2}' | cut -f1 -d.)
+
+if [ "$speed" -gt "$throttle_mb_ceiling" -o "$speed" -lt "$throttle_mb_floor" ]; then
+	echo "scrub speed $speed is not properly throttled"
+fi
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/282.out b/tests/btrfs/282.out
new file mode 100644
index 00000000..8d53e7eb
--- /dev/null
+++ b/tests/btrfs/282.out
@@ -0,0 +1,3 @@
+QA output created by 282
+wrote 2147483648/2147483648 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.39.0