* [PATCH v6] btrfs: Add new test for setting the chunk size.
@ 2021-11-24 21:12 Stefan Roesch
2021-11-28 15:55 ` Eryu Guan
0 siblings, 1 reply; 3+ messages in thread
From: Stefan Roesch @ 2021-11-24 21:12 UTC (permalink / raw)
To: fstests, linux-btrfs, kernel-team; +Cc: shr, Josef Bacik
Add new testcase for testing the new btrfs sysfs knob to change the
chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block
type>/chunk_size.
The test case implements three different cases:
- Test allocation with the default chunk size
- Test allocation after increasing the chunk size
- Test allocation when the free space is smaller than the chunk size.
Note: this test needs to force the allocation of space. It uses the
/sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
Testing:
The test has been run with volumes of different sizes.
Signed-off-by: Stefan Roesch <shr@fb.com>
Reviewed-by: Josef Bacik <josef@toxicpanda.com>
---
V6: - renamed test case from 248 to 251
- use _require_fs_sysfs
- use device name in _get_fs_sysfs_attr and _set_fs_sysfs_attr
V5: - Modify _get_fs_sysfs_attr and _set_fs_sysfs_attr to support btrfs
- Use _get_fs_sysfs_attr and _set_fs_sysfs_attr
- Remove _get_btrfs_sysfs_attr and _set_btrfs_fsysfs_attr
- Rename local functions to not use "_"
- use $AWK_PROG and $BTRFS_UTIL_PROG
- remove call to _require_scratch_size
- fixes for coding convention
- replace fail calls with echo
V4: - fixed indentation in common/btrfs
- Removed UUID code, which is no longer necessary (provided
by helper function)
- Used new helper _get_btrfs_sysfs_attr
- Direct output to /dev/null
V3: - removed tests for system block type.
- added failure case for system block type.
- renamed stripe_size to chunk_size
V2: - added new functions to common/btrfs and use them in the new test
- _require_btrfs_sysfs_attr - Make sure btrfs supports a sysfs
setting
- _get_btrfs_sysfs_attr - Read sysfs value
- _set_btrfs_sysfs_attr - Write sysfs value
- create file system of required size with _scratch_mkfs_sized
- use shortened error message
- Remove last logging message
---
common/rc | 29 ++++-
tests/btrfs/251 | 287 ++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/251.out | 11 ++
3 files changed, 323 insertions(+), 4 deletions(-)
create mode 100755 tests/btrfs/251
create mode 100644 tests/btrfs/251.out
diff --git a/common/rc b/common/rc
index 0a30a842..9522ba2e 100644
--- a/common/rc
+++ b/common/rc
@@ -4401,7 +4401,14 @@ run_fsx()
_require_fs_sysfs()
{
local attr=$1
- local dname=$(_short_dev $TEST_DEV)
+ local dname
+
+ case "$FSTYP" in
+ btrfs)
+ dname=$(findmnt -n -o UUID $TEST_DEV) ;;
+ *)
+ dname=$(_short_dev $TEST_DEV) ;;
+ esac
if [ -z "$attr" -o -z "$dname" ];then
_fail "Usage: _require_fs_sysfs <sysfs_attr_path>"
@@ -4436,10 +4443,17 @@ _set_fs_sysfs_attr()
local content="$*"
if [ ! -b "$dev" -o -z "$attr" -o -z "$content" ];then
- _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
+ _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
fi
- local dname=$(_short_dev $dev)
+ local dname
+ case "$FSTYP" in
+ btrfs)
+ dname=$(findmnt -n -o UUID ${dev}) ;;
+ *)
+ dname=$(_short_dev $dev) ;;
+ esac
+
echo "$content" > /sys/fs/${FSTYP}/${dname}/${attr}
}
@@ -4460,7 +4474,14 @@ _get_fs_sysfs_attr()
_fail "Usage: _get_fs_sysfs_attr <mounted_device> <attr>"
fi
- local dname=$(_short_dev $dev)
+ local dname
+ case "$FSTYP" in
+ btrfs)
+ dname=$(findmnt -n -o UUID ${dev}) ;;
+ *)
+ dname=$(_short_dev $dev) ;;
+ esac
+
cat /sys/fs/${FSTYP}/${dname}/${attr}
}
diff --git a/tests/btrfs/251 b/tests/btrfs/251
new file mode 100755
index 00000000..b5c4b623
--- /dev/null
+++ b/tests/btrfs/251
@@ -0,0 +1,287 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Facebook. All Rights Reserved.
+#
+# FS QA Test 250
+#
+# Test the new /sys/fs/btrfs/<uuid>/allocation/<block-type>/chunk_size
+# setting. This setting allows the admin to change the chunk size
+# setting for the next allocation.
+#
+# Test 1:
+# Allocate storage for all three block types (data, metadata and system)
+# with the default chunk size.
+#
+# Test 2:
+# Set a new chunk size to double the default size and allocate space
+# for all new block types with the new chunk size.
+#
+# Test 3:
+# Pick an allocation size that is used in a loop and make sure the last
+# allocation cannot be partially fullfilled.
+#
+# Note: Variable naming uses the following convention: if a variable name
+# ends in "_B" then its a byte value, if it ends in "_MB" then the
+# value is in megabytes.
+#
+. ./common/preamble
+_begin_fstest auto
+
+test_file="${TEST_DIR}/${seq}"
+seq=`basename $0`
+seqres="${RESULT_DIR}/${seq}"
+
+# Override the default cleanup function.
+_cleanup()
+{
+ cd /
+ rm -f "$test_file"
+}
+
+# Parse a size string which is in the format "XX.XXMib".
+#
+# Parameters:
+# - (IN) Block group type (Data, Metadata, System)
+# - (INOUT) Variable to store block group size in MB
+#
+parse_size_string() {
+ local SIZE=$(echo "$1" | $AWK_PROG 'match($0, /([0-9.]+)/) { print substr($0, RSTART, RLENGTH) }')
+ eval $2="${SIZE%.*}"
+}
+
+# Determine the size of the device in MB.
+#
+# Parameters:
+# - (INOUT) Variable to store device size in MB
+#
+device_size() {
+ btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid >> $seqres.full
+ local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
+ parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
+ eval $1=${SIZE_ALLOC%.*}
+}
+
+# Determine the free space of a block group in MB.
+#
+# Parameters:
+# - (INOUT) Variable to store free space in MB
+#
+free_space() {
+ local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
+ parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
+ parse_size_string $(echo "${SIZE}" | awk '{print $6}') SIZE_USED
+ eval $1=$(expr ${SIZE_ALLOC} - ${SIZE_USED})
+}
+
+# Determine how much space in MB has been allocated to a block group.
+#
+# Parameters:
+# - (IN) Block group type (Data, Metadata, System)
+# - (INOUT) Variable to store block group size in MB
+#
+alloc_size() {
+ local SIZE_STRING=$($BTRFS_UTIL_PROG filesystem df ${SCRATCH_MNT} -m | grep "$1" | awk '{print $3}')
+ parse_size_string ${SIZE_STRING} BLOCK_GROUP_SIZE
+ eval $2="${BLOCK_GROUP_SIZE}"
+}
+
+. ./common/filter
+_supported_fs btrfs
+_require_test
+_require_scratch
+_require_btrfs_fs_sysfs
+
+# Delete log file if it exists.
+rm -f "${seqres}.full"
+
+# Make filesystem.
+_scratch_mkfs_sized $((10 * 1024 * 1024 * 1024)) >> $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+# Check if there is sufficient sysfs support.
+_require_fs_sysfs allocation/metadata/chunk_size
+_require_fs_sysfs allocation/metadata/force_chunk_alloc
+
+# Get free space.
+free_space FREE_SPACE_MB
+device_size DEVICE_SIZE_MB
+
+echo "free space = ${FREE_SPACE_MB}MB" >> ${seqres}.full
+
+# If device is a symbolic link, get block device.
+SCRATCH_BDEV=${SCRATCH_DEV}
+if [[ -h ${SCRATCH_DEV} ]]; then
+ SCRATCH_BDEV=$(readlink -f $SCRATCH_DEV)
+fi
+
+# Get chunk sizes.
+echo "Capture default chunk sizes."
+FIRST_DATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
+FIRST_METADATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
+
+echo "Orig Data chunk size = ${FIRST_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
+echo "Orig Metaata chunk size = ${FIRST_METADATA_CHUNK_SIZE_B}" >> ${seqres}.full
+
+INIT_ALLOC_SIZE_MB=$(expr \( ${FIRST_DATA_CHUNK_SIZE_B} + ${FIRST_METADATA_CHUNK_SIZE_B} \) / 1024 / 1024)
+echo "Allocation size for initial allocation = ${INIT_ALLOC_SIZE_MB}MB" >> $seqres.full
+
+#
+# Do first allocation with the default chunk sizes for the different block
+# types.
+#
+echo "First allocation."
+alloc_size "Data" DATA_SIZE_START_MB
+alloc_size "Metadata" METADATA_SIZE_START_MB
+
+echo "Block group Data alloc size = ${DATA_SIZE_START_MB}MB" >> $seqres.full
+echo "Block group Metadata alloc size = ${METADATA_SIZE_START_MB}MB" >> $seqres.full
+
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
+
+alloc_size "Data" FIRST_DATA_SIZE_MB
+alloc_size "Metadata" FIRST_METADATA_SIZE_MB
+
+echo "First block group Data alloc size = ${FIRST_DATA_SIZE_MB}MB" >> ${seqres}.full
+echo "First block group Metadata alloc size = ${FIRST_METADATA_SIZE_MB}MB" >> ${seqres}.full
+
+free_space FREE_SPACE_AFTER
+echo "Free space after first allocation = ${FREE_SPACE_AFTER}MB" >> ${seqres}.full
+
+#
+# Do allocation with the doubled chunk sizes for the different block types.
+#
+echo "Second allocation."
+SECOND_DATA_CHUNK_SIZE_B=$(expr ${FIRST_DATA_CHUNK_SIZE_B} \* 2)
+SECOND_METADATA_CHUNK_SIZE_B=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} \* 2)
+
+echo "Second block group Data calc alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
+echo "Second block group Metadata calc alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
+
+# Stripe size is limited to 10% of device size.
+if [[ ${SECOND_DATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
+ SECOND_DATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
+fi
+if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
+ SECOND_METADATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
+fi
+
+echo "Second block group Data alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
+echo "Second block group Metadata alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
+
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${SECOND_DATA_CHUNK_SIZE_B}
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size ${SECOND_METADATA_CHUNK_SIZE_B}
+
+SECOND_DATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
+SECOND_METADATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
+
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
+echo "Allocated data chunk" >> $seqres.full
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
+echo "Allocated metadata chunk" >> $seqres.full
+
+alloc_size "Data" SECOND_DATA_SIZE_MB
+alloc_size "Metadata" SECOND_METADATA_SIZE_MB
+alloc_size "System" SECOND_SYSTEM_SIZE_MB
+
+echo "Calculate request size so last memory allocation cannot be completely fullfilled."
+free_space FREE_SPACE_MB
+
+# Find request size whose space allocation cannot be completely fullfilled.
+THIRD_DATA_CHUNK_SIZE_MB=$(expr 256)
+until [ ${THIRD_DATA_CHUNK_SIZE_MB} -gt $(expr 7 \* 1024) ]; do
+ if [ $((FREE_SPACE_MB%THIRD_DATA_CHUNK_SIZE_MB)) -ne 0 ]; then
+ break
+ fi
+ THIRD_DATA_CHUNK_SIZE_MB=$((THIRD_DATA_CHUNK_SIZE_MB+256))
+done
+
+if [[ ${THIRD_DATA_CHUNK_SIZE_MB} -eq $(expr 7 \* 1024) ]]; then
+ _fail "Cannot find allocation size for partial block allocation."
+fi
+
+THIRD_DATA_CHUNK_SIZE_B=$(expr ${THIRD_DATA_CHUNK_SIZE_MB} \* 1024 \* 1024)
+echo "Allocation size in mb = ${THIRD_DATA_CHUNK_SIZE_MB}" >> ${seqres}.full
+echo "Allocation size in bytes = ${THIRD_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
+
+#
+# Do allocation until free space is exhausted.
+#
+echo "Third allocation."
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${THIRD_DATA_CHUNK_SIZE_B}
+
+free_space FREE_SPACE_MB
+while [ $FREE_SPACE_MB -gt $THIRD_DATA_CHUNK_SIZE_MB ]
+do
+ alloc_size "Data" THIRD_DATA_SIZE_MB
+ _set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
+
+ free_space FREE_SPACE_MB
+done
+
+alloc_size "Data" FOURTH_DATA_SIZE_MB
+
+#
+# Force chunk allocation of system block type must fail.
+#
+echo "Force allocation of system block type must fail."
+_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/system/force_chunk_alloc 1 2>/dev/null
+if [ $? -ne 0 ]; then
+ _dump_err "_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc"
+fi
+
+#
+# Verification of initial allocation.
+#
+echo "Verify first allocation."
+FIRST_DATA_CHUNK_SIZE_MB=$(expr ${FIRST_DATA_CHUNK_SIZE_B} / 1024 / 1024)
+FIRST_METADATA_CHUNK_SIZE_MB=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
+
+# if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne $(expr ${FIRST_DATA_SIZE_MB}) ]]; then
+if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne ${FIRST_DATA_SIZE_MB} ]]; then
+ echo "tInitial data allocation not correct."
+fi
+
+if [[ $(expr ${FIRST_METADATA_CHUNK_SIZE_MB} + ${METADATA_SIZE_START_MB}) -ne ${FIRST_METADATA_SIZE_MB} ]]; then
+ echo "Initial metadata allocation not correct."
+fi
+
+#
+# Verification of second allocation.
+#
+echo "Verify second allocation."
+SECOND_DATA_CHUNK_SIZE_MB=$(expr ${SECOND_DATA_CHUNK_SIZE_B} / 1024 / 1024)
+SECOND_METADATA_CHUNK_SIZE_MB=$(expr ${SECOND_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
+
+if [[ ${SECOND_DATA_CHUNK_SIZE_B} -ne ${SECOND_DATA_CHUNK_SIZE_READ_B} ]]; then
+ echo "Value written to allocation/data/chunk_size and read value are different."
+fi
+
+if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -ne ${SECOND_METADATA_CHUNK_SIZE_READ_B} ]]; then
+ echo "Value written to allocation/metadata/chunk_size and read value are different."
+fi
+
+if [[ $(expr ${SECOND_DATA_CHUNK_SIZE_MB} + ${FIRST_DATA_SIZE_MB}) -ne ${SECOND_DATA_SIZE_MB} ]]; then
+ echo "Data allocation after chunk size change not correct."
+fi
+
+if [[ $(expr ${SECOND_METADATA_CHUNK_SIZE_MB} + ${FIRST_METADATA_SIZE_MB}) -ne ${SECOND_METADATA_SIZE_MB} ]]; then
+ echo "Metadata allocation after chunk size change not correct."
+fi
+
+#
+# Verification of third allocation.
+#
+echo "Verify third allocation."
+if [[ ${FREE_SPACE_MB} -gt ${THIRD_DATA_CHUNK_SIZE_MB} ]]; then
+ echo "Free space after allocating over memlimit is too high."
+fi
+
+# The + 1 is required as 1MB is always kept as free space.
+if [[ $(expr ${THIRD_DATA_CHUNK_SIZE_MB} + ${THIRD_DATA_SIZE_MB} + 1) -le $(expr ${FOURTH_DATA_SIZE_MB}) ]]; then
+ echo "Allocation until out of memory: last memory allocation size is not correct."
+fi
+
+status=0
+exit
+
diff --git a/tests/btrfs/251.out b/tests/btrfs/251.out
new file mode 100644
index 00000000..bb7b45d8
--- /dev/null
+++ b/tests/btrfs/251.out
@@ -0,0 +1,11 @@
+QA output created by 248
+Capture default chunk sizes.
+First allocation.
+Second allocation.
+Calculate request size so last memory allocation cannot be completely fullfilled.
+Third allocation.
+Force allocation of system block type must fail.
+_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc
+Verify first allocation.
+Verify second allocation.
+Verify third allocation.
--
2.30.2
^ permalink raw reply related [flat|nested] 3+ messages in thread
* Re: [PATCH v6] btrfs: Add new test for setting the chunk size.
2021-11-24 21:12 [PATCH v6] btrfs: Add new test for setting the chunk size Stefan Roesch
@ 2021-11-28 15:55 ` Eryu Guan
2021-11-29 18:19 ` Stefan Roesch
0 siblings, 1 reply; 3+ messages in thread
From: Eryu Guan @ 2021-11-28 15:55 UTC (permalink / raw)
To: Stefan Roesch; +Cc: fstests, linux-btrfs, kernel-team, Josef Bacik
On Wed, Nov 24, 2021 at 01:12:30PM -0800, Stefan Roesch wrote:
> Add new testcase for testing the new btrfs sysfs knob to change the
> chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block
> type>/chunk_size.
>
> The test case implements three different cases:
> - Test allocation with the default chunk size
> - Test allocation after increasing the chunk size
> - Test allocation when the free space is smaller than the chunk size.
>
> Note: this test needs to force the allocation of space. It uses the
> /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
>
> Testing:
> The test has been run with volumes of different sizes.
>
> Signed-off-by: Stefan Roesch <shr@fb.com>
> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
> ---
> V6: - renamed test case from 248 to 251
> - use _require_fs_sysfs
> - use device name in _get_fs_sysfs_attr and _set_fs_sysfs_attr
> V5: - Modify _get_fs_sysfs_attr and _set_fs_sysfs_attr to support btrfs
> - Use _get_fs_sysfs_attr and _set_fs_sysfs_attr
> - Remove _get_btrfs_sysfs_attr and _set_btrfs_fsysfs_attr
> - Rename local functions to not use "_"
> - use $AWK_PROG and $BTRFS_UTIL_PROG
> - remove call to _require_scratch_size
> - fixes for coding convention
> - replace fail calls with echo
> V4: - fixed indentation in common/btrfs
> - Removed UUID code, which is no longer necessary (provided
> by helper function)
> - Used new helper _get_btrfs_sysfs_attr
> - Direct output to /dev/null
> V3: - removed tests for system block type.
> - added failure case for system block type.
> - renamed stripe_size to chunk_size
> V2: - added new functions to common/btrfs and use them in the new test
> - _require_btrfs_sysfs_attr - Make sure btrfs supports a sysfs
> setting
> - _get_btrfs_sysfs_attr - Read sysfs value
> - _set_btrfs_sysfs_attr - Write sysfs value
> - create file system of required size with _scratch_mkfs_sized
> - use shortened error message
> - Remove last logging message
> ---
> common/rc | 29 ++++-
> tests/btrfs/251 | 287 ++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/251.out | 11 ++
> 3 files changed, 323 insertions(+), 4 deletions(-)
> create mode 100755 tests/btrfs/251
> create mode 100644 tests/btrfs/251.out
>
> diff --git a/common/rc b/common/rc
> index 0a30a842..9522ba2e 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -4401,7 +4401,14 @@ run_fsx()
> _require_fs_sysfs()
> {
> local attr=$1
> - local dname=$(_short_dev $TEST_DEV)
> + local dname
> +
> + case "$FSTYP" in
> + btrfs)
> + dname=$(findmnt -n -o UUID $TEST_DEV) ;;
> + *)
> + dname=$(_short_dev $TEST_DEV) ;;
> + esac
>
> if [ -z "$attr" -o -z "$dname" ];then
> _fail "Usage: _require_fs_sysfs <sysfs_attr_path>"
> @@ -4436,10 +4443,17 @@ _set_fs_sysfs_attr()
> local content="$*"
>
> if [ ! -b "$dev" -o -z "$attr" -o -z "$content" ];then
> - _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
> + _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
What is this change? Seems it adds a space between the first two tabs,
and causes git am to complain about whitespace damange.
> fi
>
> - local dname=$(_short_dev $dev)
> + local dname
> + case "$FSTYP" in
> + btrfs)
> + dname=$(findmnt -n -o UUID ${dev}) ;;
> + *)
> + dname=$(_short_dev $dev) ;;
> + esac
> +
> echo "$content" > /sys/fs/${FSTYP}/${dname}/${attr}
> }
>
> @@ -4460,7 +4474,14 @@ _get_fs_sysfs_attr()
> _fail "Usage: _get_fs_sysfs_attr <mounted_device> <attr>"
> fi
>
> - local dname=$(_short_dev $dev)
> + local dname
> + case "$FSTYP" in
> + btrfs)
> + dname=$(findmnt -n -o UUID ${dev}) ;;
> + *)
> + dname=$(_short_dev $dev) ;;
> + esac
> +
> cat /sys/fs/${FSTYP}/${dname}/${attr}
> }
>
> diff --git a/tests/btrfs/251 b/tests/btrfs/251
> new file mode 100755
> index 00000000..b5c4b623
> --- /dev/null
> +++ b/tests/btrfs/251
> @@ -0,0 +1,287 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Facebook. All Rights Reserved.
> +#
> +# FS QA Test 250
> +#
> +# Test the new /sys/fs/btrfs/<uuid>/allocation/<block-type>/chunk_size
> +# setting. This setting allows the admin to change the chunk size
> +# setting for the next allocation.
> +#
> +# Test 1:
> +# Allocate storage for all three block types (data, metadata and system)
> +# with the default chunk size.
> +#
> +# Test 2:
> +# Set a new chunk size to double the default size and allocate space
> +# for all new block types with the new chunk size.
> +#
> +# Test 3:
> +# Pick an allocation size that is used in a loop and make sure the last
> +# allocation cannot be partially fullfilled.
> +#
> +# Note: Variable naming uses the following convention: if a variable name
> +# ends in "_B" then its a byte value, if it ends in "_MB" then the
> +# value is in megabytes.
> +#
> +. ./common/preamble
> +_begin_fstest auto
> +
> +test_file="${TEST_DIR}/${seq}"
Seems it's not used in the test?
> +seq=`basename $0`
> +seqres="${RESULT_DIR}/${seq}"
> +
> +# Override the default cleanup function.
> +_cleanup()
> +{
> + cd /
> + rm -f "$test_file"
Then there's no need to remove it in cleanup
> +}
> +
> +# Parse a size string which is in the format "XX.XXMib".
> +#
> +# Parameters:
> +# - (IN) Block group type (Data, Metadata, System)
> +# - (INOUT) Variable to store block group size in MB
> +#
> +parse_size_string() {
> + local SIZE=$(echo "$1" | $AWK_PROG 'match($0, /([0-9.]+)/) { print substr($0, RSTART, RLENGTH) }')
> + eval $2="${SIZE%.*}"
The above "eval" line uses spaces as indentions not tab.
> +}
> +
> +# Determine the size of the device in MB.
> +#
> +# Parameters:
> +# - (INOUT) Variable to store device size in MB
> +#
> +device_size() {
> + btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid >> $seqres.full
> + local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
> + parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
> + eval $1=${SIZE_ALLOC%.*}
> +}
> +
> +# Determine the free space of a block group in MB.
> +#
> +# Parameters:
> +# - (INOUT) Variable to store free space in MB
> +#
> +free_space() {
> + local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
> + parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
> + parse_size_string $(echo "${SIZE}" | awk '{print $6}') SIZE_USED
> + eval $1=$(expr ${SIZE_ALLOC} - ${SIZE_USED})
> +}
> +
> +# Determine how much space in MB has been allocated to a block group.
> +#
> +# Parameters:
> +# - (IN) Block group type (Data, Metadata, System)
> +# - (INOUT) Variable to store block group size in MB
> +#
> +alloc_size() {
> + local SIZE_STRING=$($BTRFS_UTIL_PROG filesystem df ${SCRATCH_MNT} -m | grep "$1" | awk '{print $3}')
> + parse_size_string ${SIZE_STRING} BLOCK_GROUP_SIZE
> + eval $2="${BLOCK_GROUP_SIZE}"
> +}
> +
> +. ./common/filter
> +_supported_fs btrfs
> +_require_test
> +_require_scratch
> +_require_btrfs_fs_sysfs
Looks like we could drop this require, as _require_fs_sysfs below
ensures this requirement must be met.
> +
> +# Delete log file if it exists.
> +rm -f "${seqres}.full"
> +
> +# Make filesystem.
> +_scratch_mkfs_sized $((10 * 1024 * 1024 * 1024)) >> $seqres.full 2>&1
Any reason we need a 10G filesystem? Some comments would be good.
> +_scratch_mount >> $seqres.full 2>&1
> +
> +# Check if there is sufficient sysfs support.
> +_require_fs_sysfs allocation/metadata/chunk_size
> +_require_fs_sysfs allocation/metadata/force_chunk_alloc
> +
> +# Get free space.
> +free_space FREE_SPACE_MB
> +device_size DEVICE_SIZE_MB
> +
> +echo "free space = ${FREE_SPACE_MB}MB" >> ${seqres}.full
> +
> +# If device is a symbolic link, get block device.
> +SCRATCH_BDEV=${SCRATCH_DEV}
> +if [[ -h ${SCRATCH_DEV} ]]; then
> + SCRATCH_BDEV=$(readlink -f $SCRATCH_DEV)
> +fi
> +
> +# Get chunk sizes.
> +echo "Capture default chunk sizes."
> +FIRST_DATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
> +FIRST_METADATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
> +
> +echo "Orig Data chunk size = ${FIRST_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
> +echo "Orig Metaata chunk size = ${FIRST_METADATA_CHUNK_SIZE_B}" >> ${seqres}.full
> +
> +INIT_ALLOC_SIZE_MB=$(expr \( ${FIRST_DATA_CHUNK_SIZE_B} + ${FIRST_METADATA_CHUNK_SIZE_B} \) / 1024 / 1024)
> +echo "Allocation size for initial allocation = ${INIT_ALLOC_SIZE_MB}MB" >> $seqres.full
> +
> +#
> +# Do first allocation with the default chunk sizes for the different block
> +# types.
> +#
> +echo "First allocation."
> +alloc_size "Data" DATA_SIZE_START_MB
> +alloc_size "Metadata" METADATA_SIZE_START_MB
> +
> +echo "Block group Data alloc size = ${DATA_SIZE_START_MB}MB" >> $seqres.full
> +echo "Block group Metadata alloc size = ${METADATA_SIZE_START_MB}MB" >> $seqres.full
> +
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
> +
> +alloc_size "Data" FIRST_DATA_SIZE_MB
> +alloc_size "Metadata" FIRST_METADATA_SIZE_MB
> +
> +echo "First block group Data alloc size = ${FIRST_DATA_SIZE_MB}MB" >> ${seqres}.full
> +echo "First block group Metadata alloc size = ${FIRST_METADATA_SIZE_MB}MB" >> ${seqres}.full
> +
> +free_space FREE_SPACE_AFTER
> +echo "Free space after first allocation = ${FREE_SPACE_AFTER}MB" >> ${seqres}.full
> +
> +#
> +# Do allocation with the doubled chunk sizes for the different block types.
> +#
> +echo "Second allocation."
> +SECOND_DATA_CHUNK_SIZE_B=$(expr ${FIRST_DATA_CHUNK_SIZE_B} \* 2)
> +SECOND_METADATA_CHUNK_SIZE_B=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} \* 2)
> +
> +echo "Second block group Data calc alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
> +echo "Second block group Metadata calc alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
> +
> +# Stripe size is limited to 10% of device size.
> +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
> + SECOND_DATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
> +fi
> +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
> + SECOND_METADATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
> +fi
> +
> +echo "Second block group Data alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
> +echo "Second block group Metadata alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
> +
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${SECOND_DATA_CHUNK_SIZE_B}
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size ${SECOND_METADATA_CHUNK_SIZE_B}
> +
> +SECOND_DATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
> +SECOND_METADATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
> +
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
> +echo "Allocated data chunk" >> $seqres.full
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
> +echo "Allocated metadata chunk" >> $seqres.full
> +
> +alloc_size "Data" SECOND_DATA_SIZE_MB
> +alloc_size "Metadata" SECOND_METADATA_SIZE_MB
> +alloc_size "System" SECOND_SYSTEM_SIZE_MB
> +
> +echo "Calculate request size so last memory allocation cannot be completely fullfilled."
> +free_space FREE_SPACE_MB
> +
> +# Find request size whose space allocation cannot be completely fullfilled.
> +THIRD_DATA_CHUNK_SIZE_MB=$(expr 256)
> +until [ ${THIRD_DATA_CHUNK_SIZE_MB} -gt $(expr 7 \* 1024) ]; do
> + if [ $((FREE_SPACE_MB%THIRD_DATA_CHUNK_SIZE_MB)) -ne 0 ]; then
> + break
> + fi
> + THIRD_DATA_CHUNK_SIZE_MB=$((THIRD_DATA_CHUNK_SIZE_MB+256))
> +done
> +
> +if [[ ${THIRD_DATA_CHUNK_SIZE_MB} -eq $(expr 7 \* 1024) ]]; then
> + _fail "Cannot find allocation size for partial block allocation."
> +fi
> +
> +THIRD_DATA_CHUNK_SIZE_B=$(expr ${THIRD_DATA_CHUNK_SIZE_MB} \* 1024 \* 1024)
> +echo "Allocation size in mb = ${THIRD_DATA_CHUNK_SIZE_MB}" >> ${seqres}.full
> +echo "Allocation size in bytes = ${THIRD_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
> +
> +#
> +# Do allocation until free space is exhausted.
> +#
> +echo "Third allocation."
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${THIRD_DATA_CHUNK_SIZE_B}
> +
> +free_space FREE_SPACE_MB
> +while [ $FREE_SPACE_MB -gt $THIRD_DATA_CHUNK_SIZE_MB ]
> +do
> + alloc_size "Data" THIRD_DATA_SIZE_MB
> + _set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
> +
> + free_space FREE_SPACE_MB
> +done
> +
> +alloc_size "Data" FOURTH_DATA_SIZE_MB
> +
> +#
> +# Force chunk allocation of system block type must fail.
> +#
> +echo "Force allocation of system block type must fail."
> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/system/force_chunk_alloc 1 2>/dev/null
> +if [ $? -ne 0 ]; then
> + _dump_err "_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc"
> +fi
I don't think we need this check. We could just remove the
"2 > /dev/null" part from the above _set_fs_sysfs_attr and it will print
error message and break golden image on error.
Thanks,
Eryu
> +
> +#
> +# Verification of initial allocation.
> +#
> +echo "Verify first allocation."
> +FIRST_DATA_CHUNK_SIZE_MB=$(expr ${FIRST_DATA_CHUNK_SIZE_B} / 1024 / 1024)
> +FIRST_METADATA_CHUNK_SIZE_MB=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
> +
> +# if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne $(expr ${FIRST_DATA_SIZE_MB}) ]]; then
> +if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne ${FIRST_DATA_SIZE_MB} ]]; then
> + echo "tInitial data allocation not correct."
> +fi
> +
> +if [[ $(expr ${FIRST_METADATA_CHUNK_SIZE_MB} + ${METADATA_SIZE_START_MB}) -ne ${FIRST_METADATA_SIZE_MB} ]]; then
> + echo "Initial metadata allocation not correct."
> +fi
> +
> +#
> +# Verification of second allocation.
> +#
> +echo "Verify second allocation."
> +SECOND_DATA_CHUNK_SIZE_MB=$(expr ${SECOND_DATA_CHUNK_SIZE_B} / 1024 / 1024)
> +SECOND_METADATA_CHUNK_SIZE_MB=$(expr ${SECOND_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
> +
> +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -ne ${SECOND_DATA_CHUNK_SIZE_READ_B} ]]; then
> + echo "Value written to allocation/data/chunk_size and read value are different."
> +fi
> +
> +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -ne ${SECOND_METADATA_CHUNK_SIZE_READ_B} ]]; then
> + echo "Value written to allocation/metadata/chunk_size and read value are different."
> +fi
> +
> +if [[ $(expr ${SECOND_DATA_CHUNK_SIZE_MB} + ${FIRST_DATA_SIZE_MB}) -ne ${SECOND_DATA_SIZE_MB} ]]; then
> + echo "Data allocation after chunk size change not correct."
> +fi
> +
> +if [[ $(expr ${SECOND_METADATA_CHUNK_SIZE_MB} + ${FIRST_METADATA_SIZE_MB}) -ne ${SECOND_METADATA_SIZE_MB} ]]; then
> + echo "Metadata allocation after chunk size change not correct."
> +fi
> +
> +#
> +# Verification of third allocation.
> +#
> +echo "Verify third allocation."
> +if [[ ${FREE_SPACE_MB} -gt ${THIRD_DATA_CHUNK_SIZE_MB} ]]; then
> + echo "Free space after allocating over memlimit is too high."
> +fi
> +
> +# The + 1 is required as 1MB is always kept as free space.
> +if [[ $(expr ${THIRD_DATA_CHUNK_SIZE_MB} + ${THIRD_DATA_SIZE_MB} + 1) -le $(expr ${FOURTH_DATA_SIZE_MB}) ]]; then
> + echo "Allocation until out of memory: last memory allocation size is not correct."
> +fi
> +
> +status=0
> +exit
> +
> diff --git a/tests/btrfs/251.out b/tests/btrfs/251.out
> new file mode 100644
> index 00000000..bb7b45d8
> --- /dev/null
> +++ b/tests/btrfs/251.out
> @@ -0,0 +1,11 @@
> +QA output created by 248
> +Capture default chunk sizes.
> +First allocation.
> +Second allocation.
> +Calculate request size so last memory allocation cannot be completely fullfilled.
> +Third allocation.
> +Force allocation of system block type must fail.
> +_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc
> +Verify first allocation.
> +Verify second allocation.
> +Verify third allocation.
> --
> 2.30.2
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: [PATCH v6] btrfs: Add new test for setting the chunk size.
2021-11-28 15:55 ` Eryu Guan
@ 2021-11-29 18:19 ` Stefan Roesch
0 siblings, 0 replies; 3+ messages in thread
From: Stefan Roesch @ 2021-11-29 18:19 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, kernel-team, Josef Bacik
On 11/28/21 7:55 AM, Eryu Guan wrote:
> On Wed, Nov 24, 2021 at 01:12:30PM -0800, Stefan Roesch wrote:
>> Add new testcase for testing the new btrfs sysfs knob to change the
>> chunk size. The new knob uses /sys/fs/btrfs/<UUID>/allocation/<block
>> type>/chunk_size.
>>
>> The test case implements three different cases:
>> - Test allocation with the default chunk size
>> - Test allocation after increasing the chunk size
>> - Test allocation when the free space is smaller than the chunk size.
>>
>> Note: this test needs to force the allocation of space. It uses the
>> /sys/fs/btrfs/<UUID>/allocation/<block type>/force_chunk_alloc knob.
>>
>> Testing:
>> The test has been run with volumes of different sizes.
>>
>> Signed-off-by: Stefan Roesch <shr@fb.com>
>> Reviewed-by: Josef Bacik <josef@toxicpanda.com>
>> ---
>> V6: - renamed test case from 248 to 251
>> - use _require_fs_sysfs
>> - use device name in _get_fs_sysfs_attr and _set_fs_sysfs_attr
>> V5: - Modify _get_fs_sysfs_attr and _set_fs_sysfs_attr to support btrfs
>> - Use _get_fs_sysfs_attr and _set_fs_sysfs_attr
>> - Remove _get_btrfs_sysfs_attr and _set_btrfs_fsysfs_attr
>> - Rename local functions to not use "_"
>> - use $AWK_PROG and $BTRFS_UTIL_PROG
>> - remove call to _require_scratch_size
>> - fixes for coding convention
>> - replace fail calls with echo
>> V4: - fixed indentation in common/btrfs
>> - Removed UUID code, which is no longer necessary (provided
>> by helper function)
>> - Used new helper _get_btrfs_sysfs_attr
>> - Direct output to /dev/null
>> V3: - removed tests for system block type.
>> - added failure case for system block type.
>> - renamed stripe_size to chunk_size
>> V2: - added new functions to common/btrfs and use them in the new test
>> - _require_btrfs_sysfs_attr - Make sure btrfs supports a sysfs
>> setting
>> - _get_btrfs_sysfs_attr - Read sysfs value
>> - _set_btrfs_sysfs_attr - Write sysfs value
>> - create file system of required size with _scratch_mkfs_sized
>> - use shortened error message
>> - Remove last logging message
>> ---
>> common/rc | 29 ++++-
>> tests/btrfs/251 | 287 ++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/251.out | 11 ++
>> 3 files changed, 323 insertions(+), 4 deletions(-)
>> create mode 100755 tests/btrfs/251
>> create mode 100644 tests/btrfs/251.out
>>
>> diff --git a/common/rc b/common/rc
>> index 0a30a842..9522ba2e 100644
>> --- a/common/rc
>> +++ b/common/rc
>> @@ -4401,7 +4401,14 @@ run_fsx()
>> _require_fs_sysfs()
>> {
>> local attr=$1
>> - local dname=$(_short_dev $TEST_DEV)
>> + local dname
>> +
>> + case "$FSTYP" in
>> + btrfs)
>> + dname=$(findmnt -n -o UUID $TEST_DEV) ;;
>> + *)
>> + dname=$(_short_dev $TEST_DEV) ;;
>> + esac
>>
>> if [ -z "$attr" -o -z "$dname" ];then
>> _fail "Usage: _require_fs_sysfs <sysfs_attr_path>"
>> @@ -4436,10 +4443,17 @@ _set_fs_sysfs_attr()
>> local content="$*"
>>
>> if [ ! -b "$dev" -o -z "$attr" -o -z "$content" ];then
>> - _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
>> + _fail "Usage: _set_fs_sysfs_attr <mounted_device> <attr> <content>"
>
> What is this change? Seems it adds a space between the first two tabs,
> and causes git am to complain about whitespace damange.
>
Fixed.
>> fi
>>
>> - local dname=$(_short_dev $dev)
>> + local dname
>> + case "$FSTYP" in
>> + btrfs)
>> + dname=$(findmnt -n -o UUID ${dev}) ;;
>> + *)
>> + dname=$(_short_dev $dev) ;;
>> + esac
>> +
>> echo "$content" > /sys/fs/${FSTYP}/${dname}/${attr}
>> }
>>
>> @@ -4460,7 +4474,14 @@ _get_fs_sysfs_attr()
>> _fail "Usage: _get_fs_sysfs_attr <mounted_device> <attr>"
>> fi
>>
>> - local dname=$(_short_dev $dev)
>> + local dname
>> + case "$FSTYP" in
>> + btrfs)
>> + dname=$(findmnt -n -o UUID ${dev}) ;;
>> + *)
>> + dname=$(_short_dev $dev) ;;
>> + esac
>> +
>> cat /sys/fs/${FSTYP}/${dname}/${attr}
>> }
>>
>> diff --git a/tests/btrfs/251 b/tests/btrfs/251
>> new file mode 100755
>> index 00000000..b5c4b623
>> --- /dev/null
>> +++ b/tests/btrfs/251
>> @@ -0,0 +1,287 @@
>> +#!/bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +# Copyright (c) 2021 Facebook. All Rights Reserved.
>> +#
>> +# FS QA Test 250
>> +#
>> +# Test the new /sys/fs/btrfs/<uuid>/allocation/<block-type>/chunk_size
>> +# setting. This setting allows the admin to change the chunk size
>> +# setting for the next allocation.
>> +#
>> +# Test 1:
>> +# Allocate storage for all three block types (data, metadata and system)
>> +# with the default chunk size.
>> +#
>> +# Test 2:
>> +# Set a new chunk size to double the default size and allocate space
>> +# for all new block types with the new chunk size.
>> +#
>> +# Test 3:
>> +# Pick an allocation size that is used in a loop and make sure the last
>> +# allocation cannot be partially fullfilled.
>> +#
>> +# Note: Variable naming uses the following convention: if a variable name
>> +# ends in "_B" then its a byte value, if it ends in "_MB" then the
>> +# value is in megabytes.
>> +#
>> +. ./common/preamble
>> +_begin_fstest auto
>> +
>> +test_file="${TEST_DIR}/${seq}"
>
> Seems it's not used in the test?
>
Correct. I removed it.
>> +seq=`basename $0`
>> +seqres="${RESULT_DIR}/${seq}"
>> +
>> +# Override the default cleanup function.
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f "$test_file"
>
> Then there's no need to remove it in cleanup
>
Removed.
>> +}
>> +
>> +# Parse a size string which is in the format "XX.XXMib".
>> +#
>> +# Parameters:
>> +# - (IN) Block group type (Data, Metadata, System)
>> +# - (INOUT) Variable to store block group size in MB
>> +#
>> +parse_size_string() {
>> + local SIZE=$(echo "$1" | $AWK_PROG 'match($0, /([0-9.]+)/) { print substr($0, RSTART, RLENGTH) }')
>> + eval $2="${SIZE%.*}"
>
> The above "eval" line uses spaces as indentions not tab.
>
Fixed.
>> +}
>> +
>> +# Determine the size of the device in MB.
>> +#
>> +# Parameters:
>> +# - (INOUT) Variable to store device size in MB
>> +#
>> +device_size() {
>> + btrfs fi show ${SCRATCH_MNT} --mbytes | grep devid >> $seqres.full
>> + local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
>> + parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
>> + eval $1=${SIZE_ALLOC%.*}
>> +}
>> +
>> +# Determine the free space of a block group in MB.
>> +#
>> +# Parameters:
>> +# - (INOUT) Variable to store free space in MB
>> +#
>> +free_space() {
>> + local SIZE=$($BTRFS_UTIL_PROG filesystem show ${SCRATCH_MNT} --mbytes | grep devid)
>> + parse_size_string $(echo "${SIZE}" | awk '{print $4}') SIZE_ALLOC
>> + parse_size_string $(echo "${SIZE}" | awk '{print $6}') SIZE_USED
>> + eval $1=$(expr ${SIZE_ALLOC} - ${SIZE_USED})
>> +}
>> +
>> +# Determine how much space in MB has been allocated to a block group.
>> +#
>> +# Parameters:
>> +# - (IN) Block group type (Data, Metadata, System)
>> +# - (INOUT) Variable to store block group size in MB
>> +#
>> +alloc_size() {
>> + local SIZE_STRING=$($BTRFS_UTIL_PROG filesystem df ${SCRATCH_MNT} -m | grep "$1" | awk '{print $3}')
>> + parse_size_string ${SIZE_STRING} BLOCK_GROUP_SIZE
>> + eval $2="${BLOCK_GROUP_SIZE}"
>> +}
>> +
>> +. ./common/filter
>> +_supported_fs btrfs
>> +_require_test
>> +_require_scratch
>> +_require_btrfs_fs_sysfs
>
> Looks like we could drop this require, as _require_fs_sysfs below
> ensures this requirement must be met.
>
Removed.
>> +
>> +# Delete log file if it exists.
>> +rm -f "${seqres}.full"
>> +
>> +# Make filesystem.
>> +_scratch_mkfs_sized $((10 * 1024 * 1024 * 1024)) >> $seqres.full 2>&1
>
> Any reason we need a 10G filesystem? Some comments would be good.
>
I added a comment.
>> +_scratch_mount >> $seqres.full 2>&1
>> +
>> +# Check if there is sufficient sysfs support.
>> +_require_fs_sysfs allocation/metadata/chunk_size
>> +_require_fs_sysfs allocation/metadata/force_chunk_alloc
>> +
>> +# Get free space.
>> +free_space FREE_SPACE_MB
>> +device_size DEVICE_SIZE_MB
>> +
>> +echo "free space = ${FREE_SPACE_MB}MB" >> ${seqres}.full
>> +
>> +# If device is a symbolic link, get block device.
>> +SCRATCH_BDEV=${SCRATCH_DEV}
>> +if [[ -h ${SCRATCH_DEV} ]]; then
>> + SCRATCH_BDEV=$(readlink -f $SCRATCH_DEV)
>> +fi
>> +
>> +# Get chunk sizes.
>> +echo "Capture default chunk sizes."
>> +FIRST_DATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
>> +FIRST_METADATA_CHUNK_SIZE_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
>> +
>> +echo "Orig Data chunk size = ${FIRST_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
>> +echo "Orig Metaata chunk size = ${FIRST_METADATA_CHUNK_SIZE_B}" >> ${seqres}.full
>> +
>> +INIT_ALLOC_SIZE_MB=$(expr \( ${FIRST_DATA_CHUNK_SIZE_B} + ${FIRST_METADATA_CHUNK_SIZE_B} \) / 1024 / 1024)
>> +echo "Allocation size for initial allocation = ${INIT_ALLOC_SIZE_MB}MB" >> $seqres.full
>> +
>> +#
>> +# Do first allocation with the default chunk sizes for the different block
>> +# types.
>> +#
>> +echo "First allocation."
>> +alloc_size "Data" DATA_SIZE_START_MB
>> +alloc_size "Metadata" METADATA_SIZE_START_MB
>> +
>> +echo "Block group Data alloc size = ${DATA_SIZE_START_MB}MB" >> $seqres.full
>> +echo "Block group Metadata alloc size = ${METADATA_SIZE_START_MB}MB" >> $seqres.full
>> +
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
>> +
>> +alloc_size "Data" FIRST_DATA_SIZE_MB
>> +alloc_size "Metadata" FIRST_METADATA_SIZE_MB
>> +
>> +echo "First block group Data alloc size = ${FIRST_DATA_SIZE_MB}MB" >> ${seqres}.full
>> +echo "First block group Metadata alloc size = ${FIRST_METADATA_SIZE_MB}MB" >> ${seqres}.full
>> +
>> +free_space FREE_SPACE_AFTER
>> +echo "Free space after first allocation = ${FREE_SPACE_AFTER}MB" >> ${seqres}.full
>> +
>> +#
>> +# Do allocation with the doubled chunk sizes for the different block types.
>> +#
>> +echo "Second allocation."
>> +SECOND_DATA_CHUNK_SIZE_B=$(expr ${FIRST_DATA_CHUNK_SIZE_B} \* 2)
>> +SECOND_METADATA_CHUNK_SIZE_B=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} \* 2)
>> +
>> +echo "Second block group Data calc alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
>> +echo "Second block group Metadata calc alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
>> +
>> +# Stripe size is limited to 10% of device size.
>> +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
>> + SECOND_DATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
>> +fi
>> +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -gt $(expr ${DEVICE_SIZE_MB} \* 10 \* 1024 \* 1024 / 100) ]]; then
>> + SECOND_METADATA_CHUNK_SIZE_B=$(expr ${DEVICE_SIZE_MB} \* 10 / 100 \* 1024 \* 1024)
>> +fi
>> +
>> +echo "Second block group Data alloc size = ${SECOND_DATA_CHUNK_SIZE_B}" >> $seqres.full
>> +echo "Second block group Metadata alloc size = ${SECOND_METADATA_CHUNK_SIZE_B}" >> $seqres.full
>> +
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${SECOND_DATA_CHUNK_SIZE_B}
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size ${SECOND_METADATA_CHUNK_SIZE_B}
>> +
>> +SECOND_DATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size)
>> +SECOND_METADATA_CHUNK_SIZE_READ_B=$(_get_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/chunk_size)
>> +
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
>> +echo "Allocated data chunk" >> $seqres.full
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/metadata/force_chunk_alloc 1
>> +echo "Allocated metadata chunk" >> $seqres.full
>> +
>> +alloc_size "Data" SECOND_DATA_SIZE_MB
>> +alloc_size "Metadata" SECOND_METADATA_SIZE_MB
>> +alloc_size "System" SECOND_SYSTEM_SIZE_MB
>> +
>> +echo "Calculate request size so last memory allocation cannot be completely fullfilled."
>> +free_space FREE_SPACE_MB
>> +
>> +# Find request size whose space allocation cannot be completely fullfilled.
>> +THIRD_DATA_CHUNK_SIZE_MB=$(expr 256)
>> +until [ ${THIRD_DATA_CHUNK_SIZE_MB} -gt $(expr 7 \* 1024) ]; do
>> + if [ $((FREE_SPACE_MB%THIRD_DATA_CHUNK_SIZE_MB)) -ne 0 ]; then
>> + break
>> + fi
>> + THIRD_DATA_CHUNK_SIZE_MB=$((THIRD_DATA_CHUNK_SIZE_MB+256))
>> +done
>> +
>> +if [[ ${THIRD_DATA_CHUNK_SIZE_MB} -eq $(expr 7 \* 1024) ]]; then
>> + _fail "Cannot find allocation size for partial block allocation."
>> +fi
>> +
>> +THIRD_DATA_CHUNK_SIZE_B=$(expr ${THIRD_DATA_CHUNK_SIZE_MB} \* 1024 \* 1024)
>> +echo "Allocation size in mb = ${THIRD_DATA_CHUNK_SIZE_MB}" >> ${seqres}.full
>> +echo "Allocation size in bytes = ${THIRD_DATA_CHUNK_SIZE_B}" >> ${seqres}.full
>> +
>> +#
>> +# Do allocation until free space is exhausted.
>> +#
>> +echo "Third allocation."
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/chunk_size ${THIRD_DATA_CHUNK_SIZE_B}
>> +
>> +free_space FREE_SPACE_MB
>> +while [ $FREE_SPACE_MB -gt $THIRD_DATA_CHUNK_SIZE_MB ]
>> +do
>> + alloc_size "Data" THIRD_DATA_SIZE_MB
>> + _set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/data/force_chunk_alloc 1
>> +
>> + free_space FREE_SPACE_MB
>> +done
>> +
>> +alloc_size "Data" FOURTH_DATA_SIZE_MB
>> +
>> +#
>> +# Force chunk allocation of system block type must fail.
>> +#
>> +echo "Force allocation of system block type must fail."
>> +_set_fs_sysfs_attr ${SCRATCH_BDEV} allocation/system/force_chunk_alloc 1 2>/dev/null
>> +if [ $? -ne 0 ]; then
>> + _dump_err "_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc"
>> +fi
>
> I don't think we need this check. We could just remove the
> "2 > /dev/null" part from the above _set_fs_sysfs_attr and it will print
> error message and break golden image on error.
>
If we don't add this, we need to add an error message to the log like:
/common/rc: line 4457: echo: write error: No space left on device
This is pretty fragile.
> Thanks,
> Eryu
>
>> +
>> +#
>> +# Verification of initial allocation.
>> +#
>> +echo "Verify first allocation."
>> +FIRST_DATA_CHUNK_SIZE_MB=$(expr ${FIRST_DATA_CHUNK_SIZE_B} / 1024 / 1024)
>> +FIRST_METADATA_CHUNK_SIZE_MB=$(expr ${FIRST_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
>> +
>> +# if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne $(expr ${FIRST_DATA_SIZE_MB}) ]]; then
>> +if [[ $(expr ${FIRST_DATA_CHUNK_SIZE_MB} + ${DATA_SIZE_START_MB}) -ne ${FIRST_DATA_SIZE_MB} ]]; then
>> + echo "tInitial data allocation not correct."
>> +fi
>> +
>> +if [[ $(expr ${FIRST_METADATA_CHUNK_SIZE_MB} + ${METADATA_SIZE_START_MB}) -ne ${FIRST_METADATA_SIZE_MB} ]]; then
>> + echo "Initial metadata allocation not correct."
>> +fi
>> +
>> +#
>> +# Verification of second allocation.
>> +#
>> +echo "Verify second allocation."
>> +SECOND_DATA_CHUNK_SIZE_MB=$(expr ${SECOND_DATA_CHUNK_SIZE_B} / 1024 / 1024)
>> +SECOND_METADATA_CHUNK_SIZE_MB=$(expr ${SECOND_METADATA_CHUNK_SIZE_B} / 1024 / 1024)
>> +
>> +if [[ ${SECOND_DATA_CHUNK_SIZE_B} -ne ${SECOND_DATA_CHUNK_SIZE_READ_B} ]]; then
>> + echo "Value written to allocation/data/chunk_size and read value are different."
>> +fi
>> +
>> +if [[ ${SECOND_METADATA_CHUNK_SIZE_B} -ne ${SECOND_METADATA_CHUNK_SIZE_READ_B} ]]; then
>> + echo "Value written to allocation/metadata/chunk_size and read value are different."
>> +fi
>> +
>> +if [[ $(expr ${SECOND_DATA_CHUNK_SIZE_MB} + ${FIRST_DATA_SIZE_MB}) -ne ${SECOND_DATA_SIZE_MB} ]]; then
>> + echo "Data allocation after chunk size change not correct."
>> +fi
>> +
>> +if [[ $(expr ${SECOND_METADATA_CHUNK_SIZE_MB} + ${FIRST_METADATA_SIZE_MB}) -ne ${SECOND_METADATA_SIZE_MB} ]]; then
>> + echo "Metadata allocation after chunk size change not correct."
>> +fi
>> +
>> +#
>> +# Verification of third allocation.
>> +#
>> +echo "Verify third allocation."
>> +if [[ ${FREE_SPACE_MB} -gt ${THIRD_DATA_CHUNK_SIZE_MB} ]]; then
>> + echo "Free space after allocating over memlimit is too high."
>> +fi
>> +
>> +# The + 1 is required as 1MB is always kept as free space.
>> +if [[ $(expr ${THIRD_DATA_CHUNK_SIZE_MB} + ${THIRD_DATA_SIZE_MB} + 1) -le $(expr ${FOURTH_DATA_SIZE_MB}) ]]; then
>> + echo "Allocation until out of memory: last memory allocation size is not correct."
>> +fi
>> +
>> +status=0
>> +exit
>> +
>> diff --git a/tests/btrfs/251.out b/tests/btrfs/251.out
>> new file mode 100644
>> index 00000000..bb7b45d8
>> --- /dev/null
>> +++ b/tests/btrfs/251.out
>> @@ -0,0 +1,11 @@
>> +QA output created by 248
>> +Capture default chunk sizes.
>> +First allocation.
>> +Second allocation.
>> +Calculate request size so last memory allocation cannot be completely fullfilled.
>> +Third allocation.
>> +Force allocation of system block type must fail.
>> +_set_fs_sysfs_attr cannot write allocation/system/force_chunk_alloc
>> +Verify first allocation.
>> +Verify second allocation.
>> +Verify third allocation.
>> --
>> 2.30.2
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2021-11-29 18:21 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-11-24 21:12 [PATCH v6] btrfs: Add new test for setting the chunk size Stefan Roesch
2021-11-28 15:55 ` Eryu Guan
2021-11-29 18:19 ` Stefan Roesch
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).