[PATCH v3] btrfs/237: Use zone cap instead of zone size in fill_size and rest calculation

[PATCH v3] btrfs/237: Use zone cap instead of zone size in fill_size and rest calculation

[Pankaj Raghav]

This test will break when zone capacity != zone size because the
calculation of the size to be filled is done using zone_size instead of
the actual capacity available per zone. Fix it by using zone capacity.

As a zoned device can have variable capacity, use the btrfs utility to
get the zone capacity from the first data block group that the test will
be performed on.

The support to extract zone capacity was added to blkzone only from
version 2.37. So zcap will be used only when the blkzone version is
greater or equal to 2.37.

Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
---
Changes since v1:
- Add a filter that can append a 'cap' column for blkzone version <2.37
  (Naohiro)
- Extract the capacity from the zone on which the test is performed as a
  zoned device can have variable zone capacity(Naohiro)

Changes since v2:
- Use btrfs inspect-internal utility to find the first data zone address
instead of hardcoding. (Naohiro)

 common/filter   | 13 +++++++++++++
 tests/btrfs/237 | 21 +++++++++++++++++----
 2 files changed, 30 insertions(+), 4 deletions(-)

diff --git a/common/filter b/common/filter
index 257227c2..5fe86756 100644
--- a/common/filter
+++ b/common/filter
@@ -634,5 +634,18 @@ _filter_bash()
 	sed -e "s/^bash: line 1: /bash: /"
 }
 
+#
+# blkzone report added zone capacity to be printed from v2.37.
+# This filter will add an extra column 'cap' with the same value of
+# 'len'(zone size) for blkzone version < 2.37
+#
+# Before: start: 0x000100000, len 0x040000, wptr 0x000000 ..
+# After: start: 0x000100000, len 0x040000, cap 0x040000, wptr 0x000000 ..
+_filter_blkzone_report()
+{
+	$AWK_PROG -F "," 'BEGIN{OFS=",";} $3 !~ /cap/ {$2=$2","$2;} {print;}' |\
+	sed -e 's/len/cap/2'
+}
+
 # make sure this script returns success
 /bin/true
diff --git a/tests/btrfs/237 b/tests/btrfs/237
index 96940549..cb897416 100755
--- a/tests/btrfs/237
+++ b/tests/btrfs/237
@@ -35,8 +35,13 @@ get_data_bg()
 		grep -Eo "CHUNK_ITEM [[:digit:]]+" | cut -d ' ' -f 2
 }
 
-zonesize=$(cat /sys/block/$(_short_dev $SCRATCH_DEV)/queue/chunk_sectors)
-zonesize=$((zonesize << 9))
+get_data_bg_physical()
+{
+	# Assumes SINGLE data profile
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t CHUNK $SCRATCH_DEV |\
+		grep -A 4 CHUNK_ITEM | grep -A 3 'type DATA\|SINGLE' |\
+	        grep -Eo 'offset [[:digit:]]+'| cut -d ' ' -f 2
+}
 
 _scratch_mkfs >/dev/null 2>&1
 _scratch_mount -o commit=1 # 1s commit time to speed up test
@@ -49,12 +54,20 @@ if [[ "$uuid" == "" ]]; then
 	exit 1
 fi
 
+start_data_bg_phy=$(get_data_bg_physical)
+start_data_bg_phy=$((data_bg_zone >> 9))
+
+size=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
+	_filter_blkzone_report |\
+	grep -Po "cap ([0x\d]+)+" | cut -d ' ' -f 2)
+size=$((size << 9))
+
 reclaim_threshold=75
 echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
 fill_percent=$((reclaim_threshold + 2))
 rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
-fill_size=$((zonesize * fill_percent / 100))
-rest=$((zonesize * rest_percent / 100))
+fill_size=$((size * fill_percent / 100))
+rest=$((size * rest_percent / 100))
 
 # step 1, fill FS over $fillsize
 $XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
-- 
2.25.1

Re: [PATCH v3] btrfs/237: Use zone cap instead of zone size in fill_size and rest calculation

[Pankaj Raghav]

friendly ping....

On 2022-03-21 12:58, Pankaj Raghav wrote:
> This test will break when zone capacity != zone size because the
> calculation of the size to be filled is done using zone_size instead of
> the actual capacity available per zone. Fix it by using zone capacity.
> 
> As a zoned device can have variable capacity, use the btrfs utility to
> get the zone capacity from the first data block group that the test will
> be performed on.
> 
> The support to extract zone capacity was added to blkzone only from
> version 2.37. So zcap will be used only when the blkzone version is
> greater or equal to 2.37.
> 
> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>
> ---
> Changes since v1:
> - Add a filter that can append a 'cap' column for blkzone version <2.37
>   (Naohiro)
> - Extract the capacity from the zone on which the test is performed as a
>   zoned device can have variable zone capacity(Naohiro)
> 
> Changes since v2:
> - Use btrfs inspect-internal utility to find the first data zone address
> instead of hardcoding. (Naohiro)
> 
>  common/filter   | 13 +++++++++++++
>  tests/btrfs/237 | 21 +++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/common/filter b/common/filter
> index 257227c2..5fe86756 100644
> --- a/common/filter
> +++ b/common/filter
> @@ -634,5 +634,18 @@ _filter_bash()
>  	sed -e "s/^bash: line 1: /bash: /"
>  }
>  
> +#
> +# blkzone report added zone capacity to be printed from v2.37.
> +# This filter will add an extra column 'cap' with the same value of
> +# 'len'(zone size) for blkzone version < 2.37
> +#
> +# Before: start: 0x000100000, len 0x040000, wptr 0x000000 ..
> +# After: start: 0x000100000, len 0x040000, cap 0x040000, wptr 0x000000 ..
> +_filter_blkzone_report()
> +{
> +	$AWK_PROG -F "," 'BEGIN{OFS=",";} $3 !~ /cap/ {$2=$2","$2;} {print;}' |\
> +	sed -e 's/len/cap/2'
> +}
> +
>  # make sure this script returns success
>  /bin/true
> diff --git a/tests/btrfs/237 b/tests/btrfs/237
> index 96940549..cb897416 100755
> --- a/tests/btrfs/237
> +++ b/tests/btrfs/237
> @@ -35,8 +35,13 @@ get_data_bg()
>  		grep -Eo "CHUNK_ITEM [[:digit:]]+" | cut -d ' ' -f 2
>  }
>  
> -zonesize=$(cat /sys/block/$(_short_dev $SCRATCH_DEV)/queue/chunk_sectors)
> -zonesize=$((zonesize << 9))
> +get_data_bg_physical()
> +{
> +	# Assumes SINGLE data profile
> +	$BTRFS_UTIL_PROG inspect-internal dump-tree -t CHUNK $SCRATCH_DEV |\
> +		grep -A 4 CHUNK_ITEM | grep -A 3 'type DATA\|SINGLE' |\
> +	        grep -Eo 'offset [[:digit:]]+'| cut -d ' ' -f 2
> +}
>  
>  _scratch_mkfs >/dev/null 2>&1
>  _scratch_mount -o commit=1 # 1s commit time to speed up test
> @@ -49,12 +54,20 @@ if [[ "$uuid" == "" ]]; then
>  	exit 1
>  fi
>  
> +start_data_bg_phy=$(get_data_bg_physical)
> +start_data_bg_phy=$((data_bg_zone >> 9))
> +
> +size=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
> +	_filter_blkzone_report |\
> +	grep -Po "cap ([0x\d]+)+" | cut -d ' ' -f 2)
> +size=$((size << 9))
> +
>  reclaim_threshold=75
>  echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
>  fill_percent=$((reclaim_threshold + 2))
>  rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
> -fill_size=$((zonesize * fill_percent / 100))
> -rest=$((zonesize * rest_percent / 100))
> +fill_size=$((size * fill_percent / 100))
> +rest=$((size * rest_percent / 100))
>  
>  # step 1, fill FS over $fillsize
>  $XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full

-- 
Regards,
Pankaj

Re: [PATCH v3] btrfs/237: Use zone cap instead of zone size in fill_size and rest calculation

[Naohiro Aota]

On Mon, Mar 21, 2022 at 12:58:48PM +0100, Pankaj Raghav wrote:
> This test will break when zone capacity != zone size because the
> calculation of the size to be filled is done using zone_size instead of
> the actual capacity available per zone. Fix it by using zone capacity.
> 
> As a zoned device can have variable capacity, use the btrfs utility to
> get the zone capacity from the first data block group that the test will
> be performed on.
> 
> The support to extract zone capacity was added to blkzone only from
> version 2.37. So zcap will be used only when the blkzone version is
> greater or equal to 2.37.
> 
> Signed-off-by: Pankaj Raghav <p.raghav@samsung.com>

I tried this test, but it didn't work. See below.

> ---
> Changes since v1:
> - Add a filter that can append a 'cap' column for blkzone version <2.37
>   (Naohiro)
> - Extract the capacity from the zone on which the test is performed as a
>   zoned device can have variable zone capacity(Naohiro)
> 
> Changes since v2:
> - Use btrfs inspect-internal utility to find the first data zone address
> instead of hardcoding. (Naohiro)
> 
>  common/filter   | 13 +++++++++++++
>  tests/btrfs/237 | 21 +++++++++++++++++----
>  2 files changed, 30 insertions(+), 4 deletions(-)
> 
> diff --git a/common/filter b/common/filter
> index 257227c2..5fe86756 100644
> --- a/common/filter
> +++ b/common/filter
> @@ -634,5 +634,18 @@ _filter_bash()
>  	sed -e "s/^bash: line 1: /bash: /"
>  }
>  
> +#
> +# blkzone report added zone capacity to be printed from v2.37.
> +# This filter will add an extra column 'cap' with the same value of
> +# 'len'(zone size) for blkzone version < 2.37
> +#
> +# Before: start: 0x000100000, len 0x040000, wptr 0x000000 ..
> +# After: start: 0x000100000, len 0x040000, cap 0x040000, wptr 0x000000 ..
> +_filter_blkzone_report()
> +{
> +	$AWK_PROG -F "," 'BEGIN{OFS=",";} $3 !~ /cap/ {$2=$2","$2;} {print;}' |\
> +	sed -e 's/len/cap/2'
> +}
> +
>  # make sure this script returns success
>  /bin/true
> diff --git a/tests/btrfs/237 b/tests/btrfs/237
> index 96940549..cb897416 100755
> --- a/tests/btrfs/237
> +++ b/tests/btrfs/237
> @@ -35,8 +35,13 @@ get_data_bg()
>  		grep -Eo "CHUNK_ITEM [[:digit:]]+" | cut -d ' ' -f 2
>  }
>  
> -zonesize=$(cat /sys/block/$(_short_dev $SCRATCH_DEV)/queue/chunk_sectors)
> -zonesize=$((zonesize << 9))
> +get_data_bg_physical()
> +{
> +	# Assumes SINGLE data profile
> +	$BTRFS_UTIL_PROG inspect-internal dump-tree -t CHUNK $SCRATCH_DEV |\
> +		grep -A 4 CHUNK_ITEM | grep -A 3 'type DATA\|SINGLE' |\
> +	        grep -Eo 'offset [[:digit:]]+'| cut -d ' ' -f 2
> +}
>  
>  _scratch_mkfs >/dev/null 2>&1
>  _scratch_mount -o commit=1 # 1s commit time to speed up test
> @@ -49,12 +54,20 @@ if [[ "$uuid" == "" ]]; then
>  	exit 1
>  fi
>  
> +start_data_bg_phy=$(get_data_bg_physical)
> +start_data_bg_phy=$((data_bg_zone >> 9))

start_data_bg_phy=$((start_data_bg_phy >> 9)) ?

> +
> +size=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
> +	_filter_blkzone_report |\
> +	grep -Po "cap ([0x\d]+)+" | cut -d ' ' -f 2)

The pattern should be "cap 0x[[:xdigit:]]+" as it is hexadecimal. I
missed this point before.

> +size=$((size << 9))
> +
>  reclaim_threshold=75
>  echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
>  fill_percent=$((reclaim_threshold + 2))
>  rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
> -fill_size=$((zonesize * fill_percent / 100))
> -rest=$((zonesize * rest_percent / 100))
> +fill_size=$((size * fill_percent / 100))
> +rest=$((size * rest_percent / 100))
>  
>  # step 1, fill FS over $fillsize
>  $XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
> -- 
> 2.25.1
> 

Re: [PATCH v3] btrfs/237: Use zone cap instead of zone size in fill_size and rest calculation

[Pankaj Raghav]

On 2022-03-25 06:00, Naohiro Aota wrote:
>>  
>> +start_data_bg_phy=$(get_data_bg_physical)
>> +start_data_bg_phy=$((data_bg_zone >> 9))
> 
> start_data_bg_phy=$((start_data_bg_phy >> 9)) ?
> 
Oops. I fixed this in my test machine but forgot to put these while
sending my patch. Thanks.
>> +
>> +size=$($BLKZONE_PROG report -o $start_data_bg_phy -l 1 $SCRATCH_DEV |\
>> +	_filter_blkzone_report |\
>> +	grep -Po "cap ([0x\d]+)+" | cut -d ' ' -f 2)
> 
> The pattern should be "cap 0x[[:xdigit:]]+" as it is hexadecimal. I
> missed this point before.
> 
I will fix it up.
>> +size=$((size << 9))
>> +
>>  reclaim_threshold=75
>>  echo $reclaim_threshold > /sys/fs/btrfs/"$uuid"/bg_reclaim_threshold
>>  fill_percent=$((reclaim_threshold + 2))
>>  rest_percent=$((90 - fill_percent)) # make sure we're not creating a new BG
>> -fill_size=$((zonesize * fill_percent / 100))
>> -rest=$((zonesize * rest_percent / 100))
>> +fill_size=$((size * fill_percent / 100))
>> +rest=$((size * rest_percent / 100))
>>  
>>  # step 1, fill FS over $fillsize
>>  $XFS_IO_PROG -fc "pwrite 0 $fill_size" $SCRATCH_MNT/$seq.test1 >> $seqres.full
>> -- 
>> 2.25.1

-- 
Regards,
Pankaj