[PATCH v5 0/2] test i_blocks for truncated largefiles

[PATCH v5 0/2] test i_blocks for truncated largefiles

[Pavel Reichl]

Hello,

Thank you for your comments. Please see next revision.

V3
* introduced create_sizedfile() helper
* amended g/694 and g/698 to utilize this helper

V4
* copy-pasted the helper as suggested by Zorro (just proposed a different comment)
* updated the tests to use it and check its return value
* added _fixed_by_kernel_commit directive into g/694

V5
* updated helper comment as Darrick suggested
* changed the error message when failure to create test file occurs

Pavel Reichl (2):
  common: new helper to alloacate fixed size files
  generic: test i_blocks for truncated large files

 common/rc             | 23 +++++++++++++++++++
 tests/generic/694     |  8 ++++++-
 tests/generic/698     | 51 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/698.out |  2 ++
 4 files changed, 83 insertions(+), 1 deletion(-)
 create mode 100755 tests/generic/698
 create mode 100644 tests/generic/698.out

-- 
2.37.3

[PATCH v5 1/2] common: new helper to alloacate fixed size files

[Pavel Reichl]

Helper that creates files of specified size using falloc if supported,
otherwise pwrite is used.

Signed-off-by: Pavel Reichl <preichl@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 common/rc         | 23 +++++++++++++++++++++++
 tests/generic/694 |  8 +++++++-
 2 files changed, 30 insertions(+), 1 deletion(-)

diff --git a/common/rc b/common/rc
index a25cbcd0..228fcb37 100644
--- a/common/rc
+++ b/common/rc
@@ -4925,6 +4925,29 @@ hexdump()
 	_fail "Use _hexdump(), please!"
 }
 
+# Try to create a file with inode->i_blocks >= (length / blocksize).
+# There may be some small overhead, e.g. ext2 filesystem allocates a
+# substantial number of blocks to store block mappings. Those are accounted
+# to i_blocks.
+_create_file_sized()
+{
+	local length=$1
+	local file=$2
+	local tmp=`mktemp -u`
+	local ret=0
+
+	$XFS_IO_PROG -ft -c "falloc 0 $length" $file >$tmp.out 2>&1
+	ret=$?
+	if (grep -Eq "Operation not supported|command .* not found" $tmp.out);then
+		# fallocate isn't supported, fallback to general buffer write
+		$XFS_IO_PROG -ft -c "pwrite 0 $length" $file >$tmp.out 2>&1
+		ret=$?
+	fi
+	[ $ret -ne 0 ] && cat $tmp.out
+	rm -f $tmp.out
+	return $ret
+}
+
 init_rc
 
 ################################################################################
diff --git a/tests/generic/694 b/tests/generic/694
index dfd988df..c96c2154 100755
--- a/tests/generic/694
+++ b/tests/generic/694
@@ -21,6 +21,9 @@ _cleanup()
 }
 
 _supported_fs generic
+_fixed_by_kernel_commit 0c336d6e33f4 \
+	"exfat: fix incorrect loading of i_blocks for large file"
+
 _require_test
 _require_fs_space $TEST_DIR $((4 * 1024 * 1024)) #kB
 
@@ -30,7 +33,10 @@ junk_dir=$TEST_DIR/$seq
 junk_file=$junk_dir/junk
 mkdir -p $junk_dir
 
-$XFS_IO_PROG -f -c "pwrite -W 0 4G" $junk_file > /dev/null
+_create_file_sized 4G $junk_file
+if [ $? -ne 0 ]; then
+	echo "Could not create 4G test file"
+fi
 
 iblocks=`stat -c '%b' $junk_file`
 
-- 
2.37.3

[PATCH v5 2/2] generic: test i_blocks for truncated large files

[Pavel Reichl]

This is a regression test for an incorrect computation of i_blocks for
truncated files larger than 4 GiB. Bug was filed for exFAT.

Test is based on reproducer provied by Christophe Vu-Brugier as part
of kernel patch-fix submission.

Signed-off-by: Pavel Reichl <preichl@redhat.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/698     | 51 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/698.out |  2 ++
 2 files changed, 53 insertions(+)
 create mode 100755 tests/generic/698
 create mode 100644 tests/generic/698.out

diff --git a/tests/generic/698 b/tests/generic/698
new file mode 100755
index 00000000..6aeea64e
--- /dev/null
+++ b/tests/generic/698
@@ -0,0 +1,51 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022  Red Hat Inc. All Rights Reserved.
+#
+# FS QA Test 698
+#
+# Verify that i_blocks for truncated files larger than 4 GiB have correct
+# values.
+#
+# This test verifies the problem fixed in kernel with commit
+# 92fba084b79e exfat: fix i_blocks for files truncated over 4 GiB
+#
+. ./common/preamble
+. ./common/filter
+
+_begin_fstest auto
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	rm -r -f $tmp.* $junk_dir
+}
+
+_supported_fs generic
+_fixed_by_kernel_commit 92fba084b79e \
+	"exfat: fix i_blocks for files truncated over 4 GiB"
+
+_require_test
+_require_fs_space $TEST_DIR $((5 * 1024 * 1024)) #kB
+
+junk_dir=$TEST_DIR/$seq
+junk_file=$junk_dir/junk
+mkdir -p $junk_dir
+
+_create_file_sized 5G $junk_file
+if [ $? -ne 0 ]; then
+	echo "Could not create 5G test file"
+fi
+
+truncate -s 4G $junk_file
+
+block_size=`stat -c '%B' $junk_file`
+iblocks_after_truncate=`stat -c '%b' $junk_file`
+iblocks_expected=$((4 * 1024 * 1024 * 1024 / $block_size))
+
+_within_tolerance "Number of allocated blocks after truncate" $iblocks_after_truncate $iblocks_expected 1% -v
+
+status=0
+
+exit
diff --git a/tests/generic/698.out b/tests/generic/698.out
new file mode 100644
index 00000000..cbb02d37
--- /dev/null
+++ b/tests/generic/698.out
@@ -0,0 +1,2 @@
+QA output created by 698
+Number of allocated blocks after truncate is in range
-- 
2.37.3

Re: [PATCH v5 1/2] common: new helper to alloacate fixed size files

[Zorro Lang]

On Thu, Sep 22, 2022 at 08:38:35AM +0200, Pavel Reichl wrote:
> Helper that creates files of specified size using falloc if supported,
> otherwise pwrite is used.
> 
> Signed-off-by: Pavel Reichl <preichl@redhat.com>
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> ---
>  common/rc         | 23 +++++++++++++++++++++++
>  tests/generic/694 |  8 +++++++-
>  2 files changed, 30 insertions(+), 1 deletion(-)
> 
> diff --git a/common/rc b/common/rc
> index a25cbcd0..228fcb37 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -4925,6 +4925,29 @@ hexdump()
>  	_fail "Use _hexdump(), please!"
>  }
>  
> +# Try to create a file with inode->i_blocks >= (length / blocksize).
> +# There may be some small overhead, e.g. ext2 filesystem allocates a
> +# substantial number of blocks to store block mappings. Those are accounted
> +# to i_blocks.
> +_create_file_sized()
> +{
> +	local length=$1
> +	local file=$2
> +	local tmp=`mktemp -u`
> +	local ret=0
> +
> +	$XFS_IO_PROG -ft -c "falloc 0 $length" $file >$tmp.out 2>&1
> +	ret=$?
> +	if (grep -Eq "Operation not supported|command .* not found" $tmp.out);then
> +		# fallocate isn't supported, fallback to general buffer write
> +		$XFS_IO_PROG -ft -c "pwrite 0 $length" $file >$tmp.out 2>&1
> +		ret=$?
> +	fi
> +	[ $ret -ne 0 ] && cat $tmp.out
> +	rm -f $tmp.out
> +	return $ret
> +}
> +
>  init_rc
>  
>  ################################################################################
> diff --git a/tests/generic/694 b/tests/generic/694
> index dfd988df..c96c2154 100755
> --- a/tests/generic/694
> +++ b/tests/generic/694
> @@ -21,6 +21,9 @@ _cleanup()
>  }
>  
>  _supported_fs generic
> +_fixed_by_kernel_commit 0c336d6e33f4 \
> +	"exfat: fix incorrect loading of i_blocks for large file"
> +

Thanks for adding this too, this version looks good to me now.
Reviewed-by: Zorro Lang <zlang@redhat.com>

>  _require_test
>  _require_fs_space $TEST_DIR $((4 * 1024 * 1024)) #kB
>  
> @@ -30,7 +33,10 @@ junk_dir=$TEST_DIR/$seq
>  junk_file=$junk_dir/junk
>  mkdir -p $junk_dir
>  
> -$XFS_IO_PROG -f -c "pwrite -W 0 4G" $junk_file > /dev/null
> +_create_file_sized 4G $junk_file
> +if [ $? -ne 0 ]; then
> +	echo "Could not create 4G test file"
> +fi
>  
>  iblocks=`stat -c '%b' $junk_file`
>  
> -- 
> 2.37.3
> 

Re: [PATCH v5 2/2] generic: test i_blocks for truncated large files

[Zorro Lang]

On Thu, Sep 22, 2022 at 08:38:36AM +0200, Pavel Reichl wrote:
> This is a regression test for an incorrect computation of i_blocks for
> truncated files larger than 4 GiB. Bug was filed for exFAT.
> 
> Test is based on reproducer provied by Christophe Vu-Brugier as part
> of kernel patch-fix submission.
> 
> Signed-off-by: Pavel Reichl <preichl@redhat.com>
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> ---
>  tests/generic/698     | 51 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/698.out |  2 ++
>  2 files changed, 53 insertions(+)
>  create mode 100755 tests/generic/698
>  create mode 100644 tests/generic/698.out
> 
> diff --git a/tests/generic/698 b/tests/generic/698
> new file mode 100755
> index 00000000..6aeea64e
> --- /dev/null
> +++ b/tests/generic/698
> @@ -0,0 +1,51 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2022  Red Hat Inc. All Rights Reserved.
> +#
> +# FS QA Test 698
> +#
> +# Verify that i_blocks for truncated files larger than 4 GiB have correct
> +# values.
> +#
> +# This test verifies the problem fixed in kernel with commit
> +# 92fba084b79e exfat: fix i_blocks for files truncated over 4 GiB
> +#
> +. ./common/preamble
> +. ./common/filter
> +
> +_begin_fstest auto
> +
> +# Override the default cleanup function.
> +_cleanup()
> +{
> +	cd /
> +	rm -r -f $tmp.* $junk_dir
> +}
> +
> +_supported_fs generic
> +_fixed_by_kernel_commit 92fba084b79e \
> +	"exfat: fix i_blocks for files truncated over 4 GiB"
> +
> +_require_test
> +_require_fs_space $TEST_DIR $((5 * 1024 * 1024)) #kB
> +
> +junk_dir=$TEST_DIR/$seq
> +junk_file=$junk_dir/junk
> +mkdir -p $junk_dir
> +
> +_create_file_sized 5G $junk_file
> +if [ $? -ne 0 ]; then
> +	echo "Could not create 5G test file"
> +fi

Good to me,
Reviewed-by: Zorro Lang <zlang@redhat.com>

> +
> +truncate -s 4G $junk_file
> +
> +block_size=`stat -c '%B' $junk_file`
> +iblocks_after_truncate=`stat -c '%b' $junk_file`
> +iblocks_expected=$((4 * 1024 * 1024 * 1024 / $block_size))
> +
> +_within_tolerance "Number of allocated blocks after truncate" $iblocks_after_truncate $iblocks_expected 1% -v
> +
> +status=0
> +
> +exit
> diff --git a/tests/generic/698.out b/tests/generic/698.out
> new file mode 100644
> index 00000000..cbb02d37
> --- /dev/null
> +++ b/tests/generic/698.out
> @@ -0,0 +1,2 @@
> +QA output created by 698
> +Number of allocated blocks after truncate is in range
> -- 
> 2.37.3
> 

Re: [PATCH v5 1/2] common: new helper to alloacate fixed size files

[Darrick J. Wong]

On Thu, Sep 22, 2022 at 08:38:35AM +0200, Pavel Reichl wrote:
> Helper that creates files of specified size using falloc if supported,
> otherwise pwrite is used.
> 
> Signed-off-by: Pavel Reichl <preichl@redhat.com>
> Reviewed-by: Darrick J. Wong <djwong@kernel.org>

Technically speaking, I only RVB-tagged the second patch, not this first
one...

> ---
>  common/rc         | 23 +++++++++++++++++++++++
>  tests/generic/694 |  8 +++++++-
>  2 files changed, 30 insertions(+), 1 deletion(-)
> 
> diff --git a/common/rc b/common/rc
> index a25cbcd0..228fcb37 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -4925,6 +4925,29 @@ hexdump()
>  	_fail "Use _hexdump(), please!"
>  }
>  
> +# Try to create a file with inode->i_blocks >= (length / blocksize).
> +# There may be some small overhead, e.g. ext2 filesystem allocates a
> +# substantial number of blocks to store block mappings. Those are accounted
> +# to i_blocks.
> +_create_file_sized()
> +{
> +	local length=$1
> +	local file=$2
> +	local tmp=`mktemp -u`
> +	local ret=0
> +
> +	$XFS_IO_PROG -ft -c "falloc 0 $length" $file >$tmp.out 2>&1
> +	ret=$?
> +	if (grep -Eq "Operation not supported|command .* not found" $tmp.out);then
> +		# fallocate isn't supported, fallback to general buffer write
> +		$XFS_IO_PROG -ft -c "pwrite 0 $length" $file >$tmp.out 2>&1
> +		ret=$?
> +	fi
> +	[ $ret -ne 0 ] && cat $tmp.out
> +	rm -f $tmp.out
> +	return $ret
> +}
> +
>  init_rc
>  
>  ################################################################################
> diff --git a/tests/generic/694 b/tests/generic/694
> index dfd988df..c96c2154 100755
> --- a/tests/generic/694
> +++ b/tests/generic/694
> @@ -21,6 +21,9 @@ _cleanup()
>  }
>  
>  _supported_fs generic
> +_fixed_by_kernel_commit 0c336d6e33f4 \
> +	"exfat: fix incorrect loading of i_blocks for large file"
> +
>  _require_test
>  _require_fs_space $TEST_DIR $((4 * 1024 * 1024)) #kB
>  
> @@ -30,7 +33,10 @@ junk_dir=$TEST_DIR/$seq
>  junk_file=$junk_dir/junk
>  mkdir -p $junk_dir
>  
> -$XFS_IO_PROG -f -c "pwrite -W 0 4G" $junk_file > /dev/null
> +_create_file_sized 4G $junk_file
> +if [ $? -ne 0 ]; then
> +	echo "Could not create 4G test file"

...but this does look good now.
Reviewed-by: Darrick J. Wong <djwong@kernel.org>

--D

> +fi
>  
>  iblocks=`stat -c '%b' $junk_file`
>  
> -- 
> 2.37.3
> 

Re: [PATCH v5 1/2] common: new helper to alloacate fixed size files

[Pavel Reichl]

On 9/22/22 18:22, Darrick J. Wong wrote:
> On Thu, Sep 22, 2022 at 08:38:35AM +0200, Pavel Reichl wrote:
>> Helper that creates files of specified size using falloc if supported,
>> otherwise pwrite is used.
>>
>> Signed-off-by: Pavel Reichl <preichl@redhat.com>
>> Reviewed-by: Darrick J. Wong <djwong@kernel.org>
> Technically speaking, I only RVB-tagged the second patch, not this first
> one...
Oh, sorry about that - my bad.