[PATCHSET 0/4] fstests: random fixes

[PATCHSET 0/4] fstests: random fixes

[Darrick J. Wong]

Hi all,

Here's the usual batch of odd fixes for fstests.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=random-fixes

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=random-fixes

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
---
 common/xfs            |    6 +++---
 tests/generic/459     |    1 +
 tests/generic/673.out |    4 ++--
 tests/xfs/420         |   33 +++++++++++++++++++++------------
 4 files changed, 27 insertions(+), 17 deletions(-)

[PATCH 1/4] generic/459: ensure that the lvm devices have been created

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Once in a /very/ long while this test fails because _mkfs_dev can't find
the LVM thinp volume after it's been created.  Since the "solution" du
jour seems to be to sprinkle udevadm settle calls everywhere, do that
here in the hopes that will fix it.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/459 |    1 +
 1 file changed, 1 insertion(+)


diff --git a/tests/generic/459 b/tests/generic/459
index cda19e6e..57d58e55 100755
--- a/tests/generic/459
+++ b/tests/generic/459
@@ -70,6 +70,7 @@ $LVM_PROG lvcreate  --virtualsize $virtsize \
 		    -T $vgname/$poolname \
 		    -n $lvname >>$seqres.full 2>&1
 
+$UDEV_SETTLE_PROG &>/dev/null
 _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
 
 # Running the test over the original volume doesn't reproduce the problem

[PATCH 2/4] common/xfs: fix broken code in _check_xfs_filesystem

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Fix some problems with undefined variables in the scrub control code.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 common/xfs |    6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)


diff --git a/common/xfs b/common/xfs
index 053b6189..ac1d021e 100644
--- a/common/xfs
+++ b/common/xfs
@@ -568,12 +568,12 @@ _check_xfs_filesystem()
 		# before executing a scrub operation.
 		$XFS_IO_PROG -c syncfs $mntpt >> $seqres.full 2>&1
 
-		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $mntpt > $tmp.scrub 2>&1
+		"$XFS_SCRUB_PROG" -v -d -n $mntpt > $tmp.scrub 2>&1
 		if [ $? -ne 0 ]; then
 			_log_err "_check_xfs_filesystem: filesystem on $device failed scrub"
-			echo "*** xfs_scrub $scrubflag -v -d -n output ***" >> $seqres.full
+			echo "*** xfs_scrub -v -d -n output ***" >> $seqres.full
 			cat $tmp.scrub >> $seqres.full
-			echo "*** end xfs_scrub output" >> $serqres.full
+			echo "*** end xfs_scrub output" >> $seqres.full
 			ok=0
 		fi
 		rm -f $tmp.scrub

[PATCH 3/4] xfs/420: fix occasional test failures due to pagecache readahead

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Every now and then, this test fails with this golden output:

--- xfs/420.out
+++ xfs/420.out.bad
@@ -29,7 +29,7 @@
 Whence Result
 DATA   0
 HOLE   131072
-DATA   196608
+DATA   192512
 HOLE   262144
 Compare files
 c2803804acc9936eef8aab42c119bfac  SCRATCH_MNT/test-420/file1

Curiously, the file checksums always match, and it's not *forbidden* for
the page cache to have a page backing an unwritten extent that hasn't
been written.

The condition that this test cares about is that block 3 (192k-256k) are
reported by SEEK_DATA as data even if the data fork has a hole and the
COW fork has an unwritten extent.  Matthew Wilcox thinks this is a side
effect of readahead.

To fix this occasional false failure, call SEEK_DATA and SEEK_HOLE only
on the offsets that we care about.

Suggested-by: Matthew Wilcox <willy@infradead.org>
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/420 |   33 +++++++++++++++++++++------------
 1 file changed, 21 insertions(+), 12 deletions(-)


diff --git a/tests/xfs/420 b/tests/xfs/420
index 12b17588..d38772c9 100755
--- a/tests/xfs/420
+++ b/tests/xfs/420
@@ -50,6 +50,24 @@ _scratch_mount >> $seqres.full 2>&1
 testdir=$SCRATCH_MNT/test-$seq
 mkdir $testdir
 
+# pagecache readahead can sometimes cause extra pages to be inserted into the
+# file mapping where we have an unwritten extent in the COW fork.  Call lseek
+# on each $blksz offset that interests us (as opposed to the whole file) so
+# that these extra pages are not disclosed.
+#
+# The important thing we're testing is that SEEK_DATA reports block 3 as data
+# when the COW fork has an unwritten mapping and the data fork has a hole.
+exercise_lseek() {
+	echo "Seek holes and data in file1"
+	$XFS_IO_PROG -c "seek -d 0" $testdir/file1
+	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file1 | sed -e '/Whence/d'
+	echo "Seek holes and data in file2"
+	$XFS_IO_PROG -c "seek -d 0" $testdir/file2
+	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file2 | sed -e '/Whence/d'
+	$XFS_IO_PROG -c "seek -d $((3 * blksz))" $testdir/file2 | sed -e '/Whence/d'
+	$XFS_IO_PROG -c "seek -h $((4 * blksz))" $testdir/file2 | sed -e '/Whence/d'
+}
+
 blksz=65536
 nr=8
 filesize=$((blksz * nr))
@@ -83,10 +101,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
 
-echo "Seek holes and data in file1"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
-echo "Seek holes and data in file2"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
+exercise_lseek
 
 echo "Compare files"
 md5sum $testdir/file1 | _filter_scratch
@@ -102,10 +117,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
 
-echo "Seek holes and data in file1"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
-echo "Seek holes and data in file2"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
+exercise_lseek
 
 echo "Compare files"
 md5sum $testdir/file1 | _filter_scratch
@@ -121,10 +133,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
 $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
 
-echo "Seek holes and data in file1"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
-echo "Seek holes and data in file2"
-$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
+exercise_lseek
 
 echo "Compare files"
 md5sum $testdir/file1 | _filter_scratch

[PATCH 4/4] generic/673: fix golden output to reflect vfs setgid behavior

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Filipe Manana pointed out[1] that the setgid dropping behavior encoded
in this generic test is based on some outdated XFS code, and not based
on what the VFS inode attribute change functions actually do.  Now that
we're working on fixing that, we should update the golden output to
reflect what all filesystems are supposed to be doing.

[1] https://lore.kernel.org/linux-xfs/CAL3q7H47iNQ=Wmk83WcGB-KBJVOEtR9+qGczzCeXJ9Y2KCV25Q@mail.gmail.com/

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/673.out |    4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)


diff --git a/tests/generic/673.out b/tests/generic/673.out
index 8df672d6..4d18bca2 100644
--- a/tests/generic/673.out
+++ b/tests/generic/673.out
@@ -3,7 +3,7 @@ Test 1 - qa_user, non-exec file
 310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
 6666 -rwSrwSrw- SCRATCH_MNT/a
 3784de23efab7a2074c9ec66901e39e5  SCRATCH_MNT/a
-2666 -rw-rwSrw- SCRATCH_MNT/a
+666 -rw-rw-rw- SCRATCH_MNT/a
 
 Test 2 - qa_user, group-exec file
 310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
@@ -15,7 +15,7 @@ Test 3 - qa_user, user-exec file
 310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
 6766 -rwsrwSrw- SCRATCH_MNT/a
 3784de23efab7a2074c9ec66901e39e5  SCRATCH_MNT/a
-2766 -rwxrwSrw- SCRATCH_MNT/a
+766 -rwxrw-rw- SCRATCH_MNT/a
 
 Test 4 - qa_user, all-exec file
 310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a

[PATCH 5/4] xfs/076: only create files on the data device

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

This test checks that filesystems with sparse inode support can continue
to allocate inodes when free space gets fragmented.  Inodes only exist
on the data device, so we need to ensure that realtime is not enabled on
the filesystem so that the rt metadata doesn't mess with the inode usage
percentage and cause a test failure.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/076 |   13 +++++++++++--
 1 file changed, 11 insertions(+), 2 deletions(-)

diff --git a/tests/xfs/076 b/tests/xfs/076
index eac7410e..8eef1367 100755
--- a/tests/xfs/076
+++ b/tests/xfs/076
@@ -55,12 +55,21 @@ _alloc_inodes()
 
 # real QA test starts here
 
-_require_scratch
+if [ -n "$SCRATCH_RTDEV" ]; then
+	# ./check won't know we unset SCRATCH_RTDEV
+	_require_scratch_nocheck
+else
+	_require_scratch
+fi
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "fpunch"
 _require_xfs_sparse_inodes
 
-_scratch_mkfs "-d size=50m -m crc=1 -i sparse" |
+# Disable the scratch rt device to avoid test failures relating to the rt
+# bitmap consuming all the free space in our small data device.
+unset SCRATCH_RTDEV
+
+_scratch_mkfs "-d size=50m -m crc=1 -i sparse" | tee -a $seqres.full |
 	_filter_mkfs > /dev/null 2> $tmp.mkfs
 . $tmp.mkfs	# for isize
 

[PATCH 6/4] xfs/17[035]: fix intermittent failures when filesystem metadata gets large

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

These tests check that the filestreams allocator never shares an AG
across multiple streams when there's plenty of space in the filesystem.
Recent increases in metadata overhead for newer features (e.g. bigger
logs due to reflink) can throw this off, so add another AG to the
formatted filesystem to encourage it to avoid the AG with the log.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 common/filestreams |    2 +-
 tests/xfs/170      |   16 +++++++++++-----
 tests/xfs/170.out  |    8 ++++----
 tests/xfs/171      |   16 ++++++++++++----
 tests/xfs/171.out  |    8 ++++----
 tests/xfs/173      |   16 ++++++++++++----
 tests/xfs/173.out  |    8 ++++----
 7 files changed, 48 insertions(+), 26 deletions(-)

diff --git a/common/filestreams b/common/filestreams
index 8165effe..62acb47c 100644
--- a/common/filestreams
+++ b/common/filestreams
@@ -80,7 +80,7 @@ _check_for_dupes()
 
 _test_streams() {
 
-	echo "# testing $* ...."
+	echo "# testing $* ...." | tee -a $seqres.full
 	local agcount="$1"
 	local agsize="$2" # in MB
 	local stream_count="$3"
diff --git a/tests/xfs/170 b/tests/xfs/170
index 5e622d24..b9ead341 100755
--- a/tests/xfs/170
+++ b/tests/xfs/170
@@ -25,11 +25,17 @@ _check_filestreams_support || _notrun "filestreams not available"
 # test small stream, multiple I/O per file, 30s timeout
 _set_stream_timeout_centisecs 3000
 
-# test streams does a mkfs and mount
-_test_streams 8 22 4 8 3 0 0
-_test_streams 8 22 4 8 3 1 0
-_test_streams 8 22 4 8 3 0 1
-_test_streams 8 22 4 8 3 1 1
+# This test checks that the filestreams allocator never allocates space in any
+# given AG into more than one stream when there's plenty of space on the
+# filesystem.  Newer feature sets (e.g. reflink) have increased the size of
+# the log for small filesystems, so we make sure there's one more AG than
+# filestreams to encourage the allocator to skip whichever AG owns the log.
+#
+# Exercise 9x 22MB AGs, 4 filestreams, 8 files per stream, and 3MB per file.
+_test_streams 9 22 4 8 3 0 0
+_test_streams 9 22 4 8 3 1 0
+_test_streams 9 22 4 8 3 0 1
+_test_streams 9 22 4 8 3 1 1
 
 status=0
 exit
diff --git a/tests/xfs/170.out b/tests/xfs/170.out
index e71515e9..16dcb795 100644
--- a/tests/xfs/170.out
+++ b/tests/xfs/170.out
@@ -1,20 +1,20 @@
 QA output created by 170
-# testing 8 22 4 8 3 0 0 ....
+# testing 9 22 4 8 3 0 0 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 8 22 4 8 3 1 0 ....
+# testing 9 22 4 8 3 1 0 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 8 22 4 8 3 0 1 ....
+# testing 9 22 4 8 3 0 1 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 8 22 4 8 3 1 1 ....
+# testing 9 22 4 8 3 1 1 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
diff --git a/tests/xfs/171 b/tests/xfs/171
index 4412fe2f..f93b6011 100755
--- a/tests/xfs/171
+++ b/tests/xfs/171
@@ -29,10 +29,18 @@ _check_filestreams_support || _notrun "filestreams not available"
 # 100 = 78.1% full, should reliably succeed
 _set_stream_timeout_centisecs 12000
 
-_test_streams 64 16 8 100 1 1 0
-_test_streams 64 16 8 100 1 1 1
-_test_streams 64 16 8 100 1 0 0
-_test_streams 64 16 8 100 1 0 1
+# This test tries to get close to the exact point at which the filestreams
+# allocator will start to allocate space from some AG into more than one
+# stream.  Newer feature sets (e.g. reflink) have increased the size of the log
+# for small filesystems, so we make sure there's one more AG than filestreams
+# to encourage the allocator to skip whichever AG owns the log.
+#
+# This test exercises 64x 16MB AGs, 8 filestreams, 100 files per stream, and
+# 1MB per file.
+_test_streams 65 16 8 100 1 1 0
+_test_streams 65 16 8 100 1 1 1
+_test_streams 65 16 8 100 1 0 0
+_test_streams 65 16 8 100 1 0 1
 
 status=0
 exit
diff --git a/tests/xfs/171.out b/tests/xfs/171.out
index 89407cb2..73f73c90 100644
--- a/tests/xfs/171.out
+++ b/tests/xfs/171.out
@@ -1,20 +1,20 @@
 QA output created by 171
-# testing 64 16 8 100 1 1 0 ....
+# testing 65 16 8 100 1 1 0 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 64 16 8 100 1 1 1 ....
+# testing 65 16 8 100 1 1 1 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 64 16 8 100 1 0 0 ....
+# testing 65 16 8 100 1 0 0 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 64 16 8 100 1 0 1 ....
+# testing 65 16 8 100 1 0 1 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
diff --git a/tests/xfs/173 b/tests/xfs/173
index bce6ac51..6b18d919 100755
--- a/tests/xfs/173
+++ b/tests/xfs/173
@@ -26,10 +26,18 @@ _check_filestreams_support || _notrun "filestreams not available"
 # be less than or equal to half the AG count so we don't run out of AGs.
 _set_stream_timeout_centisecs 12000
 
-_test_streams 64 16 33 8 2 1 1 fail
-_test_streams 64 16 32 8 2 0 1
-_test_streams 64 16 33 8 2 0 0 fail
-_test_streams 64 16 32 8 2 1 0
+# This test checks the exact point at which the filestreams allocator will
+# start to allocate space from some AG into more than one stream.  Newer
+# feature sets (e.g. reflink) have increased the size of the log for small
+# filesystems, so we make sure there's one more AG than filestreams to
+# encourage the allocator to skip whichever AG owns the log.
+#
+# Exercise 65x 16MB AGs, 32/33 filestreams, 8 files per stream, and 2MB per
+# file.
+_test_streams 65 16 34 8 2 1 1 fail
+_test_streams 65 16 32 8 2 0 1
+_test_streams 65 16 34 8 2 0 0 fail
+_test_streams 65 16 32 8 2 1 0
 
 status=0
 exit
diff --git a/tests/xfs/173.out b/tests/xfs/173.out
index 21493057..705c352a 100644
--- a/tests/xfs/173.out
+++ b/tests/xfs/173.out
@@ -1,20 +1,20 @@
 QA output created by 173
-# testing 64 16 33 8 2 1 1 fail ....
+# testing 65 16 34 8 2 1 1 fail ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + expected failure, matching AGs
-# testing 64 16 32 8 2 0 1 ....
+# testing 65 16 32 8 2 0 1 ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + passed, streams are in seperate AGs
-# testing 64 16 33 8 2 0 0 fail ....
+# testing 65 16 34 8 2 0 0 fail ....
 # streaming
 # sync AGs...
 # checking stream AGs...
 + expected failure, matching AGs
-# testing 64 16 32 8 2 1 0 ....
+# testing 65 16 32 8 2 1 0 ....
 # streaming
 # sync AGs...
 # checking stream AGs...

Re: [PATCH 1/4] generic/459: ensure that the lvm devices have been created

[Zorro Lang]

On Tue, Mar 15, 2022 at 08:30:09PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Once in a /very/ long while this test fails because _mkfs_dev can't find
> the LVM thinp volume after it's been created.  Since the "solution" du
> jour seems to be to sprinkle udevadm settle calls everywhere, do that
> here in the hopes that will fix it.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---

Good to me,
Reviewed-by: Zorro Lang <zlang@redhat.com>

>  tests/generic/459 |    1 +
>  1 file changed, 1 insertion(+)
> 
> 
> diff --git a/tests/generic/459 b/tests/generic/459
> index cda19e6e..57d58e55 100755
> --- a/tests/generic/459
> +++ b/tests/generic/459
> @@ -70,6 +70,7 @@ $LVM_PROG lvcreate  --virtualsize $virtsize \
>  		    -T $vgname/$poolname \
>  		    -n $lvname >>$seqres.full 2>&1
>  
> +$UDEV_SETTLE_PROG &>/dev/null
>  _mkfs_dev /dev/mapper/$vgname-$lvname >>$seqres.full 2>&1
>  
>  # Running the test over the original volume doesn't reproduce the problem
> 

Re: [PATCH 2/4] common/xfs: fix broken code in _check_xfs_filesystem

[Zorro Lang]

On Tue, Mar 15, 2022 at 08:30:14PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Fix some problems with undefined variables in the scrub control code.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---

Actually from the beginning I don't know what's the $scrubflag used for, due
to it never be used in any other place of xfstests. I thought you left it for
personal debug reason, or even thought you'd like to make it as XFS_SCRUB_FLAG
to be a global parameter. Anyway I'm good if you'd like to remove it now.

Reviewed-by: Zorro Lang <zlang@redhat.com>

>  common/xfs |    6 +++---
>  1 file changed, 3 insertions(+), 3 deletions(-)
> 
> 
> diff --git a/common/xfs b/common/xfs
> index 053b6189..ac1d021e 100644
> --- a/common/xfs
> +++ b/common/xfs
> @@ -568,12 +568,12 @@ _check_xfs_filesystem()
>  		# before executing a scrub operation.
>  		$XFS_IO_PROG -c syncfs $mntpt >> $seqres.full 2>&1
>  
> -		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $mntpt > $tmp.scrub 2>&1
> +		"$XFS_SCRUB_PROG" -v -d -n $mntpt > $tmp.scrub 2>&1
>  		if [ $? -ne 0 ]; then
>  			_log_err "_check_xfs_filesystem: filesystem on $device failed scrub"
> -			echo "*** xfs_scrub $scrubflag -v -d -n output ***" >> $seqres.full
> +			echo "*** xfs_scrub -v -d -n output ***" >> $seqres.full
>  			cat $tmp.scrub >> $seqres.full
> -			echo "*** end xfs_scrub output" >> $serqres.full
> +			echo "*** end xfs_scrub output" >> $seqres.full
>  			ok=0
>  		fi
>  		rm -f $tmp.scrub
> 

Re: [PATCH 3/4] xfs/420: fix occasional test failures due to pagecache readahead

[Zorro Lang]

On Tue, Mar 15, 2022 at 08:30:20PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Every now and then, this test fails with this golden output:
> 
> --- xfs/420.out
> +++ xfs/420.out.bad
> @@ -29,7 +29,7 @@
>  Whence Result
>  DATA   0
>  HOLE   131072
> -DATA   196608
> +DATA   192512
>  HOLE   262144
>  Compare files
>  c2803804acc9936eef8aab42c119bfac  SCRATCH_MNT/test-420/file1

Looks like this part easy to cause `git am` misunderstanding[1], Hmm...
any method to deal with that?

[1]
Applying: xfs/420: fix occasional test failures due to pagecache readahead
error: 420.out: does not exist in index
Patch failed at 0001 xfs/420: fix occasional test failures due to pagecache readahead
hint: Use 'git am --show-current-patch=diff' to see the failed patch
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".

> 
> Curiously, the file checksums always match, and it's not *forbidden* for
> the page cache to have a page backing an unwritten extent that hasn't
> been written.
> 
> The condition that this test cares about is that block 3 (192k-256k) are
> reported by SEEK_DATA as data even if the data fork has a hole and the
> COW fork has an unwritten extent.  Matthew Wilcox thinks this is a side
> effect of readahead.
> 
> To fix this occasional false failure, call SEEK_DATA and SEEK_HOLE only
> on the offsets that we care about.
> 
> Suggested-by: Matthew Wilcox <willy@infradead.org>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
>  tests/xfs/420 |   33 +++++++++++++++++++++------------
>  1 file changed, 21 insertions(+), 12 deletions(-)
> 
> 
> diff --git a/tests/xfs/420 b/tests/xfs/420
> index 12b17588..d38772c9 100755
> --- a/tests/xfs/420
> +++ b/tests/xfs/420
> @@ -50,6 +50,24 @@ _scratch_mount >> $seqres.full 2>&1
>  testdir=$SCRATCH_MNT/test-$seq
>  mkdir $testdir
>  
> +# pagecache readahead can sometimes cause extra pages to be inserted into the
> +# file mapping where we have an unwritten extent in the COW fork.  Call lseek
> +# on each $blksz offset that interests us (as opposed to the whole file) so
> +# that these extra pages are not disclosed.
> +#
> +# The important thing we're testing is that SEEK_DATA reports block 3 as data
> +# when the COW fork has an unwritten mapping and the data fork has a hole.
> +exercise_lseek() {
> +	echo "Seek holes and data in file1"
> +	$XFS_IO_PROG -c "seek -d 0" $testdir/file1
> +	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file1 | sed -e '/Whence/d'
> +	echo "Seek holes and data in file2"
> +	$XFS_IO_PROG -c "seek -d 0" $testdir/file2
> +	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> +	$XFS_IO_PROG -c "seek -d $((3 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> +	$XFS_IO_PROG -c "seek -h $((4 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> +}
> +
>  blksz=65536
>  nr=8
>  filesize=$((blksz * nr))
> @@ -83,10 +101,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
>  
> -echo "Seek holes and data in file1"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> -echo "Seek holes and data in file2"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> +exercise_lseek
>  
>  echo "Compare files"
>  md5sum $testdir/file1 | _filter_scratch
> @@ -102,10 +117,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
>  
> -echo "Seek holes and data in file1"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> -echo "Seek holes and data in file2"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> +exercise_lseek
>  
>  echo "Compare files"
>  md5sum $testdir/file1 | _filter_scratch
> @@ -121,10 +133,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
>  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
>  
> -echo "Seek holes and data in file1"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> -echo "Seek holes and data in file2"
> -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> +exercise_lseek
>  
>  echo "Compare files"
>  md5sum $testdir/file1 | _filter_scratch
> 

Re: [PATCH 4/4] generic/673: fix golden output to reflect vfs setgid behavior

[Zorro Lang]

On Tue, Mar 15, 2022 at 08:30:25PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> Filipe Manana pointed out[1] that the setgid dropping behavior encoded
> in this generic test is based on some outdated XFS code, and not based
> on what the VFS inode attribute change functions actually do.  Now that
> we're working on fixing that, we should update the golden output to
> reflect what all filesystems are supposed to be doing.
> 
> [1] https://lore.kernel.org/linux-xfs/CAL3q7H47iNQ=Wmk83WcGB-KBJVOEtR9+qGczzCeXJ9Y2KCV25Q@mail.gmail.com/
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---

Thanks for making this change clear. As this's a generic test, other filesystem
might hit new failures if this test passed on them before. And hope some downstream
XFS users won't report too many customer bugs to us :) Anyway, it's good to me.

Reviewed-by: Zorro Lang <zlang@redhat.com>

>  tests/generic/673.out |    4 ++--
>  1 file changed, 2 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/tests/generic/673.out b/tests/generic/673.out
> index 8df672d6..4d18bca2 100644
> --- a/tests/generic/673.out
> +++ b/tests/generic/673.out
> @@ -3,7 +3,7 @@ Test 1 - qa_user, non-exec file
>  310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
>  6666 -rwSrwSrw- SCRATCH_MNT/a
>  3784de23efab7a2074c9ec66901e39e5  SCRATCH_MNT/a
> -2666 -rw-rwSrw- SCRATCH_MNT/a
> +666 -rw-rw-rw- SCRATCH_MNT/a
>  
>  Test 2 - qa_user, group-exec file
>  310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
> @@ -15,7 +15,7 @@ Test 3 - qa_user, user-exec file
>  310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
>  6766 -rwsrwSrw- SCRATCH_MNT/a
>  3784de23efab7a2074c9ec66901e39e5  SCRATCH_MNT/a
> -2766 -rwxrwSrw- SCRATCH_MNT/a
> +766 -rwxrw-rw- SCRATCH_MNT/a
>  
>  Test 4 - qa_user, all-exec file
>  310f146ce52077fcd3308dcbe7632bb2  SCRATCH_MNT/a
> 

Re: [PATCH 6/4] xfs/17[035]: fix intermittent failures when filesystem metadata gets large

[Zorro Lang]

On Wed, Mar 16, 2022 at 03:13:26PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> These tests check that the filestreams allocator never shares an AG
> across multiple streams when there's plenty of space in the filesystem.
> Recent increases in metadata overhead for newer features (e.g. bigger
> logs due to reflink) can throw this off, so add another AG to the
> formatted filesystem to encourage it to avoid the AG with the log.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---



>  common/filestreams |    2 +-
>  tests/xfs/170      |   16 +++++++++++-----
>  tests/xfs/170.out  |    8 ++++----
>  tests/xfs/171      |   16 ++++++++++++----
>  tests/xfs/171.out  |    8 ++++----
>  tests/xfs/173      |   16 ++++++++++++----
>  tests/xfs/173.out  |    8 ++++----

Looks good to me, and test passed after merging this patch. Just one
tiny question, the subject is "xfs/17[035]: ...", if "17[035]" mean
170, 173 and 175, but the files you changed are 170, 171 and 173 as
above :)

Reviewed-by: Zorro Lang <zlang@redhat.com>



>  7 files changed, 48 insertions(+), 26 deletions(-)
> 
> diff --git a/common/filestreams b/common/filestreams
> index 8165effe..62acb47c 100644
> --- a/common/filestreams
> +++ b/common/filestreams
> @@ -80,7 +80,7 @@ _check_for_dupes()
>  
>  _test_streams() {
>  
> -	echo "# testing $* ...."
> +	echo "# testing $* ...." | tee -a $seqres.full
>  	local agcount="$1"
>  	local agsize="$2" # in MB
>  	local stream_count="$3"
> diff --git a/tests/xfs/170 b/tests/xfs/170
> index 5e622d24..b9ead341 100755
> --- a/tests/xfs/170
> +++ b/tests/xfs/170
> @@ -25,11 +25,17 @@ _check_filestreams_support || _notrun "filestreams not available"
>  # test small stream, multiple I/O per file, 30s timeout
>  _set_stream_timeout_centisecs 3000
>  
> -# test streams does a mkfs and mount
> -_test_streams 8 22 4 8 3 0 0
> -_test_streams 8 22 4 8 3 1 0
> -_test_streams 8 22 4 8 3 0 1
> -_test_streams 8 22 4 8 3 1 1
> +# This test checks that the filestreams allocator never allocates space in any
> +# given AG into more than one stream when there's plenty of space on the
> +# filesystem.  Newer feature sets (e.g. reflink) have increased the size of
> +# the log for small filesystems, so we make sure there's one more AG than
> +# filestreams to encourage the allocator to skip whichever AG owns the log.
> +#
> +# Exercise 9x 22MB AGs, 4 filestreams, 8 files per stream, and 3MB per file.
> +_test_streams 9 22 4 8 3 0 0
> +_test_streams 9 22 4 8 3 1 0
> +_test_streams 9 22 4 8 3 0 1
> +_test_streams 9 22 4 8 3 1 1
>  
>  status=0
>  exit
> diff --git a/tests/xfs/170.out b/tests/xfs/170.out
> index e71515e9..16dcb795 100644
> --- a/tests/xfs/170.out
> +++ b/tests/xfs/170.out
> @@ -1,20 +1,20 @@
>  QA output created by 170
> -# testing 8 22 4 8 3 0 0 ....
> +# testing 9 22 4 8 3 0 0 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 8 22 4 8 3 1 0 ....
> +# testing 9 22 4 8 3 1 0 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 8 22 4 8 3 0 1 ....
> +# testing 9 22 4 8 3 0 1 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 8 22 4 8 3 1 1 ....
> +# testing 9 22 4 8 3 1 1 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
> diff --git a/tests/xfs/171 b/tests/xfs/171
> index 4412fe2f..f93b6011 100755
> --- a/tests/xfs/171
> +++ b/tests/xfs/171
> @@ -29,10 +29,18 @@ _check_filestreams_support || _notrun "filestreams not available"
>  # 100 = 78.1% full, should reliably succeed
>  _set_stream_timeout_centisecs 12000
>  
> -_test_streams 64 16 8 100 1 1 0
> -_test_streams 64 16 8 100 1 1 1
> -_test_streams 64 16 8 100 1 0 0
> -_test_streams 64 16 8 100 1 0 1
> +# This test tries to get close to the exact point at which the filestreams
> +# allocator will start to allocate space from some AG into more than one
> +# stream.  Newer feature sets (e.g. reflink) have increased the size of the log
> +# for small filesystems, so we make sure there's one more AG than filestreams
> +# to encourage the allocator to skip whichever AG owns the log.
> +#
> +# This test exercises 64x 16MB AGs, 8 filestreams, 100 files per stream, and
> +# 1MB per file.
> +_test_streams 65 16 8 100 1 1 0
> +_test_streams 65 16 8 100 1 1 1
> +_test_streams 65 16 8 100 1 0 0
> +_test_streams 65 16 8 100 1 0 1
>  
>  status=0
>  exit
> diff --git a/tests/xfs/171.out b/tests/xfs/171.out
> index 89407cb2..73f73c90 100644
> --- a/tests/xfs/171.out
> +++ b/tests/xfs/171.out
> @@ -1,20 +1,20 @@
>  QA output created by 171
> -# testing 64 16 8 100 1 1 0 ....
> +# testing 65 16 8 100 1 1 0 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 64 16 8 100 1 1 1 ....
> +# testing 65 16 8 100 1 1 1 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 64 16 8 100 1 0 0 ....
> +# testing 65 16 8 100 1 0 0 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 64 16 8 100 1 0 1 ....
> +# testing 65 16 8 100 1 0 1 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
> diff --git a/tests/xfs/173 b/tests/xfs/173
> index bce6ac51..6b18d919 100755
> --- a/tests/xfs/173
> +++ b/tests/xfs/173
> @@ -26,10 +26,18 @@ _check_filestreams_support || _notrun "filestreams not available"
>  # be less than or equal to half the AG count so we don't run out of AGs.
>  _set_stream_timeout_centisecs 12000
>  
> -_test_streams 64 16 33 8 2 1 1 fail
> -_test_streams 64 16 32 8 2 0 1
> -_test_streams 64 16 33 8 2 0 0 fail
> -_test_streams 64 16 32 8 2 1 0
> +# This test checks the exact point at which the filestreams allocator will
> +# start to allocate space from some AG into more than one stream.  Newer
> +# feature sets (e.g. reflink) have increased the size of the log for small
> +# filesystems, so we make sure there's one more AG than filestreams to
> +# encourage the allocator to skip whichever AG owns the log.
> +#
> +# Exercise 65x 16MB AGs, 32/33 filestreams, 8 files per stream, and 2MB per
> +# file.
> +_test_streams 65 16 34 8 2 1 1 fail
> +_test_streams 65 16 32 8 2 0 1
> +_test_streams 65 16 34 8 2 0 0 fail
> +_test_streams 65 16 32 8 2 1 0
>  
>  status=0
>  exit
> diff --git a/tests/xfs/173.out b/tests/xfs/173.out
> index 21493057..705c352a 100644
> --- a/tests/xfs/173.out
> +++ b/tests/xfs/173.out
> @@ -1,20 +1,20 @@
>  QA output created by 173
> -# testing 64 16 33 8 2 1 1 fail ....
> +# testing 65 16 34 8 2 1 1 fail ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + expected failure, matching AGs
> -# testing 64 16 32 8 2 0 1 ....
> +# testing 65 16 32 8 2 0 1 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + passed, streams are in seperate AGs
> -# testing 64 16 33 8 2 0 0 fail ....
> +# testing 65 16 34 8 2 0 0 fail ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
>  + expected failure, matching AGs
> -# testing 64 16 32 8 2 1 0 ....
> +# testing 65 16 32 8 2 1 0 ....
>  # streaming
>  # sync AGs...
>  # checking stream AGs...
> 

Re: [PATCH 5/4] xfs/076: only create files on the data device

[Zorro Lang]

On Wed, Mar 16, 2022 at 03:12:32PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
> 
> This test checks that filesystems with sparse inode support can continue
> to allocate inodes when free space gets fragmented.  Inodes only exist
> on the data device, so we need to ensure that realtime is not enabled on
> the filesystem so that the rt metadata doesn't mess with the inode usage
> percentage and cause a test failure.
> 
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---

Looks good to me,

Reviewed-by: Zorro Lang <zlang@redhat.com>

>  tests/xfs/076 |   13 +++++++++++--
>  1 file changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/tests/xfs/076 b/tests/xfs/076
> index eac7410e..8eef1367 100755
> --- a/tests/xfs/076
> +++ b/tests/xfs/076
> @@ -55,12 +55,21 @@ _alloc_inodes()
>  
>  # real QA test starts here
>  
> -_require_scratch
> +if [ -n "$SCRATCH_RTDEV" ]; then
> +	# ./check won't know we unset SCRATCH_RTDEV
> +	_require_scratch_nocheck
> +else
> +	_require_scratch
> +fi
>  _require_xfs_io_command "falloc"
>  _require_xfs_io_command "fpunch"
>  _require_xfs_sparse_inodes
>  
> -_scratch_mkfs "-d size=50m -m crc=1 -i sparse" |
> +# Disable the scratch rt device to avoid test failures relating to the rt
> +# bitmap consuming all the free space in our small data device.
> +unset SCRATCH_RTDEV
> +
> +_scratch_mkfs "-d size=50m -m crc=1 -i sparse" | tee -a $seqres.full |
>  	_filter_mkfs > /dev/null 2> $tmp.mkfs
>  . $tmp.mkfs	# for isize
>  
> 

Re: [PATCH 3/4] xfs/420: fix occasional test failures due to pagecache readahead

[Darrick J. Wong]

On Tue, Mar 22, 2022 at 01:13:54PM +0800, Zorro Lang wrote:
> On Tue, Mar 15, 2022 at 08:30:20PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > Every now and then, this test fails with this golden output:
> > 
> > --- xfs/420.out
> > +++ xfs/420.out.bad
> > @@ -29,7 +29,7 @@
> >  Whence Result
> >  DATA   0
> >  HOLE   131072
> > -DATA   196608
> > +DATA   192512
> >  HOLE   262144
> >  Compare files
> >  c2803804acc9936eef8aab42c119bfac  SCRATCH_MNT/test-420/file1
> 
> Looks like this part easy to cause `git am` misunderstanding[1], Hmm...
> any method to deal with that?

LOL.  Um.  I guess I'll edit the commit message to omit the diff header.

--D

> [1]
> Applying: xfs/420: fix occasional test failures due to pagecache readahead
> error: 420.out: does not exist in index
> Patch failed at 0001 xfs/420: fix occasional test failures due to pagecache readahead
> hint: Use 'git am --show-current-patch=diff' to see the failed patch
> When you have resolved this problem, run "git am --continue".
> If you prefer to skip this patch, run "git am --skip" instead.
> To restore the original branch and stop patching, run "git am --abort".
> 
> > 
> > Curiously, the file checksums always match, and it's not *forbidden* for
> > the page cache to have a page backing an unwritten extent that hasn't
> > been written.
> > 
> > The condition that this test cares about is that block 3 (192k-256k) are
> > reported by SEEK_DATA as data even if the data fork has a hole and the
> > COW fork has an unwritten extent.  Matthew Wilcox thinks this is a side
> > effect of readahead.
> > 
> > To fix this occasional false failure, call SEEK_DATA and SEEK_HOLE only
> > on the offsets that we care about.
> > 
> > Suggested-by: Matthew Wilcox <willy@infradead.org>
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> >  tests/xfs/420 |   33 +++++++++++++++++++++------------
> >  1 file changed, 21 insertions(+), 12 deletions(-)
> > 
> > 
> > diff --git a/tests/xfs/420 b/tests/xfs/420
> > index 12b17588..d38772c9 100755
> > --- a/tests/xfs/420
> > +++ b/tests/xfs/420
> > @@ -50,6 +50,24 @@ _scratch_mount >> $seqres.full 2>&1
> >  testdir=$SCRATCH_MNT/test-$seq
> >  mkdir $testdir
> >  
> > +# pagecache readahead can sometimes cause extra pages to be inserted into the
> > +# file mapping where we have an unwritten extent in the COW fork.  Call lseek
> > +# on each $blksz offset that interests us (as opposed to the whole file) so
> > +# that these extra pages are not disclosed.
> > +#
> > +# The important thing we're testing is that SEEK_DATA reports block 3 as data
> > +# when the COW fork has an unwritten mapping and the data fork has a hole.
> > +exercise_lseek() {
> > +	echo "Seek holes and data in file1"
> > +	$XFS_IO_PROG -c "seek -d 0" $testdir/file1
> > +	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file1 | sed -e '/Whence/d'
> > +	echo "Seek holes and data in file2"
> > +	$XFS_IO_PROG -c "seek -d 0" $testdir/file2
> > +	$XFS_IO_PROG -c "seek -h $((2 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> > +	$XFS_IO_PROG -c "seek -d $((3 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> > +	$XFS_IO_PROG -c "seek -h $((4 * blksz))" $testdir/file2 | sed -e '/Whence/d'
> > +}
> > +
> >  blksz=65536
> >  nr=8
> >  filesize=$((blksz * nr))
> > @@ -83,10 +101,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
> >  
> > -echo "Seek holes and data in file1"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> > -echo "Seek holes and data in file2"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> > +exercise_lseek
> >  
> >  echo "Compare files"
> >  md5sum $testdir/file1 | _filter_scratch
> > @@ -102,10 +117,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
> >  
> > -echo "Seek holes and data in file1"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> > -echo "Seek holes and data in file2"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> > +exercise_lseek
> >  
> >  echo "Compare files"
> >  md5sum $testdir/file1 | _filter_scratch
> > @@ -121,10 +133,7 @@ $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file1 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file2 >> $seqres.full 2>&1
> >  $XFS_IO_PROG -c "bmap -ev" -c "bmap -cv" $testdir/file3 >> $seqres.full 2>&1
> >  
> > -echo "Seek holes and data in file1"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file1
> > -echo "Seek holes and data in file2"
> > -$XFS_IO_PROG -c "seek -a -r 0" $testdir/file2
> > +exercise_lseek
> >  
> >  echo "Compare files"
> >  md5sum $testdir/file1 | _filter_scratch
> > 
> 

Re: [PATCH 6/4] xfs/17[035]: fix intermittent failures when filesystem metadata gets large

[Darrick J. Wong]

On Tue, Mar 22, 2022 at 03:15:56PM +0800, Zorro Lang wrote:
> On Wed, Mar 16, 2022 at 03:13:26PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> > 
> > These tests check that the filestreams allocator never shares an AG
> > across multiple streams when there's plenty of space in the filesystem.
> > Recent increases in metadata overhead for newer features (e.g. bigger
> > logs due to reflink) can throw this off, so add another AG to the
> > formatted filesystem to encourage it to avoid the AG with the log.
> > 
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> 
> 
> 
> >  common/filestreams |    2 +-
> >  tests/xfs/170      |   16 +++++++++++-----
> >  tests/xfs/170.out  |    8 ++++----
> >  tests/xfs/171      |   16 ++++++++++++----
> >  tests/xfs/171.out  |    8 ++++----
> >  tests/xfs/173      |   16 ++++++++++++----
> >  tests/xfs/173.out  |    8 ++++----
> 
> Looks good to me, and test passed after merging this patch. Just one
> tiny question, the subject is "xfs/17[035]: ...", if "17[035]" mean
> 170, 173 and 175, but the files you changed are 170, 171 and 173 as
> above :)

oops.  Clearly I suck at regex. :(

Subject: xfs/17*: fix filestreams test failures when...

--D

> Reviewed-by: Zorro Lang <zlang@redhat.com>
> 
> 
> 
> >  7 files changed, 48 insertions(+), 26 deletions(-)
> > 
> > diff --git a/common/filestreams b/common/filestreams
> > index 8165effe..62acb47c 100644
> > --- a/common/filestreams
> > +++ b/common/filestreams
> > @@ -80,7 +80,7 @@ _check_for_dupes()
> >  
> >  _test_streams() {
> >  
> > -	echo "# testing $* ...."
> > +	echo "# testing $* ...." | tee -a $seqres.full
> >  	local agcount="$1"
> >  	local agsize="$2" # in MB
> >  	local stream_count="$3"
> > diff --git a/tests/xfs/170 b/tests/xfs/170
> > index 5e622d24..b9ead341 100755
> > --- a/tests/xfs/170
> > +++ b/tests/xfs/170
> > @@ -25,11 +25,17 @@ _check_filestreams_support || _notrun "filestreams not available"
> >  # test small stream, multiple I/O per file, 30s timeout
> >  _set_stream_timeout_centisecs 3000
> >  
> > -# test streams does a mkfs and mount
> > -_test_streams 8 22 4 8 3 0 0
> > -_test_streams 8 22 4 8 3 1 0
> > -_test_streams 8 22 4 8 3 0 1
> > -_test_streams 8 22 4 8 3 1 1
> > +# This test checks that the filestreams allocator never allocates space in any
> > +# given AG into more than one stream when there's plenty of space on the
> > +# filesystem.  Newer feature sets (e.g. reflink) have increased the size of
> > +# the log for small filesystems, so we make sure there's one more AG than
> > +# filestreams to encourage the allocator to skip whichever AG owns the log.
> > +#
> > +# Exercise 9x 22MB AGs, 4 filestreams, 8 files per stream, and 3MB per file.
> > +_test_streams 9 22 4 8 3 0 0
> > +_test_streams 9 22 4 8 3 1 0
> > +_test_streams 9 22 4 8 3 0 1
> > +_test_streams 9 22 4 8 3 1 1
> >  
> >  status=0
> >  exit
> > diff --git a/tests/xfs/170.out b/tests/xfs/170.out
> > index e71515e9..16dcb795 100644
> > --- a/tests/xfs/170.out
> > +++ b/tests/xfs/170.out
> > @@ -1,20 +1,20 @@
> >  QA output created by 170
> > -# testing 8 22 4 8 3 0 0 ....
> > +# testing 9 22 4 8 3 0 0 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 8 22 4 8 3 1 0 ....
> > +# testing 9 22 4 8 3 1 0 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 8 22 4 8 3 0 1 ....
> > +# testing 9 22 4 8 3 0 1 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 8 22 4 8 3 1 1 ....
> > +# testing 9 22 4 8 3 1 1 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> > diff --git a/tests/xfs/171 b/tests/xfs/171
> > index 4412fe2f..f93b6011 100755
> > --- a/tests/xfs/171
> > +++ b/tests/xfs/171
> > @@ -29,10 +29,18 @@ _check_filestreams_support || _notrun "filestreams not available"
> >  # 100 = 78.1% full, should reliably succeed
> >  _set_stream_timeout_centisecs 12000
> >  
> > -_test_streams 64 16 8 100 1 1 0
> > -_test_streams 64 16 8 100 1 1 1
> > -_test_streams 64 16 8 100 1 0 0
> > -_test_streams 64 16 8 100 1 0 1
> > +# This test tries to get close to the exact point at which the filestreams
> > +# allocator will start to allocate space from some AG into more than one
> > +# stream.  Newer feature sets (e.g. reflink) have increased the size of the log
> > +# for small filesystems, so we make sure there's one more AG than filestreams
> > +# to encourage the allocator to skip whichever AG owns the log.
> > +#
> > +# This test exercises 64x 16MB AGs, 8 filestreams, 100 files per stream, and
> > +# 1MB per file.
> > +_test_streams 65 16 8 100 1 1 0
> > +_test_streams 65 16 8 100 1 1 1
> > +_test_streams 65 16 8 100 1 0 0
> > +_test_streams 65 16 8 100 1 0 1
> >  
> >  status=0
> >  exit
> > diff --git a/tests/xfs/171.out b/tests/xfs/171.out
> > index 89407cb2..73f73c90 100644
> > --- a/tests/xfs/171.out
> > +++ b/tests/xfs/171.out
> > @@ -1,20 +1,20 @@
> >  QA output created by 171
> > -# testing 64 16 8 100 1 1 0 ....
> > +# testing 65 16 8 100 1 1 0 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 64 16 8 100 1 1 1 ....
> > +# testing 65 16 8 100 1 1 1 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 64 16 8 100 1 0 0 ....
> > +# testing 65 16 8 100 1 0 0 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 64 16 8 100 1 0 1 ....
> > +# testing 65 16 8 100 1 0 1 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> > diff --git a/tests/xfs/173 b/tests/xfs/173
> > index bce6ac51..6b18d919 100755
> > --- a/tests/xfs/173
> > +++ b/tests/xfs/173
> > @@ -26,10 +26,18 @@ _check_filestreams_support || _notrun "filestreams not available"
> >  # be less than or equal to half the AG count so we don't run out of AGs.
> >  _set_stream_timeout_centisecs 12000
> >  
> > -_test_streams 64 16 33 8 2 1 1 fail
> > -_test_streams 64 16 32 8 2 0 1
> > -_test_streams 64 16 33 8 2 0 0 fail
> > -_test_streams 64 16 32 8 2 1 0
> > +# This test checks the exact point at which the filestreams allocator will
> > +# start to allocate space from some AG into more than one stream.  Newer
> > +# feature sets (e.g. reflink) have increased the size of the log for small
> > +# filesystems, so we make sure there's one more AG than filestreams to
> > +# encourage the allocator to skip whichever AG owns the log.
> > +#
> > +# Exercise 65x 16MB AGs, 32/33 filestreams, 8 files per stream, and 2MB per
> > +# file.
> > +_test_streams 65 16 34 8 2 1 1 fail
> > +_test_streams 65 16 32 8 2 0 1
> > +_test_streams 65 16 34 8 2 0 0 fail
> > +_test_streams 65 16 32 8 2 1 0
> >  
> >  status=0
> >  exit
> > diff --git a/tests/xfs/173.out b/tests/xfs/173.out
> > index 21493057..705c352a 100644
> > --- a/tests/xfs/173.out
> > +++ b/tests/xfs/173.out
> > @@ -1,20 +1,20 @@
> >  QA output created by 173
> > -# testing 64 16 33 8 2 1 1 fail ....
> > +# testing 65 16 34 8 2 1 1 fail ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + expected failure, matching AGs
> > -# testing 64 16 32 8 2 0 1 ....
> > +# testing 65 16 32 8 2 0 1 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + passed, streams are in seperate AGs
> > -# testing 64 16 33 8 2 0 0 fail ....
> > +# testing 65 16 34 8 2 0 0 fail ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> >  + expected failure, matching AGs
> > -# testing 64 16 32 8 2 1 0 ....
> > +# testing 65 16 32 8 2 1 0 ....
> >  # streaming
> >  # sync AGs...
> >  # checking stream AGs...
> > 
>