[PATCHSET 0/7] fstests: random fixes for v2023.05.01

[PATCHSET 0/7] fstests: random fixes for v2023.05.01

[Darrick J. Wong]

Hi all,

Here's the usual odd fixes for fstests.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

kernel git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfs-linux.git/log/?h=random-fixes

xfsprogs git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfsprogs-dev.git/log/?h=random-fixes

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
---
 ltp/fsx.c           |    9 +++++----
 src/fiemap-tester.c |   25 +++++++++++--------------
 tests/generic/094   |    5 -----
 tests/generic/225   |    5 -----
 tests/generic/724   |    2 +-
 tests/xfs/243       |   12 ++++++------
 tests/xfs/245       |    6 +++---
 tests/xfs/262       |    2 +-
 tests/xfs/272       |    4 ++--
 tests/xfs/274       |    8 ++++----
 tests/xfs/791       |    2 +-
 11 files changed, 34 insertions(+), 46 deletions(-)

[PATCH 1/7] fsx: fix indenting of columns in bad buffers report

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

When file corruption is detected, make the columns of the report line
up correctly even in the diff output.  Although the .out.bad file
contains this (with spaces to demonstrate unequivocally what happens
when tabs are formatted as 8-column indent):

OFFSET  GOOD    BAD     RANGE
0x2c000 0x0000  0xd6c1  0x00000

diffing the good and bad golden output yields poorly formatted output:

+OFFSET GOOD    BAD     RANGE
+0x2c000        0x0000  0xd6c1  0x00000

Replace the tabs with columns indented with printf width specifiers so
that the test output gets this:

OFFSET      GOOD    BAD     RANGE
0x2c000     0x0000  0xd6c1  0x0

...which always lines up the columns regardless of the user's tab
display settings or diff inserting plus signs.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 ltp/fsx.c |    9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)


diff --git a/ltp/fsx.c b/ltp/fsx.c
index c76b06ca76..ffa64cfa00 100644
--- a/ltp/fsx.c
+++ b/ltp/fsx.c
@@ -669,17 +669,18 @@ check_buffers(char *buf, unsigned offset, unsigned size)
 	if (memcmp(good_buf + offset, buf, size) != 0) {
 		prt("READ BAD DATA: offset = 0x%x, size = 0x%x, fname = %s\n",
 		    offset, size, fname);
-		prt("OFFSET\tGOOD\tBAD\tRANGE\n");
+		prt("%-10s  %-6s  %-6s  %s\n", "OFFSET", "GOOD", "BAD", "RANGE");
 		while (size > 0) {
 			c = good_buf[offset];
 			t = buf[i];
 			if (c != t) {
 			        if (n < 16) {
 					bad = short_at(&buf[i]);
-				        prt("0x%05x\t0x%04x\t0x%04x", offset,
-				            short_at(&good_buf[offset]), bad);
+				        prt("0x%-8x  0x%04x  0x%04x  0x%x\n",
+					    offset,
+					    short_at(&good_buf[offset]), bad,
+					    n);
 					op = buf[offset & 1 ? i+1 : i];
-				        prt("\t0x%05x\n", n);
 					if (op)
 						prt("operation# (mod 256) for "
 						  "the bad data may be %u\n",

[PATCH 2/7] xfs/262: remove dangerous labels

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

This test starts with a consistent filesystem and should end that way
too.  In other words, it isn't dangerous, so drop that from the tags.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/262 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/tests/xfs/262 b/tests/xfs/262
index 78db5e3421..b28a6c88b7 100755
--- a/tests/xfs/262
+++ b/tests/xfs/262
@@ -9,7 +9,7 @@
 # executable and libraries!) to see what happens.
 #
 . ./common/preamble
-_begin_fstest dangerous_fuzzers dangerous_scrub dangerous_online_repair
+_begin_fstest fuzzers scrub online_repair
 
 _register_cleanup "_cleanup" BUS
 

[PATCH 3/7] generic/724,xfs/791: adjust test preconditions for post-EOF stripe zeroing

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

I recently introduced a new fstests config with explicitly specified
stripe geometry of 128k stripe units and a stripe width of 4.  This
broke both of these tests because I hadn't counted on a few things:

1) The write to $SCRATCH_MNT/b at 768k would a 128k delalloc extent
2) This delalloc extent would extend beyond EOF
3) Increasing the file size from 832k to 1m would cause iomap to zero
   the pagecache for the parts of the delalloc extent beyond EOF
4) The newly dirtied posteof delalloc areas would get written to disk
   with a real space allocation

Under these circumstances, FIEXCHRANGE with SKIP_FILE1_HOLES sees a
written extent containing zeroes in file B between 832k and 1m.  File
A has a written extent containing 'X' in the same range, so it exchanges
the two.  When RAID geometry is disabled, the area between 832k and 1m
is usually a hole, so FIEXCHRANGE does nothing.  This causes the md5sum
of the two files to be different, and the test fails.

Fix the test by truncating B to 1m before writing anything to it.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/724 |    2 +-
 tests/xfs/791     |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/tests/generic/724 b/tests/generic/724
index 90cff8cf31..8d7dc4e12a 100755
--- a/tests/generic/724
+++ b/tests/generic/724
@@ -33,9 +33,9 @@ _require_congruent_file_oplen $SCRATCH_MNT 65536
 _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
 
 # Create the donor file
+$XFS_IO_PROG -f -c 'truncate 1m' $SCRATCH_MNT/b
 _pwrite_byte 0x59 64k 64k $SCRATCH_MNT/b >> $seqres.full
 _pwrite_byte 0x57 768k 64k $SCRATCH_MNT/b >> $seqres.full
-$XFS_IO_PROG -c 'truncate 1m' $SCRATCH_MNT/b
 
 md5sum $SCRATCH_MNT/a | _filter_scratch
 md5sum $SCRATCH_MNT/b | _filter_scratch
diff --git a/tests/xfs/791 b/tests/xfs/791
index c89bc3531e..d82314ee08 100755
--- a/tests/xfs/791
+++ b/tests/xfs/791
@@ -37,9 +37,9 @@ _require_congruent_file_oplen $SCRATCH_MNT 65536
 _pwrite_byte 0x58 0 1m $SCRATCH_MNT/a >> $seqres.full
 
 # Create the donor file
+$XFS_IO_PROG -f -c 'truncate 1m' $SCRATCH_MNT/b
 _pwrite_byte 0x59 64k 64k $SCRATCH_MNT/b >> $seqres.full
 _pwrite_byte 0x57 768k 64k $SCRATCH_MNT/b >> $seqres.full
-$XFS_IO_PROG -c 'truncate 1m' $SCRATCH_MNT/b
 sync
 
 md5sum $SCRATCH_MNT/a | _filter_scratch

[PATCH 4/7] xfs/{243,245,272,274}: ignore raid alignment flags in bmap output

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

This test doesn't care about the RAID alignment status of the mappings
that it finds; it only cares about shared and unwritten.  Ignore the
mapping stripe alignment flags in the bmapx output.  This fixes this
test when the fs has su=128k,sw=4.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/xfs/243 |   12 ++++++------
 tests/xfs/245 |    6 +++---
 tests/xfs/272 |    4 ++--
 tests/xfs/274 |    8 ++++----
 4 files changed, 15 insertions(+), 15 deletions(-)


diff --git a/tests/xfs/243 b/tests/xfs/243
index 514fa35667..2e537f3f55 100755
--- a/tests/xfs/243
+++ b/tests/xfs/243
@@ -92,16 +92,16 @@ echo "Delayed allocation CoW extents:"
 test $(_xfs_bmapx_find cow $testdir/file3 delalloc) -gt 0 || \
 	echo "Expected to find a delalloc CoW extent"
 echo "Shared data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '100000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '10[01]{4}$') -gt 0 || \
 	echo "Expected to find a shared data extent"
 echo "Unwritten data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '10000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '1[01]{4}$') -gt 0 || \
 	echo "Expected to find an unwritten data extent"
 echo "Hole data extents:"
 test $(_xfs_bmapx_find data $testdir/file3 hole) -gt 0 || \
 	echo "Expected to find a hole data extent"
 echo "Regular data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '000000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '00[01]{4}$') -gt 0 || \
 	echo "Expected to find a regular data extent"
 
 sync
@@ -115,16 +115,16 @@ echo "Real CoW extents:"
 test $(_xfs_bmapx_find cow $testdir/file3 delalloc ) -eq 0 || \
 	echo "Expected to find zero delalloc CoW extent"
 echo "Shared data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '100000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '10[01]{4}$') -gt 0 || \
 	echo "Expected to find a shared data extent"
 echo "Unwritten data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '10000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '1[01]{4}$') -gt 0 || \
 	echo "Expected to find an unwritten data extent"
 echo "Hole data extents:"
 test $(_xfs_bmapx_find data $testdir/file3 hole) -gt 0 || \
 	echo "Expected to find a hole data extent"
 echo "Regular data extents:"
-test $(_xfs_bmapx_find data $testdir/file3 '000000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file3 -E '00[01]{4}$') -gt 0 || \
 	echo "Expected to find a regular data extent"
 
 _scratch_cycle_mount
diff --git a/tests/xfs/245 b/tests/xfs/245
index 0cd0935cfa..595a5938b4 100755
--- a/tests/xfs/245
+++ b/tests/xfs/245
@@ -42,17 +42,17 @@ md5sum $testdir/file1 | _filter_scratch
 md5sum $testdir/file2 | _filter_scratch
 
 echo "Unwritten data extents"
-test $(_xfs_bmapx_find data $testdir/file1 '10000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file1 -E '1[01]{4}$') -gt 0 || \
 	echo "Expected to find an unwritten file1 extent"
 echo "Shared data extents"
-test $(_xfs_bmapx_find data $testdir/file1 '100000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file1 -E '10[01]{4}$') -gt 0 || \
 	echo "Expected to find a shared data extent"
 
 echo "Hole data extents"
 test $(_xfs_bmapx_find data $testdir/file2 'hole') -gt 0 || \
 	echo "Expected to find a hole data extent"
 echo "Shared data extents"
-test $(_xfs_bmapx_find data $testdir/file2 '100000$') -gt 0 || \
+test $(_xfs_bmapx_find data $testdir/file2 -E '10[01]{4}$') -gt 0 || \
 	echo "Expected to find a shared data extent"
 
 echo "Hole cow extents"
diff --git a/tests/xfs/272 b/tests/xfs/272
index 7ed8b95122..c68fa9d614 100755
--- a/tests/xfs/272
+++ b/tests/xfs/272
@@ -49,10 +49,10 @@ $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/urk | grep '^[[:space:]]*[0-9]*:' | grep
 
 echo "Check bmap and fsmap" | tee -a $seqres.full
 cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do
-	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total}$"
+	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total}(| [01]*)$"
 	echo "${qstr}" >> $seqres.full
 	grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full
-	found=$(grep -c "${qstr}" $TEST_DIR/fsmap)
+	found=$(grep -E -c "${qstr}" $TEST_DIR/fsmap)
 	test $found -eq 1 || echo "Unexpected output for offset ${offrange}."
 done
 
diff --git a/tests/xfs/274 b/tests/xfs/274
index dcaea68804..cd483d77bc 100755
--- a/tests/xfs/274
+++ b/tests/xfs/274
@@ -49,10 +49,10 @@ $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f1 | grep '^[[:space:]]*[0-9]*:' | grep -
 
 echo "Check f1 bmap and fsmap" | tee -a $seqres.full
 cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do
-	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$"
+	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 010[01]{4}$"
 	echo "${qstr}" >> $seqres.full
 	grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full
-	found=$(grep -c "${qstr}" $TEST_DIR/fsmap)
+	found=$(grep -E -c "${qstr}" $TEST_DIR/fsmap)
 	test $found -eq 1 || echo "Unexpected output for offset ${offrange}."
 done
 
@@ -62,10 +62,10 @@ $XFS_IO_PROG -c 'bmap -v' $SCRATCH_MNT/f2 | grep '^[[:space:]]*[0-9]*:' | grep -
 
 echo "Check f2 bmap and fsmap" | tee -a $seqres.full
 cat $TEST_DIR/bmap | while read ext offrange colon blockrange ag agrange total crap; do
-	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 0100000$"
+	qstr="^[[:space:]]*[0-9]*:[[:space:]]*[0-9]*:[0-9]*[[:space:]]*${blockrange} :[[:space:]]*${ino}[[:space:]]*${offrange}[[:space:]]*${ag}[[:space:]]*${agrange}[[:space:]]*${total} 010[01]{4}$"
 	echo "${qstr}" >> $seqres.full
 	grep "${qstr}" $TEST_DIR/fsmap >> $seqres.full
-	found=$(grep -c "${qstr}" $TEST_DIR/fsmap)
+	found=$(grep -E -c "${qstr}" $TEST_DIR/fsmap)
 	test $found -eq 1 || echo "Unexpected output for offset ${offrange}."
 done
 

[PATCH 5/7] fiemap-tester: holes can be backed by unwritten extents

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Filesystem behavior is pretty open-ended for sparse ranges (i.e. holes)
of a file that have not yet been written to.  That space can remain
unmapped, it can be mapped to written space that has been zeroed, or it
can be mapped to unwritten extents.

This program trips over that last condition on XFS.  If the file is
located on a data device with a raid stripe geometry or on a realtime
device with a realtime extent size larger than 1 filesystem block, it's
possible for unwritten areas to be backed by unwritten preallocations or
unwritten rt blocks, respectively.

Fix the test to skip unwritten extents here.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 src/fiemap-tester.c |    7 +++++++
 1 file changed, 7 insertions(+)


diff --git a/src/fiemap-tester.c b/src/fiemap-tester.c
index 3db24daa79..7e9f9fe8c1 100644
--- a/src/fiemap-tester.c
+++ b/src/fiemap-tester.c
@@ -375,6 +375,13 @@ check_hole(struct fiemap *fiemap, int fd, __u64 logical_offset, int blocksize)
 		if (logical_offset + blocksize < start)
 			break;
 
+		/*
+		 * Filesystems are allowed to fill in holes with preallocated
+		 * unwritten extents
+		 */
+		if (extent->fe_flags & FIEMAP_EXTENT_UNWRITTEN)
+			continue;
+
 		if (logical_offset >= start &&
 		    logical_offset < end) {
 

[PATCH 6/7] fiemap: FIEMAP_EXTENT_LAST denotes the last record in the recordset

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Remove this check because FIEMAP_EXTENT_LAST denotes the last space
mapping record in the recordset of space mappings attached to the file.
That last mapping might actually map space beyond EOF, in the case of
(a) speculative post-eof preallocations, (b) stripe-aligned allocations
on XFS, or (c) realtime files in XFS.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 src/fiemap-tester.c |   18 ++++--------------
 1 file changed, 4 insertions(+), 14 deletions(-)


diff --git a/src/fiemap-tester.c b/src/fiemap-tester.c
index 7e9f9fe8c1..fa085a25f1 100644
--- a/src/fiemap-tester.c
+++ b/src/fiemap-tester.c
@@ -236,7 +236,7 @@ check_flags(struct fiemap *fiemap, int blocksize)
 
 static int
 check_data(struct fiemap *fiemap, __u64 logical_offset, int blocksize,
-	   int last, int prealloc)
+	   int prealloc)
 {
 	struct fiemap_extent *extent;
 	__u64 orig_offset = logical_offset;
@@ -280,11 +280,6 @@ check_data(struct fiemap *fiemap, __u64 logical_offset, int blocksize,
 	if (!found) {
 		printf("ERROR: couldn't find extent at %llu\n",
 		       (unsigned long long)(orig_offset / blocksize));
-	} else if (last &&
-		   !(fiemap->fm_extents[c].fe_flags & FIEMAP_EXTENT_LAST)) {
-		printf("ERROR: last extent not marked as last: %llu\n",
-		       (unsigned long long)(orig_offset / blocksize));
-		found = 0;
 	}
 
 	return (!found) ? -1 : 0;
@@ -418,7 +413,7 @@ compare_fiemap_and_map(int fd, char *map, int blocks, int blocksize, int syncfil
 {
 	struct fiemap *fiemap;
 	char *fiebuf;
-	int blocks_to_map, ret, cur_extent = 0, last_data = 0;
+	int blocks_to_map, ret, cur_extent = 0;
 	__u64 map_start, map_length;
 	int i, c;
 
@@ -437,11 +432,6 @@ compare_fiemap_and_map(int fd, char *map, int blocks, int blocksize, int syncfil
 	map_start = 0;
 	map_length = blocks_to_map * blocksize;
 
-	for (i = 0; i < blocks; i++) {
-		if (map[i] != 'H')
-			last_data = i;
-	}
-
 	fiemap->fm_flags = syncfile ? FIEMAP_FLAG_SYNC : 0;
 	fiemap->fm_extent_count = blocks_to_map;
 	fiemap->fm_mapped_extents = 0;
@@ -471,7 +461,7 @@ compare_fiemap_and_map(int fd, char *map, int blocks, int blocksize, int syncfil
 			switch (map[i]) {
 			case 'D':
 				if (check_data(fiemap, logical_offset,
-					       blocksize, last_data == i, 0))
+					       blocksize, 0))
 					goto error;
 				break;
 			case 'H':
@@ -481,7 +471,7 @@ compare_fiemap_and_map(int fd, char *map, int blocks, int blocksize, int syncfil
 				break;
 			case 'P':
 				if (check_data(fiemap, logical_offset,
-					       blocksize, last_data == i, 1))
+					       blocksize, 1))
 					goto error;
 				break;
 			default:

[PATCH 7/7] generic/{094,225}: drop the file allocation unit requirements

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Drop these two test precondition requirements now that we've fixed
fiemap-tester to handle unwritten preallocations that are mapped to
unwritten parts of files and/or mapped beyond EOF.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/094 |    5 -----
 tests/generic/225 |    5 -----
 2 files changed, 10 deletions(-)


diff --git a/tests/generic/094 b/tests/generic/094
index 5c12b7310a..0d9ce8b6ee 100755
--- a/tests/generic/094
+++ b/tests/generic/094
@@ -26,11 +26,6 @@ fiemapfile=$SCRATCH_MNT/$seq.fiemap
 
 _require_test_program "fiemap-tester"
 
-# FIEMAP test doesn't like finding unwritten blocks after it punches out
-# a partial rt extent.
-test "$FSTYP" = "xfs" && \
-	_require_file_block_size_equals_fs_block_size $SCRATCH_MNT
-
 seed=`date +%s`
 
 echo "using seed $seed" >> $seqres.full
diff --git a/tests/generic/225 b/tests/generic/225
index d96382996e..a996889ecf 100755
--- a/tests/generic/225
+++ b/tests/generic/225
@@ -26,11 +26,6 @@ fiemaplog=$SCRATCH_MNT/$seq.log
 
 _require_test_program "fiemap-tester"
 
-# FIEMAP test doesn't like finding unwritten blocks after it punches out
-# a partial rt extent.
-test "$FSTYP" = "xfs" && \
-	_require_file_block_size_equals_fs_block_size $SCRATCH_MNT
-
 seed=`date +%s`
 
 echo "using seed $seed" >> $fiemaplog