btrfs read repair: more tests

btrfs read repair: more tests

[Christoph Hellwig]

Hi all,

this series adds a few more btrfs read repair tests.  Still all for user
data as I haven't managed to successfully inject corruption into metadata
yet.

[PATCH 1/4] btrfs: fix the_btrfs_get_physical invocation in btrfs-map-logical

[Christoph Hellwig]

The -b flag without an argument is not supported, so remove it.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 common/btrfs | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/common/btrfs b/common/btrfs
index c7058918..14ad890e 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -537,7 +537,7 @@ _btrfs_get_physical()
 
 	_require_command "$BTRFS_MAP_LOGICAL_PROG" btrfs-map-logical
 
-	$BTRFS_MAP_LOGICAL_PROG -b -l $logical $SCRATCH_DEV >> $seqres.full 2>&1
+	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV >> $seqres.full 2>&1
 	$BTRFS_MAP_LOGICAL_PROG -l $logical $SCRATCH_DEV | \
 		$AWK_PROG "(\$1 ~ /mirror/ && \$2 ~ /$stripe/) { print \$6 }"
 }
-- 
2.30.2

[PATCH 2/4] btrfs; add a test for impossible repair cases

[Christoph Hellwig]

Verify that a repair attempt that can't succeed because all copies are
bad returns a proper I/O error and doesn't cause any deadlocks.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/268     | 62 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/268.out |  7 +++++
 2 files changed, 69 insertions(+)
 create mode 100755 tests/btrfs/268
 create mode 100644 tests/btrfs/268.out

diff --git a/tests/btrfs/268 b/tests/btrfs/268
new file mode 100755
index 00000000..5043bb02
--- /dev/null
+++ b/tests/btrfs/268
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Christoph Hellwig.
+#
+# FS QA Test 268
+#
+# Test that btrfs read repair on a raid1 profile won't loop forever if data
+# is corrupted on both mirrors and can't be recovered.
+#
+
+. ./common/preamble
+_begin_fstest auto quick read_repair
+
+. ./common/filter
+
+_supported_fs btrfs
+_require_odirect
+_require_non_zoned_device "${SCRATCH_DEV}" # no overwrites on zoned devices
+_require_scratch_dev_pool 2
+_scratch_dev_pool_get 2
+
+echo "step 1......mkfs.btrfs"
+
+_scratch_pool_mkfs "-d raid1 -b 1G" >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 256K 0 256K" \
+	"$SCRATCH_MNT/foobar" | \
+	_filter_xfs_io_offset
+
+# ensure btrfs-map-logical sees the tree updates
+sync
+
+logical=$(_btrfs_get_first_logical $SCRATCH_MNT/foobar)
+
+physical1=$(_btrfs_get_physical ${logical} 1)
+devpath1=$(_btrfs_get_device_path ${logical} 1)
+
+physical2=$(_btrfs_get_physical ${logical} 2)
+devpath2=$(_btrfs_get_device_path ${logical} 2)
+
+physical3=$(_btrfs_get_physical ${logical} 3)
+devpath3=$(_btrfs_get_device_path ${logical} 3)
+
+_scratch_unmount
+
+echo "step 2......corrupt file extent"
+$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical1 4K" \
+	$devpath1 > /dev/null
+$XFS_IO_PROG -d -c "pwrite -S 0xba -b 4K $physical2 4K" \
+	$devpath2 > /dev/null
+
+_scratch_mount
+
+echo "step 3......try to repair"
+$XFS_IO_PROG -d -c "pread -b 4K 0 4K" $SCRATCH_MNT/foobar
+
+_scratch_unmount
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/268.out b/tests/btrfs/268.out
new file mode 100644
index 00000000..d28b37b3
--- /dev/null
+++ b/tests/btrfs/268.out
@@ -0,0 +1,7 @@
+QA output created by 268
+step 1......mkfs.btrfs
+wrote 262144/262144 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+step 2......corrupt file extent
+step 3......try to repair
+pread: Input/output error
-- 
2.30.2

[PATCH 3/4] btrfs: test checker pattern corruption on raid10

[Christoph Hellwig]

Check read repair for the case where the corruption is spread over
the different legs of a raid10 set.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/269     | 73 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/269.out | 41 +++++++++++++++++++++++++
 2 files changed, 114 insertions(+)
 create mode 100755 tests/btrfs/269
 create mode 100644 tests/btrfs/269.out

diff --git a/tests/btrfs/269 b/tests/btrfs/269
new file mode 100755
index 00000000..3c11da3f
--- /dev/null
+++ b/tests/btrfs/269
@@ -0,0 +1,73 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2022 Christoph Hellwig.
+#
+# FS QA Test 269
+#
+# Test btrfs read repair over tricky stripe boundaries on the raid10 profile:
+#
+#             | stripe 0 | stripe 2
+#   --------------------------------
+#    mirror 1 | I/O FAIL | GOOD    
+#    mirror 2 | GOOD     | CSUM FAIL
+#
+
+. ./common/preamble
+_begin_fstest auto quick read_repair
+
+. ./common/filter
+
+_supported_fs btrfs
+_require_odirect
+_require_non_zoned_device "${SCRATCH_DEV}" # no overwrites on zoned devices
+_require_scratch_dev_pool 4
+_scratch_dev_pool_get 4
+
+echo "step 1......mkfs.btrfs"
+
+_scratch_pool_mkfs "-d raid10 -b 1G" >>$seqres.full 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" \
+	"$SCRATCH_MNT/foobar" | \
+	_filter_xfs_io_offset
+
+# ensure btrfs-map-logical sees the tree updates
+sync
+
+logical=$(_btrfs_get_first_logical $SCRATCH_MNT/foobar)
+
+physical1=$(_btrfs_get_physical ${logical} 1)
+devpath1=$(_btrfs_get_device_path ${logical} 1)
+
+physical4=$(_btrfs_get_physical ${logical} 3)
+devpath4=$(_btrfs_get_device_path ${logical} 3)
+
+_scratch_unmount
+
+echo "step 2......corrupt file extent"
+
+$XFS_IO_PROG -d -c "pwrite -S 0xbd -b 64K $physical1 64K" \
+	$devpath1 > /dev/null
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $((physical4 + 65536)) 64K" \
+	$devpath4 > /dev/null
+
+_scratch_mount
+
+echo "step 3......repair the bad copy"
+
+_btrfs_direct_read_on_mirror 0 2 "$SCRATCH_MNT/foobar" 0 128K
+_btrfs_direct_read_on_mirror 1 2 "$SCRATCH_MNT/foobar" 0 128K
+
+_scratch_unmount
+
+echo "step 4......check if the repair worked"
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical1 512" $devpath1 |\
+	_filter_xfs_io_offset
+$XFS_IO_PROG -d -c "pread -v -b 512 $((physical4 + 65536)) 512" $devpath4 |\
+	_filter_xfs_io_offset
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/269.out b/tests/btrfs/269.out
new file mode 100644
index 00000000..d3ad7f07
--- /dev/null
+++ b/tests/btrfs/269.out
@@ -0,0 +1,41 @@
+QA output created by 269
+step 1......mkfs.btrfs
+wrote 131072/131072 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+step 2......corrupt file extent
+step 3......repair the bad copy
+step 4......check if the repair worked
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+read 512/512 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.30.2

[PATCH 4/4] btrfs: test read repair on a corrupted compressed extent

[Christoph Hellwig]

Exercise read repair on a corrupted compressed sector.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/270     | 79 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/270.out | 41 +++++++++++++++++++++++
 2 files changed, 120 insertions(+)
 create mode 100755 tests/btrfs/270
 create mode 100644 tests/btrfs/270.out

diff --git a/tests/btrfs/270 b/tests/btrfs/270
new file mode 100755
index 00000000..4229a02c
--- /dev/null
+++ b/tests/btrfs/270
@@ -0,0 +1,79 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
+#
+# FS QA Test 270
+#
+# Regression test for btrfs buffered read repair of compressed data.
+#
+. ./common/preamble
+_begin_fstest auto quick read_repair
+
+. ./common/filter
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_non_zoned_device "${SCRATCH_DEV}" # no overwrites on zoned devices
+_require_scratch_dev_pool 2
+_scratch_dev_pool_get 2
+
+get_physical()
+{
+	local logical=$1
+	local stripe=$2
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+		grep $logical -A 6 | \
+		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$6 }"
+}
+
+get_devid()
+{
+	local logical=$1
+	local stripe=$2
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+		grep $logical -A 6 | \
+		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$4 }"
+}
+
+get_device_path()
+{
+	local devid=$1
+	echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}"
+}
+
+
+echo "step 1......mkfs.btrfs"
+_check_minimal_fs_size $(( 1024 * 1024 * 1024 ))
+_scratch_pool_mkfs "-d raid1 -b 1G" >>$seqres.full 2>&1
+_scratch_mount -ocompress
+
+# Create a file with all data being compressed
+$XFS_IO_PROG -f -c "pwrite -S 0xaa -W -b 128K 0 128K" \
+	"$SCRATCH_MNT/foobar" | _filter_xfs_io_offset
+
+logical_in_btrfs=$(_btrfs_get_first_logical $SCRATCH_MNT/foobar)
+physical=$(get_physical ${logical_in_btrfs} 1)
+devid=$(get_devid ${logical_in_btrfs} 1)
+devpath=$(get_device_path ${devid})
+
+_scratch_unmount
+echo "step 2......corrupt file extent"
+echo " corrupt stripe #1, devid $devid devpath $devpath physical $physical" \
+	>> $seqres.full
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical 64K" $devpath > /dev/null
+
+_scratch_mount
+
+echo "step 3......repair the bad copy"
+_btrfs_buffered_read_on_mirror 1 2 "$SCRATCH_MNT/foobar" 0 128K
+
+_scratch_unmount
+
+echo "step 4......check if the repair worked"
+$XFS_IO_PROG -c "pread -v -b 512 $physical 512" $devpath |\
+	_filter_xfs_io_offset
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/270.out b/tests/btrfs/270.out
new file mode 100644
index 00000000..53a80692
--- /dev/null
+++ b/tests/btrfs/270.out
@@ -0,0 +1,41 @@
+QA output created by 270
+step 1......mkfs.btrfs
+wrote 131072/131072 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+step 2......corrupt file extent
+step 3......repair the bad copy
+step 4......check if the repair worked
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+XXXXXXXX:  aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa  ................
+read 512/512 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
-- 
2.30.2

Re: [PATCH 4/4] btrfs: test read repair on a corrupted compressed extent

[Christoph Hellwig]

So while this test properly documents the current behavior, it failed
to grasp how broken that behavior ist: the current read repair code
writes back the uncompressed data to disk even for a compressed extent,
and this test verified the behavior.

Below is a correct test that fails on current mainline.  I'll send fixes
but right now they depend on a lot of prep work.

---
From 6b6c505f75c6c7cc15359f14053b1db43e3d3091 Mon Sep 17 00:00:00 2001
From: Christoph Hellwig <hch@lst.de>
Date: Wed, 22 Jun 2022 06:55:36 +0200
Subject: btrfs: test read repair on a corrupted compressed extent

Exercise read repair on a corrupted compressed sector.

Signed-off-by: Christoph Hellwig <hch@lst.de>
---
 tests/btrfs/270     | 82 +++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/270.out |  7 ++++
 2 files changed, 89 insertions(+)
 create mode 100755 tests/btrfs/270
 create mode 100644 tests/btrfs/270.out

diff --git a/tests/btrfs/270 b/tests/btrfs/270
new file mode 100755
index 00000000..5b73fb15
--- /dev/null
+++ b/tests/btrfs/270
@@ -0,0 +1,82 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
+#
+# FS QA Test 270
+#
+# Regression test for btrfs buffered read repair of compressed data.
+#
+. ./common/preamble
+_begin_fstest auto quick read_repair compress
+
+. ./common/filter
+
+_supported_fs btrfs
+_require_btrfs_command inspect-internal dump-tree
+_require_non_zoned_device "${SCRATCH_DEV}" # no overwrites on zoned devices
+_require_scratch_dev_pool 2
+_scratch_dev_pool_get 2
+
+get_physical()
+{
+	local logical=$1
+	local stripe=$2
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+		grep $logical -A 6 | \
+		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$6 }"
+}
+
+get_devid()
+{
+	local logical=$1
+	local stripe=$2
+	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+		grep $logical -A 6 | \
+		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$4 }"
+}
+
+get_device_path()
+{
+	local devid=$1
+	echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}"
+}
+
+
+echo "step 1......mkfs.btrfs"
+_check_minimal_fs_size $(( 1024 * 1024 * 1024 ))
+_scratch_pool_mkfs "-d raid1 -b 1G" >>$seqres.full 2>&1
+_scratch_mount -ocompress
+
+# Create a file with all data being compressed
+$XFS_IO_PROG -f -c "pwrite -S 0xaa -W -b 128K 0 128K" \
+	"$SCRATCH_MNT/foobar" | _filter_xfs_io_offset
+
+logical_in_btrfs=$(_btrfs_get_first_logical $SCRATCH_MNT/foobar)
+physical=$(get_physical ${logical_in_btrfs} 1)
+devid=$(get_devid ${logical_in_btrfs} 1)
+devpath=$(get_device_path ${devid})
+
+_scratch_unmount
+echo "step 2......corrupt file extent"
+echo " corrupt stripe #1, devid $devid devpath $devpath physical $physical" \
+	>> $seqres.full
+dd if=$devpath of=$TEST_DIR/$seq.dump.good skip=$physical bs=1 count=4096 \
+	2>/dev/null
+$XFS_IO_PROG -c "pwrite -S 0xbb -b 4K $physical 4K" $devpath > /dev/null
+
+_scratch_mount
+
+echo "step 3......repair the bad copy"
+_btrfs_buffered_read_on_mirror 1 2 "$SCRATCH_MNT/foobar" 0 128K
+
+_scratch_unmount
+
+echo "step 4......check if the repair worked"
+dd if=$devpath of=$TEST_DIR/$seq.dump skip=$physical bs=1 count=4096 \
+	2>/dev/null
+cmp -bl $TEST_DIR/$seq.dump.good $TEST_DIR/$seq.dump
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/270.out b/tests/btrfs/270.out
new file mode 100644
index 00000000..6d744c02
--- /dev/null
+++ b/tests/btrfs/270.out
@@ -0,0 +1,7 @@
+QA output created by 270
+step 1......mkfs.btrfs
+wrote 131072/131072 bytes
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+step 2......corrupt file extent
+step 3......repair the bad copy
+step 4......check if the repair worked
-- 
2.30.2

Re: [PATCH 4/4] btrfs: test read repair on a corrupted compressed extent

[Zorro Lang]

On Wed, Jun 22, 2022 at 11:21:40AM +0200, Christoph Hellwig wrote:
> So while this test properly documents the current behavior, it failed
> to grasp how broken that behavior ist: the current read repair code
> writes back the uncompressed data to disk even for a compressed extent,
> and this test verified the behavior.
> 
> Below is a correct test that fails on current mainline.  I'll send fixes
> but right now they depend on a lot of prep work.
> 
> ---
> From 6b6c505f75c6c7cc15359f14053b1db43e3d3091 Mon Sep 17 00:00:00 2001
> From: Christoph Hellwig <hch@lst.de>
> Date: Wed, 22 Jun 2022 06:55:36 +0200
> Subject: btrfs: test read repair on a corrupted compressed extent
> 
> Exercise read repair on a corrupted compressed sector.
> 
> Signed-off-by: Christoph Hellwig <hch@lst.de>
> ---
>  tests/btrfs/270     | 82 +++++++++++++++++++++++++++++++++++++++++++++
>  tests/btrfs/270.out |  7 ++++
>  2 files changed, 89 insertions(+)
>  create mode 100755 tests/btrfs/270
>  create mode 100644 tests/btrfs/270.out
> 
> diff --git a/tests/btrfs/270 b/tests/btrfs/270
> new file mode 100755
> index 00000000..5b73fb15
> --- /dev/null
> +++ b/tests/btrfs/270
> @@ -0,0 +1,82 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
                   ^^^^^^^^^^^
Is it a wrong copy&paste ?

> +#
> +# FS QA Test 270
> +#
> +# Regression test for btrfs buffered read repair of compressed data.

If this's a regression test, I'd like to see the fix be reviewed/acked
at first :)

Thanks,
Zorro

> +#
> +. ./common/preamble
> +_begin_fstest auto quick read_repair compress
> +
> +. ./common/filter
> +
> +_supported_fs btrfs
> +_require_btrfs_command inspect-internal dump-tree
> +_require_non_zoned_device "${SCRATCH_DEV}" # no overwrites on zoned devices
> +_require_scratch_dev_pool 2
> +_scratch_dev_pool_get 2
> +
> +get_physical()
> +{
> +	local logical=$1
> +	local stripe=$2
> +	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> +		grep $logical -A 6 | \
> +		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$6 }"
> +}
> +
> +get_devid()
> +{
> +	local logical=$1
> +	local stripe=$2
> +	$BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> +		grep $logical -A 6 | \
> +		$AWK_PROG "(\$1 ~ /stripe/ && \$3 ~ /devid/ && \$2 ~ /$stripe/) { print \$4 }"
> +}
> +
> +get_device_path()
> +{
> +	local devid=$1
> +	echo "$SCRATCH_DEV_POOL" | $AWK_PROG "{print \$$devid}"
> +}
> +
> +
> +echo "step 1......mkfs.btrfs"
> +_check_minimal_fs_size $(( 1024 * 1024 * 1024 ))
> +_scratch_pool_mkfs "-d raid1 -b 1G" >>$seqres.full 2>&1
> +_scratch_mount -ocompress
> +
> +# Create a file with all data being compressed
> +$XFS_IO_PROG -f -c "pwrite -S 0xaa -W -b 128K 0 128K" \
> +	"$SCRATCH_MNT/foobar" | _filter_xfs_io_offset
> +
> +logical_in_btrfs=$(_btrfs_get_first_logical $SCRATCH_MNT/foobar)
> +physical=$(get_physical ${logical_in_btrfs} 1)
> +devid=$(get_devid ${logical_in_btrfs} 1)
> +devpath=$(get_device_path ${devid})
> +
> +_scratch_unmount
> +echo "step 2......corrupt file extent"
> +echo " corrupt stripe #1, devid $devid devpath $devpath physical $physical" \
> +	>> $seqres.full
> +dd if=$devpath of=$TEST_DIR/$seq.dump.good skip=$physical bs=1 count=4096 \
> +	2>/dev/null
> +$XFS_IO_PROG -c "pwrite -S 0xbb -b 4K $physical 4K" $devpath > /dev/null
> +
> +_scratch_mount
> +
> +echo "step 3......repair the bad copy"
> +_btrfs_buffered_read_on_mirror 1 2 "$SCRATCH_MNT/foobar" 0 128K
> +
> +_scratch_unmount
> +
> +echo "step 4......check if the repair worked"
> +dd if=$devpath of=$TEST_DIR/$seq.dump skip=$physical bs=1 count=4096 \
> +	2>/dev/null
> +cmp -bl $TEST_DIR/$seq.dump.good $TEST_DIR/$seq.dump
> +
> +_scratch_dev_pool_put
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/270.out b/tests/btrfs/270.out
> new file mode 100644
> index 00000000..6d744c02
> --- /dev/null
> +++ b/tests/btrfs/270.out
> @@ -0,0 +1,7 @@
> +QA output created by 270
> +step 1......mkfs.btrfs
> +wrote 131072/131072 bytes
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +step 2......corrupt file extent
> +step 3......repair the bad copy
> +step 4......check if the repair worked
> -- 
> 2.30.2
> 

Re: [PATCH 4/4] btrfs: test read repair on a corrupted compressed extent

[Christoph Hellwig]

On Wed, Jun 22, 2022 at 08:41:18PM +0800, Zorro Lang wrote:
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
>                    ^^^^^^^^^^^
> Is it a wrong copy&paste ?

A lot of the test is copied from btrfs/141, so I wanted to keep the
copyright intact.

> > +#
> > +# FS QA Test 270
> > +#
> > +# Regression test for btrfs buffered read repair of compressed data.
> 
> If this's a regression test, I'd like to see the fix be reviewed/acked
> at first :)

Heh.  Actually I'm not sure it is a regression, this was copy and
pasted as well.  I think it actually has been broken since day 1.

Re: [PATCH 4/4] btrfs: test read repair on a corrupted compressed extent

[Zorro Lang]

On Wed, Jun 22, 2022 at 03:07:45PM +0200, Christoph Hellwig wrote:
> On Wed, Jun 22, 2022 at 08:41:18PM +0800, Zorro Lang wrote:
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +# Copyright (c) 2017 Liu Bo.  All Rights Reserved.
> >                    ^^^^^^^^^^^
> > Is it a wrong copy&paste ?
> 
> A lot of the test is copied from btrfs/141, so I wanted to keep the
> copyright intact.

Oh, there're some cases copy from old cases in fstests, but they are changed
a little to have different coverage. Generally they metioned they copy from
which one case (number). I don't mind if you'd like to keep original
Copyright name, but the copyright time is old for a new test case, I don't
know if we'd better to update it. Hmm... sorry, not good at these copyright
things...

> 
> > > +#
> > > +# FS QA Test 270
> > > +#
> > > +# Regression test for btrfs buffered read repair of compressed data.
> > 
> > If this's a regression test, I'd like to see the fix be reviewed/acked
> > at first :)
> 
> Heh.  Actually I'm not sure it is a regression, this was copy and
> pasted as well.  I think it actually has been broken since day 1.
> 

Re: btrfs read repair: more tests

[Zorro Lang]

On Wed, Jun 22, 2022 at 06:58:40AM +0200, Christoph Hellwig wrote:
> Hi all,
> 
> this series adds a few more btrfs read repair tests.  Still all for user
> data as I haven't managed to successfully inject corruption into metadata
> yet.

This patchset is good to me (from fstests side). If there's not more review
points or objections from btrfs list, I'll merge it. Welcome more review.

Thanks,
Zorro

>