* [PATCH 0/6] Regression test for btrfs read repair
@ 2017-05-09 17:56 Liu Bo
2017-05-09 17:56 ` [PATCH 1/6] fstests: add _filter_filefrag Liu Bo
` (5 more replies)
0 siblings, 6 replies; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
This set is adding four regression test case for btrfs read repair, and both
directIO read and buffered read are tested.
Patch 1 and 2 is adding two helpers to summarize the common code among the
following test cases in patch 3-6.
Liu Bo (6):
fstests: add _filter_filefrag
fstests: add _get_current_dmesg
fstests: regression test for btrfs dio read repair
fstests: regression test for btrfs buffered read's repair
fstests: regression test for nocsum dio read's repair
fstests: regression test for nocsum buffered read's repair
common/filter | 19 +++++++
common/rc | 9 ++-
tests/btrfs/140 | 115 ++++++++++++++++++++++++++++++++++++++
tests/btrfs/140.out | 39 +++++++++++++
tests/btrfs/141 | 116 +++++++++++++++++++++++++++++++++++++++
tests/btrfs/141.out | 39 +++++++++++++
tests/btrfs/142 | 137 ++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/142.out | 39 +++++++++++++
tests/btrfs/143 | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/143.out | 39 +++++++++++++
tests/btrfs/group | 4 ++
11 files changed, 709 insertions(+), 2 deletions(-)
create mode 100755 tests/btrfs/140
create mode 100644 tests/btrfs/140.out
create mode 100755 tests/btrfs/141
create mode 100644 tests/btrfs/141.out
create mode 100755 tests/btrfs/142
create mode 100644 tests/btrfs/142.out
create mode 100755 tests/btrfs/143
create mode 100644 tests/btrfs/143.out
--
2.5.0
^ permalink raw reply [flat|nested] 12+ messages in thread
* [PATCH 1/6] fstests: add _filter_filefrag
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-09 17:56 ` [PATCH 2/6] fstests: add _get_current_dmesg Liu Bo
` (4 subsequent siblings)
5 siblings, 0 replies; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
_filter_filefrag is a helper function to filter filefrag's output and it can be
used to get a file's file offset and physical offset.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
common/filter | 19 +++++++++++++++++++
1 file changed, 19 insertions(+)
diff --git a/common/filter b/common/filter
index 5fd71a8..1ef342b 100644
--- a/common/filter
+++ b/common/filter
@@ -418,5 +418,24 @@ _filter_ovl_dirs()
-e "s,$OVL_WORK,OVL_WORK,g"
}
+# interpret filefrag output,
+# eg. "physical 1234, length 10, logical 5678" -> "1234#10#5678"
+_filter_filefrag()
+{
+ perl -ne '
+ if (/blocks? of (\d+) bytes/) {
+ $blocksize = $1;
+ next
+ }
+ ($ext, $logical, $physical, $length) =
+ (/^\s*(\d+):\s+(\d+)..\s+\d+:\s+(\d+)..\s+\d+:\s+(\d+):/)
+ or next;
+ ($flags) = /.*:\s*(\S*)$/;
+ print $physical * $blocksize, "#",
+ $length * $blocksize, "#",
+ $logical * $blocksize, "#",
+ $flags, "\n"'
+}
+
# make sure this script returns success
/bin/true
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 2/6] fstests: add _get_current_dmesg
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
2017-05-09 17:56 ` [PATCH 1/6] fstests: add _filter_filefrag Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-10 10:40 ` Eryu Guan
2017-05-09 17:56 ` [PATCH 3/6] fstests: regression test for btrfs dio read repair Liu Bo
` (3 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
_get_current_dmesg can be used to grep customized pattern.
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
common/rc | 9 +++++++--
1 file changed, 7 insertions(+), 2 deletions(-)
diff --git a/common/rc b/common/rc
index 78a2101..111ed69 100644
--- a/common/rc
+++ b/common/rc
@@ -3215,6 +3215,12 @@ _get_device_size()
grep `_short_dev $1` /proc/partitions | awk '{print $3}'
}
+# get current dmesg log
+_get_current_dmesg()
+{
+ dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac
+}
+
# check dmesg log for WARNING/Oops/etc.
_check_dmesg()
{
@@ -3230,8 +3236,7 @@ _check_dmesg()
# search the dmesg log of last run of $seqnum for possible failures
# use sed \cregexpc address type, since $seqnum contains "/"
- dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | \
- tac | $filter >$seqres.dmesg
+ _get_current_dmesg | $filter >$seqres.dmesg
grep -q -e "kernel BUG at" \
-e "WARNING:" \
-e "BUG:" \
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 3/6] fstests: regression test for btrfs dio read repair
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
2017-05-09 17:56 ` [PATCH 1/6] fstests: add _filter_filefrag Liu Bo
2017-05-09 17:56 ` [PATCH 2/6] fstests: add _get_current_dmesg Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-10 10:53 ` Eryu Guan
2017-05-09 17:56 ` [PATCH 4/6] fstests: regression test for btrfs buffered read's repair Liu Bo
` (2 subsequent siblings)
5 siblings, 1 reply; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
This case tests whether dio read can repair the bad copy if we have
a good copy.
Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
introduced the regression.
The upstream fix is
Btrfs: fix invalid dereference in btrfs_retry_endio
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
tests/btrfs/140 | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/140.out | 39 ++++++++++++++++++
tests/btrfs/group | 1 +
3 files changed, 155 insertions(+)
create mode 100755 tests/btrfs/140
create mode 100644 tests/btrfs/140.out
diff --git a/tests/btrfs/140 b/tests/btrfs/140
new file mode 100755
index 0000000..09a9939
--- /dev/null
+++ b/tests/btrfs/140
@@ -0,0 +1,115 @@
+#! /bin/bash
+# FS QA Test 140
+#
+# Regression test for btrfs DIO read's repair during read.
+#
+# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
+# introduced the regression.
+# The upstream fix is
+# Btrfs: fix invalid dereference in btrfs_retry_endio
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch_dev_pool 2
+
+_require_btrfs_command inspect-internal dump-tree
+_require_command "$FILEFRAG_PROG" filefrag
+_require_odirect
+
+get_physical()
+{
+ # $1 is logical address
+ # print chunk tree and find devid 2 which is $SCRATCH_DEV
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+ grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
+}
+
+_scratch_dev_pool_get 2
+# step 1, create a raid1 btrfs which contains one 128k file.
+echo "step 1......mkfs.btrfs" >>$seqres.full
+
+mkfs_opts="-d raid1 -b 1G"
+_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+
+# -o nospace_cache makes sure data is written to the start position of the data
+# chunk
+_scratch_mount -o nospace_cache
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+
+# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
+# one in $SCRATCH_DEV_POOL
+echo "step 2......corrupt file extent" >>$seqres.full
+
+${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
+logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
+physical_on_scratch=`get_physical ${logical_in_btrfs}`
+
+_scratch_unmount
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_mount
+
+# step 3, 128k dio read (this read can repair bad copy)
+echo "step 3......repair the bad copy" >>$seqres.full
+
+# since raid1 consists of two copies, and the following read may read the good
+# copy directly, so lets loop 10 times here and discard output that dio reads
+# give
+for i in `seq 1 10`; do
+ $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
+ _get_current_dmesg | grep -q -e "csum failed" && break
+done
+
+_scratch_unmount
+
+# check if the repair works
+$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/140.out b/tests/btrfs/140.out
new file mode 100644
index 0000000..c8565f5
--- /dev/null
+++ b/tests/btrfs/140.out
@@ -0,0 +1,39 @@
+QA output created by 140
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+read 512/512 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 9d4b80b..1cb9c98 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -141,3 +141,4 @@
137 auto quick send
138 auto compress
139 auto qgroup
+140 auto quick
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 4/6] fstests: regression test for btrfs buffered read's repair
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
` (2 preceding siblings ...)
2017-05-09 17:56 ` [PATCH 3/6] fstests: regression test for btrfs dio read repair Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-09 17:56 ` [PATCH 5/6] fstests: regression test for nocsum dio " Liu Bo
2017-05-09 17:56 ` [PATCH 6/6] fstests: regression test for nocsum buffered " Liu Bo
5 siblings, 0 replies; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
This case tests whether buffered read can repair the bad copy if we
have a good copy.
Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed
and drop checks") introduced the regression.
The upstream fix is
Btrfs: bring back repair during read
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
tests/btrfs/141 | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/141.out | 39 ++++++++++++++++++
tests/btrfs/group | 1 +
3 files changed, 156 insertions(+)
create mode 100755 tests/btrfs/141
create mode 100644 tests/btrfs/141.out
diff --git a/tests/btrfs/141 b/tests/btrfs/141
new file mode 100755
index 0000000..70f2952
--- /dev/null
+++ b/tests/btrfs/141
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 141
+#
+# Regression test for btrfs buffered read's repair during read.
+#
+# Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed
+# and drop checks") introduced the regression.
+#
+# The upstream fix is
+# Btrfs: bring back repair during read
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch_dev_pool 2
+
+_require_btrfs_command inspect-internal dump-tree
+_require_command "$FILEFRAG_PROG" filefrag
+
+get_physical()
+{
+ # $1 is logical address
+ # print chunk tree and find devid 2 which is $SCRATCH_DEV
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+ grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
+}
+
+_scratch_dev_pool_get 2
+# step 1, create a raid1 btrfs which contains one 128k file.
+echo "step 1......mkfs.btrfs" >>$seqres.full
+
+mkfs_opts="-d raid1 -b 1G"
+_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+
+# -o nospace_cache makes sure data is written to the start position of the data
+# chunk
+_scratch_mount -o nospace_cache
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+
+# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
+# one in $SCRATCH_DEV_POOL
+echo "step 2......corrupt file extent" >>$seqres.full
+
+${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
+logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
+physical_on_scratch=`get_physical ${logical_in_btrfs}`
+
+_scratch_unmount
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_mount
+
+# step 3, 128k buffered read (this read can repair bad copy)
+echo "step 3......repair the bad copy" >>$seqres.full
+
+# since raid1 consists of two copies, and the following read may read the good
+# copy directly, lets loop 10 times here and discard output that buffered reads
+# give
+for i in `seq 1 10`; do
+ echo 3 > /proc/sys/vm/drop_caches
+ $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
+ _get_current_dmesg | grep -q -e "csum failed" && break;
+done
+
+_scratch_unmount
+
+# check if the repair works
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/141.out b/tests/btrfs/141.out
new file mode 100644
index 0000000..2e08fc2
--- /dev/null
+++ b/tests/btrfs/141.out
@@ -0,0 +1,39 @@
+QA output created by 141
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+read 512/512 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 1cb9c98..7b380e5 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -142,3 +142,4 @@
138 auto compress
139 auto qgroup
140 auto quick
+141 auto quick
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 5/6] fstests: regression test for nocsum dio read's repair
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
` (3 preceding siblings ...)
2017-05-09 17:56 ` [PATCH 4/6] fstests: regression test for btrfs buffered read's repair Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-09 17:56 ` [PATCH 6/6] fstests: regression test for nocsum buffered " Liu Bo
5 siblings, 0 replies; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
introduced this regression. It'd cause 'Segmentation fault' error.
The upstream fix is
Btrfs: fix segment fault when doing dio read
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
tests/btrfs/142 | 137 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/142.out | 39 +++++++++++++++
tests/btrfs/group | 1 +
3 files changed, 177 insertions(+)
create mode 100755 tests/btrfs/142
create mode 100644 tests/btrfs/142.out
diff --git a/tests/btrfs/142 b/tests/btrfs/142
new file mode 100755
index 0000000..953ef0a
--- /dev/null
+++ b/tests/btrfs/142
@@ -0,0 +1,137 @@
+#! /bin/bash
+# FS QA Test 142
+#
+# Regression test for btrfs DIO read's repair during read without checksum.
+#
+# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
+# introduced this regression. It'd cause 'Segmentation fault' error.
+#
+# The upstream fix is
+# Btrfs: fix segment fault when doing dio read
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch_dev_pool 2
+
+_require_btrfs_command inspect-internal dump-tree
+_require_command "$FILEFRAG_PROG" filefrag
+
+get_physical()
+{
+ # $1 is logical address
+ # print chunk tree and find devid 2 which is $SCRATCH_DEV
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+ grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
+}
+
+
+SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
+
+start_fail()
+{
+ echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 2 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > $DEBUGFS_MNT/fail_make_request/verbose
+ echo 1 > $SYSFS_BDEV/make-it-fail
+}
+
+stop_fail()
+{
+ echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 0 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > $SYSFS_BDEV/make-it-fail
+}
+
+_scratch_dev_pool_get 2
+# step 1, create a raid1 btrfs which contains one 128k file.
+echo "step 1......mkfs.btrfs" >>$seqres.full
+
+mkfs_opts="-d raid1 -b 1G"
+_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+
+# -o nospace_cache makes sure data is written to the start position of the data
+# chunk
+_scratch_mount -o nospace_cache,nodatasum
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+
+# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
+# one in $SCRATCH_DEV_POOL
+echo "step 2......corrupt file extent" >>$seqres.full
+
+${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
+logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
+physical_on_scratch=`get_physical ${logical_in_btrfs}`
+
+_scratch_unmount
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_mount -o nospace_cache,nodatasum
+
+# step 3, 128k dio read (this read can repair bad copy)
+echo "step 3......repair the bad copy" >>$seqres.full
+
+# since raid1 consists of two copies, and the following read may read the good
+# copy directly, so lets loop 10 times here and discard output that dio reads
+# give
+for i in `seq 1 10`; do
+ # start_fail only fails the following dio read so the repair is
+ # supposed to work.
+ start_fail
+ $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
+ stop_fail
+ _get_current_dmesg | grep -q -e "direct IO failed" && break
+done
+
+_scratch_unmount
+
+# check if the repair works
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/142.out b/tests/btrfs/142.out
new file mode 100644
index 0000000..d8be7fb
--- /dev/null
+++ b/tests/btrfs/142.out
@@ -0,0 +1,39 @@
+QA output created by 142
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+read 512/512 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 7b380e5..a4f97cd 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -143,3 +143,4 @@
139 auto qgroup
140 auto quick
141 auto quick
+142 auto quick
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* [PATCH 6/6] fstests: regression test for nocsum buffered read's repair
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
` (4 preceding siblings ...)
2017-05-09 17:56 ` [PATCH 5/6] fstests: regression test for nocsum dio " Liu Bo
@ 2017-05-09 17:56 ` Liu Bo
2017-05-10 10:56 ` Eryu Guan
5 siblings, 1 reply; 12+ messages in thread
From: Liu Bo @ 2017-05-09 17:56 UTC (permalink / raw)
To: fstests; +Cc: linux-btrfs, Filipe Manana, Eryu Guan
This is to test whether buffered read retry-repair code is able to work in
raid1 case as expected.
Please note that without checksum, btrfs doesn't know if the data used to
repair is correct, so repair is more of resync which makes sure that both
of the copy has the same content.
Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and drop
checks") introduced the regression.
The upstream fix is
Btrfs: bring back repair during read
Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
---
tests/btrfs/143 | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/143.out | 39 +++++++++++++
tests/btrfs/group | 1 +
3 files changed, 195 insertions(+)
create mode 100755 tests/btrfs/143
create mode 100644 tests/btrfs/143.out
diff --git a/tests/btrfs/143 b/tests/btrfs/143
new file mode 100755
index 0000000..5263e78
--- /dev/null
+++ b/tests/btrfs/143
@@ -0,0 +1,155 @@
+#! /bin/bash
+# FS QA Test 143
+#
+# Regression test for btrfs buffered read's repair during read without checksum.
+#
+# This is to test whether buffered read retry-repair code is able to work in
+# raid1 case as expected.
+#
+# Please note that without checksum, btrfs doesn't know if the data used to
+# repair is correct, so repair is more of resync which makes sure that both
+# of the copy has the same content.
+#
+# Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and drop
+# checks") introduced the regression.
+#
+# The upstream fix is
+# Btrfs: bring back repair during read
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Liu Bo. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch_dev_pool 2
+
+_require_btrfs_command inspect-internal dump-tree
+_require_command "$FILEFRAG_PROG" filefrag
+
+_check_repair()
+{
+ filter=${1:-cat}
+ dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac | $filter | grep -q -e "read error corrected"
+ if [ $? -eq 0 ]; then
+ echo 1
+ else
+ echo 0
+ fi
+}
+
+get_physical()
+{
+ # $1 is logical address
+ # print chunk tree and find devid 2 which is $SCRATCH_DEV
+ $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
+ grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
+}
+
+SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
+
+start_fail()
+{
+ echo 100 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 4 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > $DEBUGFS_MNT/fail_make_request/verbose
+ echo 1 > $SYSFS_BDEV/make-it-fail
+}
+
+stop_fail()
+{
+ echo 0 > $DEBUGFS_MNT/fail_make_request/probability
+ echo 0 > $DEBUGFS_MNT/fail_make_request/times
+ echo 0 > $SYSFS_BDEV/make-it-fail
+}
+
+_scratch_dev_pool_get 2
+# step 1, create a raid1 btrfs which contains one 128k file.
+echo "step 1......mkfs.btrfs" >>$seqres.full
+
+mkfs_opts="-d raid1 -b 1G"
+_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
+
+# -o nospace_cache makes sure data is written to the start position of the data
+# chunk
+_scratch_mount -o nospace_cache,nodatasum
+
+$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
+
+# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
+# one in $SCRATCH_DEV_POOL
+echo "step 2......corrupt file extent" >>$seqres.full
+
+${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
+logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
+physical_on_scratch=`get_physical ${logical_in_btrfs}`
+
+_scratch_unmount
+$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_mount -o nospace_cache
+
+# step 3, 128k buffered read (this read can repair bad copy)
+echo "step 3......repair the bad copy" >>$seqres.full
+
+# since raid1 consists of two copies, and the following read may read the good
+# copy directly, so lets loop 10 times here and discard output that buffered
+# reads give
+for i in `seq 1 10`; do
+ # start_fail only fails the following buffered read so the repair is
+ # supposed to work.
+ echo 3 > /proc/sys/vm/drop_caches
+ start_fail
+ $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
+ stop_fail
+ _get_current_dmesg | grep -q -e "read error corrected" && break
+done
+
+_scratch_unmount
+
+# check if the repair works
+$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
+
+_scratch_dev_pool_put
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/143.out b/tests/btrfs/143.out
new file mode 100644
index 0000000..97bf45c
--- /dev/null
+++ b/tests/btrfs/143.out
@@ -0,0 +1,39 @@
+QA output created by 143
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
+read 512/512 bytes at offset 136708096
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a4f97cd..6f19619 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -144,3 +144,4 @@
140 auto quick
141 auto quick
142 auto quick
+143 auto quick
--
2.5.0
^ permalink raw reply related [flat|nested] 12+ messages in thread
* Re: [PATCH 2/6] fstests: add _get_current_dmesg
2017-05-09 17:56 ` [PATCH 2/6] fstests: add _get_current_dmesg Liu Bo
@ 2017-05-10 10:40 ` Eryu Guan
0 siblings, 0 replies; 12+ messages in thread
From: Eryu Guan @ 2017-05-10 10:40 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Filipe Manana
On Tue, May 09, 2017 at 11:56:07AM -0600, Liu Bo wrote:
> _get_current_dmesg can be used to grep customized pattern.
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
I can't apply this patch on top of current master, perhaps it needs a
rebase :)
> ---
> common/rc | 9 +++++++--
> 1 file changed, 7 insertions(+), 2 deletions(-)
>
> diff --git a/common/rc b/common/rc
> index 78a2101..111ed69 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -3215,6 +3215,12 @@ _get_device_size()
> grep `_short_dev $1` /proc/partitions | awk '{print $3}'
> }
>
> +# get current dmesg log
> +_get_current_dmesg()
> +{
> + dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac
> +}
> +
> # check dmesg log for WARNING/Oops/etc.
> _check_dmesg()
> {
> @@ -3230,8 +3236,7 @@ _check_dmesg()
>
> # search the dmesg log of last run of $seqnum for possible failures
> # use sed \cregexpc address type, since $seqnum contains "/"
This second line of comment should go to _get_current_dmesg.
Thanks,
Eryu
> - dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | \
> - tac | $filter >$seqres.dmesg
> + _get_current_dmesg | $filter >$seqres.dmesg
> grep -q -e "kernel BUG at" \
> -e "WARNING:" \
> -e "BUG:" \
> --
> 2.5.0
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 3/6] fstests: regression test for btrfs dio read repair
2017-05-09 17:56 ` [PATCH 3/6] fstests: regression test for btrfs dio read repair Liu Bo
@ 2017-05-10 10:53 ` Eryu Guan
2017-05-16 17:48 ` Liu Bo
0 siblings, 1 reply; 12+ messages in thread
From: Eryu Guan @ 2017-05-10 10:53 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Filipe Manana
On Tue, May 09, 2017 at 11:56:08AM -0600, Liu Bo wrote:
> This case tests whether dio read can repair the bad copy if we have
> a good copy.
>
> Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> introduced the regression.
>
> The upstream fix is
> Btrfs: fix invalid dereference in btrfs_retry_endio
I noticed this is in upstream now, you can refer to it along with hash
tag too.
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> ---
> tests/btrfs/140 | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/140.out | 39 ++++++++++++++++++
> tests/btrfs/group | 1 +
> 3 files changed, 155 insertions(+)
> create mode 100755 tests/btrfs/140
> create mode 100644 tests/btrfs/140.out
>
> diff --git a/tests/btrfs/140 b/tests/btrfs/140
> new file mode 100755
> index 0000000..09a9939
> --- /dev/null
> +++ b/tests/btrfs/140
> @@ -0,0 +1,115 @@
> +#! /bin/bash
> +# FS QA Test 140
> +#
> +# Regression test for btrfs DIO read's repair during read.
> +#
> +# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> +# introduced the regression.
> +# The upstream fix is
> +# Btrfs: fix invalid dereference in btrfs_retry_endio
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Liu Bo. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
> +
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$FILEFRAG_PROG" filefrag
> +_require_odirect
> +
> +get_physical()
> +{
> + # $1 is logical address
> + # print chunk tree and find devid 2 which is $SCRATCH_DEV
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
> +}
> +
> +_scratch_dev_pool_get 2
> +# step 1, create a raid1 btrfs which contains one 128k file.
> +echo "step 1......mkfs.btrfs" >>$seqres.full
> +
> +mkfs_opts="-d raid1 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# -o nospace_cache makes sure data is written to the start position of the data
> +# chunk
> +_scratch_mount -o nospace_cache
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
> +
> +# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
> +# one in $SCRATCH_DEV_POOL
> +echo "step 2......corrupt file extent" >>$seqres.full
> +
> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
> +logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
> +physical_on_scratch=`get_physical ${logical_in_btrfs}`
> +
> +_scratch_unmount
> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
> +
> +_scratch_mount
> +
> +# step 3, 128k dio read (this read can repair bad copy)
> +echo "step 3......repair the bad copy" >>$seqres.full
> +
> +# since raid1 consists of two copies, and the following read may read the good
> +# copy directly, so lets loop 10 times here and discard output that dio reads
> +# give
> +for i in `seq 1 10`; do
> + $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
> + _get_current_dmesg | grep -q -e "csum failed" && break
> +done
Half of the time I got test failure because pread from SCRATCH_DEV read
0xbb instead of 0xaa on v4.11 kernel (bug should be fixed there), tested
on two different hosts and could hit failure on both hosts.
Similar failure happened to all the 4 tests randomly. I thought it was
because "csum failed" was never hit, so I tried a "while true; do" loop,
and that did fix the btrfs/140 failure for me, but then btrfs/141 would
loop forever sometimes.
On the other hand, the tests from your last post always passed on the
same test host, but I didn't see anything particular would make this
difference..
Can you please take a look? Thanks!
Eryu
> +
> +_scratch_unmount
> +
> +# check if the repair works
> +$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
> +
> +_scratch_dev_pool_put
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/140.out b/tests/btrfs/140.out
> new file mode 100644
> index 0000000..c8565f5
> --- /dev/null
> +++ b/tests/btrfs/140.out
> @@ -0,0 +1,39 @@
> +QA output created by 140
> +wrote 131072/131072 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 65536/65536 bytes at offset 136708096
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +read 512/512 bytes at offset 136708096
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index 9d4b80b..1cb9c98 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -141,3 +141,4 @@
> 137 auto quick send
> 138 auto compress
> 139 auto qgroup
> +140 auto quick
> --
> 2.5.0
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 6/6] fstests: regression test for nocsum buffered read's repair
2017-05-09 17:56 ` [PATCH 6/6] fstests: regression test for nocsum buffered " Liu Bo
@ 2017-05-10 10:56 ` Eryu Guan
0 siblings, 0 replies; 12+ messages in thread
From: Eryu Guan @ 2017-05-10 10:56 UTC (permalink / raw)
To: Liu Bo; +Cc: fstests, linux-btrfs, Filipe Manana
On Tue, May 09, 2017 at 11:56:11AM -0600, Liu Bo wrote:
> This is to test whether buffered read retry-repair code is able to work in
> raid1 case as expected.
>
> Please note that without checksum, btrfs doesn't know if the data used to
> repair is correct, so repair is more of resync which makes sure that both
> of the copy has the same content.
>
> Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and drop
> checks") introduced the regression.
>
> The upstream fix is
> Btrfs: bring back repair during read
btrfs/14[1-3] all could refer to the upstream patch along with its
commit id.
>
> Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> ---
> tests/btrfs/143 | 155 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/143.out | 39 +++++++++++++
> tests/btrfs/group | 1 +
> 3 files changed, 195 insertions(+)
> create mode 100755 tests/btrfs/143
> create mode 100644 tests/btrfs/143.out
>
> diff --git a/tests/btrfs/143 b/tests/btrfs/143
> new file mode 100755
> index 0000000..5263e78
> --- /dev/null
> +++ b/tests/btrfs/143
> @@ -0,0 +1,155 @@
> +#! /bin/bash
> +# FS QA Test 143
> +#
> +# Regression test for btrfs buffered read's repair during read without checksum.
> +#
> +# This is to test whether buffered read retry-repair code is able to work in
> +# raid1 case as expected.
> +#
> +# Please note that without checksum, btrfs doesn't know if the data used to
> +# repair is correct, so repair is more of resync which makes sure that both
> +# of the copy has the same content.
> +#
> +# Commit 20a7db8ab3f2 ("btrfs: add dummy callback for readpage_io_failed and drop
> +# checks") introduced the regression.
> +#
> +# The upstream fix is
> +# Btrfs: bring back repair during read
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Liu Bo. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_scratch_dev_pool 2
btrfs/142 and btrfs/143 need a "_require_fail_make_request"
> +
> +_require_btrfs_command inspect-internal dump-tree
> +_require_command "$FILEFRAG_PROG" filefrag
> +
> +_check_repair()
> +{
> + filter=${1:-cat}
> + dmesg | tac | sed -ne "0,\#run fstests $seqnum at $date_time#p" | tac | $filter | grep -q -e "read error corrected"
> + if [ $? -eq 0 ]; then
> + echo 1
> + else
> + echo 0
> + fi
> +}
This function can be removed.
Thanks,
Eryu
> +
> +get_physical()
> +{
> + # $1 is logical address
> + # print chunk tree and find devid 2 which is $SCRATCH_DEV
> + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> + grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
> +}
> +
> +SYSFS_BDEV=`_sysfs_dev $SCRATCH_DEV`
> +
> +start_fail()
> +{
> + echo 100 > $DEBUGFS_MNT/fail_make_request/probability
> + echo 4 > $DEBUGFS_MNT/fail_make_request/times
> + echo 0 > $DEBUGFS_MNT/fail_make_request/verbose
> + echo 1 > $SYSFS_BDEV/make-it-fail
> +}
> +
> +stop_fail()
> +{
> + echo 0 > $DEBUGFS_MNT/fail_make_request/probability
> + echo 0 > $DEBUGFS_MNT/fail_make_request/times
> + echo 0 > $SYSFS_BDEV/make-it-fail
> +}
> +
> +_scratch_dev_pool_get 2
> +# step 1, create a raid1 btrfs which contains one 128k file.
> +echo "step 1......mkfs.btrfs" >>$seqres.full
> +
> +mkfs_opts="-d raid1 -b 1G"
> +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> +
> +# -o nospace_cache makes sure data is written to the start position of the data
> +# chunk
> +_scratch_mount -o nospace_cache,nodatasum
> +
> +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
> +
> +# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
> +# one in $SCRATCH_DEV_POOL
> +echo "step 2......corrupt file extent" >>$seqres.full
> +
> +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
> +logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
> +physical_on_scratch=`get_physical ${logical_in_btrfs}`
> +
> +_scratch_unmount
> +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
> +
> +_scratch_mount -o nospace_cache
> +
> +# step 3, 128k buffered read (this read can repair bad copy)
> +echo "step 3......repair the bad copy" >>$seqres.full
> +
> +# since raid1 consists of two copies, and the following read may read the good
> +# copy directly, so lets loop 10 times here and discard output that buffered
> +# reads give
> +for i in `seq 1 10`; do
> + # start_fail only fails the following buffered read so the repair is
> + # supposed to work.
> + echo 3 > /proc/sys/vm/drop_caches
> + start_fail
> + $XFS_IO_PROG -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
> + stop_fail
> + _get_current_dmesg | grep -q -e "read error corrected" && break
> +done
> +
> +_scratch_unmount
> +
> +# check if the repair works
> +$XFS_IO_PROG -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
> +
> +_scratch_dev_pool_put
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/143.out b/tests/btrfs/143.out
> new file mode 100644
> index 0000000..97bf45c
> --- /dev/null
> +++ b/tests/btrfs/143.out
> @@ -0,0 +1,39 @@
> +QA output created by 143
> +wrote 131072/131072 bytes at offset 0
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +wrote 65536/65536 bytes at offset 136708096
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> +08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> +read 512/512 bytes at offset 136708096
> +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index a4f97cd..6f19619 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -144,3 +144,4 @@
> 140 auto quick
> 141 auto quick
> 142 auto quick
> +143 auto quick
> --
> 2.5.0
>
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 3/6] fstests: regression test for btrfs dio read repair
2017-05-10 10:53 ` Eryu Guan
@ 2017-05-16 17:48 ` Liu Bo
2017-05-17 4:59 ` Liu Bo
0 siblings, 1 reply; 12+ messages in thread
From: Liu Bo @ 2017-05-16 17:48 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Filipe Manana
On Wed, May 10, 2017 at 06:53:26PM +0800, Eryu Guan wrote:
> On Tue, May 09, 2017 at 11:56:08AM -0600, Liu Bo wrote:
> > This case tests whether dio read can repair the bad copy if we have
> > a good copy.
> >
> > Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> > introduced the regression.
> >
> > The upstream fix is
> > Btrfs: fix invalid dereference in btrfs_retry_endio
>
> I noticed this is in upstream now, you can refer to it along with hash
> tag too.
>
> >
> > Signed-off-by: Liu Bo <bo.li.liu@oracle.com>
> > ---
> > tests/btrfs/140 | 115 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> > tests/btrfs/140.out | 39 ++++++++++++++++++
> > tests/btrfs/group | 1 +
> > 3 files changed, 155 insertions(+)
> > create mode 100755 tests/btrfs/140
> > create mode 100644 tests/btrfs/140.out
> >
> > diff --git a/tests/btrfs/140 b/tests/btrfs/140
> > new file mode 100755
> > index 0000000..09a9939
> > --- /dev/null
> > +++ b/tests/btrfs/140
> > @@ -0,0 +1,115 @@
> > +#! /bin/bash
> > +# FS QA Test 140
> > +#
> > +# Regression test for btrfs DIO read's repair during read.
> > +#
> > +# Commit 2dabb3248453 ("Btrfs: Direct I/O read: Work on sectorsized blocks")
> > +# introduced the regression.
> > +# The upstream fix is
> > +# Btrfs: fix invalid dereference in btrfs_retry_endio
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017 Liu Bo. All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1 # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > + cd /
> > + rm -f $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# remove previous $seqres.full before test
> > +rm -f $seqres.full
> > +
> > +# real QA test starts here
> > +
> > +# Modify as appropriate.
> > +_supported_fs btrfs
> > +_supported_os Linux
> > +_require_scratch_dev_pool 2
> > +
> > +_require_btrfs_command inspect-internal dump-tree
> > +_require_command "$FILEFRAG_PROG" filefrag
> > +_require_odirect
> > +
> > +get_physical()
> > +{
> > + # $1 is logical address
> > + # print chunk tree and find devid 2 which is $SCRATCH_DEV
> > + $BTRFS_UTIL_PROG inspect-internal dump-tree -t 3 $SCRATCH_DEV | \
> > + grep $1 -A 6 | awk '($1 ~ /stripe/ && $3 ~ /devid/ && $4 ~ /1/) { print $6 }'
> > +}
> > +
> > +_scratch_dev_pool_get 2
> > +# step 1, create a raid1 btrfs which contains one 128k file.
> > +echo "step 1......mkfs.btrfs" >>$seqres.full
> > +
> > +mkfs_opts="-d raid1 -b 1G"
> > +_scratch_pool_mkfs $mkfs_opts >>$seqres.full 2>&1
> > +
> > +# -o nospace_cache makes sure data is written to the start position of the data
> > +# chunk
> > +_scratch_mount -o nospace_cache
> > +
> > +$XFS_IO_PROG -f -d -c "pwrite -S 0xaa -b 128K 0 128K" "$SCRATCH_MNT/foobar" | _filter_xfs_io
> > +
> > +# step 2, corrupt the first 64k of one copy (on SCRATCH_DEV which is the first
> > +# one in $SCRATCH_DEV_POOL
> > +echo "step 2......corrupt file extent" >>$seqres.full
> > +
> > +${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar >> $seqres.full
> > +logical_in_btrfs=`${FILEFRAG_PROG} -v $SCRATCH_MNT/foobar | _filter_filefrag | cut -d '#' -f 1`
> > +physical_on_scratch=`get_physical ${logical_in_btrfs}`
> > +
> > +_scratch_unmount
> > +$XFS_IO_PROG -d -c "pwrite -S 0xbb -b 64K $physical_on_scratch 64K" $SCRATCH_DEV | _filter_xfs_io
> > +
> > +_scratch_mount
> > +
> > +# step 3, 128k dio read (this read can repair bad copy)
> > +echo "step 3......repair the bad copy" >>$seqres.full
> > +
> > +# since raid1 consists of two copies, and the following read may read the good
> > +# copy directly, so lets loop 10 times here and discard output that dio reads
> > +# give
> > +for i in `seq 1 10`; do
> > + $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
> > + _get_current_dmesg | grep -q -e "csum failed" && break
> > +done
>
> Half of the time I got test failure because pread from SCRATCH_DEV read
> 0xbb instead of 0xaa on v4.11 kernel (bug should be fixed there), tested
> on two different hosts and could hit failure on both hosts.
>
> Similar failure happened to all the 4 tests randomly. I thought it was
> because "csum failed" was never hit, so I tried a "while true; do" loop,
> and that did fix the btrfs/140 failure for me, but then btrfs/141 would
> loop forever sometimes.
>
> On the other hand, the tests from your last post always passed on the
> same test host, but I didn't see anything particular would make this
> difference..
>
> Can you please take a look? Thanks!
>
Oh, sorry for the trouble, it's all due to the same reason, that
is, the stripe read balance in btrfs simply looks at
(current->pid % num_stripes) and picks up a stripe to read from.
Since I put the bad data on stripe 1 in raid1 profile, we need an
odd $pid to trigger the checksum failures, but I have no idea how
to certainly get a task with odd pid number in one shot, so I'll
just use "while true; do" for now, and update it later if I find
a solution.
Thanks,
-liubo
> Eryu
>
> > +
> > +_scratch_unmount
> > +
> > +# check if the repair works
> > +$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
> > +
> > +_scratch_dev_pool_put
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/btrfs/140.out b/tests/btrfs/140.out
> > new file mode 100644
> > index 0000000..c8565f5
> > --- /dev/null
> > +++ b/tests/btrfs/140.out
> > @@ -0,0 +1,39 @@
> > +QA output created by 140
> > +wrote 131072/131072 bytes at offset 0
> > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > +wrote 65536/65536 bytes at offset 136708096
> > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > +08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > +read 512/512 bytes at offset 136708096
> > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > diff --git a/tests/btrfs/group b/tests/btrfs/group
> > index 9d4b80b..1cb9c98 100644
> > --- a/tests/btrfs/group
> > +++ b/tests/btrfs/group
> > @@ -141,3 +141,4 @@
> > 137 auto quick send
> > 138 auto compress
> > 139 auto qgroup
> > +140 auto quick
> > --
> > 2.5.0
> >
^ permalink raw reply [flat|nested] 12+ messages in thread
* Re: [PATCH 3/6] fstests: regression test for btrfs dio read repair
2017-05-16 17:48 ` Liu Bo
@ 2017-05-17 4:59 ` Liu Bo
0 siblings, 0 replies; 12+ messages in thread
From: Liu Bo @ 2017-05-17 4:59 UTC (permalink / raw)
To: Eryu Guan; +Cc: fstests, linux-btrfs, Filipe Manana
On Tue, May 16, 2017 at 11:48:46AM -0600, Liu Bo wrote:
> On Wed, May 10, 2017 at 06:53:26PM +0800, Eryu Guan wrote:
> > On Tue, May 09, 2017 at 11:56:08AM -0600, Liu Bo wrote:
[...]
> > > +
> > > +# step 3, 128k dio read (this read can repair bad copy)
> > > +echo "step 3......repair the bad copy" >>$seqres.full
> > > +
> > > +# since raid1 consists of two copies, and the following read may read the good
> > > +# copy directly, so lets loop 10 times here and discard output that dio reads
> > > +# give
> > > +for i in `seq 1 10`; do
> > > + $XFS_IO_PROG -d -c "pread -b 128K 0 128K" "$SCRATCH_MNT/foobar" > /dev/null
> > > + _get_current_dmesg | grep -q -e "csum failed" && break
> > > +done
> >
> > Half of the time I got test failure because pread from SCRATCH_DEV read
> > 0xbb instead of 0xaa on v4.11 kernel (bug should be fixed there), tested
> > on two different hosts and could hit failure on both hosts.
> >
> > Similar failure happened to all the 4 tests randomly. I thought it was
> > because "csum failed" was never hit, so I tried a "while true; do" loop,
> > and that did fix the btrfs/140 failure for me, but then btrfs/141 would
> > loop forever sometimes.
> >
> > On the other hand, the tests from your last post always passed on the
> > same test host, but I didn't see anything particular would make this
> > difference..
> >
> > Can you please take a look? Thanks!
> >
>
> Oh, sorry for the trouble, it's all due to the same reason, that
> is, the stripe read balance in btrfs simply looks at
> (current->pid % num_stripes) and picks up a stripe to read from.
>
> Since I put the bad data on stripe 1 in raid1 profile, we need an
> odd $pid to trigger the checksum failures, but I have no idea how
> to certainly get a task with odd pid number in one shot, so I'll
> just use "while true; do" for now, and update it later if I find
> a solution.
>
(Originally I thought that 'loop forever' was due to bad luck so that the reader
always gets an evenly %pid.)
I figured out why running ./check btrfs/14[0-1] would end up looping on 141
forever, it turns out that csum errors got printed by btrfs_warn_rl which has a
global rate limit, running 140 will drain the rate limit so running 141 won't
have csum errors output in dmesg and it loops forever since 'grep' couldn't find
anything.
Obviously that forever thing is not acceptable, so..here is the workaround.
Since I've put the bad copy on stripe #1 while the good copy lies on stripe #0,
in that 'while true; do' loop, the bad copy gets access when (the reader's pid %
2 == 1) is true, thus we could check the reader's pid instead of doing grep in
dmesg. It's probably fragile though.
Thanks,
-liubo
> Thanks,
>
> -liubo
> > Eryu
> >
> > > +
> > > +_scratch_unmount
> > > +
> > > +# check if the repair works
> > > +$XFS_IO_PROG -d -c "pread -v -b 512 $physical_on_scratch 512" $SCRATCH_DEV | _filter_xfs_io
> > > +
> > > +_scratch_dev_pool_put
> > > +# success, all done
> > > +status=0
> > > +exit
> > > diff --git a/tests/btrfs/140.out b/tests/btrfs/140.out
> > > new file mode 100644
> > > index 0000000..c8565f5
> > > --- /dev/null
> > > +++ b/tests/btrfs/140.out
> > > @@ -0,0 +1,39 @@
> > > +QA output created by 140
> > > +wrote 131072/131072 bytes at offset 0
> > > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > > +wrote 65536/65536 bytes at offset 136708096
> > > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > > +08260000: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260010: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260020: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260030: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260040: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260050: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260060: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260070: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260080: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260090: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082600f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260100: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260110: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260120: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260130: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260140: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260150: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260160: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260170: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260180: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +08260190: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601a0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601b0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601c0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601d0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601e0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +082601f0: aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa aa ................
> > > +read 512/512 bytes at offset 136708096
> > > +XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
> > > diff --git a/tests/btrfs/group b/tests/btrfs/group
> > > index 9d4b80b..1cb9c98 100644
> > > --- a/tests/btrfs/group
> > > +++ b/tests/btrfs/group
> > > @@ -141,3 +141,4 @@
> > > 137 auto quick send
> > > 138 auto compress
> > > 139 auto qgroup
> > > +140 auto quick
> > > --
> > > 2.5.0
> > >
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 12+ messages in thread
end of thread, other threads:[~2017-05-17 5:01 UTC | newest]
Thread overview: 12+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-05-09 17:56 [PATCH 0/6] Regression test for btrfs read repair Liu Bo
2017-05-09 17:56 ` [PATCH 1/6] fstests: add _filter_filefrag Liu Bo
2017-05-09 17:56 ` [PATCH 2/6] fstests: add _get_current_dmesg Liu Bo
2017-05-10 10:40 ` Eryu Guan
2017-05-09 17:56 ` [PATCH 3/6] fstests: regression test for btrfs dio read repair Liu Bo
2017-05-10 10:53 ` Eryu Guan
2017-05-16 17:48 ` Liu Bo
2017-05-17 4:59 ` Liu Bo
2017-05-09 17:56 ` [PATCH 4/6] fstests: regression test for btrfs buffered read's repair Liu Bo
2017-05-09 17:56 ` [PATCH 5/6] fstests: regression test for nocsum dio " Liu Bo
2017-05-09 17:56 ` [PATCH 6/6] fstests: regression test for nocsum buffered " Liu Bo
2017-05-10 10:56 ` Eryu Guan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.