* [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
@ 2018-06-01 1:34 Qu Wenruo
2018-06-05 10:42 ` Anand Jain
` (2 more replies)
0 siblings, 3 replies; 8+ messages in thread
From: Qu Wenruo @ 2018-06-01 1:34 UTC (permalink / raw)
To: linux-btrfs, fstests
This is a long existing bug (from 2012) but exposed by a reporter
recently, that when compressed extent without data csum get written to
device-replace target device, the written data is in fact uncompressed data
other than the original compressed data.
And since btrfs still consider the data is compressed and will try to read it
as compressed, it can cause read error.
The root cause is located, and one RFC patch already sent to fix it,
titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
(The RFC is only for the extra possible way to fix the bug, the fix
itself should work without problem)
Reported-by: James Harvey <jamespharvey20@gmail.com>
Signed-off-by: Qu Wenruo <wqu@suse.com>
---
tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/161.out | 2 +
tests/btrfs/group | 1 +
3 files changed, 94 insertions(+)
create mode 100755 tests/btrfs/161
create mode 100644 tests/btrfs/161.out
diff --git a/tests/btrfs/161 b/tests/btrfs/161
new file mode 100755
index 00000000..d4a2b474
--- /dev/null
+++ b/tests/btrfs/161
@@ -0,0 +1,91 @@
+#! /bin/bash
+# FS QA Test 161
+#
+# Test if btrfs will corrupt compressed data extent without data csum
+# by replacing it with uncompressed data, when doing replacing device.
+#
+# This could be fixed by the following RFC patch:
+# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
+#
+#-----------------------------------------------------------------------
+# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_test
+_require_scratch_dev_pool 2
+_require_scratch_dev_pool_equal_size
+
+
+_scratch_dev_pool_get 1
+_spare_dev_get
+_scratch_pool_mkfs >> $seqres.full 2>&1
+
+# Create nodatasum inode
+_scratch_mount "-o nodatasum"
+touch $SCRATCH_MNT/nodatasum_file
+_scratch_remount "datasum,compress"
+_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
+
+# Write the compressed data back to disk
+sync
+
+# Replace the device
+_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
+
+_scratch_unmount
+
+_mount $SPARE_DEV $SCRATCH_MNT
+
+# Since now the compressed extent contains *UNCOMPRESSED* data, reading it will
+# easily trigger a EIO error
+cat $SCRATCH_MNT/nodatasum_file > /dev/null
+
+_scratch_unmount
+_spare_dev_put
+_scratch_dev_pool_put
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
new file mode 100644
index 00000000..1752a243
--- /dev/null
+++ b/tests/btrfs/161.out
@@ -0,0 +1,2 @@
+QA output created by 161
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index f04ee8d5..f900b3d0 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -163,3 +163,4 @@
158 auto quick raid scrub
159 auto quick
160 auto quick
+161 auto quick replace
--
2.17.0
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-01 1:34 [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device Qu Wenruo
@ 2018-06-05 10:42 ` Anand Jain
2018-06-05 11:39 ` Qu Wenruo
2018-06-07 6:21 ` Eryu Guan
2018-06-28 5:11 ` Nikolay Borisov
2 siblings, 1 reply; 8+ messages in thread
From: Anand Jain @ 2018-06-05 10:42 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs, fstests
On 06/01/2018 09:34 AM, Qu Wenruo wrote:
> This is a long existing bug (from 2012) but exposed by a reporter
> recently, that when compressed extent without data csum get written to
> device-replace target device, the written data is in fact uncompressed data
> other than the original compressed data.
>
> And since btrfs still consider the data is compressed and will try to read it
> as compressed, it can cause read error.
>
> The root cause is located, and one RFC patch already sent to fix it,
> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> (The RFC is only for the extra possible way to fix the bug, the fix
> itself should work without problem)
>
> Reported-by: James Harvey <jamespharvey20@gmail.com>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
> ---
> tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/161.out | 2 +
> tests/btrfs/group | 1 +
> 3 files changed, 94 insertions(+)
> create mode 100755 tests/btrfs/161
> create mode 100644 tests/btrfs/161.out
>
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index 00000000..d4a2b474
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,91 @@
> +#! /bin/bash
> +# FS QA Test 161
> +#
> +# Test if btrfs will corrupt compressed data extent without data csum
> +# by replacing it with uncompressed data, when doing replacing device.
> +#
> +# This could be fixed by the following RFC patch:
> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_test
> +_require_scratch_dev_pool 2
> +_require_scratch_dev_pool_equal_size
> +
> +
> +_scratch_dev_pool_get 1
> +_spare_dev_get
> +_scratch_pool_mkfs >> $seqres.full 2>&1
> +
> +# Create nodatasum inode
> +_scratch_mount "-o nodatasum"
> +touch $SCRATCH_MNT/nodatasum_file
> +_scratch_remount "datasum,compress"
> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +# Write the compressed data back to disk
> +sync
> +
> +# Replace the device
> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
> +
> +_scratch_unmount
> +
> +_mount $SPARE_DEV $SCRATCH_MNT
> +
> +# Since now the compressed extent contains *UNCOMPRESSED* data, reading it will
> +# easily trigger a EIO error
This sentence is confusing. Are you referring to the bug?
what is compressed extent containing uncompressed data?
Thanks, Anand
> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +_scratch_unmount
> +_spare_dev_put
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index 00000000..1752a243
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,2 @@
> +QA output created by 161
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5..f900b3d0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -163,3 +163,4 @@
> 158 auto quick raid scrub
> 159 auto quick
> 160 auto quick
> +161 auto quick replace
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-05 10:42 ` Anand Jain
@ 2018-06-05 11:39 ` Qu Wenruo
0 siblings, 0 replies; 8+ messages in thread
From: Qu Wenruo @ 2018-06-05 11:39 UTC (permalink / raw)
To: Anand Jain, Qu Wenruo, linux-btrfs, fstests
[-- Attachment #1.1: Type: text/plain, Size: 5356 bytes --]
On 2018年06月05日 18:42, Anand Jain wrote:
>
>
> On 06/01/2018 09:34 AM, Qu Wenruo wrote:
>> This is a long existing bug (from 2012) but exposed by a reporter
>> recently, that when compressed extent without data csum get written to
>> device-replace target device, the written data is in fact uncompressed
>> data
>> other than the original compressed data.
>>
>> And since btrfs still consider the data is compressed and will try to
>> read it
>> as compressed, it can cause read error.
>>
>> The root cause is located, and one RFC patch already sent to fix it,
>> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device
>> replace".
>> (The RFC is only for the extra possible way to fix the bug, the fix
>> itself should work without problem)
>>
>> Reported-by: James Harvey <jamespharvey20@gmail.com>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>> ---
>> tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/161.out | 2 +
>> tests/btrfs/group | 1 +
>> 3 files changed, 94 insertions(+)
>> create mode 100755 tests/btrfs/161
>> create mode 100644 tests/btrfs/161.out
>>
>> diff --git a/tests/btrfs/161 b/tests/btrfs/161
>> new file mode 100755
>> index 00000000..d4a2b474
>> --- /dev/null
>> +++ b/tests/btrfs/161
>> @@ -0,0 +1,91 @@
>> +#! /bin/bash
>> +# FS QA Test 161
>> +#
>> +# Test if btrfs will corrupt compressed data extent without data csum
>> +# by replacing it with uncompressed data, when doing replacing device.
>> +#
>> +# This could be fixed by the following RFC patch:
>> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_test
>> +_require_scratch_dev_pool 2
>> +_require_scratch_dev_pool_equal_size
>> +
>> +
>> +_scratch_dev_pool_get 1
>> +_spare_dev_get
>> +_scratch_pool_mkfs >> $seqres.full 2>&1
>> +
>> +# Create nodatasum inode
>> +_scratch_mount "-o nodatasum"
>> +touch $SCRATCH_MNT/nodatasum_file
>> +_scratch_remount "datasum,compress"
>> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
>> +
>> +# Write the compressed data back to disk
>> +sync
>> +
>> +# Replace the device
>> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
>> +
>> +_scratch_unmount
>> +
>> +_mount $SPARE_DEV $SCRATCH_MNT
>> +
>> +# Since now the compressed extent contains *UNCOMPRESSED* data,
>> reading it will
>> +# easily trigger a EIO error
>
> This sentence is confusing. Are you referring to the bug?
Yes.
> what is compressed extent containing uncompressed data?
The data extent has lzo compression flag.
While its on-disk data is in fact the uncompressed data.
With the test case, just check the on-disk data extent, you'll find
that's all 0xcd, other the lzo compressed data.
Thanks,
Qu
>
> Thanks, Anand
>
>> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
>> +
>> +_scratch_unmount
>> +_spare_dev_put
>> +_scratch_dev_pool_put
>> +
>> +echo "Silence is golden"
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
>> new file mode 100644
>> index 00000000..1752a243
>> --- /dev/null
>> +++ b/tests/btrfs/161.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 161
>> +Silence is golden
>> diff --git a/tests/btrfs/group b/tests/btrfs/group
>> index f04ee8d5..f900b3d0 100644
>> --- a/tests/btrfs/group
>> +++ b/tests/btrfs/group
>> @@ -163,3 +163,4 @@
>> 158 auto quick raid scrub
>> 159 auto quick
>> 160 auto quick
>> +161 auto quick replace
>>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-01 1:34 [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device Qu Wenruo
2018-06-05 10:42 ` Anand Jain
@ 2018-06-07 6:21 ` Eryu Guan
2018-06-07 6:58 ` Qu Wenruo
2018-06-28 5:11 ` Nikolay Borisov
2 siblings, 1 reply; 8+ messages in thread
From: Eryu Guan @ 2018-06-07 6:21 UTC (permalink / raw)
To: Qu Wenruo; +Cc: linux-btrfs, fstests
On Fri, Jun 01, 2018 at 09:34:48AM +0800, Qu Wenruo wrote:
> This is a long existing bug (from 2012) but exposed by a reporter
> recently, that when compressed extent without data csum get written to
> device-replace target device, the written data is in fact uncompressed data
> other than the original compressed data.
>
> And since btrfs still consider the data is compressed and will try to read it
> as compressed, it can cause read error.
>
> The root cause is located, and one RFC patch already sent to fix it,
> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> (The RFC is only for the extra possible way to fix the bug, the fix
> itself should work without problem)
>
> Reported-by: James Harvey <jamespharvey20@gmail.com>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
Looks fine to me overall, some minor issues inline. But I'd really like
an explicit ACK from btrfs folks.
> ---
> tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/161.out | 2 +
> tests/btrfs/group | 1 +
> 3 files changed, 94 insertions(+)
> create mode 100755 tests/btrfs/161
> create mode 100644 tests/btrfs/161.out
>
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index 00000000..d4a2b474
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,91 @@
> +#! /bin/bash
> +# FS QA Test 161
> +#
> +# Test if btrfs will corrupt compressed data extent without data csum
> +# by replacing it with uncompressed data, when doing replacing device.
> +#
> +# This could be fixed by the following RFC patch:
> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_test
This is not needed, there's no point running fsck on $TEST_DEV after
test when it's not used at all.
> +_require_scratch_dev_pool 2
> +_require_scratch_dev_pool_equal_size
> +
> +
> +_scratch_dev_pool_get 1
> +_spare_dev_get
> +_scratch_pool_mkfs >> $seqres.full 2>&1
> +
> +# Create nodatasum inode
> +_scratch_mount "-o nodatasum"
> +touch $SCRATCH_MNT/nodatasum_file
> +_scratch_remount "datasum,compress"
> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +# Write the compressed data back to disk
> +sync
> +
> +# Replace the device
> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
> +
> +_scratch_unmount
> +
> +_mount $SPARE_DEV $SCRATCH_MNT
Better to explain the cycle mount with comments.
> +
> +# Since now the compressed extent contains *UNCOMPRESSED* data, reading it will
> +# easily trigger a EIO error
> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +_scratch_unmount
> +_spare_dev_put
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index 00000000..1752a243
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,2 @@
> +QA output created by 161
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5..f900b3d0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -163,3 +163,4 @@
> 158 auto quick raid scrub
> 159 auto quick
> 160 auto quick
> +161 auto quick replace
Anand introduced a new 'volume' group in commit 144c8463d38b ("btrfs:
introduce btrfs/volume group"), all 'replace' tests should be in
'volume' group now.
Thanks,
Eryu
> --
> 2.17.0
>
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-07 6:21 ` Eryu Guan
@ 2018-06-07 6:58 ` Qu Wenruo
0 siblings, 0 replies; 8+ messages in thread
From: Qu Wenruo @ 2018-06-07 6:58 UTC (permalink / raw)
To: Eryu Guan, Qu Wenruo; +Cc: linux-btrfs, fstests
[-- Attachment #1.1: Type: text/plain, Size: 5842 bytes --]
On 2018年06月07日 14:21, Eryu Guan wrote:
> On Fri, Jun 01, 2018 at 09:34:48AM +0800, Qu Wenruo wrote:
>> This is a long existing bug (from 2012) but exposed by a reporter
>> recently, that when compressed extent without data csum get written to
>> device-replace target device, the written data is in fact uncompressed data
>> other than the original compressed data.
>>
>> And since btrfs still consider the data is compressed and will try to read it
>> as compressed, it can cause read error.
>>
>> The root cause is located, and one RFC patch already sent to fix it,
>> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
>> (The RFC is only for the extra possible way to fix the bug, the fix
>> itself should work without problem)
>>
>> Reported-by: James Harvey <jamespharvey20@gmail.com>
>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>
> Looks fine to me overall, some minor issues inline. But I'd really like
> an explicit ACK from btrfs folks.
>
>> ---
>> tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
>> tests/btrfs/161.out | 2 +
>> tests/btrfs/group | 1 +
>> 3 files changed, 94 insertions(+)
>> create mode 100755 tests/btrfs/161
>> create mode 100644 tests/btrfs/161.out
>>
>> diff --git a/tests/btrfs/161 b/tests/btrfs/161
>> new file mode 100755
>> index 00000000..d4a2b474
>> --- /dev/null
>> +++ b/tests/btrfs/161
>> @@ -0,0 +1,91 @@
>> +#! /bin/bash
>> +# FS QA Test 161
>> +#
>> +# Test if btrfs will corrupt compressed data extent without data csum
>> +# by replacing it with uncompressed data, when doing replacing device.
>> +#
>> +# This could be fixed by the following RFC patch:
>> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
>> +#
>> +#-----------------------------------------------------------------------
>> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
>> +#
>> +# This program is free software; you can redistribute it and/or
>> +# modify it under the terms of the GNU General Public License as
>> +# published by the Free Software Foundation.
>> +#
>> +# This program is distributed in the hope that it would be useful,
>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
>> +# GNU General Public License for more details.
>> +#
>> +# You should have received a copy of the GNU General Public License
>> +# along with this program; if not, write the Free Software Foundation,
>> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
>> +#-----------------------------------------------------------------------
>> +#
>> +
>> +seq=`basename $0`
>> +seqres=$RESULT_DIR/$seq
>> +echo "QA output created by $seq"
>> +
>> +here=`pwd`
>> +tmp=/tmp/$$
>> +status=1 # failure is the default!
>> +trap "_cleanup; exit \$status" 0 1 2 3 15
>> +
>> +_cleanup()
>> +{
>> + cd /
>> + rm -f $tmp.*
>> +}
>> +
>> +# get standard environment, filters and checks
>> +. ./common/rc
>> +. ./common/filter
>> +
>> +# remove previous $seqres.full before test
>> +rm -f $seqres.full
>> +
>> +# real QA test starts here
>> +
>> +# Modify as appropriate.
>> +_supported_fs btrfs
>> +_supported_os Linux
>> +_require_test
>
> This is not needed, there's no point running fsck on $TEST_DEV after
> test when it's not used at all.
Forgot to remove that line.
>
>> +_require_scratch_dev_pool 2
>> +_require_scratch_dev_pool_equal_size
>> +
>> +
>> +_scratch_dev_pool_get 1
>> +_spare_dev_get
>> +_scratch_pool_mkfs >> $seqres.full 2>&1
>> +
>> +# Create nodatasum inode
>> +_scratch_mount "-o nodatasum"
>> +touch $SCRATCH_MNT/nodatasum_file
>> +_scratch_remount "datasum,compress"
>> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
>> +
>> +# Write the compressed data back to disk
>> +sync
>> +
>> +# Replace the device
>> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
>> +
>> +_scratch_unmount
>> +
>> +_mount $SPARE_DEV $SCRATCH_MNT
>
> Better to explain the cycle mount with comments.
It's used to drop the page cache so that next read will fetch data from
disk other than using the cache.
I'll add the comment in the next version.
>
>> +
>> +# Since now the compressed extent contains *UNCOMPRESSED* data, reading it will
>> +# easily trigger a EIO error
>> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
>> +
>> +_scratch_unmount
>> +_spare_dev_put
>> +_scratch_dev_pool_put
>> +
>> +echo "Silence is golden"
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
>> new file mode 100644
>> index 00000000..1752a243
>> --- /dev/null
>> +++ b/tests/btrfs/161.out
>> @@ -0,0 +1,2 @@
>> +QA output created by 161
>> +Silence is golden
>> diff --git a/tests/btrfs/group b/tests/btrfs/group
>> index f04ee8d5..f900b3d0 100644
>> --- a/tests/btrfs/group
>> +++ b/tests/btrfs/group
>> @@ -163,3 +163,4 @@
>> 158 auto quick raid scrub
>> 159 auto quick
>> 160 auto quick
>> +161 auto quick replace
>
> Anand introduced a new 'volume' group in commit 144c8463d38b ("btrfs:
> introduce btrfs/volume group"), all 'replace' tests should be in
> 'volume' group now.
Makes sense.
Thanks,
Qu
>
> Thanks,
> Eryu
>
>> --
>> 2.17.0
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe fstests" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at http://vger.kernel.org/majordomo-info.html
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 488 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-01 1:34 [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device Qu Wenruo
2018-06-05 10:42 ` Anand Jain
2018-06-07 6:21 ` Eryu Guan
@ 2018-06-28 5:11 ` Nikolay Borisov
2018-06-28 5:34 ` Eryu Guan
2 siblings, 1 reply; 8+ messages in thread
From: Nikolay Borisov @ 2018-06-28 5:11 UTC (permalink / raw)
To: Qu Wenruo, linux-btrfs, fstests
On 1.06.2018 04:34, Qu Wenruo wrote:
> This is a long existing bug (from 2012) but exposed by a reporter
> recently, that when compressed extent without data csum get written to
> device-replace target device, the written data is in fact uncompressed data
> other than the original compressed data.
>
> And since btrfs still consider the data is compressed and will try to read it
> as compressed, it can cause read error.
>
> The root cause is located, and one RFC patch already sent to fix it,
> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> (The RFC is only for the extra possible way to fix the bug, the fix
> itself should work without problem)
>
> Reported-by: James Harvey <jamespharvey20@gmail.com>
> Signed-off-by: Qu Wenruo <wqu@suse.com>
Reviewed-by: Nikolay Borisov <nborisov@suse.com>
> ---
> tests/btrfs/161 | 91 +++++++++++++++++++++++++++++++++++++++++++++
> tests/btrfs/161.out | 2 +
> tests/btrfs/group | 1 +
> 3 files changed, 94 insertions(+)
> create mode 100755 tests/btrfs/161
> create mode 100644 tests/btrfs/161.out
>
> diff --git a/tests/btrfs/161 b/tests/btrfs/161
> new file mode 100755
> index 00000000..d4a2b474
> --- /dev/null
> +++ b/tests/btrfs/161
> @@ -0,0 +1,91 @@
> +#! /bin/bash
> +# FS QA Test 161
> +#
> +# Test if btrfs will corrupt compressed data extent without data csum
> +# by replacing it with uncompressed data, when doing replacing device.
> +#
> +# This could be fixed by the following RFC patch:
> +# "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace"
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (C) 2018 SUSE Linux Products GmbH. All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1 # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> + cd /
> + rm -f $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# remove previous $seqres.full before test
> +rm -f $seqres.full
> +
> +# real QA test starts here
> +
> +# Modify as appropriate.
> +_supported_fs btrfs
> +_supported_os Linux
> +_require_test
> +_require_scratch_dev_pool 2
> +_require_scratch_dev_pool_equal_size
> +
> +
> +_scratch_dev_pool_get 1
> +_spare_dev_get
> +_scratch_pool_mkfs >> $seqres.full 2>&1
> +
> +# Create nodatasum inode
> +_scratch_mount "-o nodatasum"
> +touch $SCRATCH_MNT/nodatasum_file
> +_scratch_remount "datasum,compress"
> +_pwrite_byte 0xcd 0 128K $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +# Write the compressed data back to disk
> +sync
> +
> +# Replace the device
> +_run_btrfs_util_prog replace start -Bf 1 $SPARE_DEV $SCRATCH_MNT
> +
> +_scratch_unmount
> +
> +_mount $SPARE_DEV $SCRATCH_MNT
> +
> +# Since now the compressed extent contains *UNCOMPRESSED* data, reading it will
> +# easily trigger a EIO error
> +cat $SCRATCH_MNT/nodatasum_file > /dev/null
> +
> +_scratch_unmount
> +_spare_dev_put
> +_scratch_dev_pool_put
> +
> +echo "Silence is golden"
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
> new file mode 100644
> index 00000000..1752a243
> --- /dev/null
> +++ b/tests/btrfs/161.out
> @@ -0,0 +1,2 @@
> +QA output created by 161
> +Silence is golden
> diff --git a/tests/btrfs/group b/tests/btrfs/group
> index f04ee8d5..f900b3d0 100644
> --- a/tests/btrfs/group
> +++ b/tests/btrfs/group
> @@ -163,3 +163,4 @@
> 158 auto quick raid scrub
> 159 auto quick
> 160 auto quick
> +161 auto quick replace
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-28 5:11 ` Nikolay Borisov
@ 2018-06-28 5:34 ` Eryu Guan
2018-06-28 6:31 ` Nikolay Borisov
0 siblings, 1 reply; 8+ messages in thread
From: Eryu Guan @ 2018-06-28 5:34 UTC (permalink / raw)
To: Nikolay Borisov; +Cc: Qu Wenruo, linux-btrfs, fstests
On Thu, Jun 28, 2018 at 08:11:00AM +0300, Nikolay Borisov wrote:
>
>
> On 1.06.2018 04:34, Qu Wenruo wrote:
> > This is a long existing bug (from 2012) but exposed by a reporter
> > recently, that when compressed extent without data csum get written to
> > device-replace target device, the written data is in fact uncompressed data
> > other than the original compressed data.
> >
> > And since btrfs still consider the data is compressed and will try to read it
> > as compressed, it can cause read error.
> >
> > The root cause is located, and one RFC patch already sent to fix it,
> > titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
> > (The RFC is only for the extra possible way to fix the bug, the fix
> > itself should work without problem)
> >
> > Reported-by: James Harvey <jamespharvey20@gmail.com>
> > Signed-off-by: Qu Wenruo <wqu@suse.com>
>
> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
Thanks for the review! I assume the v3 patch also passes your review :)
Eryu
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device
2018-06-28 5:34 ` Eryu Guan
@ 2018-06-28 6:31 ` Nikolay Borisov
0 siblings, 0 replies; 8+ messages in thread
From: Nikolay Borisov @ 2018-06-28 6:31 UTC (permalink / raw)
To: Eryu Guan; +Cc: Qu Wenruo, linux-btrfs, fstests
On 28.06.2018 08:34, Eryu Guan wrote:
> On Thu, Jun 28, 2018 at 08:11:00AM +0300, Nikolay Borisov wrote:
>>
>>
>> On 1.06.2018 04:34, Qu Wenruo wrote:
>>> This is a long existing bug (from 2012) but exposed by a reporter
>>> recently, that when compressed extent without data csum get written to
>>> device-replace target device, the written data is in fact uncompressed data
>>> other than the original compressed data.
>>>
>>> And since btrfs still consider the data is compressed and will try to read it
>>> as compressed, it can cause read error.
>>>
>>> The root cause is located, and one RFC patch already sent to fix it,
>>> titled "[PATCH RFC] btrfs: scrub: Don't use inode pages for device replace".
>>> (The RFC is only for the extra possible way to fix the bug, the fix
>>> itself should work without problem)
>>>
>>> Reported-by: James Harvey <jamespharvey20@gmail.com>
>>> Signed-off-by: Qu Wenruo <wqu@suse.com>
>>
>> Reviewed-by: Nikolay Borisov <nborisov@suse.com>
>
> Thanks for the review! I assume the v3 patch also passes your review :)
Yes, I just saw that you requested an ack from a btrfs developer some
time ago and this test didn't move forward, hence i replied. But yes,
it's a valid test for btrfs.
>
> Eryu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at http://vger.kernel.org/majordomo-info.html
>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2018-06-28 6:31 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-06-01 1:34 [PATCH] fstests: btrfs: Test if btrfs will corrupt nodatasum compressed extent when replacing device Qu Wenruo
2018-06-05 10:42 ` Anand Jain
2018-06-05 11:39 ` Qu Wenruo
2018-06-07 6:21 ` Eryu Guan
2018-06-07 6:58 ` Qu Wenruo
2018-06-28 5:11 ` Nikolay Borisov
2018-06-28 5:34 ` Eryu Guan
2018-06-28 6:31 ` Nikolay Borisov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).