* [PATCH 1/2] btrfs: add verify chattr support for send/receive test
@ 2018-05-09 2:13 Howard McLauchlan
2018-05-09 2:13 ` [PATCH 2/2] xfstests: btrfs, test send's ability to punch holes and prealloc extents Howard McLauchlan
0 siblings, 1 reply; 2+ messages in thread
From: Howard McLauchlan @ 2018-05-09 2:13 UTC (permalink / raw)
To: linux-btrfs
Cc: Chris Mason, Josef Bacik, David Sterba, Filipe Manana,
Omar Sandoval, Howard McLauchlan
From: Howard McLauchlan <hmclauchlan@fb.com>
This test aims to verify correct behaviour with chattr operations and
btrfs send/receive. The intent is to check general correctness as well
as special interactions with troublesome flags(immutable, append only).
This test is motivated by a bug in btrfs which demonstrates a lack of
chattr support in btrfs send/receive.
A kernel patch to fix this can be found at:
btrfs: add chattr support for send/receive
The accompanying userspace patch can be found at:
btrfs-progs: add chattr support for send/receive
Signed-off-by: Howard McLauchlan <hmclauchlan@fb.com>
---
tests/btrfs/161 | 202 ++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/161.out | 38 +++++++++
tests/btrfs/group | 1 +
3 files changed, 241 insertions(+)
create mode 100755 tests/btrfs/161
create mode 100644 tests/btrfs/161.out
diff --git a/tests/btrfs/161 b/tests/btrfs/161
new file mode 100755
index 00000000..6c30a5e2
--- /dev/null
+++ b/tests/btrfs/161
@@ -0,0 +1,202 @@
+#! /bin/bash
+# FS QA Test 161
+#
+# This test verifies the correct behaviour of chattr support for btrfs
+# send/receive; 6 cases will be tested:
+# 1. New inode created with an inode flag set
+# 2. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
+# chattr -a
+# pwrite something
+# chattr +a
+# 3. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
+# chattr -a
+# pwrite something
+# chattr +d
+# 4. Existing inode is written to with sequence:
+# setfattr something
+# chattr +a
+# 5. Existing inode with BTRFS_INODE_APPEND is written to with sequence:
+# chattr -a
+# setfattr something
+# setfattr something else
+# chattr +a
+# 6. As above, but with pwrite instead of setfattr
+# The goal of 5 and 6 is not to test correctness, but to ensure we
+# don't send extra chattrs that are unnecessary.
+#
+# We verify the md5sum of the snapshots in the receive directory to ensure file
+# contents have changed appropriately. We also observe the flags changing (or
+# not changing) as appropriate.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2018 Facebook. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ cd /
+ rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+# Modify as appropriate.
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+
+rm -rf $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >>$seqres.full 2>&1
+_scratch_mount
+
+# Create receive directory
+mkdir $SCRATCH_MNT/receive
+
+# Create test file and set chattr flag
+_run_btrfs_util_prog subvolume create $SCRATCH_MNT/parent
+$XFS_IO_PROG -f -c "pwrite -S 0xaa 0K 32K" $SCRATCH_MNT/parent/foo | _filter_xfs_io
+$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
+
+# Send/Receive initial snapshot
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/old_parent
+_run_btrfs_util_prog send --stream-version 2 -f $SCRATCH_MNT/out $SCRATCH_MNT/old_parent
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify post-send content and flags
+echo "post-send file digest for old_parent:"
+md5sum $SCRATCH_MNT/old_parent/foo | _filter_scratch
+echo "post-send file flag for old_parent:"
+lsattr $SCRATCH_MNT/receive/old_parent/foo | _filter_scratch
+
+# Make change
+$CHATTR_PROG -a $SCRATCH_MNT/parent/foo
+$XFS_IO_PROG -f -c "pwrite -S 0xab 0K 32K" $SCRATCH_MNT/parent/foo | _filter_xfs_io
+$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
+
+# Send incremental change
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/child_1
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/old_parent -f $SCRATCH_MNT/out \
+$SCRATCH_MNT/child_1
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify
+echo "post-send file digest for child_1:"
+md5sum $SCRATCH_MNT/child_1/foo | _filter_scratch
+echo "post-send file flag for child_1:"
+lsattr $SCRATCH_MNT/receive/child_1/foo | _filter_scratch
+
+# Make change
+$CHATTR_PROG -a $SCRATCH_MNT/parent/foo
+$XFS_IO_PROG -f -c "pwrite -S 0xac 0K 32K" $SCRATCH_MNT/parent/foo | _filter_xfs_io
+$CHATTR_PROG +d $SCRATCH_MNT/parent/foo
+
+# Send incremental change
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/child_2
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/old_parent -f $SCRATCH_MNT/out \
+$SCRATCH_MNT/child_2
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify
+echo "post-send file digest for child_2:"
+md5sum $SCRATCH_MNT/child_2/foo | _filter_scratch
+echo "post-send file flag for child_2:"
+lsattr $SCRATCH_MNT/receive/child_2/foo | _filter_scratch
+
+# Send fattr, then chattr
+$SETFATTR_PROG -n user.test -v 0xaaaa $SCRATCH_MNT/parent/foo
+$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
+
+# Send incremental change against child_2
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/child_3
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/child_2 -f $SCRATCH_MNT/out \
+$SCRATCH_MNT/child_3
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify
+echo "post-send file flag for child_3:"
+lsattr $SCRATCH_MNT/receive/child_3/foo | _filter_scratch
+echo "post-send xattr for child_3:"
+$GETFATTR_PROG --absolute-names -d $SCRATCH_MNT/receive/child_3/foo | _filter_scratch
+
+# chattr, send fattr, then chattr
+$CHATTR_PROG -a $SCRATCH_MNT/parent/foo
+$SETFATTR_PROG -n user.test_1 -v 0xbbbb $SCRATCH_MNT/parent/foo
+$SETFATTR_PROG -n user.test -v 0xcccc $SCRATCH_MNT/parent/foo
+$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
+
+
+# Send incremental change
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/child_4
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/old_parent -f $SCRATCH_MNT/out \
+$SCRATCH_MNT/child_4
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify
+echo "post-send file flag for child_4:"
+lsattr $SCRATCH_MNT/receive/child_4/foo | _filter_scratch
+echo "post-send xattr for child_4:"
+$GETFATTR_PROG --absolute-names -d $SCRATCH_MNT/receive/child_4/foo | _filter_scratch
+
+# Technically we've hit this path, but to be sure...
+$CHATTR_PROG -a $SCRATCH_MNT/parent/foo
+$XFS_IO_PROG -f -c "pwrite -S 0xac 64k 128k" $SCRATCH_MNT/parent/foo | _filter_xfs_io
+$XFS_IO_PROG -f -c "pwrite -S 0xac 1024k 1056k" $SCRATCH_MNT/parent/foo | _filter_xfs_io
+$CHATTR_PROG +a $SCRATCH_MNT/parent/foo
+
+# Send incremental change
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT/parent \
+ $SCRATCH_MNT/child_5
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/old_parent -f $SCRATCH_MNT/out \
+$SCRATCH_MNT/child_5
+_run_btrfs_util_prog receive -f $SCRATCH_MNT/out $SCRATCH_MNT/receive
+
+# Verify
+echo "post-send file flag for child_5:"
+lsattr $SCRATCH_MNT/receive/child_5/foo | _filter_scratch
+
+# optional stuff if your test has verbose output to help resolve problems
+
+#echo "If failure, check $seqres.full (this) and $seqres.full.ok (reference)"
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/161.out b/tests/btrfs/161.out
new file mode 100644
index 00000000..945d6db5
--- /dev/null
+++ b/tests/btrfs/161.out
@@ -0,0 +1,38 @@
+QA output created by 161
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+post-send file digest for old_parent:
+c28418534a020122aca59fd3ff9581b5 SCRATCH_MNT/old_parent/foo
+post-send file flag for old_parent:
+-----a------------ SCRATCH_MNT/receive/old_parent/foo
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+post-send file digest for child_1:
+8e3ba8330fc5309ed7d2f447c13d9e71 SCRATCH_MNT/child_1/foo
+post-send file flag for child_1:
+-----a------------ SCRATCH_MNT/receive/child_1/foo
+wrote 32768/32768 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+post-send file digest for child_2:
+5671126b7326949ea0f14d5a875084ac SCRATCH_MNT/child_2/foo
+post-send file flag for child_2:
+------d----------- SCRATCH_MNT/receive/child_2/foo
+post-send file flag for child_3:
+-----ad----------- SCRATCH_MNT/receive/child_3/foo
+post-send xattr for child_3:
+# file: SCRATCH_MNT/receive/child_3/foo
+user.test=0sqqo=
+
+post-send file flag for child_4:
+-----ad----------- SCRATCH_MNT/receive/child_4/foo
+post-send xattr for child_4:
+# file: SCRATCH_MNT/receive/child_4/foo
+user.test=0szMw=
+user.test_1=0su7s=
+
+wrote 131072/131072 bytes at offset 65536
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1081344/1081344 bytes at offset 1048576
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+post-send file flag for child_5:
+-----ad----------- SCRATCH_MNT/receive/child_5/foo
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ba766f6b..939f1f7e 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -162,3 +162,4 @@
157 auto quick raid
158 auto quick raid scrub
159 auto quick
+161 auto quick send
--
2.17.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
* [PATCH 2/2] xfstests: btrfs, test send's ability to punch holes and prealloc extents
2018-05-09 2:13 [PATCH 1/2] btrfs: add verify chattr support for send/receive test Howard McLauchlan
@ 2018-05-09 2:13 ` Howard McLauchlan
0 siblings, 0 replies; 2+ messages in thread
From: Howard McLauchlan @ 2018-05-09 2:13 UTC (permalink / raw)
To: linux-btrfs
Cc: Chris Mason, Josef Bacik, David Sterba, Filipe Manana,
Omar Sandoval, Filipe Manana, Howard McLauchlan
From: Filipe Manana <fdmanana@gmail.com>
This test verifies that after an incremental btrfs send the
replicated file has the same exact hole and data structure as in
the origin filesystem. This didn't use to be the case before the
send stream version 2 - holes were sent as write operations of 0
valued bytes instead of punching holes with the fallocate system
call, and pre-allocated extents were sent as well as write
operations of 0 valued bytes instead of intructions for the
receiver to use the fallocate system call.
It also checks that prealloc extents that lie beyond the file's
size are replicated by an incremental send.
[Howard: rebased on kernel v4.17-rc4, btrfs progs v4.16.1]
Signed-off-by: Howard McLauchlan <linux@hmclauchlan.com>
Signed-off-by: Filipe David Borba Manana <fdmanana@gmail.com>
---
common/rc | 10 ++++
tests/btrfs/160 | 121 ++++++++++++++++++++++++++++++++++++++++++++
tests/btrfs/160.out | 42 +++++++++++++++
tests/btrfs/group | 1 +
4 files changed, 174 insertions(+)
create mode 100755 tests/btrfs/160
create mode 100644 tests/btrfs/160.out
diff --git a/common/rc b/common/rc
index 9ffab7fd..e2e9904a 100644
--- a/common/rc
+++ b/common/rc
@@ -3802,6 +3802,16 @@ _require_scratch_feature()
esac
}
+_require_btrfs_stream_version_2()
+{
+ $BTRFS_UTIL_PROG send 2>&1 | \
+ grep '^[ \t]*\--stream-version[ \t]\+.*' > /dev/null 2>&1
+ if [ $? -ne 0 ]; then
+ _notrun "Missing btrfs-progs send -a command line option, skipped this test"
+ fi
+}
+
+
init_rc
################################################################################
diff --git a/tests/btrfs/160 b/tests/btrfs/160
new file mode 100755
index 00000000..e4b7264c
--- /dev/null
+++ b/tests/btrfs/160
@@ -0,0 +1,121 @@
+#! /bin/bash
+# FS QA Test No. btrfs/160
+#
+# Verify that after an incremental btrfs send the replicated file has
+# the same exact hole and data structure as in the origin filesystem.
+# This didn't use to be the case before the send stream version 2 -
+# holes were sent as write operations of 0 valued bytes instead of punching
+# holes with the fallocate system call, and pre-allocated extents were sent
+# as well as write operations of 0 valued bytes instead of intructions for
+# the receiver to use the fallocate system call. Also check that prealloc
+# extents that lie beyond the file's size are replicated by an incremental
+# send.
+#
+# More specifically, this structure preserving guarantee was added by the
+# following linux kernel commits:
+#
+# Btrfs: send, use fallocate command to punch holes
+# Btrfs: send, use fallocate command to allocate extents
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2014 Filipe Manana. All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+tmp=/tmp/$$
+status=1 # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+ rm -fr $send_files_dir
+ rm -fr $tmp
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/punch
+
+# real QA test starts here
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_fssum
+_require_xfs_io_command "fiemap"
+_require_btrfs_stream_version_2
+
+send_files_dir=$TEST_DIR/btrfs-test-$seq
+
+rm -f $seqres.full
+rm -fr $send_files_dir
+mkdir $send_files_dir
+
+_scratch_mkfs >/dev/null 2>&1
+_scratch_mount
+
+$XFS_IO_PROG -f -c "pwrite -S 0x01 -b 300000 0 300000" $SCRATCH_MNT/foo \
+ | _filter_xfs_io
+
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap1
+
+$XFS_IO_PROG -c "fpunch 100000 50000" $SCRATCH_MNT/foo
+$XFS_IO_PROG -c "falloc 100000 50000" $SCRATCH_MNT/foo
+$XFS_IO_PROG -c "pwrite -S 0xff -b 1000 120000 1000" $SCRATCH_MNT/foo \
+ | _filter_xfs_io
+$XFS_IO_PROG -c "fpunch 250000 20000" $SCRATCH_MNT/foo
+
+$XFS_IO_PROG -c "falloc -k 300000 1000000" $SCRATCH_MNT/foo
+$XFS_IO_PROG -c "falloc -k 9000000 2000000" $SCRATCH_MNT/foo
+
+_run_btrfs_util_prog subvolume snapshot -r $SCRATCH_MNT $SCRATCH_MNT/mysnap2
+
+_run_btrfs_util_prog send --stream-version 2 -f $send_files_dir/1.snap \
+$SCRATCH_MNT/mysnap1
+_run_btrfs_util_prog send --stream-version 2 -p $SCRATCH_MNT/mysnap1 \
+-f $send_files_dir/2.snap $SCRATCH_MNT/mysnap2
+
+md5sum $SCRATCH_MNT/mysnap2/foo | _filter_scratch
+# List all hole and data segments.
+$XFS_IO_PROG -r -c "seek -r -a 0" $SCRATCH_MNT/mysnap2/foo
+# List all extents, we're interested here in prealloc extents that lie beyond
+# the file's size.
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/mysnap2/foo | _filter_fiemap
+
+_scratch_unmount
+_check_scratch_fs
+
+_scratch_mkfs >/dev/null 2>&1
+_scratch_mount
+
+_run_btrfs_util_prog receive -f $send_files_dir/1.snap $SCRATCH_MNT
+_run_btrfs_util_prog receive -f $send_files_dir/2.snap $SCRATCH_MNT
+
+md5sum $SCRATCH_MNT/mysnap2/foo | _filter_scratch
+# List all hole and data segments.
+$XFS_IO_PROG -r -c "seek -r -a 0" $SCRATCH_MNT/mysnap2/foo
+# List all extents, we're interested here in prealloc extents that lie beyond
+# the file's size.
+$XFS_IO_PROG -r -c "fiemap -v" $SCRATCH_MNT/mysnap2/foo | _filter_fiemap
+
+_check_scratch_fs
+
+status=0
+exit
diff --git a/tests/btrfs/160.out b/tests/btrfs/160.out
new file mode 100644
index 00000000..020f1356
--- /dev/null
+++ b/tests/btrfs/160.out
@@ -0,0 +1,42 @@
+QA output created by 160
+wrote 300000/300000 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 1000/1000 bytes at offset 120000
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+5a88aa0da62c8999c16515050947a70a SCRATCH_MNT/mysnap2/foo
+Whence Result
+DATA 0
+HOLE 102400
+DATA 118784
+HOLE 122880
+DATA 147456
+HOLE 253952
+DATA 266240
+HOLE 300000
+0: [0..199]: data
+1: [200..231]: unwritten
+2: [232..239]: data
+3: [240..287]: unwritten
+4: [288..495]: data
+5: [496..519]: hole
+6: [520..591]: data
+7: [592..2543]: unwritten
+8: [2544..17575]: hole
+9: [17576..21487]: unwritten
+5a88aa0da62c8999c16515050947a70a SCRATCH_MNT/mysnap2/foo
+Whence Result
+DATA 0
+HOLE 102400
+DATA 118784
+HOLE 122880
+DATA 147456
+HOLE 253952
+DATA 266240
+HOLE 300000
+0: [0..199]: data
+1: [200..231]: unwritten
+2: [232..239]: data
+3: [240..287]: unwritten
+4: [288..495]: data
+5: [496..519]: hole
+6: [520..591]: data
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 939f1f7e..28d06840 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -162,4 +162,5 @@
157 auto quick raid
158 auto quick raid scrub
159 auto quick
+160 auto quick send
161 auto quick send
--
2.17.0
^ permalink raw reply related [flat|nested] 2+ messages in thread
end of thread, other threads:[~2018-05-09 2:13 UTC | newest]
Thread overview: 2+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-05-09 2:13 [PATCH 1/2] btrfs: add verify chattr support for send/receive test Howard McLauchlan
2018-05-09 2:13 ` [PATCH 2/2] xfstests: btrfs, test send's ability to punch holes and prealloc extents Howard McLauchlan
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.