[PATCH 0/4] miscellaneous tests

[PATCH 0/4] miscellaneous tests

[Darrick J. Wong]

Miscellaneous test fixes, a new reflink test, and some more stress tests
for the online scrubber.

--D

[PATCH 1/4] xfs/348: classify this as a fuzzer test

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

xfs/348 is a fuzzer test since it calls xfs_db to break the scratch fs,
so put it in the fuzzers group.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/group |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/tests/xfs/group b/tests/xfs/group
index e2dfae2..eaa65b1 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -337,7 +337,7 @@
 345 auto quick clone
 346 auto quick clone
 347 auto quick clone
-348 auto quick repair
+348 auto quick fuzzers repair
 349 dangerous_fuzzers scrub
 350 dangerous_fuzzers dangerous_scrub dangerous_repair
 351 dangerous_fuzzers dangerous_scrub dangerous_online_repair

Re: [PATCH 1/4] xfs/348: classify this as a fuzzer test

[Christoph Hellwig]

Looks fine,

Reviewed-by: Christoph Hellwig <hch@lst.de>

[PATCH 2/4] xfs: minor fixes to the fsmap tests

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Refactor the fsmap tests slightly to use new helpers, and
fix some minor output scanning issues.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/273 |    3 +--
 tests/xfs/275 |    4 +---
 tests/xfs/276 |    2 +-
 3 files changed, 3 insertions(+), 6 deletions(-)


diff --git a/tests/xfs/273 b/tests/xfs/273
index 1a4ee93..f784a35 100755
--- a/tests/xfs/273
+++ b/tests/xfs/273
@@ -50,8 +50,7 @@ _require_xfs_io_command "fsmap"
 rm -f "$seqres.full"
 
 echo "Format and mount"
-_scratch_mkfs > "$seqres.full" 2>&1
-_scratch_populate >> $seqres.full
+_scratch_populate_cached nofill > $seqres.full 2>&1
 
 echo "Compare fsmap" | tee -a $seqres.full
 _scratch_mount
diff --git a/tests/xfs/275 b/tests/xfs/275
index cc6d353..3ccfd77 100755
--- a/tests/xfs/275
+++ b/tests/xfs/275
@@ -42,11 +42,9 @@ _cleanup()
 # real QA test starts here
 _supported_os Linux
 _supported_fs xfs
+_require_logdev
 _require_scratch
 _require_xfs_io_command "fsmap"
-if [ "$USE_EXTERNAL" != "yes" ] || [ -z "$SCRATCH_LOGDEV" ]; then
-	_notrun "Need external log device"
-fi
 
 rm -f "$seqres.full"
 
diff --git a/tests/xfs/276 b/tests/xfs/276
index cea91c9..e39b5d5 100755
--- a/tests/xfs/276
+++ b/tests/xfs/276
@@ -77,7 +77,7 @@ done
 
 echo "Check device field of FS metadata and realtime file"
 data_dev=$(grep 'static fs metadata' $TEST_DIR/fsmap | head -n 1 | awk '{print $2}')
-rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}')
+rt_dev=$(grep "${ino}[[:space:]]*[0-9]*\.\.[0-9]*[[:space:]]*[0-9]*$" $TEST_DIR/fsmap | head -n 1 | awk '{print $2}')
 test "${data_dev}" != "${rt_dev}" || echo "data ${data_dev} realtime ${rt_dev}?"
 
 # success, all done

[PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

If we reflink a file with N blocks to another file one block at a time,
does the destination file end up with the same number of extents as the
source file?  In other words, does the filesystem succeed at combining
adjacent mappings into a maximal extents?

Signed-off-by: Darrick J. Wong <djwong@djwong.org>
---
 tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/930.out |   11 +++++
 tests/generic/group   |    1 
 3 files changed, 118 insertions(+)
 create mode 100755 tests/generic/930
 create mode 100644 tests/generic/930.out


diff --git a/tests/generic/930 b/tests/generic/930
new file mode 100755
index 0000000..15d8cbf
--- /dev/null
+++ b/tests/generic/930
@@ -0,0 +1,106 @@
+#! /bin/bash
+# FS QA Test No. 930
+#
+# Check that reflinking adjacent blocks in a file produces a single
+# block mapping extent.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+	wait
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs generic
+_require_scratch_reflink
+_require_fiemap
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+blocks=32
+blksz=65536
+sz=$((blocks * blksz))
+
+echo "Create the original files"
+$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
+_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
+seq 0 $blksz $((sz - blksz)) | while read offset; do
+	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
+done
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+
+echo "Check extent counts"
+f1=$(_count_extents $testdir/file1)
+f2=$(_count_extents $testdir/file2)
+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
+
+# Did the fs combine the extent mappings when we made f2?
+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
+
+_scratch_cycle_mount
+
+echo "Compare files after remounting"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+
+echo "Check extent counts"
+f1=$(_count_extents $testdir/file1)
+f2=$(_count_extents $testdir/file2)
+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
+
+# Are the mappings still combined?
+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/930.out b/tests/generic/930.out
new file mode 100644
index 0000000..556108a
--- /dev/null
+++ b/tests/generic/930.out
@@ -0,0 +1,11 @@
+QA output created by 930
+Format and mount
+Create the original files
+Compare files
+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
+Check extent counts
+Compare files after remounting
+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
+Check extent counts
diff --git a/tests/generic/group b/tests/generic/group
index d0bc47d..4b4df66 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -411,3 +411,4 @@
 406 auto quick dangerous
 407 auto quick clone metadata
 408 auto quick clone dedupe metadata
+930 auto quick clone

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Eryu Guan]

On Fri, Feb 24, 2017 at 05:12:57PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> If we reflink a file with N blocks to another file one block at a time,
> does the destination file end up with the same number of extents as the
> source file?  In other words, does the filesystem succeed at combining
> adjacent mappings into a maximal extents?

I'm not sure if this is a standard behavior and applies to btrfs too?
But btrfs is failing this test now:

+f1 (1) != f2 (32)
+s1 (1) != s2 (32)

Fix test or btrfs? I'm taking it if btrfs is the one to be fixed :)

Thanks,
Eryu

> 
> Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> ---
>  tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/930.out |   11 +++++
>  tests/generic/group   |    1 
>  3 files changed, 118 insertions(+)
>  create mode 100755 tests/generic/930
>  create mode 100644 tests/generic/930.out
> 
> 
> diff --git a/tests/generic/930 b/tests/generic/930
> new file mode 100755
> index 0000000..15d8cbf
> --- /dev/null
> +++ b/tests/generic/930
> @@ -0,0 +1,106 @@
> +#! /bin/bash
> +# FS QA Test No. 930
> +#
> +# Check that reflinking adjacent blocks in a file produces a single
> +# block mapping extent.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +	wait
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs generic
> +_require_scratch_reflink
> +_require_fiemap
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +blocks=32
> +blksz=65536
> +sz=$((blocks * blksz))
> +
> +echo "Create the original files"
> +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> +seq 0 $blksz $((sz - blksz)) | while read offset; do
> +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> +done
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Did the fs combine the extent mappings when we made f2?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +_scratch_cycle_mount
> +
> +echo "Compare files after remounting"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Are the mappings still combined?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/930.out b/tests/generic/930.out
> new file mode 100644
> index 0000000..556108a
> --- /dev/null
> +++ b/tests/generic/930.out
> @@ -0,0 +1,11 @@
> +QA output created by 930
> +Format and mount
> +Create the original files
> +Compare files
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> +Compare files after remounting
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> diff --git a/tests/generic/group b/tests/generic/group
> index d0bc47d..4b4df66 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -411,3 +411,4 @@
>  406 auto quick dangerous
>  407 auto quick clone metadata
>  408 auto quick clone dedupe metadata
> +930 auto quick clone
> 

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

On Tue, Feb 28, 2017 at 04:15:02PM +0800, Eryu Guan wrote:
> On Fri, Feb 24, 2017 at 05:12:57PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > If we reflink a file with N blocks to another file one block at a time,
> > does the destination file end up with the same number of extents as the
> > source file?  In other words, does the filesystem succeed at combining
> > adjacent mappings into a maximal extents?
> 
> I'm not sure if this is a standard behavior and applies to btrfs too?
> But btrfs is failing this test now:
> 
> +f1 (1) != f2 (32)
> +s1 (1) != s2 (32)
> 
> Fix test or btrfs? I'm taking it if btrfs is the one to be fixed :)

btrfs has that weird behavior where it doesn't merge the adjacent
extents at all (at least not according to FIEMAP) until you remount the
filesystem.  After the remount it's fine, but... WTF? :)

So yes, the test is working as designed.  btrfs needs fixing, or I guess
worst case we can _notrun it on btrfs.

Snark aside, it was intended originally to make sure that XFS is
properly merging the extent records together; then it occurred to me to
rewrite it with fiemap and make it one of the generic reflink tests so
that ocfs2 can get tested too.

--D

> 
> Thanks,
> Eryu
> 
> > 
> > Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> > ---
> >  tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/930.out |   11 +++++
> >  tests/generic/group   |    1 
> >  3 files changed, 118 insertions(+)
> >  create mode 100755 tests/generic/930
> >  create mode 100644 tests/generic/930.out
> > 
> > 
> > diff --git a/tests/generic/930 b/tests/generic/930
> > new file mode 100755
> > index 0000000..15d8cbf
> > --- /dev/null
> > +++ b/tests/generic/930
> > @@ -0,0 +1,106 @@
> > +#! /bin/bash
> > +# FS QA Test No. 930
> > +#
> > +# Check that reflinking adjacent blocks in a file produces a single
> > +# block mapping extent.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.*
> > +	wait
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +. ./common/reflink
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs generic
> > +_require_scratch_reflink
> > +_require_fiemap
> > +
> > +echo "Format and mount"
> > +_scratch_mkfs > $seqres.full 2>&1
> > +_scratch_mount >> $seqres.full 2>&1
> > +
> > +testdir=$SCRATCH_MNT/test-$seq
> > +mkdir $testdir
> > +
> > +blocks=32
> > +blksz=65536
> > +sz=$((blocks * blksz))
> > +
> > +echo "Create the original files"
> > +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> > +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> > +seq 0 $blksz $((sz - blksz)) | while read offset; do
> > +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> > +done
> > +
> > +echo "Compare files"
> > +md5sum $testdir/file1 | _filter_scratch
> > +md5sum $testdir/file2 | _filter_scratch
> > +
> > +echo "Check extent counts"
> > +f1=$(_count_extents $testdir/file1)
> > +f2=$(_count_extents $testdir/file2)
> > +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> > +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> > +
> > +# Did the fs combine the extent mappings when we made f2?
> > +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> > +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> > +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> > +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> > +
> > +_scratch_cycle_mount
> > +
> > +echo "Compare files after remounting"
> > +md5sum $testdir/file1 | _filter_scratch
> > +md5sum $testdir/file2 | _filter_scratch
> > +
> > +echo "Check extent counts"
> > +f1=$(_count_extents $testdir/file1)
> > +f2=$(_count_extents $testdir/file2)
> > +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> > +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> > +
> > +# Are the mappings still combined?
> > +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> > +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> > +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> > +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/generic/930.out b/tests/generic/930.out
> > new file mode 100644
> > index 0000000..556108a
> > --- /dev/null
> > +++ b/tests/generic/930.out
> > @@ -0,0 +1,11 @@
> > +QA output created by 930
> > +Format and mount
> > +Create the original files
> > +Compare files
> > +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> > +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> > +Check extent counts
> > +Compare files after remounting
> > +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> > +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> > +Check extent counts
> > diff --git a/tests/generic/group b/tests/generic/group
> > index d0bc47d..4b4df66 100644
> > --- a/tests/generic/group
> > +++ b/tests/generic/group
> > @@ -411,3 +411,4 @@
> >  406 auto quick dangerous
> >  407 auto quick clone metadata
> >  408 auto quick clone dedupe metadata
> > +930 auto quick clone
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Eryu Guan]

On Tue, Feb 28, 2017 at 08:31:50AM -0800, Darrick J. Wong wrote:
> On Tue, Feb 28, 2017 at 04:15:02PM +0800, Eryu Guan wrote:
> > On Fri, Feb 24, 2017 at 05:12:57PM -0800, Darrick J. Wong wrote:
> > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > > 
> > > If we reflink a file with N blocks to another file one block at a time,
> > > does the destination file end up with the same number of extents as the
> > > source file?  In other words, does the filesystem succeed at combining
> > > adjacent mappings into a maximal extents?
> > 
> > I'm not sure if this is a standard behavior and applies to btrfs too?
> > But btrfs is failing this test now:
> > 
> > +f1 (1) != f2 (32)
> > +s1 (1) != s2 (32)
> > 
> > Fix test or btrfs? I'm taking it if btrfs is the one to be fixed :)
> 
> btrfs has that weird behavior where it doesn't merge the adjacent
> extents at all (at least not according to FIEMAP) until you remount the
> filesystem.  After the remount it's fine, but... WTF? :)
> 
> So yes, the test is working as designed.  btrfs needs fixing, or I guess
> worst case we can _notrun it on btrfs.

Thanks for the explanation! I'll take it as it is at this moment then.

> 
> Snark aside, it was intended originally to make sure that XFS is
> properly merging the extent records together; then it occurred to me to
> rewrite it with fiemap and make it one of the generic reflink tests so
> that ocfs2 can get tested too.

Ah, that reminds me that I need to do testing with ocfs2 too, and test
passed :)

Thanks,
Eryu

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Qu Wenruo]

Hi Darrick

At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> If we reflink a file with N blocks to another file one block at a time,
> does the destination file end up with the same number of extents as the
> source file?  In other words, does the filesystem succeed at combining
> adjacent mappings into a maximal extents?

Btrfs can't pass the test since it doesn't merge such mapping, not only 
the fiemap result, but also on-disk extents.

IIRC ocfs also merges it, so I wonder if this is a documented behavior 
to merge any extent maps that are adjacent.

Further more, if a fs has a limit on extent size on disk, e.g 128M.
But 2 file extents are also adjacent on disk, then should fiemap return 
a large 256M map or just 2 adjacent 128M maps?

Thanks,
Qu
>
> Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> ---
>  tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/930.out |   11 +++++
>  tests/generic/group   |    1
>  3 files changed, 118 insertions(+)
>  create mode 100755 tests/generic/930
>  create mode 100644 tests/generic/930.out
>
>
> diff --git a/tests/generic/930 b/tests/generic/930
> new file mode 100755
> index 0000000..15d8cbf
> --- /dev/null
> +++ b/tests/generic/930
> @@ -0,0 +1,106 @@
> +#! /bin/bash
> +# FS QA Test No. 930
> +#
> +# Check that reflinking adjacent blocks in a file produces a single
> +# block mapping extent.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +	wait
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs generic
> +_require_scratch_reflink
> +_require_fiemap
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +blocks=32
> +blksz=65536
> +sz=$((blocks * blksz))
> +
> +echo "Create the original files"
> +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> +seq 0 $blksz $((sz - blksz)) | while read offset; do
> +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> +done
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Did the fs combine the extent mappings when we made f2?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +_scratch_cycle_mount
> +
> +echo "Compare files after remounting"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Are the mappings still combined?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/930.out b/tests/generic/930.out
> new file mode 100644
> index 0000000..556108a
> --- /dev/null
> +++ b/tests/generic/930.out
> @@ -0,0 +1,11 @@
> +QA output created by 930
> +Format and mount
> +Create the original files
> +Compare files
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> +Compare files after remounting
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> diff --git a/tests/generic/group b/tests/generic/group
> index d0bc47d..4b4df66 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -411,3 +411,4 @@
>  406 auto quick dangerous
>  407 auto quick clone metadata
>  408 auto quick clone dedupe metadata
> +930 auto quick clone
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwICaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=zLZW67c3IwvZjBGr4cIg8v5ywQZq5TdY4rYizcb5vLI&s=N2J_WqHQvVybd3Oz82W7gXLb3rC204UQHpktWS1e0Tc&e= 
>
>


--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwICaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=zLZW67c3IwvZjBGr4cIg8v5ywQZq5TdY4rYizcb5vLI&s=N2J_WqHQvVybd3Oz82W7gXLb3rC204UQHpktWS1e0Tc&e= 

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Qu Wenruo]

Hi Darrick

At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
>
> If we reflink a file with N blocks to another file one block at a time,
> does the destination file end up with the same number of extents as the
> source file?  In other words, does the filesystem succeed at combining
> adjacent mappings into a maximal extents?

Btrfs can't pass the test since it doesn't merge such mapping, not only 
the fiemap result, but also on-disk extents.

IIRC ocfs also merges it, so I wonder if this is a documented behavior 
to merge any extent maps that are adjacent.

Further more, if a fs has a limit on extent size on disk, e.g 128M.
But 2 file extents are also adjacent on disk, then should fiemap return 
a large 256M map or just 2 adjacent 128M maps?

Thanks,
Qu
>
> Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> ---
>  tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/930.out |   11 +++++
>  tests/generic/group   |    1
>  3 files changed, 118 insertions(+)
>  create mode 100755 tests/generic/930
>  create mode 100644 tests/generic/930.out
>
>
> diff --git a/tests/generic/930 b/tests/generic/930
> new file mode 100755
> index 0000000..15d8cbf
> --- /dev/null
> +++ b/tests/generic/930
> @@ -0,0 +1,106 @@
> +#! /bin/bash
> +# FS QA Test No. 930
> +#
> +# Check that reflinking adjacent blocks in a file produces a single
> +# block mapping extent.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +	wait
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs generic
> +_require_scratch_reflink
> +_require_fiemap
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +blocks=32
> +blksz=65536
> +sz=$((blocks * blksz))
> +
> +echo "Create the original files"
> +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> +seq 0 $blksz $((sz - blksz)) | while read offset; do
> +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> +done
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Did the fs combine the extent mappings when we made f2?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +_scratch_cycle_mount
> +
> +echo "Compare files after remounting"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file2 | _filter_scratch
> +
> +echo "Check extent counts"
> +f1=$(_count_extents $testdir/file1)
> +f2=$(_count_extents $testdir/file2)
> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> +
> +# Are the mappings still combined?
> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/930.out b/tests/generic/930.out
> new file mode 100644
> index 0000000..556108a
> --- /dev/null
> +++ b/tests/generic/930.out
> @@ -0,0 +1,11 @@
> +QA output created by 930
> +Format and mount
> +Create the original files
> +Compare files
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> +Compare files after remounting
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> +Check extent counts
> diff --git a/tests/generic/group b/tests/generic/group
> index d0bc47d..4b4df66 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -411,3 +411,4 @@
>  406 auto quick dangerous
>  407 auto quick clone metadata
>  408 auto quick clone dedupe metadata
> +930 auto quick clone
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
> Hi Darrick
> 
> At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> >From: Darrick J. Wong <darrick.wong@oracle.com>
> >
> >If we reflink a file with N blocks to another file one block at a time,
> >does the destination file end up with the same number of extents as the
> >source file?  In other words, does the filesystem succeed at combining
> >adjacent mappings into a maximal extents?
> 
> Btrfs can't pass the test since it doesn't merge such mapping, not only the
> fiemap result, but also on-disk extents.

Weirdly, I observed that if you unmount and remount the btrfs it
actually /will/ merge the extents.

> IIRC ocfs also merges it, so I wonder if this is a documented behavior to
> merge any extent maps that are adjacent.

Generally, yes you'd expect that two logically and physically contiguous
extents to be merged together.  That might change a bit for compressed
or encrypted extents.

> Further more, if a fs has a limit on extent size on disk, e.g 128M.
> But 2 file extents are also adjacent on disk, then should fiemap return a
> large 256M map or just 2 adjacent 128M maps?

It's up to the fs, really... XFS will merge the extent records (up to
maximum extent size) and then unmerge the shared/not-shared FIEMAP
records for accurate shared extent reporting.

--D

> 
> Thanks,
> Qu
> >
> >Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> >---
> > tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
> > tests/generic/930.out |   11 +++++
> > tests/generic/group   |    1
> > 3 files changed, 118 insertions(+)
> > create mode 100755 tests/generic/930
> > create mode 100644 tests/generic/930.out
> >
> >
> >diff --git a/tests/generic/930 b/tests/generic/930
> >new file mode 100755
> >index 0000000..15d8cbf
> >--- /dev/null
> >+++ b/tests/generic/930
> >@@ -0,0 +1,106 @@
> >+#! /bin/bash
> >+# FS QA Test No. 930
> >+#
> >+# Check that reflinking adjacent blocks in a file produces a single
> >+# block mapping extent.
> >+#
> >+#-----------------------------------------------------------------------
> >+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> >+#
> >+# This program is free software; you can redistribute it and/or
> >+# modify it under the terms of the GNU General Public License as
> >+# published by the Free Software Foundation.
> >+#
> >+# This program is distributed in the hope that it would be useful,
> >+# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >+# GNU General Public License for more details.
> >+#
> >+# You should have received a copy of the GNU General Public License
> >+# along with this program; if not, write the Free Software Foundation,
> >+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> >+#-----------------------------------------------------------------------
> >+#
> >+
> >+seq=`basename $0`
> >+seqres=$RESULT_DIR/$seq
> >+echo "QA output created by $seq"
> >+
> >+here=`pwd`
> >+tmp=/tmp/$$
> >+status=1	# failure is the default!
> >+trap "_cleanup; exit \$status" 0 1 2 3 7 15
> >+
> >+_cleanup()
> >+{
> >+	cd /
> >+	rm -rf $tmp.*
> >+	wait
> >+}
> >+
> >+# get standard environment, filters and checks
> >+. ./common/rc
> >+. ./common/filter
> >+. ./common/reflink
> >+
> >+# real QA test starts here
> >+_supported_os Linux
> >+_supported_fs generic
> >+_require_scratch_reflink
> >+_require_fiemap
> >+
> >+echo "Format and mount"
> >+_scratch_mkfs > $seqres.full 2>&1
> >+_scratch_mount >> $seqres.full 2>&1
> >+
> >+testdir=$SCRATCH_MNT/test-$seq
> >+mkdir $testdir
> >+
> >+blocks=32
> >+blksz=65536
> >+sz=$((blocks * blksz))
> >+
> >+echo "Create the original files"
> >+$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> >+_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> >+seq 0 $blksz $((sz - blksz)) | while read offset; do
> >+	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> >+done
> >+
> >+echo "Compare files"
> >+md5sum $testdir/file1 | _filter_scratch
> >+md5sum $testdir/file2 | _filter_scratch
> >+
> >+echo "Check extent counts"
> >+f1=$(_count_extents $testdir/file1)
> >+f2=$(_count_extents $testdir/file2)
> >+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+
> >+# Did the fs combine the extent mappings when we made f2?
> >+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >+
> >+_scratch_cycle_mount
> >+
> >+echo "Compare files after remounting"
> >+md5sum $testdir/file1 | _filter_scratch
> >+md5sum $testdir/file2 | _filter_scratch
> >+
> >+echo "Check extent counts"
> >+f1=$(_count_extents $testdir/file1)
> >+f2=$(_count_extents $testdir/file2)
> >+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+
> >+# Are the mappings still combined?
> >+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >+
> >+# success, all done
> >+status=0
> >+exit
> >diff --git a/tests/generic/930.out b/tests/generic/930.out
> >new file mode 100644
> >index 0000000..556108a
> >--- /dev/null
> >+++ b/tests/generic/930.out
> >@@ -0,0 +1,11 @@
> >+QA output created by 930
> >+Format and mount
> >+Create the original files
> >+Compare files
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >+Check extent counts
> >+Compare files after remounting
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >+Check extent counts
> >diff --git a/tests/generic/group b/tests/generic/group
> >index d0bc47d..4b4df66 100644
> >--- a/tests/generic/group
> >+++ b/tests/generic/group
> >@@ -411,3 +411,4 @@
> > 406 auto quick dangerous
> > 407 auto quick clone metadata
> > 408 auto quick clone dedupe metadata
> >+930 auto quick clone
> >
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=ji4pI6Gjs_DSzSPcr-Q-vhH1VbaXdHyL5tPzJN-yBpM&s=TJXD12aledcnxTXpOf-WLJ3bxvj-aDDTXUm7mpaUs1U&e= 
> >
> >
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=ji4pI6Gjs_DSzSPcr-Q-vhH1VbaXdHyL5tPzJN-yBpM&s=TJXD12aledcnxTXpOf-WLJ3bxvj-aDDTXUm7mpaUs1U&e= 
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=ji4pI6Gjs_DSzSPcr-Q-vhH1VbaXdHyL5tPzJN-yBpM&s=TJXD12aledcnxTXpOf-WLJ3bxvj-aDDTXUm7mpaUs1U&e= 

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
> Hi Darrick
> 
> At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> >From: Darrick J. Wong <darrick.wong@oracle.com>
> >
> >If we reflink a file with N blocks to another file one block at a time,
> >does the destination file end up with the same number of extents as the
> >source file?  In other words, does the filesystem succeed at combining
> >adjacent mappings into a maximal extents?
> 
> Btrfs can't pass the test since it doesn't merge such mapping, not only the
> fiemap result, but also on-disk extents.

Weirdly, I observed that if you unmount and remount the btrfs it
actually /will/ merge the extents.

> IIRC ocfs also merges it, so I wonder if this is a documented behavior to
> merge any extent maps that are adjacent.

Generally, yes you'd expect that two logically and physically contiguous
extents to be merged together.  That might change a bit for compressed
or encrypted extents.

> Further more, if a fs has a limit on extent size on disk, e.g 128M.
> But 2 file extents are also adjacent on disk, then should fiemap return a
> large 256M map or just 2 adjacent 128M maps?

It's up to the fs, really... XFS will merge the extent records (up to
maximum extent size) and then unmerge the shared/not-shared FIEMAP
records for accurate shared extent reporting.

--D

> 
> Thanks,
> Qu
> >
> >Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> >---
> > tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
> > tests/generic/930.out |   11 +++++
> > tests/generic/group   |    1
> > 3 files changed, 118 insertions(+)
> > create mode 100755 tests/generic/930
> > create mode 100644 tests/generic/930.out
> >
> >
> >diff --git a/tests/generic/930 b/tests/generic/930
> >new file mode 100755
> >index 0000000..15d8cbf
> >--- /dev/null
> >+++ b/tests/generic/930
> >@@ -0,0 +1,106 @@
> >+#! /bin/bash
> >+# FS QA Test No. 930
> >+#
> >+# Check that reflinking adjacent blocks in a file produces a single
> >+# block mapping extent.
> >+#
> >+#-----------------------------------------------------------------------
> >+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> >+#
> >+# This program is free software; you can redistribute it and/or
> >+# modify it under the terms of the GNU General Public License as
> >+# published by the Free Software Foundation.
> >+#
> >+# This program is distributed in the hope that it would be useful,
> >+# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >+# GNU General Public License for more details.
> >+#
> >+# You should have received a copy of the GNU General Public License
> >+# along with this program; if not, write the Free Software Foundation,
> >+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> >+#-----------------------------------------------------------------------
> >+#
> >+
> >+seq=`basename $0`
> >+seqres=$RESULT_DIR/$seq
> >+echo "QA output created by $seq"
> >+
> >+here=`pwd`
> >+tmp=/tmp/$$
> >+status=1	# failure is the default!
> >+trap "_cleanup; exit \$status" 0 1 2 3 7 15
> >+
> >+_cleanup()
> >+{
> >+	cd /
> >+	rm -rf $tmp.*
> >+	wait
> >+}
> >+
> >+# get standard environment, filters and checks
> >+. ./common/rc
> >+. ./common/filter
> >+. ./common/reflink
> >+
> >+# real QA test starts here
> >+_supported_os Linux
> >+_supported_fs generic
> >+_require_scratch_reflink
> >+_require_fiemap
> >+
> >+echo "Format and mount"
> >+_scratch_mkfs > $seqres.full 2>&1
> >+_scratch_mount >> $seqres.full 2>&1
> >+
> >+testdir=$SCRATCH_MNT/test-$seq
> >+mkdir $testdir
> >+
> >+blocks=32
> >+blksz=65536
> >+sz=$((blocks * blksz))
> >+
> >+echo "Create the original files"
> >+$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> >+_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> >+seq 0 $blksz $((sz - blksz)) | while read offset; do
> >+	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> >+done
> >+
> >+echo "Compare files"
> >+md5sum $testdir/file1 | _filter_scratch
> >+md5sum $testdir/file2 | _filter_scratch
> >+
> >+echo "Check extent counts"
> >+f1=$(_count_extents $testdir/file1)
> >+f2=$(_count_extents $testdir/file2)
> >+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+
> >+# Did the fs combine the extent mappings when we made f2?
> >+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >+
> >+_scratch_cycle_mount
> >+
> >+echo "Compare files after remounting"
> >+md5sum $testdir/file1 | _filter_scratch
> >+md5sum $testdir/file2 | _filter_scratch
> >+
> >+echo "Check extent counts"
> >+f1=$(_count_extents $testdir/file1)
> >+f2=$(_count_extents $testdir/file2)
> >+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >+
> >+# Are the mappings still combined?
> >+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >+
> >+# success, all done
> >+status=0
> >+exit
> >diff --git a/tests/generic/930.out b/tests/generic/930.out
> >new file mode 100644
> >index 0000000..556108a
> >--- /dev/null
> >+++ b/tests/generic/930.out
> >@@ -0,0 +1,11 @@
> >+QA output created by 930
> >+Format and mount
> >+Create the original files
> >+Compare files
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >+Check extent counts
> >+Compare files after remounting
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >+Check extent counts
> >diff --git a/tests/generic/group b/tests/generic/group
> >index d0bc47d..4b4df66 100644
> >--- a/tests/generic/group
> >+++ b/tests/generic/group
> >@@ -411,3 +411,4 @@
> > 406 auto quick dangerous
> > 407 auto quick clone metadata
> > 408 auto quick clone dedupe metadata
> >+930 auto quick clone
> >
> >--
> >To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >the body of a message to majordomo@vger.kernel.org
> >More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> 
> 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Qu Wenruo]

At 03/21/2017 10:23 PM, Darrick J. Wong wrote:
> On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
>> Hi Darrick
>>
>> At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
>>> From: Darrick J. Wong <darrick.wong@oracle.com>
>>>
>>> If we reflink a file with N blocks to another file one block at a time,
>>> does the destination file end up with the same number of extents as the
>>> source file?  In other words, does the filesystem succeed at combining
>>> adjacent mappings into a maximal extents?
>>
>> Btrfs can't pass the test since it doesn't merge such mapping, not only the
>> fiemap result, but also on-disk extents.
>
> Weirdly, I observed that if you unmount and remount the btrfs it
> actually /will/ merge the extents.

I tried that, but it's even more weird, after remount, fiemap still 
return split ones.
But if call fiemap again after about 1 second, it reports one single 
large extent.
(if calling 2nd fiemap just after 1st one, 2nd one will still return 
split extent map)

While on disk, the extents are split indeed.

So it's a btrfs bug then.

>
>> IIRC ocfs also merges it, so I wonder if this is a documented behavior to
>> merge any extent maps that are adjacent.
>
> Generally, yes you'd expect that two logically and physically contiguous
> extents to be merged together.  That might change a bit for compressed
> or encrypted extents.
>
>> Further more, if a fs has a limit on extent size on disk, e.g 128M.
>> But 2 file extents are also adjacent on disk, then should fiemap return a
>> large 256M map or just 2 adjacent 128M maps?
>
> It's up to the fs, really... XFS will merge the extent records (up to
> maximum extent size) and then unmerge the shared/not-shared FIEMAP
> records for accurate shared extent reporting.

BTW, are XFS extents on disk are merged when doing reflink, or just 
merge the extent record at fiemap time?

Further more, is there any tool like btrfs-debug-tree to show *every* 
btree block and its content data?
I am very interested in XFS on-disk and runtime implementation.

Thanks,
Qu
>
> --D
>
>>
>> Thanks,
>> Qu
>>>
>>> Signed-off-by: Darrick J. Wong <djwong@djwong.org>
>>> ---
>>> tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
>>> tests/generic/930.out |   11 +++++
>>> tests/generic/group   |    1
>>> 3 files changed, 118 insertions(+)
>>> create mode 100755 tests/generic/930
>>> create mode 100644 tests/generic/930.out
>>>
>>>
>>> diff --git a/tests/generic/930 b/tests/generic/930
>>> new file mode 100755
>>> index 0000000..15d8cbf
>>> --- /dev/null
>>> +++ b/tests/generic/930
>>> @@ -0,0 +1,106 @@
>>> +#! /bin/bash
>>> +# FS QA Test No. 930
>>> +#
>>> +# Check that reflinking adjacent blocks in a file produces a single
>>> +# block mapping extent.
>>> +#
>>> +#-----------------------------------------------------------------------
>>> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
>>> +#
>>> +# This program is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU General Public License as
>>> +# published by the Free Software Foundation.
>>> +#
>>> +# This program is distributed in the hope that it would be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program; if not, write the Free Software Foundation,
>>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>>> +#-----------------------------------------------------------------------
>>> +#
>>> +
>>> +seq=`basename $0`
>>> +seqres=$RESULT_DIR/$seq
>>> +echo "QA output created by $seq"
>>> +
>>> +here=`pwd`
>>> +tmp=/tmp/$$
>>> +status=1	# failure is the default!
>>> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
>>> +
>>> +_cleanup()
>>> +{
>>> +	cd /
>>> +	rm -rf $tmp.*
>>> +	wait
>>> +}
>>> +
>>> +# get standard environment, filters and checks
>>> +. ./common/rc
>>> +. ./common/filter
>>> +. ./common/reflink
>>> +
>>> +# real QA test starts here
>>> +_supported_os Linux
>>> +_supported_fs generic
>>> +_require_scratch_reflink
>>> +_require_fiemap
>>> +
>>> +echo "Format and mount"
>>> +_scratch_mkfs > $seqres.full 2>&1
>>> +_scratch_mount >> $seqres.full 2>&1
>>> +
>>> +testdir=$SCRATCH_MNT/test-$seq
>>> +mkdir $testdir
>>> +
>>> +blocks=32
>>> +blksz=65536
>>> +sz=$((blocks * blksz))
>>> +
>>> +echo "Create the original files"
>>> +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
>>> +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
>>> +seq 0 $blksz $((sz - blksz)) | while read offset; do
>>> +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
>>> +done
>>> +
>>> +echo "Compare files"
>>> +md5sum $testdir/file1 | _filter_scratch
>>> +md5sum $testdir/file2 | _filter_scratch
>>> +
>>> +echo "Check extent counts"
>>> +f1=$(_count_extents $testdir/file1)
>>> +f2=$(_count_extents $testdir/file2)
>>> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +
>>> +# Did the fs combine the extent mappings when we made f2?
>>> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
>>> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
>>> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
>>> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
>>> +
>>> +_scratch_cycle_mount
>>> +
>>> +echo "Compare files after remounting"
>>> +md5sum $testdir/file1 | _filter_scratch
>>> +md5sum $testdir/file2 | _filter_scratch
>>> +
>>> +echo "Check extent counts"
>>> +f1=$(_count_extents $testdir/file1)
>>> +f2=$(_count_extents $testdir/file2)
>>> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +
>>> +# Are the mappings still combined?
>>> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
>>> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
>>> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
>>> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
>>> +
>>> +# success, all done
>>> +status=0
>>> +exit
>>> diff --git a/tests/generic/930.out b/tests/generic/930.out
>>> new file mode 100644
>>> index 0000000..556108a
>>> --- /dev/null
>>> +++ b/tests/generic/930.out
>>> @@ -0,0 +1,11 @@
>>> +QA output created by 930
>>> +Format and mount
>>> +Create the original files
>>> +Compare files
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
>>> +Check extent counts
>>> +Compare files after remounting
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
>>> +Check extent counts
>>> diff --git a/tests/generic/group b/tests/generic/group
>>> index d0bc47d..4b4df66 100644
>>> --- a/tests/generic/group
>>> +++ b/tests/generic/group
>>> @@ -411,3 +411,4 @@
>>> 406 auto quick dangerous
>>> 407 auto quick clone metadata
>>> 408 auto quick clone dedupe metadata
>>> +930 auto quick clone
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwICaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=z1VmSNXz_gLYaq0M5JuctwCrED9Ush_X3N5rXd1JsNk&s=Ji40SqPW8DR-ta9h_tDKuKamXkBYf1LrWxSHHECDNjE&e= 
>>>
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwICaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=z1VmSNXz_gLYaq0M5JuctwCrED9Ush_X3N5rXd1JsNk&s=Ji40SqPW8DR-ta9h_tDKuKamXkBYf1LrWxSHHECDNjE&e= 
>
>


--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwICaQ&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=z1VmSNXz_gLYaq0M5JuctwCrED9Ush_X3N5rXd1JsNk&s=Ji40SqPW8DR-ta9h_tDKuKamXkBYf1LrWxSHHECDNjE&e= 

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Qu Wenruo]

At 03/21/2017 10:23 PM, Darrick J. Wong wrote:
> On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
>> Hi Darrick
>>
>> At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
>>> From: Darrick J. Wong <darrick.wong@oracle.com>
>>>
>>> If we reflink a file with N blocks to another file one block at a time,
>>> does the destination file end up with the same number of extents as the
>>> source file?  In other words, does the filesystem succeed at combining
>>> adjacent mappings into a maximal extents?
>>
>> Btrfs can't pass the test since it doesn't merge such mapping, not only the
>> fiemap result, but also on-disk extents.
>
> Weirdly, I observed that if you unmount and remount the btrfs it
> actually /will/ merge the extents.

I tried that, but it's even more weird, after remount, fiemap still 
return split ones.
But if call fiemap again after about 1 second, it reports one single 
large extent.
(if calling 2nd fiemap just after 1st one, 2nd one will still return 
split extent map)

While on disk, the extents are split indeed.

So it's a btrfs bug then.

>
>> IIRC ocfs also merges it, so I wonder if this is a documented behavior to
>> merge any extent maps that are adjacent.
>
> Generally, yes you'd expect that two logically and physically contiguous
> extents to be merged together.  That might change a bit for compressed
> or encrypted extents.
>
>> Further more, if a fs has a limit on extent size on disk, e.g 128M.
>> But 2 file extents are also adjacent on disk, then should fiemap return a
>> large 256M map or just 2 adjacent 128M maps?
>
> It's up to the fs, really... XFS will merge the extent records (up to
> maximum extent size) and then unmerge the shared/not-shared FIEMAP
> records for accurate shared extent reporting.

BTW, are XFS extents on disk are merged when doing reflink, or just 
merge the extent record at fiemap time?

Further more, is there any tool like btrfs-debug-tree to show *every* 
btree block and its content data?
I am very interested in XFS on-disk and runtime implementation.

Thanks,
Qu
>
> --D
>
>>
>> Thanks,
>> Qu
>>>
>>> Signed-off-by: Darrick J. Wong <djwong@djwong.org>
>>> ---
>>> tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
>>> tests/generic/930.out |   11 +++++
>>> tests/generic/group   |    1
>>> 3 files changed, 118 insertions(+)
>>> create mode 100755 tests/generic/930
>>> create mode 100644 tests/generic/930.out
>>>
>>>
>>> diff --git a/tests/generic/930 b/tests/generic/930
>>> new file mode 100755
>>> index 0000000..15d8cbf
>>> --- /dev/null
>>> +++ b/tests/generic/930
>>> @@ -0,0 +1,106 @@
>>> +#! /bin/bash
>>> +# FS QA Test No. 930
>>> +#
>>> +# Check that reflinking adjacent blocks in a file produces a single
>>> +# block mapping extent.
>>> +#
>>> +#-----------------------------------------------------------------------
>>> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
>>> +#
>>> +# This program is free software; you can redistribute it and/or
>>> +# modify it under the terms of the GNU General Public License as
>>> +# published by the Free Software Foundation.
>>> +#
>>> +# This program is distributed in the hope that it would be useful,
>>> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
>>> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>>> +# GNU General Public License for more details.
>>> +#
>>> +# You should have received a copy of the GNU General Public License
>>> +# along with this program; if not, write the Free Software Foundation,
>>> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
>>> +#-----------------------------------------------------------------------
>>> +#
>>> +
>>> +seq=`basename $0`
>>> +seqres=$RESULT_DIR/$seq
>>> +echo "QA output created by $seq"
>>> +
>>> +here=`pwd`
>>> +tmp=/tmp/$$
>>> +status=1	# failure is the default!
>>> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
>>> +
>>> +_cleanup()
>>> +{
>>> +	cd /
>>> +	rm -rf $tmp.*
>>> +	wait
>>> +}
>>> +
>>> +# get standard environment, filters and checks
>>> +. ./common/rc
>>> +. ./common/filter
>>> +. ./common/reflink
>>> +
>>> +# real QA test starts here
>>> +_supported_os Linux
>>> +_supported_fs generic
>>> +_require_scratch_reflink
>>> +_require_fiemap
>>> +
>>> +echo "Format and mount"
>>> +_scratch_mkfs > $seqres.full 2>&1
>>> +_scratch_mount >> $seqres.full 2>&1
>>> +
>>> +testdir=$SCRATCH_MNT/test-$seq
>>> +mkdir $testdir
>>> +
>>> +blocks=32
>>> +blksz=65536
>>> +sz=$((blocks * blksz))
>>> +
>>> +echo "Create the original files"
>>> +$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
>>> +_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
>>> +seq 0 $blksz $((sz - blksz)) | while read offset; do
>>> +	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
>>> +done
>>> +
>>> +echo "Compare files"
>>> +md5sum $testdir/file1 | _filter_scratch
>>> +md5sum $testdir/file2 | _filter_scratch
>>> +
>>> +echo "Check extent counts"
>>> +f1=$(_count_extents $testdir/file1)
>>> +f2=$(_count_extents $testdir/file2)
>>> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +
>>> +# Did the fs combine the extent mappings when we made f2?
>>> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
>>> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
>>> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
>>> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
>>> +
>>> +_scratch_cycle_mount
>>> +
>>> +echo "Compare files after remounting"
>>> +md5sum $testdir/file1 | _filter_scratch
>>> +md5sum $testdir/file2 | _filter_scratch
>>> +
>>> +echo "Check extent counts"
>>> +f1=$(_count_extents $testdir/file1)
>>> +f2=$(_count_extents $testdir/file2)
>>> +s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
>>> +
>>> +# Are the mappings still combined?
>>> +test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
>>> +test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
>>> +test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
>>> +test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
>>> +
>>> +# success, all done
>>> +status=0
>>> +exit
>>> diff --git a/tests/generic/930.out b/tests/generic/930.out
>>> new file mode 100644
>>> index 0000000..556108a
>>> --- /dev/null
>>> +++ b/tests/generic/930.out
>>> @@ -0,0 +1,11 @@
>>> +QA output created by 930
>>> +Format and mount
>>> +Create the original files
>>> +Compare files
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
>>> +Check extent counts
>>> +Compare files after remounting
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
>>> +de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
>>> +Check extent counts
>>> diff --git a/tests/generic/group b/tests/generic/group
>>> index d0bc47d..4b4df66 100644
>>> --- a/tests/generic/group
>>> +++ b/tests/generic/group
>>> @@ -411,3 +411,4 @@
>>> 406 auto quick dangerous
>>> 407 auto quick clone metadata
>>> 408 auto quick clone dedupe metadata
>>> +930 auto quick clone
>>>
>>> --
>>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>>> the body of a message to majordomo@vger.kernel.org
>>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>>>
>>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

On Wed, Mar 22, 2017 at 04:13:16PM +0800, Qu Wenruo wrote:
> 
> 
> At 03/21/2017 10:23 PM, Darrick J. Wong wrote:
> >On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
> >>Hi Darrick
> >>
> >>At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> >>>From: Darrick J. Wong <darrick.wong@oracle.com>
> >>>
> >>>If we reflink a file with N blocks to another file one block at a time,
> >>>does the destination file end up with the same number of extents as the
> >>>source file?  In other words, does the filesystem succeed at combining
> >>>adjacent mappings into a maximal extents?
> >>
> >>Btrfs can't pass the test since it doesn't merge such mapping, not only the
> >>fiemap result, but also on-disk extents.
> >
> >Weirdly, I observed that if you unmount and remount the btrfs it
> >actually /will/ merge the extents.
> 
> I tried that, but it's even more weird, after remount, fiemap still return
> split ones.
> But if call fiemap again after about 1 second, it reports one single large
> extent.
> (if calling 2nd fiemap just after 1st one, 2nd one will still return split
> extent map)
> 
> While on disk, the extents are split indeed.

That's .... weirder than I was expecting.

> So it's a btrfs bug then.
> 
> >
> >>IIRC ocfs also merges it, so I wonder if this is a documented behavior to
> >>merge any extent maps that are adjacent.
> >
> >Generally, yes you'd expect that two logically and physically contiguous
> >extents to be merged together.  That might change a bit for compressed
> >or encrypted extents.
> >
> >>Further more, if a fs has a limit on extent size on disk, e.g 128M.
> >>But 2 file extents are also adjacent on disk, then should fiemap return a
> >>large 256M map or just 2 adjacent 128M maps?
> >
> >It's up to the fs, really... XFS will merge the extent records (up to
> >maximum extent size) and then unmerge the shared/not-shared FIEMAP
> >records for accurate shared extent reporting.
> 
> BTW, are XFS extents on disk are merged when doing reflink, or just merge
> the extent record at fiemap time?

Extents are always merged at the earliest possible time.

> Further more, is there any tool like btrfs-debug-tree to show *every* btree
> block and its content data?
> I am very interested in XFS on-disk and runtime implementation.

You can step through them manually with xfs_db, but there's no way to
dump a btree.  Hmm, that would be useful, I'll suggest implementing that.

--D

> Thanks,
> Qu
> >
> >--D
> >
> >>
> >>Thanks,
> >>Qu
> >>>
> >>>Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> >>>---
> >>>tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
> >>>tests/generic/930.out |   11 +++++
> >>>tests/generic/group   |    1
> >>>3 files changed, 118 insertions(+)
> >>>create mode 100755 tests/generic/930
> >>>create mode 100644 tests/generic/930.out
> >>>
> >>>
> >>>diff --git a/tests/generic/930 b/tests/generic/930
> >>>new file mode 100755
> >>>index 0000000..15d8cbf
> >>>--- /dev/null
> >>>+++ b/tests/generic/930
> >>>@@ -0,0 +1,106 @@
> >>>+#! /bin/bash
> >>>+# FS QA Test No. 930
> >>>+#
> >>>+# Check that reflinking adjacent blocks in a file produces a single
> >>>+# block mapping extent.
> >>>+#
> >>>+#-----------------------------------------------------------------------
> >>>+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> >>>+#
> >>>+# This program is free software; you can redistribute it and/or
> >>>+# modify it under the terms of the GNU General Public License as
> >>>+# published by the Free Software Foundation.
> >>>+#
> >>>+# This program is distributed in the hope that it would be useful,
> >>>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >>>+# GNU General Public License for more details.
> >>>+#
> >>>+# You should have received a copy of the GNU General Public License
> >>>+# along with this program; if not, write the Free Software Foundation,
> >>>+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> >>>+#-----------------------------------------------------------------------
> >>>+#
> >>>+
> >>>+seq=`basename $0`
> >>>+seqres=$RESULT_DIR/$seq
> >>>+echo "QA output created by $seq"
> >>>+
> >>>+here=`pwd`
> >>>+tmp=/tmp/$$
> >>>+status=1	# failure is the default!
> >>>+trap "_cleanup; exit \$status" 0 1 2 3 7 15
> >>>+
> >>>+_cleanup()
> >>>+{
> >>>+	cd /
> >>>+	rm -rf $tmp.*
> >>>+	wait
> >>>+}
> >>>+
> >>>+# get standard environment, filters and checks
> >>>+. ./common/rc
> >>>+. ./common/filter
> >>>+. ./common/reflink
> >>>+
> >>>+# real QA test starts here
> >>>+_supported_os Linux
> >>>+_supported_fs generic
> >>>+_require_scratch_reflink
> >>>+_require_fiemap
> >>>+
> >>>+echo "Format and mount"
> >>>+_scratch_mkfs > $seqres.full 2>&1
> >>>+_scratch_mount >> $seqres.full 2>&1
> >>>+
> >>>+testdir=$SCRATCH_MNT/test-$seq
> >>>+mkdir $testdir
> >>>+
> >>>+blocks=32
> >>>+blksz=65536
> >>>+sz=$((blocks * blksz))
> >>>+
> >>>+echo "Create the original files"
> >>>+$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> >>>+_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> >>>+seq 0 $blksz $((sz - blksz)) | while read offset; do
> >>>+	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> >>>+done
> >>>+
> >>>+echo "Compare files"
> >>>+md5sum $testdir/file1 | _filter_scratch
> >>>+md5sum $testdir/file2 | _filter_scratch
> >>>+
> >>>+echo "Check extent counts"
> >>>+f1=$(_count_extents $testdir/file1)
> >>>+f2=$(_count_extents $testdir/file2)
> >>>+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+
> >>>+# Did the fs combine the extent mappings when we made f2?
> >>>+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >>>+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >>>+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >>>+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >>>+
> >>>+_scratch_cycle_mount
> >>>+
> >>>+echo "Compare files after remounting"
> >>>+md5sum $testdir/file1 | _filter_scratch
> >>>+md5sum $testdir/file2 | _filter_scratch
> >>>+
> >>>+echo "Check extent counts"
> >>>+f1=$(_count_extents $testdir/file1)
> >>>+f2=$(_count_extents $testdir/file2)
> >>>+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+
> >>>+# Are the mappings still combined?
> >>>+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >>>+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >>>+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >>>+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >>>+
> >>>+# success, all done
> >>>+status=0
> >>>+exit
> >>>diff --git a/tests/generic/930.out b/tests/generic/930.out
> >>>new file mode 100644
> >>>index 0000000..556108a
> >>>--- /dev/null
> >>>+++ b/tests/generic/930.out
> >>>@@ -0,0 +1,11 @@
> >>>+QA output created by 930
> >>>+Format and mount
> >>>+Create the original files
> >>>+Compare files
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >>>+Check extent counts
> >>>+Compare files after remounting
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >>>+Check extent counts
> >>>diff --git a/tests/generic/group b/tests/generic/group
> >>>index d0bc47d..4b4df66 100644
> >>>--- a/tests/generic/group
> >>>+++ b/tests/generic/group
> >>>@@ -411,3 +411,4 @@
> >>>406 auto quick dangerous
> >>>407 auto quick clone metadata
> >>>408 auto quick clone dedupe metadata
> >>>+930 auto quick clone
> >>>
> >>>--
> >>>To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>>the body of a message to majordomo@vger.kernel.org
> >>>More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=A8c3Ik9WSm051Nlsx60vc8A2ldgZm-hj9p14kKHly-E&s=B_WnYcS5fSIGD15swW2LqVnQ-bcchDHPJ_Y2V8qbef4&e= 
> >>>
> >>>
> >>
> >>
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=A8c3Ik9WSm051Nlsx60vc8A2ldgZm-hj9p14kKHly-E&s=B_WnYcS5fSIGD15swW2LqVnQ-bcchDHPJ_Y2V8qbef4&e= 
> >
> >
> 
> 
--
To unsubscribe from this list: send the line "unsubscribe fstests" in
the body of a message to majordomo@vger.kernel.org
More majordomo info at  https://urldefense.proofpoint.com/v2/url?u=http-3A__vger.kernel.org_majordomo-2Dinfo.html&d=DwIBAg&c=RoP1YumCXCgaWHvlZYR8PQcxBKCX5YTpkKY057SbK10&r=-yMrTV4jriXR7ieyzzjV-QgHBD0UDw8ixoR77aMeAHE&m=A8c3Ik9WSm051Nlsx60vc8A2ldgZm-hj9p14kKHly-E&s=B_WnYcS5fSIGD15swW2LqVnQ-bcchDHPJ_Y2V8qbef4&e= 

Re: [PATCH 3/4] reflink: test adjacency of reflinked blocks

[Darrick J. Wong]

On Wed, Mar 22, 2017 at 04:13:16PM +0800, Qu Wenruo wrote:
> 
> 
> At 03/21/2017 10:23 PM, Darrick J. Wong wrote:
> >On Tue, Mar 21, 2017 at 03:14:36PM +0800, Qu Wenruo wrote:
> >>Hi Darrick
> >>
> >>At 02/25/2017 09:12 AM, Darrick J. Wong wrote:
> >>>From: Darrick J. Wong <darrick.wong@oracle.com>
> >>>
> >>>If we reflink a file with N blocks to another file one block at a time,
> >>>does the destination file end up with the same number of extents as the
> >>>source file?  In other words, does the filesystem succeed at combining
> >>>adjacent mappings into a maximal extents?
> >>
> >>Btrfs can't pass the test since it doesn't merge such mapping, not only the
> >>fiemap result, but also on-disk extents.
> >
> >Weirdly, I observed that if you unmount and remount the btrfs it
> >actually /will/ merge the extents.
> 
> I tried that, but it's even more weird, after remount, fiemap still return
> split ones.
> But if call fiemap again after about 1 second, it reports one single large
> extent.
> (if calling 2nd fiemap just after 1st one, 2nd one will still return split
> extent map)
> 
> While on disk, the extents are split indeed.

That's .... weirder than I was expecting.

> So it's a btrfs bug then.
> 
> >
> >>IIRC ocfs also merges it, so I wonder if this is a documented behavior to
> >>merge any extent maps that are adjacent.
> >
> >Generally, yes you'd expect that two logically and physically contiguous
> >extents to be merged together.  That might change a bit for compressed
> >or encrypted extents.
> >
> >>Further more, if a fs has a limit on extent size on disk, e.g 128M.
> >>But 2 file extents are also adjacent on disk, then should fiemap return a
> >>large 256M map or just 2 adjacent 128M maps?
> >
> >It's up to the fs, really... XFS will merge the extent records (up to
> >maximum extent size) and then unmerge the shared/not-shared FIEMAP
> >records for accurate shared extent reporting.
> 
> BTW, are XFS extents on disk are merged when doing reflink, or just merge
> the extent record at fiemap time?

Extents are always merged at the earliest possible time.

> Further more, is there any tool like btrfs-debug-tree to show *every* btree
> block and its content data?
> I am very interested in XFS on-disk and runtime implementation.

You can step through them manually with xfs_db, but there's no way to
dump a btree.  Hmm, that would be useful, I'll suggest implementing that.

--D

> Thanks,
> Qu
> >
> >--D
> >
> >>
> >>Thanks,
> >>Qu
> >>>
> >>>Signed-off-by: Darrick J. Wong <djwong@djwong.org>
> >>>---
> >>>tests/generic/930     |  106 +++++++++++++++++++++++++++++++++++++++++++++++++
> >>>tests/generic/930.out |   11 +++++
> >>>tests/generic/group   |    1
> >>>3 files changed, 118 insertions(+)
> >>>create mode 100755 tests/generic/930
> >>>create mode 100644 tests/generic/930.out
> >>>
> >>>
> >>>diff --git a/tests/generic/930 b/tests/generic/930
> >>>new file mode 100755
> >>>index 0000000..15d8cbf
> >>>--- /dev/null
> >>>+++ b/tests/generic/930
> >>>@@ -0,0 +1,106 @@
> >>>+#! /bin/bash
> >>>+# FS QA Test No. 930
> >>>+#
> >>>+# Check that reflinking adjacent blocks in a file produces a single
> >>>+# block mapping extent.
> >>>+#
> >>>+#-----------------------------------------------------------------------
> >>>+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> >>>+#
> >>>+# This program is free software; you can redistribute it and/or
> >>>+# modify it under the terms of the GNU General Public License as
> >>>+# published by the Free Software Foundation.
> >>>+#
> >>>+# This program is distributed in the hope that it would be useful,
> >>>+# but WITHOUT ANY WARRANTY; without even the implied warranty of
> >>>+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >>>+# GNU General Public License for more details.
> >>>+#
> >>>+# You should have received a copy of the GNU General Public License
> >>>+# along with this program; if not, write the Free Software Foundation,
> >>>+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> >>>+#-----------------------------------------------------------------------
> >>>+#
> >>>+
> >>>+seq=`basename $0`
> >>>+seqres=$RESULT_DIR/$seq
> >>>+echo "QA output created by $seq"
> >>>+
> >>>+here=`pwd`
> >>>+tmp=/tmp/$$
> >>>+status=1	# failure is the default!
> >>>+trap "_cleanup; exit \$status" 0 1 2 3 7 15
> >>>+
> >>>+_cleanup()
> >>>+{
> >>>+	cd /
> >>>+	rm -rf $tmp.*
> >>>+	wait
> >>>+}
> >>>+
> >>>+# get standard environment, filters and checks
> >>>+. ./common/rc
> >>>+. ./common/filter
> >>>+. ./common/reflink
> >>>+
> >>>+# real QA test starts here
> >>>+_supported_os Linux
> >>>+_supported_fs generic
> >>>+_require_scratch_reflink
> >>>+_require_fiemap
> >>>+
> >>>+echo "Format and mount"
> >>>+_scratch_mkfs > $seqres.full 2>&1
> >>>+_scratch_mount >> $seqres.full 2>&1
> >>>+
> >>>+testdir=$SCRATCH_MNT/test-$seq
> >>>+mkdir $testdir
> >>>+
> >>>+blocks=32
> >>>+blksz=65536
> >>>+sz=$((blocks * blksz))
> >>>+
> >>>+echo "Create the original files"
> >>>+$XFS_IO_PROG -f -c "falloc 0 $sz" $testdir/file1 >> $seqres.full
> >>>+_pwrite_byte 0x61 0 $sz $testdir/file1 >> $seqres.full
> >>>+seq 0 $blksz $((sz - blksz)) | while read offset; do
> >>>+	_reflink_range $testdir/file1 $offset $testdir/file2 $offset $blksz >> $seqres.full
> >>>+done
> >>>+
> >>>+echo "Compare files"
> >>>+md5sum $testdir/file1 | _filter_scratch
> >>>+md5sum $testdir/file2 | _filter_scratch
> >>>+
> >>>+echo "Check extent counts"
> >>>+f1=$(_count_extents $testdir/file1)
> >>>+f2=$(_count_extents $testdir/file2)
> >>>+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+
> >>>+# Did the fs combine the extent mappings when we made f2?
> >>>+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >>>+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >>>+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >>>+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >>>+
> >>>+_scratch_cycle_mount
> >>>+
> >>>+echo "Compare files after remounting"
> >>>+md5sum $testdir/file1 | _filter_scratch
> >>>+md5sum $testdir/file2 | _filter_scratch
> >>>+
> >>>+echo "Check extent counts"
> >>>+f1=$(_count_extents $testdir/file1)
> >>>+f2=$(_count_extents $testdir/file2)
> >>>+s1=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file1 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+s2=$($XFS_IO_PROG -c 'fiemap -v' $testdir/file2 | awk '{print $5}' | grep -c '0x.*[2367aAbBfF]...$')
> >>>+
> >>>+# Are the mappings still combined?
> >>>+test $f1 -eq $f2 || echo "f1 ($f1) != f2 ($f2)"
> >>>+test $s1 -eq $s2 || echo "s1 ($s1) != s2 ($s2)"
> >>>+test $f1 -eq $s1 || echo "f1 ($f1) != s1 ($f1)"
> >>>+test $f2 -eq $s2 || echo "f2 ($f2) != s2 ($f2)"
> >>>+
> >>>+# success, all done
> >>>+status=0
> >>>+exit
> >>>diff --git a/tests/generic/930.out b/tests/generic/930.out
> >>>new file mode 100644
> >>>index 0000000..556108a
> >>>--- /dev/null
> >>>+++ b/tests/generic/930.out
> >>>@@ -0,0 +1,11 @@
> >>>+QA output created by 930
> >>>+Format and mount
> >>>+Create the original files
> >>>+Compare files
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >>>+Check extent counts
> >>>+Compare files after remounting
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file1
> >>>+de89461b64701958984c95d1bfb0065a  SCRATCH_MNT/test-930/file2
> >>>+Check extent counts
> >>>diff --git a/tests/generic/group b/tests/generic/group
> >>>index d0bc47d..4b4df66 100644
> >>>--- a/tests/generic/group
> >>>+++ b/tests/generic/group
> >>>@@ -411,3 +411,4 @@
> >>>406 auto quick dangerous
> >>>407 auto quick clone metadata
> >>>408 auto quick clone dedupe metadata
> >>>+930 auto quick clone
> >>>
> >>>--
> >>>To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>>the body of a message to majordomo@vger.kernel.org
> >>>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >>>
> >>>
> >>
> >>
> >>--
> >>To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> >>the body of a message to majordomo@vger.kernel.org
> >>More majordomo info at  http://vger.kernel.org/majordomo-info.html
> >
> >
> 
> 

[PATCH 4/4] xfs: stress test xfs_scrub

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Three new tests:

- Repair files that are mapped into memory in running programs
- Run scrub -n concurrently with fsstress
- Run scrub -y concurrently with fsstress

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/1375     |   69 +++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1375.out |    3 ++
 tests/xfs/1376     |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1376.out |    4 ++
 tests/xfs/1377     |   84 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1377.out |    4 ++
 tests/xfs/group    |    3 ++
 7 files changed, 250 insertions(+)
 create mode 100755 tests/xfs/1375
 create mode 100644 tests/xfs/1375.out
 create mode 100755 tests/xfs/1376
 create mode 100644 tests/xfs/1376.out
 create mode 100755 tests/xfs/1377
 create mode 100644 tests/xfs/1377.out


diff --git a/tests/xfs/1375 b/tests/xfs/1375
new file mode 100755
index 0000000..d3119ef
--- /dev/null
+++ b/tests/xfs/1375
@@ -0,0 +1,69 @@
+#! /bin/bash
+# FS QA Test No. 1375
+#
+# Copy xfs_scrub to the scratch device, then run xfs_scrub in forced
+# repair mode (which will rebuild the data forks of the running scrub
+# executable and libraries!) to see what happens.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_command /usr/bin/ldd "ldd"
+_require_scrub
+_require_scratch
+_require_xfs_io_error_injection "force_repair"
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+cp $XFS_SCRUB_PROG $SCRATCH_MNT/xfs_scrub
+ldd $XFS_SCRUB_PROG | sed -e '/\//!d;/linux-gate/d;/=>/ {s/.*=>[[:blank:]]*\([^[:blank:]]*\).*/\1/};s/[[:blank:]]*\([^[:blank:]]*\) (.*)/\1/' | while read lib; do
+	cp $lib $SCRATCH_MNT/
+done
+LD_LIBRARY_PATH=$SCRATCH_MNT ldd $SCRATCH_MNT/xfs_scrub >> $seqres.full
+
+echo "Force online repairs"
+XFS_SCRUB_FORCE_REPAIR=1 LD_LIBRARY_PATH=$SCRATCH_MNT $SCRATCH_MNT/xfs_scrub -dTvy $SCRATCH_MNT >> $seqres.full
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1375.out b/tests/xfs/1375.out
new file mode 100644
index 0000000..dd7025e
--- /dev/null
+++ b/tests/xfs/1375.out
@@ -0,0 +1,3 @@
+QA output created by 1375
+Format and populate
+Force online repairs
diff --git a/tests/xfs/1376 b/tests/xfs/1376
new file mode 100755
index 0000000..f037b43
--- /dev/null
+++ b/tests/xfs/1376
@@ -0,0 +1,83 @@
+#! /bin/bash
+# FS QA Test No. 1376
+#
+# Race fio and xfs_scrub for a while to see if we crash or livelock.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_scrub
+_require_scratch
+_require_command "$KILLALL_PROG" killall
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+
+STRESS_DIR="$SCRATCH_MNT/testdir"
+mkdir -p $STRESS_DIR
+
+cpus=$((16 * LOAD_FACTOR))
+$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
+$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
+
+killstress() {
+	sleep $(( 60 * LOAD_FACTOR ))
+	$KILLALL_PROG -q $FSSTRESS_PROG
+}
+
+echo "Concurrent scrub"
+start=$(date +%s)
+end=$((start + (60 * LOAD_FACTOR) ))
+killstress &
+echo "Scrub started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
+while [ "$(date +%s)" -lt "$end" ]; do
+	timeout -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full 2>&1
+done
+
+echo "Test done"
+echo "Scrub finished at $(date)" >> $seqres.full
+$KILLALL_PROG -q $FSSTRESS_PROG
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1376.out b/tests/xfs/1376.out
new file mode 100644
index 0000000..155f424
--- /dev/null
+++ b/tests/xfs/1376.out
@@ -0,0 +1,4 @@
+QA output created by 1376
+Format and populate
+Concurrent scrub
+Test done
diff --git a/tests/xfs/1377 b/tests/xfs/1377
new file mode 100755
index 0000000..1e0814c
--- /dev/null
+++ b/tests/xfs/1377
@@ -0,0 +1,84 @@
+#! /bin/bash
+# FS QA Test No. 1377
+#
+# Race fio and xfs_scrub -y for a while to see if we crash or livelock.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_scrub
+_require_scratch
+_require_command "$KILLALL_PROG" killall
+_require_xfs_io_error_injection "force_repair"
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+
+STRESS_DIR="$SCRATCH_MNT/testdir"
+mkdir -p $STRESS_DIR
+
+cpus=$((16 * LOAD_FACTOR))
+$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
+$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
+
+killstress() {
+	sleep $(( 60 * LOAD_FACTOR ))
+	$KILLALL_PROG -q $FSSTRESS_PROG
+}
+
+echo "Concurrent repair"
+start=$(date +%s)
+end=$((start + (60 * LOAD_FACTOR) ))
+killstress &
+echo "Repair started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
+while [ "$(date +%s)" -lt "$end" ]; do
+	XFS_SCRUB_FORCE_REPAIR=1 timeout -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -y $SCRATCH_MNT >> $seqres.full
+done
+
+echo "Test done"
+echo "Repair finished at $(date)" >> $seqres.full
+$KILLALL_PROG -q $FSSTRESS_PROG
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1377.out b/tests/xfs/1377.out
new file mode 100644
index 0000000..0454a87
--- /dev/null
+++ b/tests/xfs/1377.out
@@ -0,0 +1,4 @@
+QA output created by 1377
+Format and populate
+Concurrent repair
+Test done
diff --git a/tests/xfs/group b/tests/xfs/group
index eaa65b1..7fce0a0 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -408,3 +408,6 @@
 416 dangerous_fuzzers dangerous_scrub dangerous_repair
 417 dangerous_fuzzers dangerous_scrub dangerous_online_repair
 418 dangerous_fuzzers dangerous_scrub dangerous_repair
+1375 dangerous_fuzzers dangerous_scrub dangerous_online_repair
+1376 dangerous_fuzzers dangerous_scrub
+1377 dangerous_fuzzers dangerous_scrub dangerous_online_repair

Re: [PATCH 4/4] xfs: stress test xfs_scrub

[Eryu Guan]

On Fri, Feb 24, 2017 at 05:13:04PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Three new tests:
> 
> - Repair files that are mapped into memory in running programs
> - Run scrub -n concurrently with fsstress
> - Run scrub -y concurrently with fsstress
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/xfs/1375     |   69 +++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/1375.out |    3 ++
>  tests/xfs/1376     |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/1376.out |    4 ++
>  tests/xfs/1377     |   84 ++++++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/1377.out |    4 ++
>  tests/xfs/group    |    3 ++
>  7 files changed, 250 insertions(+)
>  create mode 100755 tests/xfs/1375
>  create mode 100644 tests/xfs/1375.out
>  create mode 100755 tests/xfs/1376
>  create mode 100644 tests/xfs/1376.out
>  create mode 100755 tests/xfs/1377
>  create mode 100644 tests/xfs/1377.out
> 
> 
> diff --git a/tests/xfs/1375 b/tests/xfs/1375
> new file mode 100755
> index 0000000..d3119ef
> --- /dev/null
> +++ b/tests/xfs/1375
> @@ -0,0 +1,69 @@
> +#! /bin/bash
> +# FS QA Test No. 1375
> +#
> +# Copy xfs_scrub to the scratch device, then run xfs_scrub in forced
> +# repair mode (which will rebuild the data forks of the running scrub
> +# executable and libraries!) to see what happens.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/fuzzy
> +. ./common/inject
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs xfs
> +_require_command /usr/bin/ldd "ldd"

Introduce a new LDD_PROG? So that the path of ldd is set by
set_prog_path not hardcoded.

> +_require_scrub
> +_require_scratch
> +_require_xfs_io_error_injection "force_repair"

Seems error injection is not used in this test.

> +
> +echo "Format and populate"
> +_scratch_mkfs > "$seqres.full" 2>&1
> +_scratch_mount
> +cp $XFS_SCRUB_PROG $SCRATCH_MNT/xfs_scrub
> +ldd $XFS_SCRUB_PROG | sed -e '/\//!d;/linux-gate/d;/=>/ {s/.*=>[[:blank:]]*\([^[:blank:]]*\).*/\1/};s/[[:blank:]]*\([^[:blank:]]*\) (.*)/\1/' | while read lib; do
> +	cp $lib $SCRATCH_MNT/
> +done
> +LD_LIBRARY_PATH=$SCRATCH_MNT ldd $SCRATCH_MNT/xfs_scrub >> $seqres.full
> +
> +echo "Force online repairs"
> +XFS_SCRUB_FORCE_REPAIR=1 LD_LIBRARY_PATH=$SCRATCH_MNT $SCRATCH_MNT/xfs_scrub -dTvy $SCRATCH_MNT >> $seqres.full
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/1375.out b/tests/xfs/1375.out
> new file mode 100644
> index 0000000..dd7025e
> --- /dev/null
> +++ b/tests/xfs/1375.out
> @@ -0,0 +1,3 @@
> +QA output created by 1375
> +Format and populate
> +Force online repairs
> diff --git a/tests/xfs/1376 b/tests/xfs/1376
> new file mode 100755
> index 0000000..f037b43
> --- /dev/null
> +++ b/tests/xfs/1376
> @@ -0,0 +1,83 @@
> +#! /bin/bash
> +# FS QA Test No. 1376
> +#
> +# Race fio and xfs_scrub for a while to see if we crash or livelock.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/fuzzy
> +. ./common/inject
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs xfs
> +_require_scrub
> +_require_scratch
> +_require_command "$KILLALL_PROG" killall
> +
> +echo "Format and populate"
> +_scratch_mkfs > "$seqres.full" 2>&1
> +_scratch_mount
> +
> +STRESS_DIR="$SCRATCH_MNT/testdir"
> +mkdir -p $STRESS_DIR
> +
> +cpus=$((16 * LOAD_FACTOR))

_require_test_program "feature"
then
cpus=$((`src/feature -o` * LOAD_FACTOR)) ?

> +$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
> +$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
> +
> +killstress() {
> +	sleep $(( 60 * LOAD_FACTOR ))

There's a TIME_FACTOR for this, I think.

> +	$KILLALL_PROG -q $FSSTRESS_PROG
> +}
> +
> +echo "Concurrent scrub"
> +start=$(date +%s)
> +end=$((start + (60 * LOAD_FACTOR) ))
> +killstress &
> +echo "Scrub started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
> +while [ "$(date +%s)" -lt "$end" ]; do
> +	timeout -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full 2>&1

Introduce TIMEOUT_PROG and _require it?

> +done
> +
> +echo "Test done"
> +echo "Scrub finished at $(date)" >> $seqres.full
> +$KILLALL_PROG -q $FSSTRESS_PROG
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/1376.out b/tests/xfs/1376.out
> new file mode 100644
> index 0000000..155f424
> --- /dev/null
> +++ b/tests/xfs/1376.out
> @@ -0,0 +1,4 @@
> +QA output created by 1376
> +Format and populate
> +Concurrent scrub
> +Test done
> diff --git a/tests/xfs/1377 b/tests/xfs/1377
> new file mode 100755
> index 0000000..1e0814c
> --- /dev/null
> +++ b/tests/xfs/1377
> @@ -0,0 +1,84 @@
> +#! /bin/bash
> +# FS QA Test No. 1377
> +#
> +# Race fio and xfs_scrub -y for a while to see if we crash or livelock.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> +#-----------------------------------------------------------------------
> +#
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> +
> +_cleanup()
> +{
> +	cd /
> +	rm -rf $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +. ./common/fuzzy
> +. ./common/inject
> +
> +# real QA test starts here
> +_supported_os Linux
> +_supported_fs xfs
> +_require_scrub
> +_require_scratch
> +_require_command "$KILLALL_PROG" killall
> +_require_xfs_io_error_injection "force_repair"

Seems not used either.

Comments to 1376 also apply to 1377 :)

Thanks,
Eryu

Re: [PATCH 4/4] xfs: stress test xfs_scrub

[Darrick J. Wong]

On Tue, Feb 28, 2017 at 04:46:12PM +0800, Eryu Guan wrote:
> On Fri, Feb 24, 2017 at 05:13:04PM -0800, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Three new tests:
> > 
> > - Repair files that are mapped into memory in running programs
> > - Run scrub -n concurrently with fsstress
> > - Run scrub -y concurrently with fsstress
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/xfs/1375     |   69 +++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/1375.out |    3 ++
> >  tests/xfs/1376     |   83 +++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/1376.out |    4 ++
> >  tests/xfs/1377     |   84 ++++++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/1377.out |    4 ++
> >  tests/xfs/group    |    3 ++
> >  7 files changed, 250 insertions(+)
> >  create mode 100755 tests/xfs/1375
> >  create mode 100644 tests/xfs/1375.out
> >  create mode 100755 tests/xfs/1376
> >  create mode 100644 tests/xfs/1376.out
> >  create mode 100755 tests/xfs/1377
> >  create mode 100644 tests/xfs/1377.out
> > 
> > 
> > diff --git a/tests/xfs/1375 b/tests/xfs/1375
> > new file mode 100755
> > index 0000000..d3119ef
> > --- /dev/null
> > +++ b/tests/xfs/1375
> > @@ -0,0 +1,69 @@
> > +#! /bin/bash
> > +# FS QA Test No. 1375
> > +#
> > +# Copy xfs_scrub to the scratch device, then run xfs_scrub in forced
> > +# repair mode (which will rebuild the data forks of the running scrub
> > +# executable and libraries!) to see what happens.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +. ./common/fuzzy
> > +. ./common/inject
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs xfs
> > +_require_command /usr/bin/ldd "ldd"
> 
> Introduce a new LDD_PROG? So that the path of ldd is set by
> set_prog_path not hardcoded.

Ok.

> > +_require_scrub
> > +_require_scratch
> > +_require_xfs_io_error_injection "force_repair"
> 
> Seems error injection is not used in this test.

It is -- xfs_scrub -d will turn it on if XFS_SCRUB_FORCE_REPAIR=1 is set.
I'll add a comment stating that xfs_scrub turns it on directly.

> > +
> > +echo "Format and populate"
> > +_scratch_mkfs > "$seqres.full" 2>&1
> > +_scratch_mount
> > +cp $XFS_SCRUB_PROG $SCRATCH_MNT/xfs_scrub
> > +ldd $XFS_SCRUB_PROG | sed -e '/\//!d;/linux-gate/d;/=>/ {s/.*=>[[:blank:]]*\([^[:blank:]]*\).*/\1/};s/[[:blank:]]*\([^[:blank:]]*\) (.*)/\1/' | while read lib; do
> > +	cp $lib $SCRATCH_MNT/
> > +done
> > +LD_LIBRARY_PATH=$SCRATCH_MNT ldd $SCRATCH_MNT/xfs_scrub >> $seqres.full
> > +
> > +echo "Force online repairs"
> > +XFS_SCRUB_FORCE_REPAIR=1 LD_LIBRARY_PATH=$SCRATCH_MNT $SCRATCH_MNT/xfs_scrub -dTvy $SCRATCH_MNT >> $seqres.full
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/xfs/1375.out b/tests/xfs/1375.out
> > new file mode 100644
> > index 0000000..dd7025e
> > --- /dev/null
> > +++ b/tests/xfs/1375.out
> > @@ -0,0 +1,3 @@
> > +QA output created by 1375
> > +Format and populate
> > +Force online repairs
> > diff --git a/tests/xfs/1376 b/tests/xfs/1376
> > new file mode 100755
> > index 0000000..f037b43
> > --- /dev/null
> > +++ b/tests/xfs/1376
> > @@ -0,0 +1,83 @@
> > +#! /bin/bash
> > +# FS QA Test No. 1376
> > +#
> > +# Race fio and xfs_scrub for a while to see if we crash or livelock.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +. ./common/fuzzy
> > +. ./common/inject
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs xfs
> > +_require_scrub
> > +_require_scratch
> > +_require_command "$KILLALL_PROG" killall
> > +
> > +echo "Format and populate"
> > +_scratch_mkfs > "$seqres.full" 2>&1
> > +_scratch_mount
> > +
> > +STRESS_DIR="$SCRATCH_MNT/testdir"
> > +mkdir -p $STRESS_DIR
> > +
> > +cpus=$((16 * LOAD_FACTOR))
> 
> _require_test_program "feature"
> then
> cpus=$((`src/feature -o` * LOAD_FACTOR)) ?

Oh, did not know about that... though the point was to run 16 io threads per
load_factor...

...otoh I guess it's not such a big deal to change that to 4 threads per cpu.

> > +$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
> > +$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
> > +
> > +killstress() {
> > +	sleep $(( 60 * LOAD_FACTOR ))
> 
> There's a TIME_FACTOR for this, I think.

D'oh, yes.  Thank you. :)

> > +	$KILLALL_PROG -q $FSSTRESS_PROG
> > +}
> > +
> > +echo "Concurrent scrub"
> > +start=$(date +%s)
> > +end=$((start + (60 * LOAD_FACTOR) ))
> > +killstress &
> > +echo "Scrub started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
> > +while [ "$(date +%s)" -lt "$end" ]; do
> > +	timeout -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full 2>&1
> 
> Introduce TIMEOUT_PROG and _require it?

Ok.

> > +done
> > +
> > +echo "Test done"
> > +echo "Scrub finished at $(date)" >> $seqres.full
> > +$KILLALL_PROG -q $FSSTRESS_PROG
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/xfs/1376.out b/tests/xfs/1376.out
> > new file mode 100644
> > index 0000000..155f424
> > --- /dev/null
> > +++ b/tests/xfs/1376.out
> > @@ -0,0 +1,4 @@
> > +QA output created by 1376
> > +Format and populate
> > +Concurrent scrub
> > +Test done
> > diff --git a/tests/xfs/1377 b/tests/xfs/1377
> > new file mode 100755
> > index 0000000..1e0814c
> > --- /dev/null
> > +++ b/tests/xfs/1377
> > @@ -0,0 +1,84 @@
> > +#! /bin/bash
> > +# FS QA Test No. 1377
> > +#
> > +# Race fio and xfs_scrub -y for a while to see if we crash or livelock.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
> > +#-----------------------------------------------------------------------
> > +#
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 7 15
> > +
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -rf $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +. ./common/fuzzy
> > +. ./common/inject
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_supported_fs xfs
> > +_require_scrub
> > +_require_scratch
> > +_require_command "$KILLALL_PROG" killall
> > +_require_xfs_io_error_injection "force_repair"
> 
> Seems not used either.
> 
> Comments to 1376 also apply to 1377 :)

Fixed.  I'll send a v2 patch shortly.
Thank you for the review!

--D

> 
> Thanks,
> Eryu
> --
> To unsubscribe from this list: send the line "unsubscribe fstests" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 4/4] xfs: stress test xfs_scrub

[Darrick J. Wong]

Three new tests:

- Repair files that are mapped into memory in running programs
- Run scrub -n concurrently with fsstress
- Run scrub -y concurrently with fsstress

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: fix checks for external programs per eryu guan's recommendations
---
 common/config      |    2 +
 tests/xfs/1375     |   70 ++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1375.out |    3 ++
 tests/xfs/1376     |   85 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1376.out |    4 ++
 tests/xfs/1377     |   87 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/1377.out |    4 ++
 tests/xfs/group    |    3 ++
 8 files changed, 258 insertions(+)
 create mode 100755 tests/xfs/1375
 create mode 100644 tests/xfs/1375.out
 create mode 100755 tests/xfs/1376
 create mode 100644 tests/xfs/1376.out
 create mode 100755 tests/xfs/1377
 create mode 100644 tests/xfs/1377.out

diff --git a/common/config b/common/config
index 3ef401b..5f3202b 100644
--- a/common/config
+++ b/common/config
@@ -205,6 +205,8 @@ export SETRICHACL_PROG="`set_prog_path setrichacl`"
 export KEYCTL_PROG="`set_prog_path keyctl`"
 export XZ_PROG="`set_prog_path xz`"
 export FLOCK_PROG="`set_prog_path flock`"
+export LDD_PROG="`set_prog_path ldd`"
+export TIMEOUT_PROG="`set_prog_path timeout`"
 
 # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
 # newer systems have udevadm command but older systems like RHEL5 don't.
diff --git a/tests/xfs/1375 b/tests/xfs/1375
new file mode 100755
index 0000000..f1b0a84
--- /dev/null
+++ b/tests/xfs/1375
@@ -0,0 +1,70 @@
+#! /bin/bash
+# FS QA Test No. 1375
+#
+# Copy xfs_scrub to the scratch device, then run xfs_scrub in forced
+# repair mode (which will rebuild the data forks of the running scrub
+# executable and libraries!) to see what happens.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_command "$LDD_PROG" ldd
+_require_scrub
+_require_scratch
+# xfs_scrub will turn on error injection itself
+_require_xfs_io_error_injection "force_repair"
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+cp $XFS_SCRUB_PROG $SCRATCH_MNT/xfs_scrub
+$LDD_PROG $XFS_SCRUB_PROG | sed -e '/\//!d;/linux-gate/d;/=>/ {s/.*=>[[:blank:]]*\([^[:blank:]]*\).*/\1/};s/[[:blank:]]*\([^[:blank:]]*\) (.*)/\1/' | while read lib; do
+	cp $lib $SCRATCH_MNT/
+done
+LD_LIBRARY_PATH=$SCRATCH_MNT $LDD_PROG $SCRATCH_MNT/xfs_scrub >> $seqres.full
+
+echo "Force online repairs"
+XFS_SCRUB_FORCE_REPAIR=1 LD_LIBRARY_PATH=$SCRATCH_MNT $SCRATCH_MNT/xfs_scrub -dTvy $SCRATCH_MNT >> $seqres.full
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1375.out b/tests/xfs/1375.out
new file mode 100644
index 0000000..dd7025e
--- /dev/null
+++ b/tests/xfs/1375.out
@@ -0,0 +1,3 @@
+QA output created by 1375
+Format and populate
+Force online repairs
diff --git a/tests/xfs/1376 b/tests/xfs/1376
new file mode 100755
index 0000000..91ed8dd
--- /dev/null
+++ b/tests/xfs/1376
@@ -0,0 +1,85 @@
+#! /bin/bash
+# FS QA Test No. 1376
+#
+# Race fio and xfs_scrub for a while to see if we crash or livelock.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_test_program "feature"
+_require_command "$KILLALL_PROG" killall
+_require_command "$TIMEOUT_PROG" timeout
+_require_scrub
+_require_scratch
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+
+STRESS_DIR="$SCRATCH_MNT/testdir"
+mkdir -p $STRESS_DIR
+
+cpus=$(( $(src/feature -o) * 4 * LOAD_FACTOR))
+$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
+$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
+
+killstress() {
+	sleep $(( 60 * TIME_FACTOR ))
+	$KILLALL_PROG -q $FSSTRESS_PROG
+}
+
+echo "Concurrent scrub"
+start=$(date +%s)
+end=$((start + (60 * LOAD_FACTOR) ))
+killstress &
+echo "Scrub started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
+while [ "$(date +%s)" -lt "$end" ]; do
+	$TIMEOUT_PROG -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full 2>&1
+done
+
+echo "Test done"
+echo "Scrub finished at $(date)" >> $seqres.full
+$KILLALL_PROG -q $FSSTRESS_PROG
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1376.out b/tests/xfs/1376.out
new file mode 100644
index 0000000..155f424
--- /dev/null
+++ b/tests/xfs/1376.out
@@ -0,0 +1,4 @@
+QA output created by 1376
+Format and populate
+Concurrent scrub
+Test done
diff --git a/tests/xfs/1377 b/tests/xfs/1377
new file mode 100755
index 0000000..601b1dc
--- /dev/null
+++ b/tests/xfs/1377
@@ -0,0 +1,87 @@
+#! /bin/bash
+# FS QA Test No. 1377
+#
+# Race fio and xfs_scrub -y for a while to see if we crash or livelock.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017 Oracle, Inc.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1335  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 7 15
+
+_cleanup()
+{
+	cd /
+	rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/fuzzy
+. ./common/inject
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_test_program "feature"
+_require_command "$KILLALL_PROG" killall
+_require_command "$TIMEOUT_PROG" timeout
+_require_scrub
+_require_scratch
+# xfs_scrub will turn on error injection itself
+_require_xfs_io_error_injection "force_repair"
+
+echo "Format and populate"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount
+
+STRESS_DIR="$SCRATCH_MNT/testdir"
+mkdir -p $STRESS_DIR
+
+cpus=$(( $(src/feature -o) * 4 * LOAD_FACTOR))
+$FSSTRESS_PROG -d $STRESS_DIR -p $cpus -n $((cpus * 100000)) $FSSTRESS_AVOID >/dev/null 2>&1 &
+$XFS_SCRUB_PROG -d -T -v -n $SCRATCH_MNT >> $seqres.full
+
+killstress() {
+	sleep $(( 60 * TIME_FACTOR ))
+	$KILLALL_PROG -q $FSSTRESS_PROG
+}
+
+echo "Concurrent repair"
+start=$(date +%s)
+end=$((start + (60 * LOAD_FACTOR) ))
+killstress &
+echo "Repair started at $(date --date="@${start}"), ending at $(date --date="@${end}")" >> $seqres.full
+while [ "$(date +%s)" -lt "$end" ]; do
+	XFS_SCRUB_FORCE_REPAIR=1 $TIMEOUT_PROG -s TERM $(( end - $(date +%s) + 2 )) $XFS_SCRUB_PROG -d -T -v -y $SCRATCH_MNT >> $seqres.full
+done
+
+echo "Test done"
+echo "Repair finished at $(date)" >> $seqres.full
+$KILLALL_PROG -q $FSSTRESS_PROG
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/1377.out b/tests/xfs/1377.out
new file mode 100644
index 0000000..0454a87
--- /dev/null
+++ b/tests/xfs/1377.out
@@ -0,0 +1,4 @@
+QA output created by 1377
+Format and populate
+Concurrent repair
+Test done
diff --git a/tests/xfs/group b/tests/xfs/group
index de1c39b..0b6c8cf 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -410,3 +410,6 @@
 416 dangerous_fuzzers dangerous_scrub dangerous_repair
 417 dangerous_fuzzers dangerous_scrub dangerous_online_repair
 418 dangerous_fuzzers dangerous_scrub dangerous_repair
+1375 dangerous_fuzzers dangerous_scrub dangerous_online_repair
+1376 dangerous_fuzzers dangerous_scrub
+1377 dangerous_fuzzers dangerous_scrub dangerous_online_repair

Re: [PATCH v2 4/4] xfs: stress test xfs_scrub

[Eryu Guan]

On Tue, Feb 28, 2017 at 12:57:10PM -0800, Darrick J. Wong wrote:
> Three new tests:
> 
> - Repair files that are mapped into memory in running programs
> - Run scrub -n concurrently with fsstress
> - Run scrub -y concurrently with fsstress
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> v2: fix checks for external programs per eryu guan's recommendations
> ---
[snip]
> +
> +killstress() {
> +	sleep $(( 60 * TIME_FACTOR ))
> +	$KILLALL_PROG -q $FSSTRESS_PROG
> +}
> +
> +echo "Concurrent scrub"
> +start=$(date +%s)
> +end=$((start + (60 * LOAD_FACTOR) ))

In 137[67], I think this should be TIME_FACTOR too. I can fix them at
commit time if this change looks OK to you.

Thanks,
Eryu

Re: [PATCH v2 4/4] xfs: stress test xfs_scrub

[Darrick J. Wong]

On Wed, Mar 01, 2017 at 11:33:16AM +0800, Eryu Guan wrote:
> On Tue, Feb 28, 2017 at 12:57:10PM -0800, Darrick J. Wong wrote:
> > Three new tests:
> > 
> > - Repair files that are mapped into memory in running programs
> > - Run scrub -n concurrently with fsstress
> > - Run scrub -y concurrently with fsstress
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> > v2: fix checks for external programs per eryu guan's recommendations
> > ---
> [snip]
> > +
> > +killstress() {
> > +	sleep $(( 60 * TIME_FACTOR ))
> > +	$KILLALL_PROG -q $FSSTRESS_PROG
> > +}
> > +
> > +echo "Concurrent scrub"
> > +start=$(date +%s)
> > +end=$((start + (60 * LOAD_FACTOR) ))
> 
> In 137[67], I think this should be TIME_FACTOR too. I can fix them at
> commit time if this change looks OK to you.

Ugh, I thought I fixed those too.  Yes, the change is correct, thank you.

--D

> 
> Thanks,
> Eryu