[PATCH v2 0/5] Btrfs in-band de-duplication tests cases

[PATCH v2 0/5] Btrfs in-band de-duplication tests cases

[Qu Wenruo]

Since we are push btrfs in-band de-duplication for v4.6, it's better to
add test cases for this new feature.

Except the first basic function test, the rest are all regression test
which we found during the development.
We also found some bugs from the generic test, but we need some xfstests
option allowing us to enable dedup for any test case.
(We did it by hack _scratch_mount and _test_mount to enable dedup for
any test case)

Use the sequence number starts from 200 to avoid any possible conflicts.
The new script returns some hole number which is not that proper for
such related test case set.
Hopes it's not too hard for maintainer to modify the sequence number.

Qu Wenruo (5):
  fstests: rename _require_btrfs to _require_btrfs_subcommand
  fstests: btrfs: Add basic test for btrfs in-band de-duplication
  fstests: btrfs: Add testcase for btrfs dedup enable disable race test
  fstests: btrfs: Add per inode dedup flag test
  fstests: btrfs: Test inband dedup with balance.

 common/defrag       |   8 ++++
 common/rc           |   2 +-
 tests/btrfs/004     |   2 +-
 tests/btrfs/048     |   1 +
 tests/btrfs/059     |   1 +
 tests/btrfs/200     | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/200.out |  19 ++++++++
 tests/btrfs/201     | 100 +++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   1 +
 tests/btrfs/202     | 116 ++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/202.out |  15 +++++++
 tests/btrfs/203     |  91 ++++++++++++++++++++++++++++++++++++++
 tests/btrfs/203.out |   3 ++
 tests/btrfs/group   |   4 ++
 14 files changed, 486 insertions(+), 2 deletions(-)
 create mode 100755 tests/btrfs/200
 create mode 100644 tests/btrfs/200.out
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out
 create mode 100755 tests/btrfs/202
 create mode 100644 tests/btrfs/202.out
 create mode 100755 tests/btrfs/203
 create mode 100644 tests/btrfs/203.out

-- 
2.7.1

[PATCH 1/5] fstests: Add support to check btrfs sysfs features

[Qu Wenruo]

Btrfs has its sysfs interface showing what features current kernel/btrfs
module support.

Add _require_btrfs_kernel_feature() to check such interface.

Also rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/rc       | 2 +-
 tests/btrfs/004 | 2 +-
 tests/btrfs/048 | 1 +
 tests/btrfs/059 | 1 +
 4 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/common/rc b/common/rc
index af16c81..ff57862 100644
--- a/common/rc
+++ b/common/rc
@@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
 }
 
 # We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
 {
 	cmd=$1
 	_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
 _supported_os Linux
 _require_scratch
 _require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
 _require_command "/usr/sbin/filefrag" filefrag
 
 rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index c2cb4a6..d15346a 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,6 +48,7 @@ _supported_os Linux
 _require_test
 _require_scratch
 _require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 send_files_dir=$TEST_DIR/btrfs-test-$seq
 
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index b9a6ef4..6e7f7ee 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,6 +51,7 @@ _supported_os Linux
 _require_test
 _require_scratch
 _require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 rm -f $seqres.full
 
-- 
2.7.1

[PATCH 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand

[Qu Wenruo]

Rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion, as all other _require_btrfs_* has a quite clear suffix, like
_require_btrfs_mkfs_feature() or _require_btrfs_fs_feature().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/rc       | 2 +-
 tests/btrfs/004 | 2 +-
 tests/btrfs/048 | 1 +
 tests/btrfs/059 | 1 +
 4 files changed, 4 insertions(+), 2 deletions(-)

diff --git a/common/rc b/common/rc
index af16c81..ff57862 100644
--- a/common/rc
+++ b/common/rc
@@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
 }
 
 # We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
 {
 	cmd=$1
 	_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
 _supported_os Linux
 _require_scratch
 _require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
 _require_command "/usr/sbin/filefrag" filefrag
 
 rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index c2cb4a6..d15346a 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,6 +48,7 @@ _supported_os Linux
 _require_test
 _require_scratch
 _require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 send_files_dir=$TEST_DIR/btrfs-test-$seq
 
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index b9a6ef4..6e7f7ee 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,6 +51,7 @@ _supported_os Linux
 _require_test
 _require_scratch
 _require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 rm -f $seqres.full
 
-- 
2.7.1

[PATCH 2/5] fstests: btrfs: Add basic test for btrfs in-band de-duplication

[Qu Wenruo]

Add basic test for btrfs in-band de-duplication, including:
1) Enable
2) Re-enable
3) On disk extents are refering to same bytenr
4) Disable

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/defrag       |   8 ++++
 tests/btrfs/200     | 125 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/200.out |  19 ++++++++
 tests/btrfs/group   |   1 +
 4 files changed, 153 insertions(+)
 create mode 100755 tests/btrfs/200
 create mode 100644 tests/btrfs/200.out

diff --git a/common/defrag b/common/defrag
index 942593e..34cc822 100644
--- a/common/defrag
+++ b/common/defrag
@@ -47,6 +47,14 @@ _extent_count()
 	$XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l| $AWK_PROG '{print $1}'
 }
 
+_uniq_extent_count()
+{
+	file=$1
+	$XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
+		$AWK_PROG '{print $3}' | sort | uniq | wc -l
+}
+
 _check_extent_count()
 {
 	min=$1
diff --git a/tests/btrfs/200 b/tests/btrfs/200
new file mode 100755
index 0000000..f2ff542
--- /dev/null
+++ b/tests/btrfs/200
@@ -0,0 +1,125 @@
+#! /bin/bash
+# FS QA Test 200
+#
+# Basic btrfs inband dedup test, including:
+# 1) Enable
+# 2) Uniq file extent number
+# 3) Re-enable
+# 4) Disable
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedup, it will have at least 2 extents
+file_size=$(( 256 * 1024 * 1024 ))
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedup_test()
+{
+	backend=$1
+	dedup_bs=$2
+
+	_run_btrfs_util_prog dedup enable -s $backend -b $dedup_bs $SCRATCH_MNT
+	$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
+		$SCRATCH_MNT/initial_block | _filter_xfs_io
+
+	# sync to ensure dedup hash is added into dedup pool
+	sync
+	$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $file_size" \
+		$SCRATCH_MNT/real_file | _filter_xfs_io
+	# sync again to ensure data are all written to disk and
+	# we can get stable extent map
+	sync
+
+	# Test if real_file is de-duplicated
+	nr_uniq_extents=$(_uniq_extent_count $SCRATCH_MNT/real_file)
+	nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
+
+	echo "uniq/total: $nr_uniq_extents/$nr_total_extents" >> $seqres.full
+	# Allow a small amount of dedup miss, as commit interval or
+	# memory pressure may break a dedup_bs block and cause
+	# smalll extent which won't go through dedup routine
+	if [ $nr_uniq_extents -ge $(( $nr_total_extents * 5 / 100 )) ]; then
+		echo "Too high dedup failure rate"
+	fi
+
+	# Also check the md5sum to ensure data is not corrupted
+	md5=$(_md5_checksum $SCRATCH_MNT/real_file)
+	if [ $md5 != $init_md5 ]; then
+		echo "File after in-band de-duplication is corrupted"
+	fi
+}
+
+# Create the initial file and calculate its checksum without dedup
+$XFS_IO_PROG -f -c "pwrite 0 $file_size" $SCRATCH_MNT/csum_file | \
+	_filter_xfs_io
+init_md5=$(_md5_checksum $SCRATCH_MNT/csum_file)
+echo "md5 of the initial file is $init_md5" >> $seqres.full
+
+# Test inmemory dedup first, use 64K dedup bs to keep compatibility
+# with 64K page size
+do_dedup_test inmemory 64K
+
+# Test ondisk backend, and re-enable function
+do_dedup_test ondisk 64K
+
+# Test 128K(default) dedup bs
+do_dedup_test inmemory 128K
+do_dedup_test ondisk 128K
+
+# Check dedup disable
+_run_btrfs_util_prog dedup disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out
new file mode 100644
index 0000000..4b185be
--- /dev/null
+++ b/tests/btrfs/200.out
@@ -0,0 +1,19 @@
+QA output created by 200
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index a2fa412..0b7354b 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -119,3 +119,4 @@
 116 auto quick metadata
 117 auto quick send clone
 118 auto quick snapshot metadata
+200 auto dedup
-- 
2.7.1

[PATCH 3/5] fstests: btrfs: Add testcase for btrfs dedup enable disable race test

[Qu Wenruo]

Add test case to check btrfs dedup enable/disable race.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/btrfs/201     | 100 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   1 +
 tests/btrfs/group   |   1 +
 3 files changed, 102 insertions(+)
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out

diff --git a/tests/btrfs/201 b/tests/btrfs/201
new file mode 100755
index 0000000..4bcad13
--- /dev/null
+++ b/tests/btrfs/201
@@ -0,0 +1,100 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Basic btrfs inband dedup enable/disable race test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	kill $trigger_work &> /dev/null
+	kill $fsstress_work &> /dev/null
+	wait
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# Use 64K dedup size to keep compatibility for 64K page size
+dedup_bs=64K
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+mkdir -p $SCRATCH_MNT/stressdir
+
+fsstress_work()
+{
+	$FSSTRESS_PROG $(_scale_fsstress_args -p 8 -n 5000) $FSSTRESS_AVOID \
+		-d $SCRATCH_MNT/stressdir > /dev/null 2>&1
+}
+
+trigger_work()
+{
+	while true; do
+		_run_btrfs_util_prog dedup enable -s inmemory \
+			-b $dedup_bs $SCRATCH_MNT
+		sleep 5
+		_run_btrfs_util_prog dedup disable $SCRATCH_MNT
+		sleep 5
+		_run_btrfs_util_prog dedup enable -s ondisk \
+			-b $dedup_bs $SCRATCH_MNT
+		sleep 5
+		_run_btrfs_util_prog dedup disable $SCRATCH_MNT
+		sleep 5
+	done
+}
+
+fsstress_work &
+fsstress_pid=$!
+
+trigger_work &
+trigger_pid=$!
+
+wait $fsstress_pid
+kill $trigger_pid
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/201.out b/tests/btrfs/201.out
new file mode 100644
index 0000000..b8969af
--- /dev/null
+++ b/tests/btrfs/201.out
@@ -0,0 +1 @@
+QA output created by 201
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 0b7354b..76ebea7 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -120,3 +120,4 @@
 117 auto quick send clone
 118 auto quick snapshot metadata
 200 auto dedup
+201 auto dedup
-- 
2.7.1

[PATCH 4/5] fstests: btrfs: Add per inode dedup flag test

[Qu Wenruo]

This test will check per inode dedup flag.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/btrfs/202     | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/202.out |  15 +++++++
 tests/btrfs/group   |   1 +
 3 files changed, 132 insertions(+)
 create mode 100755 tests/btrfs/202
 create mode 100644 tests/btrfs/202.out

diff --git a/tests/btrfs/202 b/tests/btrfs/202
new file mode 100755
index 0000000..3e5d470
--- /dev/null
+++ b/tests/btrfs/202
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 202
+#
+# Btrfs per inode dedup flag test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedup
+_require_btrfs_subcommand property
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedup, it will have at least 2 extents
+file_size=$(( 256 * 1024 * 1024 ))
+dedup_bs=$(( 64 * 1024 ))
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+# Return 0 for not deduped at all , return 1 for part or full deduped
+test_file_deduped () {
+	file=$1
+
+	nr_uniq_extents=$(_uniq_extent_count $file)
+	nr_total_extents=$(_extent_count $file)
+
+	if [ $nr_uniq_extents -eq $nr_total_extents ]; then
+		echo "not de-duplicated"
+	else
+		echo "de-duplicated"
+	fi
+}
+
+dedup_write_file () {
+	file=$1
+	size=$2
+
+	$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $size" $file | _filter_xfs_io
+}
+
+print_result () {
+	file=$1
+
+	echo "$(basename $file): $(test_file_deduped $file)"
+}
+_run_btrfs_util_prog dedup enable -b $dedup_bs $SCRATCH_MNT
+touch $SCRATCH_MNT/dedup_file
+touch $SCRATCH_MNT/no_dedup_file
+mkdir $SCRATCH_MNT/dedup_dir
+mkdir $SCRATCH_MNT/no_dedup_dir
+
+_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_file dedup disable
+_run_btrfs_util_prog property set $SCRATCH_MNT/no_dedup_dir dedup disable
+
+dedup_write_file $SCRATCH_MNT/tmp $dedup_bs
+# sync to ensure hash is added to dedup tree
+sync
+
+dedup_write_file $SCRATCH_MNT/dedup_file $file_size
+dedup_write_file $SCRATCH_MNT/no_dedup_file $file_size
+dedup_write_file $SCRATCH_MNT/dedup_dir/dedup_dir_default_file $file_size
+dedup_write_file $SCRATCH_MNT/no_dedup_dir/no_dedup_dir_default_file $file_size
+
+print_result $SCRATCH_MNT/dedup_file
+print_result $SCRATCH_MNT/no_dedup_file
+print_result $SCRATCH_MNT/dedup_dir/dedup_dir_default_file
+print_result $SCRATCH_MNT/no_dedup_dir/no_dedup_dir_default_file
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/202.out b/tests/btrfs/202.out
new file mode 100644
index 0000000..ced9e88
--- /dev/null
+++ b/tests/btrfs/202.out
@@ -0,0 +1,15 @@
+QA output created by 202
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+dedup_file: de-duplicated
+no_dedup_file: not de-duplicated
+dedup_dir_default_file: de-duplicated
+no_dedup_dir_default_file: not de-duplicated
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 76ebea7..0c03cf1 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -121,3 +121,4 @@
 118 auto quick snapshot metadata
 200 auto dedup
 201 auto dedup
+202 auto dedup
-- 
2.7.1

[PATCH 5/5] fstests: btrfs: Test inband dedup with balance.

[Qu Wenruo]

Btrfs balance will reloate date extent, but its hash is removed too late
at run_delayed_ref() time, which will cause extent ref increased
increased during balance, cause either find_data_references() gives
WARN_ON() or even run_delayed_refs() fails and cause transaction abort.

Add such concurrency test for inband dedup and balance.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/btrfs/203     | 91 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/203.out |  3 ++
 tests/btrfs/group   |  1 +
 3 files changed, 95 insertions(+)
 create mode 100755 tests/btrfs/203
 create mode 100644 tests/btrfs/203.out

diff --git a/tests/btrfs/203 b/tests/btrfs/203
new file mode 100755
index 0000000..19dc55c
--- /dev/null
+++ b/tests/btrfs/203
@@ -0,0 +1,91 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Btrfs reflink with balance concurrency test
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	kill $balance_pid &> /dev/null
+	wait
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_cp_reflink
+_require_btrfs_subcommand dedup
+_require_btrfs_fs_feature dedup
+_require_btrfs_mkfs_feature dedup
+
+dedup_bs=$(( 128 * 1024 ))
+file=$SCRATCH_MNT/foo
+nr=20000
+
+_scratch_mkfs "-O dedup" >> $seqres.full 2>&1
+_scratch_mount
+
+_run_btrfs_util_prog dedup enable -b $dedup_bs $SCRATCH_MNT
+
+# create the initial file
+$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" $file | _filter_xfs_io
+
+# make sure hash is added into hash pool
+sync
+
+_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
+balance_pid=$!
+
+for n in $(seq 1 $nr); do
+	$XFS_IO_PROG -f -c "pwrite -b $dedup_bs 0 $dedup_bs" \
+		${file}_${n} > /dev/null 2>&1
+done
+
+kill $balance_pid &> /dev/null
+wait
+
+# Sometimes even we killed $balance_pid and wait returned,
+# balance may still be running, use balance cancel to wait it.
+_run_btrfs_util_prog balance cancel $SCRATCH_MNT &> /dev/null
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/203.out b/tests/btrfs/203.out
new file mode 100644
index 0000000..404394c
--- /dev/null
+++ b/tests/btrfs/203.out
@@ -0,0 +1,3 @@
+QA output created by 203
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 0c03cf1..fa90f33 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -122,3 +122,4 @@
 200 auto dedup
 201 auto dedup
 202 auto dedup
+203 auto dedup balance
-- 
2.7.1

Re: [PATCH 1/5] fstests: Add support to check btrfs sysfs features

[Filipe Manana]

On Wed, Feb 24, 2016 at 6:35 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
> Btrfs has its sysfs interface showing what features current kernel/btrfs
> module support.
>
> Add _require_btrfs_kernel_feature() to check such interface.


I think you sent the wrong patch. This doesn't add such a function and
the changes are exactly the same as in:

[PATCH 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand

>
> Also rename _require_btrfs() to _require_btrfs_subcommand() to avoid
> confusion.

So if there's a dedicated patch to do that already (the one I
mentioned above), why do it here again? (and should be a separate
patch anyway, since it's unrelated)

>
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>  common/rc       | 2 +-
>  tests/btrfs/004 | 2 +-
>  tests/btrfs/048 | 1 +
>  tests/btrfs/059 | 1 +
>  4 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/common/rc b/common/rc
> index af16c81..ff57862 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
>  }
>
>  # We check for btrfs and (optionally) features of the btrfs command
> -_require_btrfs()
> +_require_btrfs_subcommand()
>  {
>         cmd=$1
>         _require_command "$BTRFS_UTIL_PROG" btrfs
> diff --git a/tests/btrfs/004 b/tests/btrfs/004
> index 905770a..2ce628e 100755
> --- a/tests/btrfs/004
> +++ b/tests/btrfs/004
> @@ -51,7 +51,7 @@ _supported_fs btrfs
>  _supported_os Linux
>  _require_scratch
>  _require_no_large_scratch_dev
> -_require_btrfs inspect-internal
> +_require_btrfs_subcommand inspect-internal
>  _require_command "/usr/sbin/filefrag" filefrag
>
>  rm -f $seqres.full
> diff --git a/tests/btrfs/048 b/tests/btrfs/048
> index c2cb4a6..d15346a 100755
> --- a/tests/btrfs/048
> +++ b/tests/btrfs/048
> @@ -48,6 +48,7 @@ _supported_os Linux
>  _require_test
>  _require_scratch
>  _require_btrfs "property"
> +_require_btrfs_subcommand "property"
>
>  send_files_dir=$TEST_DIR/btrfs-test-$seq
>
> diff --git a/tests/btrfs/059 b/tests/btrfs/059
> index b9a6ef4..6e7f7ee 100755
> --- a/tests/btrfs/059
> +++ b/tests/btrfs/059
> @@ -51,6 +51,7 @@ _supported_os Linux
>  _require_test
>  _require_scratch
>  _require_btrfs "property"
> +_require_btrfs_subcommand "property"
>
>  rm -f $seqres.full
>
> --
> 2.7.1
>
>
>
> --
> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html



-- 
Filipe David Manana,

"Reasonable men adapt themselves to the world.
 Unreasonable men adapt the world to themselves.
 That's why all progress depends on unreasonable men."

Re: [PATCH 1/5] fstests: Add support to check btrfs sysfs features

[Qu Wenruo]

Filipe Manana wrote on 2016/02/24 07:27 +0000:
> On Wed, Feb 24, 2016 at 6:35 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Btrfs has its sysfs interface showing what features current kernel/btrfs
>> module support.
>>
>> Add _require_btrfs_kernel_feature() to check such interface.
>
>
> I think you sent the wrong patch. This doesn't add such a function and
> the changes are exactly the same as in:
>
> [PATCH 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand

Oh, this is one old and deprecated patch.
I forgot to cleanup the dir...

Please ignore this one.

The other one, "[PATCH 1/5] fstests: rename _require_btrfs to
  _require_btrfs_subcommand" is the correct one.

As fstests already provide _btrfs_require_fs_feature().

I'll send the patchset.

Thanks,
Qu
>
>>
>> Also rename _require_btrfs() to _require_btrfs_subcommand() to avoid
>> confusion.
>
> So if there's a dedicated patch to do that already (the one I
> mentioned above), why do it here again? (and should be a separate
> patch anyway, since it's unrelated)
>
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>>   common/rc       | 2 +-
>>   tests/btrfs/004 | 2 +-
>>   tests/btrfs/048 | 1 +
>>   tests/btrfs/059 | 1 +
>>   4 files changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/common/rc b/common/rc
>> index af16c81..ff57862 100644
>> --- a/common/rc
>> +++ b/common/rc
>> @@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
>>   }
>>
>>   # We check for btrfs and (optionally) features of the btrfs command
>> -_require_btrfs()
>> +_require_btrfs_subcommand()
>>   {
>>          cmd=$1
>>          _require_command "$BTRFS_UTIL_PROG" btrfs
>> diff --git a/tests/btrfs/004 b/tests/btrfs/004
>> index 905770a..2ce628e 100755
>> --- a/tests/btrfs/004
>> +++ b/tests/btrfs/004
>> @@ -51,7 +51,7 @@ _supported_fs btrfs
>>   _supported_os Linux
>>   _require_scratch
>>   _require_no_large_scratch_dev
>> -_require_btrfs inspect-internal
>> +_require_btrfs_subcommand inspect-internal
>>   _require_command "/usr/sbin/filefrag" filefrag
>>
>>   rm -f $seqres.full
>> diff --git a/tests/btrfs/048 b/tests/btrfs/048
>> index c2cb4a6..d15346a 100755
>> --- a/tests/btrfs/048
>> +++ b/tests/btrfs/048
>> @@ -48,6 +48,7 @@ _supported_os Linux
>>   _require_test
>>   _require_scratch
>>   _require_btrfs "property"
>> +_require_btrfs_subcommand "property"
>>
>>   send_files_dir=$TEST_DIR/btrfs-test-$seq
>>
>> diff --git a/tests/btrfs/059 b/tests/btrfs/059
>> index b9a6ef4..6e7f7ee 100755
>> --- a/tests/btrfs/059
>> +++ b/tests/btrfs/059
>> @@ -51,6 +51,7 @@ _supported_os Linux
>>   _require_test
>>   _require_scratch
>>   _require_btrfs "property"
>> +_require_btrfs_subcommand "property"
>>
>>   rm -f $seqres.full
>>
>> --
>> 2.7.1
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>

Re: [PATCH 1/5] fstests: Add support to check btrfs sysfs features

[Qu Wenruo]

Filipe Manana wrote on 2016/02/24 07:27 +0000:
> On Wed, Feb 24, 2016 at 6:35 AM, Qu Wenruo <quwenruo@cn.fujitsu.com> wrote:
>> Btrfs has its sysfs interface showing what features current kernel/btrfs
>> module support.
>>
>> Add _require_btrfs_kernel_feature() to check such interface.
>
>
> I think you sent the wrong patch. This doesn't add such a function and
> the changes are exactly the same as in:
>
> [PATCH 1/5] fstests: rename _require_btrfs to _require_btrfs_subcommand

Oh, this is one old and deprecated patch.
I forgot to cleanup the dir...

Please ignore this one.

The other one, "[PATCH 1/5] fstests: rename _require_btrfs to
  _require_btrfs_subcommand" is the correct one.

As fstests already provide _btrfs_require_fs_feature().

I'll send the patchset.

Thanks,
Qu
>
>>
>> Also rename _require_btrfs() to _require_btrfs_subcommand() to avoid
>> confusion.
>
> So if there's a dedicated patch to do that already (the one I
> mentioned above), why do it here again? (and should be a separate
> patch anyway, since it's unrelated)
>
>>
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>>   common/rc       | 2 +-
>>   tests/btrfs/004 | 2 +-
>>   tests/btrfs/048 | 1 +
>>   tests/btrfs/059 | 1 +
>>   4 files changed, 4 insertions(+), 2 deletions(-)
>>
>> diff --git a/common/rc b/common/rc
>> index af16c81..ff57862 100644
>> --- a/common/rc
>> +++ b/common/rc
>> @@ -2706,7 +2706,7 @@ _require_deletable_scratch_dev_pool()
>>   }
>>
>>   # We check for btrfs and (optionally) features of the btrfs command
>> -_require_btrfs()
>> +_require_btrfs_subcommand()
>>   {
>>          cmd=$1
>>          _require_command "$BTRFS_UTIL_PROG" btrfs
>> diff --git a/tests/btrfs/004 b/tests/btrfs/004
>> index 905770a..2ce628e 100755
>> --- a/tests/btrfs/004
>> +++ b/tests/btrfs/004
>> @@ -51,7 +51,7 @@ _supported_fs btrfs
>>   _supported_os Linux
>>   _require_scratch
>>   _require_no_large_scratch_dev
>> -_require_btrfs inspect-internal
>> +_require_btrfs_subcommand inspect-internal
>>   _require_command "/usr/sbin/filefrag" filefrag
>>
>>   rm -f $seqres.full
>> diff --git a/tests/btrfs/048 b/tests/btrfs/048
>> index c2cb4a6..d15346a 100755
>> --- a/tests/btrfs/048
>> +++ b/tests/btrfs/048
>> @@ -48,6 +48,7 @@ _supported_os Linux
>>   _require_test
>>   _require_scratch
>>   _require_btrfs "property"
>> +_require_btrfs_subcommand "property"
>>
>>   send_files_dir=$TEST_DIR/btrfs-test-$seq
>>
>> diff --git a/tests/btrfs/059 b/tests/btrfs/059
>> index b9a6ef4..6e7f7ee 100755
>> --- a/tests/btrfs/059
>> +++ b/tests/btrfs/059
>> @@ -51,6 +51,7 @@ _supported_os Linux
>>   _require_test
>>   _require_scratch
>>   _require_btrfs "property"
>> +_require_btrfs_subcommand "property"
>>
>>   rm -f $seqres.full
>>
>> --
>> 2.7.1
>>
>>
>>
>> --
>> To unsubscribe from this list: send the line "unsubscribe linux-btrfs" in
>> the body of a message to majordomo@vger.kernel.org
>> More majordomo info at  http://vger.kernel.org/majordomo-info.html
>
>
>