[PATCH v6 0/6] Btrfs in-band de-duplication test cases

[PATCH v6 0/6] Btrfs in-band de-duplication test cases

[Qu Wenruo]

Btrfs in-band de-duplication test case for in-memory backend.
With extra option ALWAYS_ENABLE_BTRFS_FEATURE macro to enable dedupe/quota
for all test cases.
This is quite handy to hugely increase the coverage without introducing a lot
new test cases.

v6:
  Introduce ALWAYS_ENABLE_BTRFS_FEATURE macro to enable dedupe/quota for all
  existing test cases.
v5:
  Due to kernel ioctl change, add FORCE flag for "dedupe enable" ioctl call.
v4:
  Due to kernel patchset re-organization, remove on-disk backend test cases
v3:
  Add new test cases for on-disk backend with metadata balance
v2:
  Add new test cases for on-disk backend with full balance


Qu Wenruo (6):
  fstests: common: Introduce _post_mount_hook for btrfs
  fstests: common: rename _require_btrfs to _require_btrfs_subcommand
  fstests: Add btrfs dedupe post mount hook
  fstests: btrfs: Add basic test for btrfs in-band de-duplication
  fstests: btrfs: Add testcase for btrfs dedupe and metadata balance
    race test
  fstests: btrfs: Test inband dedupe with data balance.

 common/defrag       |  13 ++++++
 common/rc           |  41 ++++++++++++++++++-
 tests/btrfs/004     |   2 +-
 tests/btrfs/048     |   2 +-
 tests/btrfs/059     |   2 +-
 tests/btrfs/200     | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/200.out |  22 ++++++++++
 tests/btrfs/201     | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   2 +
 tests/btrfs/203     | 110 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/203.out |   3 ++
 tests/btrfs/group   |   3 ++
 12 files changed, 424 insertions(+), 4 deletions(-)
 create mode 100755 tests/btrfs/200
 create mode 100644 tests/btrfs/200.out
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out
 create mode 100755 tests/btrfs/203
 create mode 100644 tests/btrfs/203.out

-- 
2.7.4

[PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs

[Qu Wenruo]

Introduce _post_mount_hook(), which will be executed after mounting
scratch/test.

It's quite useful for fs(OK, only btrfs yet, again) which needs to
use ioctl other than mount option to enable some of its feature.

Now only btrfs quota needs this hook to allow enabling quota to be
enabled for *ALL* existing test cases.

This should dramatically improve the test coverage to expose quota
related bugs.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/rc | 23 +++++++++++++++++++++++
 1 file changed, 23 insertions(+)

diff --git a/common/rc b/common/rc
index 23c007a..631397f 100644
--- a/common/rc
+++ b/common/rc
@@ -321,6 +321,27 @@ _overlay_scratch_unmount()
 	$UMOUNT_PROG $SCRATCH_MNT
 }
 
+_run_btrfs_post_mount_hook()
+{
+	mnt_point=$1
+	for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
+		if [ $n == "quota" -o $n == "qgroup" ]; then
+			# Quota can be enabled for several times
+			# and won't cause bug
+			_run_btrfs_util_prog quota enable $mnt_point
+		fi
+	done
+}
+
+_post_mount_hook()
+{
+	mnt_point=$1
+
+	if [ $FSTYP == "btrfs" -a -v ALWAYS_ENABLE_BTRFS_FEATURE ]; then
+		_run_btrfs_post_mount_hook $mnt_point
+	fi
+}
+
 _scratch_mount()
 {
     if [ "$FSTYP" == "overlay" ]; then
@@ -328,6 +349,7 @@ _scratch_mount()
         return $?
     fi
     _mount -t $FSTYP `_scratch_mount_options $*`
+    _post_mount_hook $SCRATCH_MNT
 }
 
 _scratch_unmount()
@@ -377,6 +399,7 @@ _test_mount()
     fi
     _test_options mount
     _mount -t $FSTYP $TEST_OPTIONS $TEST_FS_MOUNT_OPTS $SELINUX_MOUNT_OPTIONS $* $TEST_DEV $TEST_DIR
+    _post_mount_hook $TEST_DIR
 }
 
 _test_unmount()
-- 
2.7.4

[PATCH v6 2/6] fstests: common: rename _require_btrfs to _require_btrfs_subcommand

[Qu Wenruo]

Rename _require_btrfs() to _require_btrfs_subcommand() to avoid
confusion, as all other _require_btrfs_* has a quite clear suffix, like
_require_btrfs_mkfs_feature() or _require_btrfs_fs_feature().

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/rc       | 2 +-
 tests/btrfs/004 | 2 +-
 tests/btrfs/048 | 2 +-
 tests/btrfs/059 | 2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/common/rc b/common/rc
index 631397f..636cba6 100644
--- a/common/rc
+++ b/common/rc
@@ -3001,7 +3001,7 @@ _require_deletable_scratch_dev_pool()
 }
 
 # We check for btrfs and (optionally) features of the btrfs command
-_require_btrfs()
+_require_btrfs_subcommand()
 {
 	cmd=$1
 	_require_command "$BTRFS_UTIL_PROG" btrfs
diff --git a/tests/btrfs/004 b/tests/btrfs/004
index 905770a..2ce628e 100755
--- a/tests/btrfs/004
+++ b/tests/btrfs/004
@@ -51,7 +51,7 @@ _supported_fs btrfs
 _supported_os Linux
 _require_scratch
 _require_no_large_scratch_dev
-_require_btrfs inspect-internal
+_require_btrfs_subcommand inspect-internal
 _require_command "/usr/sbin/filefrag" filefrag
 
 rm -f $seqres.full
diff --git a/tests/btrfs/048 b/tests/btrfs/048
index 0b907b0..ac731d1 100755
--- a/tests/btrfs/048
+++ b/tests/btrfs/048
@@ -48,7 +48,7 @@ _supported_fs btrfs
 _supported_os Linux
 _require_test
 _require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 send_files_dir=$TEST_DIR/btrfs-test-$seq
 
diff --git a/tests/btrfs/059 b/tests/btrfs/059
index 8f106d2..fd67ebb 100755
--- a/tests/btrfs/059
+++ b/tests/btrfs/059
@@ -51,7 +51,7 @@ _supported_fs btrfs
 _supported_os Linux
 _require_test
 _require_scratch
-_require_btrfs "property"
+_require_btrfs_subcommand "property"
 
 rm -f $seqres.full
 
-- 
2.7.4

[PATCH v6 3/6] fstests: Add btrfs dedupe post mount hook

[Qu Wenruo]

Now fstests can run any test cases with btrfs inband-dedupe enabled.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/rc | 20 ++++++++++++++++++--
 1 file changed, 18 insertions(+), 2 deletions(-)

diff --git a/common/rc b/common/rc
index 636cba6..e0da69b 100644
--- a/common/rc
+++ b/common/rc
@@ -326,10 +326,26 @@ _run_btrfs_post_mount_hook()
 	mnt_point=$1
 	for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
 		if [ $n == "quota" -o $n == "qgroup" ]; then
-			# Quota can be enabled for several times
-			# and won't cause bug
+			# Re-enable quota won't cause anything wrong
 			_run_btrfs_util_prog quota enable $mnt_point
 		fi
+		if [ $n == "dedupe" -o $n == "inband-dedupe" -o $n == "dedupe-inband" ]; then
+			# Need to check if kernel/progs support dedupe
+			_require_btrfs_fs_feature dedupe
+			_require_btrfs_subcommand dedupe
+
+			# Check if dedupe is already enabled, or we will
+			# overwrite previous on-disk backend config
+			$BTRFS_UTIL_PROG dedupe status $mnt_point | grep -q "Enabled"
+			if [ $1 -eq 0 ]; then
+				continue
+			fi
+
+			# This will enable dedupe with default options, so
+			# btrfs dedupe test cases needs to use '-f' option to
+			# override default config.
+			_run_btrfs_util_prog dedupe enable $mnt_point
+		fi
 	done
 }
 
-- 
2.7.4

[PATCH v6 4/6] fstests: btrfs: Add basic test for btrfs in-band de-duplication

[Qu Wenruo]

Add basic test for btrfs in-band de-duplication(inmemory backend), including:
1) Enable
3) Dedup rate
4) File correctness
5) Disable

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 common/defrag       |  13 ++++++
 tests/btrfs/200     | 116 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/200.out |  22 ++++++++++
 tests/btrfs/group   |   1 +
 4 files changed, 152 insertions(+)
 create mode 100755 tests/btrfs/200
 create mode 100644 tests/btrfs/200.out

diff --git a/common/defrag b/common/defrag
index 986b4bf..73c7a99 100644
--- a/common/defrag
+++ b/common/defrag
@@ -59,6 +59,19 @@ _extent_count()
 	$XFS_IO_PROG -c "fiemap" $1 | tail -n +2 | grep -v hole | wc -l| $AWK_PROG '{print $1}'
 }
 
+# Get the number of unique file extents
+# Unique file extents means they have different ondisk bytenr
+# Some filesystem supports reflinkat() or in-band de-dup can create
+# a file whose all file extents points to the same ondisk bytenr
+# this can be used to test if such reflinkat() or in-band de-dup works
+_extent_count_uniq()
+{
+	file=$1
+	$XFS_IO_PROG -c "fiemap" $file >> $seqres.full 2>&1
+	$XFS_IO_PROG -c "fiemap" $file | tail -n +2 | grep -v hole |\
+		$AWK_PROG '{print $3}' | sort | uniq | wc -l
+}
+
 _check_extent_count()
 {
 	min=$1
diff --git a/tests/btrfs/200 b/tests/btrfs/200
new file mode 100755
index 0000000..5a190dd
--- /dev/null
+++ b/tests/btrfs/200
@@ -0,0 +1,116 @@
+#! /bin/bash
+# FS QA Test 200
+#
+# Basic btrfs inband dedupe test for inmemory backend
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/defrag
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# File size is twice the maximum file extent of btrfs
+# So even fallbacked to non-dedupe, it will have at least 2 extents
+file_size=256m
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_test()
+{
+	dedupe_bs=$1
+
+	echo "Testing inmemory dedupe backend with block size $dedupe_bs"
+	_run_btrfs_util_prog dedupe enable -f -s inmemory -b $dedupe_bs \
+		$SCRATCH_MNT
+	# do sync write to ensure dedupe hash is added into dedupe pool
+	$XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $dedupe_bs" -c "fsync"\
+		$SCRATCH_MNT/initial_block | _filter_xfs_io
+
+	# do sync write to ensure we can get stable fiemap later
+	$XFS_IO_PROG -f -c "pwrite -b $dedupe_bs 0 $file_size" -c "fsync"\
+		$SCRATCH_MNT/real_file | _filter_xfs_io
+
+	# Test if real_file is de-duplicated
+	nr_uniq_extents=$(_extent_count_uniq $SCRATCH_MNT/real_file)
+	nr_total_extents=$(_extent_count $SCRATCH_MNT/real_file)
+	nr_deduped_extents=$(($nr_total_extents - $nr_uniq_extents))
+
+	echo "deduped/total: $nr_deduped_extents/$nr_total_extents" \
+		>> $seqres.full
+	# Allow a small amount of dedupe miss, as commit interval or
+	# memory pressure may break a dedupe_bs block and cause
+	# small extent which won't go through dedupe routine
+	_within_tolerance "number of deduped extents" $nr_deduped_extents \
+		$nr_total_extents 5% -v
+
+	# Also check the md5sum to ensure data is not corrupted
+	md5=$(_md5_checksum $SCRATCH_MNT/real_file)
+	echo "md5sum: $md5"
+}
+
+# Test inmemory dedupe first, use 64K dedupe bs to keep compatibility
+# with 64K page size
+do_dedupe_test 64K
+
+# Test 128K(default) dedupe bs
+do_dedupe_test 128K
+
+# Test 1M dedupe bs
+do_dedupe_test 1M
+
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
+# Check dedupe disable
+_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/200.out b/tests/btrfs/200.out
new file mode 100644
index 0000000..e09e573
--- /dev/null
+++ b/tests/btrfs/200.out
@@ -0,0 +1,22 @@
+QA output created by 200
+Testing inmemory dedupe backend with block size 64K
+wrote 65536/65536 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
+Testing inmemory dedupe backend with block size 128K
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
+Testing inmemory dedupe backend with block size 1M
+wrote 1048576/1048576 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+wrote 268435456/268435456 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
+number of deduped extents is in range
+md5sum: a30e0f3f1b0884081de11d4357811c2e
diff --git a/tests/btrfs/group b/tests/btrfs/group
index f3a7a4f..ea9c36b 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -133,3 +133,4 @@
 128 auto quick send
 129 auto quick send
 130 auto clone send
+200 auto ib-dedupe
-- 
2.7.4

[PATCH v6 5/6] fstests: btrfs: Add testcase for btrfs dedupe and metadata balance race test

[Qu Wenruo]

Btrfs balance with inband dedupe enable/disable will expose a lot of
hidden dedupe bug:

1) Enable/disable race bug
2) Btrfs dedupe tree balance corrupted delayed_ref
3) Btrfs disable and balance will cause balance BUG_ON

Reported-by: Satoru Takeuchi <takeuchi_satoru@jp.fujitsu.com>
Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/btrfs/201     | 112 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/201.out |   2 +
 tests/btrfs/group   |   1 +
 3 files changed, 115 insertions(+)
 create mode 100755 tests/btrfs/201
 create mode 100644 tests/btrfs/201.out

diff --git a/tests/btrfs/201 b/tests/btrfs/201
new file mode 100755
index 0000000..152e9f7
--- /dev/null
+++ b/tests/btrfs/201
@@ -0,0 +1,112 @@
+#! /bin/bash
+# FS QA Test 201
+#
+# Btrfs inband dedup enable/disable race with metadata balance
+#
+# This tests will test the following bugs exposed in development:
+# 1) enable/disable race
+# 2) tree balance cause delayed ref corruption
+# 3) disable and balance cause BUG_ON
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	killall $FSSTRESS_PROG &> /dev/null
+	kill $trigger_pid &> /dev/null
+	kill $balance_pid &> /dev/null
+	wait
+
+	# See comment later
+	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+# Use 64K dedupe size to keep compatibility for 64K page size
+dedupe_bs=64K
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+mkdir -p $SCRATCH_MNT/stressdir
+
+runtime=$((60 * $TIME_FACTOR))
+
+trigger_work()
+{
+	while true; do
+		_run_btrfs_util_prog dedupe enable -f -s inmemory \
+			-b $dedupe_bs $SCRATCH_MNT
+		sleep 1
+		_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+		sleep 1
+	done
+}
+
+# redirect all output, as error output like 'balance cancelled by user'
+# will populuate the golden output.
+_btrfs_stress_balance -m $SCRATCH_MNT &> /dev/null &
+balance_pid=$!
+
+$FSSTRESS_PROG $(_scale_fsstress_args -p 1 -n 10000000) $FSSTRESS_AVOID \
+	-d $SCRATCH_MNT/stressdir > /dev/null 2>&1 &
+
+trigger_work &
+trigger_pid=$!
+
+sleep $runtime
+killall $FSSTRESS_PROG &> /dev/null
+kill $trigger_pid &> /dev/null
+kill $balance_pid &> /dev/null
+wait
+
+# Manually stop balance as it's possible balance is still running for a short
+# time. And we don't want to populate $seqres.full, so call $BTRFS_UTIL_PROG
+# directly
+$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+echo "Silence is golden"
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/201.out b/tests/btrfs/201.out
new file mode 100644
index 0000000..5ac973f
--- /dev/null
+++ b/tests/btrfs/201.out
@@ -0,0 +1,2 @@
+QA output created by 201
+Silence is golden
diff --git a/tests/btrfs/group b/tests/btrfs/group
index ea9c36b..2e539d0 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -134,3 +134,4 @@
 129 auto quick send
 130 auto clone send
 200 auto ib-dedupe
+201 auto ib-dedupe
-- 
2.7.4

[PATCH v6 6/6] fstests: btrfs: Test inband dedupe with data balance.

[Qu Wenruo]

Btrfs balance will reloate date extent, but its hash is removed too late
at run_delayed_ref() time, which will cause extent ref increased
during balance, cause either find_data_references() gives WARN_ON()
or even run_delayed_refs() fails and cause transaction abort.

Add such concurrency test for inband dedupe and data balance.

Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
---
 tests/btrfs/203     | 110 ++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/btrfs/203.out |   3 ++
 tests/btrfs/group   |   1 +
 3 files changed, 114 insertions(+)
 create mode 100755 tests/btrfs/203
 create mode 100644 tests/btrfs/203.out

diff --git a/tests/btrfs/203 b/tests/btrfs/203
new file mode 100755
index 0000000..96af734
--- /dev/null
+++ b/tests/btrfs/203
@@ -0,0 +1,110 @@
+#! /bin/bash
+# FS QA Test 203
+#
+# Btrfs inband dedupe with balance concurrency test
+#
+# This can spot inband dedupe error which will increase delayed ref on
+# an data extent inside RO block group
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2016 Fujitsu.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+#
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	kill $populate_pid &> /dev/null
+	kill $balance_pid &> /dev/null
+	wait
+	# Check later comment for reason
+	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+	rm -f $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/reflink
+
+# remove previous $seqres.full before test
+rm -f $seqres.full
+
+# real QA test starts here
+
+_supported_fs btrfs
+_supported_os Linux
+_require_scratch
+_require_cp_reflink
+_require_btrfs_subcommand dedupe
+_require_btrfs_fs_feature dedupe
+
+dedupe_bs=128k
+file_size_in_kilo=4096
+init_file=$SCRATCH_MNT/foo
+run_time=$((60 * $TIME_FACTOR))
+
+_scratch_mkfs >> $seqres.full 2>&1
+_scratch_mount
+
+do_dedupe_balance_test()
+{
+	_run_btrfs_util_prog dedupe enable -f -b $dedupe_bs -s inmemory \
+		$SCRATCH_MNT
+
+	# create the initial file and fill hash pool
+	$XFS_IO_PROG -f -c "pwrite -S 0x0 -b $dedupe_bs 0 $dedupe_bs" -c "fsync" \
+		$init_file | _filter_xfs_io
+
+	_btrfs_stress_balance $SCRATCH_MNT >/dev/null 2>&1 &
+	balance_pid=$!
+
+	# Populate fs with all 0 data, to trigger enough in-band dedupe work
+	# to race with balance
+	_populate_fs -n 5 -f 10000000 -d 1 -r $SCRATCH_MNT \
+		-s $file_size_in_kilo &> /dev/null &
+	populate_pid=$!
+
+	sleep $run_time
+
+	kill $populate_pid
+	kill $balance_pid
+	wait
+
+	# Sometimes even we killed $balance_pid and wait returned,
+	# balance may still be running, use balance cancel to wait it.
+	# As this is just a workaround, we don't want it pollute seqres
+	# so call $BTRFS_UTIL_PROG directly
+	$BTRFS_UTIL_PROG balance cancel $SCRATCH_MNT &> /dev/null
+
+	rm $SCRATCH_MNT/* -rf &> /dev/null
+	_run_btrfs_util_prog dedupe disable $SCRATCH_MNT
+}
+
+do_dedupe_balance_test
+
+# success, all done
+status=0
+exit
diff --git a/tests/btrfs/203.out b/tests/btrfs/203.out
new file mode 100644
index 0000000..404394c
--- /dev/null
+++ b/tests/btrfs/203.out
@@ -0,0 +1,3 @@
+QA output created by 203
+wrote 131072/131072 bytes at offset 0
+XXX Bytes, X ops; XX:XX:XX.X (XXX YYY/sec and XXX ops/sec)
diff --git a/tests/btrfs/group b/tests/btrfs/group
index 2e539d0..c35b919 100644
--- a/tests/btrfs/group
+++ b/tests/btrfs/group
@@ -135,3 +135,4 @@
 130 auto clone send
 200 auto ib-dedupe
 201 auto ib-dedupe
+203 auto ib-dedupe balance
-- 
2.7.4

Re: [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs

[Dave Chinner]

On Wed, Sep 14, 2016 at 09:55:22AM +0800, Qu Wenruo wrote:
> Introduce _post_mount_hook(), which will be executed after mounting
> scratch/test.
> 
> It's quite useful for fs(OK, only btrfs yet, again) which needs to
> use ioctl other than mount option to enable some of its feature.

Just implement a _btrfs_mount() function (similar to
_overlay_mount()) to do btrfs specific things at mount time.
> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
> ---
>  common/rc | 23 +++++++++++++++++++++++
>  1 file changed, 23 insertions(+)
> 
> diff --git a/common/rc b/common/rc
> index 23c007a..631397f 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -321,6 +321,27 @@ _overlay_scratch_unmount()
>  	$UMOUNT_PROG $SCRATCH_MNT
>  }
>  
> +_run_btrfs_post_mount_hook()
> +{
> +	mnt_point=$1
> +	for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do

What's this magic, undefined, undocumented variable?

Cheers,

Dave.
-- 
Dave Chinner
david@fromorbit.com

Re: [PATCH v6 1/6] fstests: common: Introduce _post_mount_hook for btrfs

[Qu Wenruo]

At 09/15/2016 12:24 PM, Dave Chinner wrote:
> On Wed, Sep 14, 2016 at 09:55:22AM +0800, Qu Wenruo wrote:
>> Introduce _post_mount_hook(), which will be executed after mounting
>> scratch/test.
>>
>> It's quite useful for fs(OK, only btrfs yet, again) which needs to
>> use ioctl other than mount option to enable some of its feature.
>
> Just implement a _btrfs_mount() function (similar to
> _overlay_mount()) to do btrfs specific things at mount time.
>> Signed-off-by: Qu Wenruo <quwenruo@cn.fujitsu.com>
>> ---
>>  common/rc | 23 +++++++++++++++++++++++
>>  1 file changed, 23 insertions(+)
>>
>> diff --git a/common/rc b/common/rc
>> index 23c007a..631397f 100644
>> --- a/common/rc
>> +++ b/common/rc
>> @@ -321,6 +321,27 @@ _overlay_scratch_unmount()
>>  	$UMOUNT_PROG $SCRATCH_MNT
>>  }
>>
>> +_run_btrfs_post_mount_hook()
>> +{
>> +	mnt_point=$1
>> +	for n in $ALWAYS_ENABLE_BTRFS_FEATURE; do
>
> What's this magic, undefined, undocumented variable?

Yes this reminds me.

The biggest problem is, where is the document of all these variables?

common/config and config/examples all lacks variables like 
TIME_FACTOR/LOAD_FACTOR.

Or it's the time to doc them all?

Thanks,
Qu
>
> Cheers,
>
> Dave.
>