[PATCH v3 00/17] generic: add some mmap CoW tests

[PATCH v3 00/17] generic: add some mmap CoW tests

[Shiyang Ruan]

Changes since v1:
  - Introduce _require_scratch_delalloc() to check if supports delay allocation
  - Remove useless _require_odirect and _require_cp_reflink in some cases

This series is to add missing mmap CoW tests in some sets of testcases:
Set 1. CoW on shared blocks which are interleaving reflinked by two files.
  - P1:            add mmap test
Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
   the destination file.
  - P2 ~ P4, P6:   add mmap test
Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
   the source file.
  - P12, P13:      add missing buffered-io and direct-io test for mixed blocks
  - P8 ~ P11, P14: add mmap test
Set 4. races tests
  - P15: race between reflink and mmap reading
  - P16: race between mmap writing to and reflink source file
  - P17: race between mmap writing to and reflink target file

Shiyang Ruan (17):
  generic: add mmap CoW test for ranges of two shared files
  generic: add mmap CoW test for regular&destination extents
  generic: add mmap CoW test for unwritten&destination extents
  generic: add mmap CoW test for holes&destination extents
  common/rc: Introduce _require_scratch_delalloc()
  generic: add mmap CoW test for delalloc&destination extents
  generic: add mmap CoW test for mixed&destination extents
  generic: add mmap CoW test for regular&source extents
  generic: add mmap CoW test for unwritten&source extents
  generic: add mmap CoW test for holes&source extents
  generic: add mmap CoW test for delalloc&source extents
  generic: add buffered-io CoW test for mixed&source extents
  generic: add direct-io CoW test for mixed&source extents
  generic: add mmap CoW test for mixed&source extents
  generic: add race test between reflink and mmap read
  generic: add race test that mmap write to source of reflink
  generic: add race test that mmap write to target of reflink

 common/rc             | 13 ++++++++
 common/reflink        | 62 +++++++++++++++++++++++++++++++++++++
 tests/generic/194     |  1 +
 tests/generic/195     |  1 +
 tests/generic/199     |  1 +
 tests/generic/200     |  1 +
 tests/generic/293     |  1 +
 tests/generic/295     |  1 +
 tests/generic/540     |  1 +
 tests/generic/541     |  1 +
 tests/generic/542     |  1 +
 tests/generic/543     |  1 +
 tests/generic/900     | 69 +++++++++++++++++++++++++++++++++++++++++
 tests/generic/900.out | 14 +++++++++
 tests/generic/901     | 59 +++++++++++++++++++++++++++++++++++
 tests/generic/901.out | 12 ++++++++
 tests/generic/902     | 59 +++++++++++++++++++++++++++++++++++
 tests/generic/902.out | 12 ++++++++
 tests/generic/903     | 59 +++++++++++++++++++++++++++++++++++
 tests/generic/903.out | 12 ++++++++
 tests/generic/904     | 62 +++++++++++++++++++++++++++++++++++++
 tests/generic/904.out | 12 ++++++++
 tests/generic/905     | 67 ++++++++++++++++++++++++++++++++++++++++
 tests/generic/905.out | 12 ++++++++
 tests/generic/906     | 60 ++++++++++++++++++++++++++++++++++++
 tests/generic/906.out | 12 ++++++++
 tests/generic/907     | 62 +++++++++++++++++++++++++++++++++++++
 tests/generic/907.out | 12 ++++++++
 tests/generic/908     | 62 +++++++++++++++++++++++++++++++++++++
 tests/generic/908.out | 12 ++++++++
 tests/generic/909     | 65 ++++++++++++++++++++++++++++++++++++++
 tests/generic/909.out | 12 ++++++++
 tests/generic/910     | 65 ++++++++++++++++++++++++++++++++++++++
 tests/generic/910.out | 12 ++++++++
 tests/generic/911     | 66 +++++++++++++++++++++++++++++++++++++++
 tests/generic/911.out | 12 ++++++++
 tests/generic/912     | 66 +++++++++++++++++++++++++++++++++++++++
 tests/generic/912.out | 12 ++++++++
 tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/913.out |  5 +++
 tests/generic/914     | 64 ++++++++++++++++++++++++++++++++++++++
 tests/generic/914.out |  4 +++
 tests/generic/915     | 64 ++++++++++++++++++++++++++++++++++++++
 tests/generic/915.out |  4 +++
 tests/xfs/172         | 12 +-------
 tests/xfs/223         |  1 +
 tests/xfs/224         |  1 +
 tests/xfs/228         |  1 +
 tests/xfs/230         |  1 +
 tests/xfs/257         |  1 +
 tests/xfs/258         |  1 +
 51 files changed, 1284 insertions(+), 11 deletions(-)
 create mode 100755 tests/generic/900
 create mode 100644 tests/generic/900.out
 create mode 100755 tests/generic/901
 create mode 100644 tests/generic/901.out
 create mode 100755 tests/generic/902
 create mode 100644 tests/generic/902.out
 create mode 100755 tests/generic/903
 create mode 100644 tests/generic/903.out
 create mode 100755 tests/generic/904
 create mode 100644 tests/generic/904.out
 create mode 100755 tests/generic/905
 create mode 100644 tests/generic/905.out
 create mode 100755 tests/generic/906
 create mode 100644 tests/generic/906.out
 create mode 100755 tests/generic/907
 create mode 100644 tests/generic/907.out
 create mode 100755 tests/generic/908
 create mode 100644 tests/generic/908.out
 create mode 100755 tests/generic/909
 create mode 100644 tests/generic/909.out
 create mode 100755 tests/generic/910
 create mode 100644 tests/generic/910.out
 create mode 100755 tests/generic/911
 create mode 100644 tests/generic/911.out
 create mode 100755 tests/generic/912
 create mode 100644 tests/generic/912.out
 create mode 100755 tests/generic/913
 create mode 100644 tests/generic/913.out
 create mode 100755 tests/generic/914
 create mode 100644 tests/generic/914.out
 create mode 100755 tests/generic/915
 create mode 100644 tests/generic/915.out

-- 
2.34.1

[PATCH v3 01/17] generic: add mmap CoW test for ranges of two shared files

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents. (MMAP version of generic/185,183)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/900     | 69 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/900.out | 14 +++++++++
 2 files changed, 83 insertions(+)
 create mode 100755 tests/generic/900
 create mode 100644 tests/generic/900.out

diff --git a/tests/generic/900 b/tests/generic/900
new file mode 100755
index 00000000..de76cd00
--- /dev/null
+++ b/tests/generic/900
@@ -0,0 +1,69 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 900
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents. (MMAP version of generic/185,183)
+#   - Create two files
+#   - Reflink the odd blocks of the first file into a third file.
+#   - Reflink the even blocks of the second file into the third file.
+#   - mmap CoW across the halfway mark.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_pwrite_byte 0x61 0 $filesize $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $filesize $testdir/file2 >> $seqres.full
+seq 0 2 $((nr-1)) | while read f; do
+	_reflink_range $testdir/file1 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
+	_pwrite_byte 0x61 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
+done
+seq 1 2 $((nr-1)) | while read f; do
+	_reflink_range $testdir/file2 $((blksz * f)) $testdir/file3 $((blksz * f)) $blksz >> $seqres.full
+	_pwrite_byte 0x62 $((blksz * f)) $blksz $testdir/file3.chk >> $seqres.full
+done
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file2 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/900.out b/tests/generic/900.out
new file mode 100644
index 00000000..b2569ab0
--- /dev/null
+++ b/tests/generic/900.out
@@ -0,0 +1,14 @@
+QA output created by 900
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
+b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-900/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-900/file1
+b83f9394092e15bdcda585cd8e776dc6  SCRATCH_MNT/test-900/file2
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-900/file3.chk
-- 
2.34.1

[PATCH v3 02/17] generic: add mmap CoW test for regular&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some regular, some not.
(MMAP version of generic/197,196)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/901     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/901.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/901
 create mode 100644 tests/generic/901.out

diff --git a/tests/generic/901 b/tests/generic/901
new file mode 100755
index 00000000..b9f4fbc0
--- /dev/null
+++ b/tests/generic/901
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 901
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some regular, some not.
+# (MMAP version of generic/197,196)
+#   - Create two files.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/901.out b/tests/generic/901.out
new file mode 100644
index 00000000..c3983e87
--- /dev/null
+++ b/tests/generic/901.out
@@ -0,0 +1,12 @@
+QA output created by 901
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3
+d3959a68638c50af07f0258e032cb554  SCRATCH_MNT/test-901/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-901/file1
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-901/file3.chk
-- 
2.34.1

[PATCH v3 03/17] generic: add mmap CoW test for unwritten&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some unwritten, some not.
(MMAP version of generic/189,188)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/902     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/902.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/902
 create mode 100644 tests/generic/902.out

diff --git a/tests/generic/902 b/tests/generic/902
new file mode 100755
index 00000000..457b039a
--- /dev/null
+++ b/tests/generic/902
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 902
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some unwritten, some not.
+# (MMAP version of generic/189,188)
+#   - Create a file and fallocate a second file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/902.out b/tests/generic/902.out
new file mode 100644
index 00000000..bc951255
--- /dev/null
+++ b/tests/generic/902.out
@@ -0,0 +1,12 @@
+QA output created by 902
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-902/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-902/file1
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-902/file3.chk
-- 
2.34.1

[PATCH v3 04/17] generic: add mmap CoW test for holes&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some holes, some not.
(MMAP version of generic/191,190)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/903     | 59 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/903.out | 12 +++++++++
 2 files changed, 71 insertions(+)
 create mode 100755 tests/generic/903
 create mode 100644 tests/generic/903.out

diff --git a/tests/generic/903 b/tests/generic/903
new file mode 100755
index 00000000..d0de3f26
--- /dev/null
+++ b/tests/generic/903
@@ -0,0 +1,59 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 903
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some holes, some not.
+# (MMAP version of generic/191,190)
+#   - Create a file and truncate a second file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/903.out b/tests/generic/903.out
new file mode 100644
index 00000000..bf57acc7
--- /dev/null
+++ b/tests/generic/903.out
@@ -0,0 +1,12 @@
+QA output created by 903
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-903/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-903/file1
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3
+09101629908f9bdd5d178e7ce20bb1bb  SCRATCH_MNT/test-903/file3.chk
-- 
2.34.1

[PATCH v3 05/17] common/rc: Introduce _require_scratch_delalloc()

[Shiyang Ruan]

Check whether delalloc is supported on current device.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/rc         | 13 +++++++++++++
 tests/generic/194 |  1 +
 tests/generic/195 |  1 +
 tests/generic/199 |  1 +
 tests/generic/200 |  1 +
 tests/generic/293 |  1 +
 tests/generic/295 |  1 +
 tests/generic/540 |  1 +
 tests/generic/541 |  1 +
 tests/generic/542 |  1 +
 tests/generic/543 |  1 +
 tests/xfs/172     | 12 +-----------
 tests/xfs/223     |  1 +
 tests/xfs/224     |  1 +
 tests/xfs/228     |  1 +
 tests/xfs/230     |  1 +
 tests/xfs/257     |  1 +
 tests/xfs/258     |  1 +
 18 files changed, 30 insertions(+), 11 deletions(-)

diff --git a/common/rc b/common/rc
index 7973ceb5..a419b3ba 100644
--- a/common/rc
+++ b/common/rc
@@ -1748,6 +1748,19 @@ _require_scratch_16T_support()
 	esac
 }
 
+# Require scratch fs supports delay allocation.
+_require_scratch_delalloc()
+{
+	_require_command "$FILEFRAG_PROG" filefrag
+
+	_scratch_mkfs > $seqres.full
+	_scratch_mount
+	$XFS_IO_PROG -f -c 'pwrite 0 64k' $SCRATCH_MNT/testy &> /dev/null
+	$FILEFRAG_PROG -v $SCRATCH_MNT/testy 2>&1 | grep -q delalloc || \
+		_notrun "test requires delayed allocation buffered writes"
+	_scratch_unmount
+}
+
 # this test needs a test partition - check we're ok & mount it
 #
 _require_test()
diff --git a/tests/generic/194 b/tests/generic/194
index ce8c31ac..ff76438d 100755
--- a/tests/generic/194
+++ b/tests/generic/194
@@ -28,6 +28,7 @@ _cleanup()
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_odirect
 
diff --git a/tests/generic/195 b/tests/generic/195
index da502c39..e087b99c 100755
--- a/tests/generic/195
+++ b/tests/generic/195
@@ -28,6 +28,7 @@ _cleanup()
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/generic/199 b/tests/generic/199
index ab13f58e..2a8cafcc 100755
--- a/tests/generic/199
+++ b/tests/generic/199
@@ -31,6 +31,7 @@ _cleanup()
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "fpunch"
 _require_cp_reflink
diff --git a/tests/generic/200 b/tests/generic/200
index deadfeaa..a1a78ef4 100755
--- a/tests/generic/200
+++ b/tests/generic/200
@@ -31,6 +31,7 @@ _cleanup()
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "fpunch"
 _require_cp_reflink
diff --git a/tests/generic/293 b/tests/generic/293
index 74231b4d..0f1d8416 100755
--- a/tests/generic/293
+++ b/tests/generic/293
@@ -24,6 +24,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/generic/295 b/tests/generic/295
index b921c79b..f66c1805 100755
--- a/tests/generic/295
+++ b/tests/generic/295
@@ -24,6 +24,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_odirect
 
diff --git a/tests/generic/540 b/tests/generic/540
index 69cf635f..38e00f97 100755
--- a/tests/generic/540
+++ b/tests/generic/540
@@ -26,6 +26,7 @@ _begin_fstest auto quick clone
 # real QA test starts here
 _supported_fs generic
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/generic/541 b/tests/generic/541
index 925f346e..89b2adad 100755
--- a/tests/generic/541
+++ b/tests/generic/541
@@ -26,6 +26,7 @@ _begin_fstest auto quick clone
 # real QA test starts here
 _supported_fs generic
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/generic/542 b/tests/generic/542
index e3e627a8..e7682f59 100755
--- a/tests/generic/542
+++ b/tests/generic/542
@@ -26,6 +26,7 @@ _begin_fstest auto quick clone
 # real QA test starts here
 _supported_fs generic
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/generic/543 b/tests/generic/543
index 047bedd5..624cfc41 100755
--- a/tests/generic/543
+++ b/tests/generic/543
@@ -26,6 +26,7 @@ _begin_fstest auto quick clone
 # real QA test starts here
 _supported_fs generic
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 
 echo "Format and mount"
diff --git a/tests/xfs/172 b/tests/xfs/172
index c0495305..56c2583b 100755
--- a/tests/xfs/172
+++ b/tests/xfs/172
@@ -16,7 +16,6 @@ _begin_fstest rw filestreams
 
 # real QA test starts here
 _supported_fs xfs
-_require_command "$FILEFRAG_PROG" filefrag
 _require_scratch
 
 # The first _test_streams call sets up the filestreams allocator to fail and
@@ -36,16 +35,7 @@ _require_scratch
 # rescan thresholds and will keep pushing out the expiration time, with the
 # result that the filestreams allocator succeeds in maintaining the streams.
 # The test considers this a failure.
-#
-# Make sure that a regular buffered write produces delalloc reservations.
-# This effectively disables the test for files with extent size hints or DAX
-# mode set.
-_scratch_mkfs > $seqres.full
-_scratch_mount
-$XFS_IO_PROG -f -c 'pwrite 0 64k' $SCRATCH_MNT/testy &> /dev/null
-$FILEFRAG_PROG -v $SCRATCH_MNT/testy 2>&1 | grep -q delalloc || \
-	_notrun "test requires delayed allocation buffered writes"
-_scratch_unmount
+_require_scratch_delalloc
 
 _check_filestreams_support || _notrun "filestreams not available"
 
diff --git a/tests/xfs/223 b/tests/xfs/223
index f002f1de..11dbad14 100755
--- a/tests/xfs/223
+++ b/tests/xfs/223
@@ -22,6 +22,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 _require_odirect
diff --git a/tests/xfs/224 b/tests/xfs/224
index 11a54fa3..f8bab07e 100755
--- a/tests/xfs/224
+++ b/tests/xfs/224
@@ -22,6 +22,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 
diff --git a/tests/xfs/228 b/tests/xfs/228
index 5c644c0f..f2f2f6a9 100755
--- a/tests/xfs/228
+++ b/tests/xfs/228
@@ -25,6 +25,7 @@ _begin_fstest auto quick clone punch
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 _require_xfs_io_command "fpunch"
diff --git a/tests/xfs/230 b/tests/xfs/230
index 53e8dd78..15f6b684 100755
--- a/tests/xfs/230
+++ b/tests/xfs/230
@@ -25,6 +25,7 @@ _begin_fstest auto quick clone punch
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 _require_xfs_io_command "fpunch"
diff --git a/tests/xfs/257 b/tests/xfs/257
index acd55c22..6a58f0ac 100755
--- a/tests/xfs/257
+++ b/tests/xfs/257
@@ -25,6 +25,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 
diff --git a/tests/xfs/258 b/tests/xfs/258
index 25e8eb60..2865cdf9 100755
--- a/tests/xfs/258
+++ b/tests/xfs/258
@@ -25,6 +25,7 @@ _begin_fstest auto quick clone
 
 # real QA test starts here
 _require_scratch_reflink
+_require_scratch_delalloc
 _require_xfs_io_command "falloc"
 _require_xfs_io_command "cowextsize"
 _require_odirect
-- 
2.34.1

[PATCH v3 06/17] generic: add mmap CoW test for delalloc&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, some delalloc, some not.
(MMAP version of generic/195,194)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/904     | 62 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/904.out | 12 +++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/generic/904
 create mode 100644 tests/generic/904.out

diff --git a/tests/generic/904 b/tests/generic/904
new file mode 100755
index 00000000..428dbbf4
--- /dev/null
+++ b/tests/generic/904
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 904
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, some delalloc, some not.
+# (MMAP version of generic/195,194)
+#   - Create a file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - Buffered write the even blocks of the second file.
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_weave_reflink_holes_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/904.out b/tests/generic/904.out
new file mode 100644
index 00000000..cc993394
--- /dev/null
+++ b/tests/generic/904.out
@@ -0,0 +1,12 @@
+QA output created by 904
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3
+fa50dba51826899c372464a153cb2117  SCRATCH_MNT/test-904/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-904/file1
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3
+55968a95dfc0120df4e8485576514320  SCRATCH_MNT/test-904/file3.chk
-- 
2.34.1

[PATCH v3 07/17] generic: add mmap CoW test for mixed&destination extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW range originally
covers multiple extents, mixed with reflinked, unwritten, hole, regular and
delalloc blocks.
(MMAP version of generic/200,199)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/905     | 67 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/905.out | 12 ++++++++
 2 files changed, 79 insertions(+)
 create mode 100755 tests/generic/905
 create mode 100644 tests/generic/905.out

diff --git a/tests/generic/905 b/tests/generic/905
new file mode 100755
index 00000000..8178f630
--- /dev/null
+++ b/tests/generic/905
@@ -0,0 +1,67 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 905
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+# (MMAP version of generic/200,199)
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_weave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_weave_reflink_rainbow_delalloc $blksz $nr $testdir/file3 >> $seqres.full
+# now cow
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file3.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file3.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/905.out b/tests/generic/905.out
new file mode 100644
index 00000000..6fb0be5f
--- /dev/null
+++ b/tests/generic/905.out
@@ -0,0 +1,12 @@
+QA output created by 905
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-905/file3.chk
+mmap CoW across the transition
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-905/file1
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-905/file3.chk
-- 
2.34.1

[PATCH v3 08/17] generic: add mmap CoW test for regular&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers regular unshared and regular shared blocks.
(MMAP version of generic/284,287)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/906     | 60 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/906.out | 12 +++++++++
 2 files changed, 72 insertions(+)
 create mode 100755 tests/generic/906
 create mode 100644 tests/generic/906.out

diff --git a/tests/generic/906 b/tests/generic/906
new file mode 100755
index 00000000..953348e8
--- /dev/null
+++ b/tests/generic/906
@@ -0,0 +1,60 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 906
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers regular unshared and regular shared blocks.
+# (MMAP version of generic/284,287)
+#   - Create two files.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_regular $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/906.out b/tests/generic/906.out
new file mode 100644
index 00000000..4949769b
--- /dev/null
+++ b/tests/generic/906.out
@@ -0,0 +1,12 @@
+QA output created by 906
+Format and mount
+Create the original files
+Compare files
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-906/file1.chk
+mmap CoW across the transition
+Compare files
+2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-906/file3
+2dc2b131303b2f70ddb480778caac8b7  SCRATCH_MNT/test-906/file1.chk
-- 
2.34.1

[PATCH v3 09/17] generic: add mmap CoW test for unwritten&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers unwritten and regular shared blocks.
(MMAP version of generic/289,290)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/907     | 62 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/907.out | 12 +++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/generic/907
 create mode 100644 tests/generic/907.out

diff --git a/tests/generic/907 b/tests/generic/907
new file mode 100755
index 00000000..61c9293a
--- /dev/null
+++ b/tests/generic/907
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 907
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers unwritten and regular shared blocks.
+# (MMAP version of generic/289,290)
+#   - Create two files.
+#   - fallocate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_unwritten $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/907.out b/tests/generic/907.out
new file mode 100644
index 00000000..c18b849c
--- /dev/null
+++ b/tests/generic/907.out
@@ -0,0 +1,12 @@
+QA output created by 907
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-907/file1.chk
+mmap CoW across the transition
+Compare files
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-907/file3
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-907/file1.chk
-- 
2.34.1

[PATCH v3 10/17] generic: add mmap CoW test for holes&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers holes and regular shared blocks.
(MMAP version of generic/291,292)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/908     | 62 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/908.out | 12 +++++++++
 2 files changed, 74 insertions(+)
 create mode 100755 tests/generic/908
 create mode 100644 tests/generic/908.out

diff --git a/tests/generic/908 b/tests/generic/908
new file mode 100755
index 00000000..ad8de699
--- /dev/null
+++ b/tests/generic/908
@@ -0,0 +1,62 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 908
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers holes and regular shared blocks.
+# (MMAP version of generic/291,292)
+#   - Create two files.
+#   - Truncate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/908.out b/tests/generic/908.out
new file mode 100644
index 00000000..c8c670ac
--- /dev/null
+++ b/tests/generic/908.out
@@ -0,0 +1,12 @@
+QA output created by 908
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-908/file1.chk
+mmap CoW across the transition
+Compare files
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-908/file3
+e8cf59797f0b3758c8878c368bb4cf7e  SCRATCH_MNT/test-908/file1.chk
-- 
2.34.1

[PATCH v3 11/17] generic: add mmap CoW test for delalloc&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode to the source file when the CoW range
covers delalloc blocks and regular shared blocks.
(MMAP version of generic/293,295)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/909     | 65 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/909.out | 12 ++++++++
 2 files changed, 77 insertions(+)
 create mode 100755 tests/generic/909
 create mode 100644 tests/generic/909.out

diff --git a/tests/generic/909 b/tests/generic/909
new file mode 100755
index 00000000..88e032b6
--- /dev/null
+++ b/tests/generic/909
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 909
+#
+# Ensuring that copy on write in mmap mode to the source file when the
+# CoW range covers delalloc blocks and regular shared blocks.
+# (MMAP version of generic/293,295)
+#   - Create two files.
+#   - Truncate the first file.
+#   - Write the odd blocks of the first file.
+#   - Reflink the odd blocks of the first file into the second file.
+#   - Write the even blocks of the first file.
+#   - mmap CoW the first file across the halfway mark, starting with the
+#     regular extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_holes $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_holes_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/909.out b/tests/generic/909.out
new file mode 100644
index 00000000..4f9a5d33
--- /dev/null
+++ b/tests/generic/909.out
@@ -0,0 +1,12 @@
+QA output created by 909
+Format and mount
+Create the original files
+Compare files
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
+b8a8a88d4c143f79900c4b4e79aa3e37  SCRATCH_MNT/test-909/file1.chk
+mmap CoW across the transition
+Compare files
+35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1
+ce2023f765eba19677517c511886d5c9  SCRATCH_MNT/test-909/file3
+35e2170e14665b780c6dec328d6a263b  SCRATCH_MNT/test-909/file1.chk
-- 
2.34.1

[PATCH v3 12/17] generic: add buffered-io CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in buffered mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        | 51 +++++++++++++++++++++++++++++++++
 tests/generic/910     | 65 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/910.out | 12 ++++++++
 3 files changed, 128 insertions(+)
 create mode 100755 tests/generic/910
 create mode 100644 tests/generic/910.out

diff --git a/common/reflink b/common/reflink
index ee60398e..68dbdedd 100644
--- a/common/reflink
+++ b/common/reflink
@@ -426,3 +426,54 @@ _sweave_reflink_holes_delalloc() {
 		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile.chk
 	done
 }
+
+# Create a file of interleaved holes, unwritten blocks, regular blocks, and
+# reflinked blocks
+_sweave_reflink_rainbow() {
+	blksz=$1
+	nr=$2
+	sfile=$3
+	dfile=$4
+
+	$XFS_IO_PROG -f -c "truncate $((blksz * nr))" $sfile
+	_pwrite_byte 0x00 0 $((blksz * nr)) $sfile.chk
+	_pwrite_byte 0x61 0 $((blksz * nr)) $dfile
+	seq 0 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 0 blocks are reflinked
+	seq 0 5 $((nr - 1)) | while read i; do
+		_reflink_range $sfile $((blksz * i)) $dfile $((blksz * i)) $blksz
+		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 1 blocks are unwritten
+	seq 1 5 $((nr - 1)) | while read i; do
+		$XFS_IO_PROG -f -c "falloc $((blksz * i)) $blksz" $sfile
+		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 2 blocks are holes
+	seq 2 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 3 blocks are regular
+	seq 3 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile.chk
+	done
+	# 4 blocks will be delalloc later
+}
+
+# For a file created with _sweave_reflink_rainbow, fill the holes with delalloc
+# extents
+_sweave_reflink_rainbow_delalloc() {
+	blksz=$1
+	nr=$2
+	dfile=$3
+
+	# 4 blocks are delalloc (do later)
+	seq 4 5 $((nr - 1)) | while read i; do
+		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile
+		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile.chk
+	done
+}
diff --git a/tests/generic/910 b/tests/generic/910
new file mode 100755
index 00000000..32628060
--- /dev/null
+++ b/tests/generic/910
@@ -0,0 +1,65 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 910
+#
+# Ensuring that copy on write in buffered mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+$XFS_IO_PROG -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/910.out b/tests/generic/910.out
new file mode 100644
index 00000000..12956584
--- /dev/null
+++ b/tests/generic/910.out
@@ -0,0 +1,12 @@
+QA output created by 910
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1.chk
+CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1.chk
-- 
2.34.1

[PATCH v3 13/17] generic: add direct-io CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in direct-io mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 tests/generic/911     | 66 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/911.out | 12 ++++++++
 2 files changed, 78 insertions(+)
 create mode 100755 tests/generic/911
 create mode 100644 tests/generic/911.out

diff --git a/tests/generic/911 b/tests/generic/911
new file mode 100755
index 00000000..01baa49a
--- /dev/null
+++ b/tests/generic/911
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 911
+#
+# Ensuring that copy on write in direct-io mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - directio CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+_require_odirect
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "directio CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+$XFS_IO_PROG -d -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/911.out b/tests/generic/911.out
new file mode 100644
index 00000000..e098fe3c
--- /dev/null
+++ b/tests/generic/911.out
@@ -0,0 +1,12 @@
+QA output created by 911
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-911/file1.chk
+directio CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-911/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-911/file1.chk
-- 
2.34.1

[PATCH v3 14/17] generic: add mmap CoW test for mixed&source extents

[Shiyang Ruan]

Ensuring that copy on write in mmap mode works when the CoW
range originally covers multiple extents, mixed with reflinked, unwritten,
hole, regular and delalloc blocks.

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/912     | 66 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/912.out | 12 ++++++++
 2 files changed, 78 insertions(+)
 create mode 100755 tests/generic/912
 create mode 100644 tests/generic/912.out

diff --git a/tests/generic/912 b/tests/generic/912
new file mode 100755
index 00000000..83e08482
--- /dev/null
+++ b/tests/generic/912
@@ -0,0 +1,66 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 912
+#
+# Ensuring that copy on write in mmap mode works when the CoW
+# range originally covers multiple extents, mixed with reflinked, unwritten,
+# hole, regular and delalloc blocks.
+#   - Create a file with the following repeating sequence of blocks:
+#     1. reflinked
+#     2. unwritten
+#     3. hole
+#     4. regular block
+#     5. delalloc
+#   - mmap CoW across the halfway mark, starting with the unwritten extent.
+#   - Check that the files are now different where we say they're different.
+#
+. ./common/preamble
+_begin_fstest auto quick clone punch
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_scratch_delalloc
+_require_xfs_io_command "falloc"
+_require_xfs_io_command "fpunch"
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+mkdir $testdir
+
+echo "Create the original files"
+blksz=65536
+nr=64
+filesize=$((blksz * nr))
+_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+echo "mmap CoW across the transition"
+cowoff=$((filesize / 4))
+cowsz=$((filesize / 2))
+_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
+# now cow
+mmapsz=$((cowoff + cowsz))
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1 >> $seqres.full
+_mwrite_byte 0x63 $cowoff $cowsz $mmapsz $testdir/file1.chk >> $seqres.full
+_scratch_cycle_mount
+
+echo "Compare files"
+md5sum $testdir/file1 | _filter_scratch
+md5sum $testdir/file3 | _filter_scratch
+md5sum $testdir/file1.chk | _filter_scratch
+
+# success, all done
+status=0
diff --git a/tests/generic/912.out b/tests/generic/912.out
new file mode 100644
index 00000000..32f25a1d
--- /dev/null
+++ b/tests/generic/912.out
@@ -0,0 +1,12 @@
+QA output created by 912
+Format and mount
+Create the original files
+Compare files
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
+6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-912/file1.chk
+mmap CoW across the transition
+Compare files
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1
+bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-912/file3
+26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-912/file1.chk
-- 
2.34.1

[PATCH v3 15/17] generic: add race test between reflink and mmap read

[Shiyang Ruan]

Test for races or FS corruption between reflink and mmap reading the
target file. (MMAP version of generic/164,165)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
---
 common/reflink        | 11 +++++++
 tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/913.out |  5 +++
 3 files changed, 88 insertions(+)
 create mode 100755 tests/generic/913
 create mode 100644 tests/generic/913.out

diff --git a/common/reflink b/common/reflink
index 68dbdedd..455260c6 100644
--- a/common/reflink
+++ b/common/reflink
@@ -186,6 +186,17 @@ _read_range() {
 	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
 }
 
+# Prints a range of a file as a hex dump
+_mread_range() {
+	file="$1"
+	offset="$2"
+	len="$3"
+	xfs_io_args="$4"
+
+	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
+		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
+}
+
 # Compare ranges of two files
 _compare_range() {
 	file1="$1"
diff --git a/tests/generic/913 b/tests/generic/913
new file mode 100755
index 00000000..f709c36c
--- /dev/null
+++ b/tests/generic/913
@@ -0,0 +1,72 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 913
+#
+# Test for races or FS corruption between reflink and mmap reading the
+# target file. (MMAP version of generic/164,165)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_cp_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=512
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize files"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
+_cp_reflink $testdir/file1 $testdir/file3
+_scratch_cycle_mount
+
+fbytes() {
+	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
+}
+
+reader() {
+	while [ ! -e $finished_file ]; do
+		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
+	done
+}
+
+echo "Reflink and mmap reread the files!"
+reader &
+for i in `seq 1 2`; do
+	seq $nr_loops -1 0 | while read i; do
+		_reflink_range  $testdir/file1 $((i * blksz)) \
+				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && break
+	done
+	seq $nr_loops -1 0 | while read i; do
+		_reflink_range  $testdir/file2 $((i * blksz)) \
+				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && break
+	done
+done
+echo "Finished reflinking"
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/913.out b/tests/generic/913.out
new file mode 100644
index 00000000..a34df6ce
--- /dev/null
+++ b/tests/generic/913.out
@@ -0,0 +1,5 @@
+QA output created by 913
+Format and mount
+Initialize files
+Reflink and mmap reread the files!
+Finished reflinking
-- 
2.34.1

[PATCH v3 16/17] generic: add race test that mmap write to source of reflink

[Shiyang Ruan]

Test for races or FS corruption when mmap writing to a file that's also
the source of a reflink operation. (MMAP version of generic/167,166)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/914     | 64 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/914.out |  4 +++
 2 files changed, 68 insertions(+)
 create mode 100755 tests/generic/914
 create mode 100644 tests/generic/914.out

diff --git a/tests/generic/914 b/tests/generic/914
new file mode 100755
index 00000000..c51c44c7
--- /dev/null
+++ b/tests/generic/914
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 914
+#
+# Test for races or FS corruption when mmap writing to a file that's also
+# the source of a reflink operation. (MMAP version of generic/167,166)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+_require_cp_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=1024
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize file"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_scratch_cycle_mount
+
+# Snapshot creator...
+snappy() {
+	n=0
+	while [ ! -e $finished_file ]; do
+		out="$(_cp_reflink $testdir/file1 $testdir/snap_$n 2>&1)"
+		res=$?
+		echo "$out" | grep -q "No space left" && break
+		test -n "$out" && echo "$out"
+		test $res -ne 0 && break
+		n=$((n + 1))
+	done
+}
+
+echo "Snapshot a file undergoing mmap rewrite"
+snappy &
+seq $nr_loops -1 0 | while read i; do
+	$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
+		-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file1 >> $seqres.full
+done
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/914.out b/tests/generic/914.out
new file mode 100644
index 00000000..1472055f
--- /dev/null
+++ b/tests/generic/914.out
@@ -0,0 +1,4 @@
+QA output created by 914
+Format and mount
+Initialize file
+Snapshot a file undergoing mmap rewrite
-- 
2.34.1

[PATCH v3 17/17] generic: add race test that mmap write to target of reflink

[Shiyang Ruan]

Test for races or FS corruption when mmap writing to a file that's also
the target of a reflink operation. (MMAP version of generic/168,170)

Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/915     | 64 +++++++++++++++++++++++++++++++++++++++++++
 tests/generic/915.out |  4 +++
 2 files changed, 68 insertions(+)
 create mode 100755 tests/generic/915
 create mode 100644 tests/generic/915.out

diff --git a/tests/generic/915 b/tests/generic/915
new file mode 100755
index 00000000..365f304c
--- /dev/null
+++ b/tests/generic/915
@@ -0,0 +1,64 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 915
+#
+# Test for races or FS corruption when mmap writing to a file that's also
+# the target of a reflink operation. (MMAP version of generic/168,170)
+#
+. ./common/preamble
+_begin_fstest auto clone
+
+_register_cleanup "_cleanup" BUS
+
+# Import common functions.
+. ./common/filter
+. ./common/reflink
+
+# real QA test starts here
+_require_scratch_reflink
+
+echo "Format and mount"
+_scratch_mkfs > $seqres.full 2>&1
+_scratch_mount >> $seqres.full 2>&1
+
+testdir=$SCRATCH_MNT/test-$seq
+finished_file=/tmp/finished
+rm -rf $finished_file
+mkdir $testdir
+
+loops=1024
+nr_loops=$((loops - 1))
+blksz=65536
+
+echo "Initialize files"
+echo >> $seqres.full
+_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
+_scratch_cycle_mount
+
+# mmap overwriter...
+overwrite() {
+	while [ ! -e $finished_file ]; do
+		seq $nr_loops -1 0 | while read i; do
+			$XFS_IO_PROG -f -c "mmap -rw $((i * blksz)) $blksz" \
+				-c "mwrite -S 0x63 $((i * blksz)) $blksz" $testdir/file2 >> $seqres.full
+		done
+	done
+}
+
+echo "Reflink and mmap write the target"
+overwrite &
+seq 1 10 | while read j; do
+	seq 0 $nr_loops | while read i; do
+		_reflink_range  $testdir/file1 $((i * blksz)) \
+				$testdir/file2 $((i * blksz)) $blksz >> $seqres.full
+		[ $? -ne 0 ] && exit
+	done
+done
+touch $finished_file
+wait
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/915.out b/tests/generic/915.out
new file mode 100644
index 00000000..4e77847b
--- /dev/null
+++ b/tests/generic/915.out
@@ -0,0 +1,4 @@
+QA output created by 915
+Format and mount
+Initialize files
+Reflink and mmap write the target
-- 
2.34.1

Re: [PATCH v3 00/17] generic: add some mmap CoW tests

[Eryu Guan]

On Tue, Dec 14, 2021 at 04:18:57PM +0800, Shiyang Ruan wrote:
> Changes since v1:
>   - Introduce _require_scratch_delalloc() to check if supports delay allocation
>   - Remove useless _require_odirect and _require_cp_reflink in some cases
> 
> This series is to add missing mmap CoW tests in some sets of testcases:
> Set 1. CoW on shared blocks which are interleaving reflinked by two files.
>   - P1:            add mmap test
> Set 2. CoW a range covers shared blocks and multiple kind of unshared blocks of
>    the destination file.
>   - P2 ~ P4, P6:   add mmap test
> Set 3. CoW a range covers shared blocks and multiple kind of unshared blocks of
>    the source file.
>   - P12, P13:      add missing buffered-io and direct-io test for mixed blocks
>   - P8 ~ P11, P14: add mmap test
> Set 4. races tests
>   - P15: race between reflink and mmap reading
>   - P16: race between mmap writing to and reflink source file
>   - P17: race between mmap writing to and reflink target file

Looks good to me overall, thanks!. But it's a large patchset, I'd like
wait for another week for feedbacks, and will merge it next week if
there's no further comments

Thanks,
Eryu

> 
> Shiyang Ruan (17):
>   generic: add mmap CoW test for ranges of two shared files
>   generic: add mmap CoW test for regular&destination extents
>   generic: add mmap CoW test for unwritten&destination extents
>   generic: add mmap CoW test for holes&destination extents
>   common/rc: Introduce _require_scratch_delalloc()
>   generic: add mmap CoW test for delalloc&destination extents
>   generic: add mmap CoW test for mixed&destination extents
>   generic: add mmap CoW test for regular&source extents
>   generic: add mmap CoW test for unwritten&source extents
>   generic: add mmap CoW test for holes&source extents
>   generic: add mmap CoW test for delalloc&source extents
>   generic: add buffered-io CoW test for mixed&source extents
>   generic: add direct-io CoW test for mixed&source extents
>   generic: add mmap CoW test for mixed&source extents
>   generic: add race test between reflink and mmap read
>   generic: add race test that mmap write to source of reflink
>   generic: add race test that mmap write to target of reflink
> 
>  common/rc             | 13 ++++++++
>  common/reflink        | 62 +++++++++++++++++++++++++++++++++++++
>  tests/generic/194     |  1 +
>  tests/generic/195     |  1 +
>  tests/generic/199     |  1 +
>  tests/generic/200     |  1 +
>  tests/generic/293     |  1 +
>  tests/generic/295     |  1 +
>  tests/generic/540     |  1 +
>  tests/generic/541     |  1 +
>  tests/generic/542     |  1 +
>  tests/generic/543     |  1 +
>  tests/generic/900     | 69 +++++++++++++++++++++++++++++++++++++++++
>  tests/generic/900.out | 14 +++++++++
>  tests/generic/901     | 59 +++++++++++++++++++++++++++++++++++
>  tests/generic/901.out | 12 ++++++++
>  tests/generic/902     | 59 +++++++++++++++++++++++++++++++++++
>  tests/generic/902.out | 12 ++++++++
>  tests/generic/903     | 59 +++++++++++++++++++++++++++++++++++
>  tests/generic/903.out | 12 ++++++++
>  tests/generic/904     | 62 +++++++++++++++++++++++++++++++++++++
>  tests/generic/904.out | 12 ++++++++
>  tests/generic/905     | 67 ++++++++++++++++++++++++++++++++++++++++
>  tests/generic/905.out | 12 ++++++++
>  tests/generic/906     | 60 ++++++++++++++++++++++++++++++++++++
>  tests/generic/906.out | 12 ++++++++
>  tests/generic/907     | 62 +++++++++++++++++++++++++++++++++++++
>  tests/generic/907.out | 12 ++++++++
>  tests/generic/908     | 62 +++++++++++++++++++++++++++++++++++++
>  tests/generic/908.out | 12 ++++++++
>  tests/generic/909     | 65 ++++++++++++++++++++++++++++++++++++++
>  tests/generic/909.out | 12 ++++++++
>  tests/generic/910     | 65 ++++++++++++++++++++++++++++++++++++++
>  tests/generic/910.out | 12 ++++++++
>  tests/generic/911     | 66 +++++++++++++++++++++++++++++++++++++++
>  tests/generic/911.out | 12 ++++++++
>  tests/generic/912     | 66 +++++++++++++++++++++++++++++++++++++++
>  tests/generic/912.out | 12 ++++++++
>  tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/913.out |  5 +++
>  tests/generic/914     | 64 ++++++++++++++++++++++++++++++++++++++
>  tests/generic/914.out |  4 +++
>  tests/generic/915     | 64 ++++++++++++++++++++++++++++++++++++++
>  tests/generic/915.out |  4 +++
>  tests/xfs/172         | 12 +-------
>  tests/xfs/223         |  1 +
>  tests/xfs/224         |  1 +
>  tests/xfs/228         |  1 +
>  tests/xfs/230         |  1 +
>  tests/xfs/257         |  1 +
>  tests/xfs/258         |  1 +
>  51 files changed, 1284 insertions(+), 11 deletions(-)
>  create mode 100755 tests/generic/900
>  create mode 100644 tests/generic/900.out
>  create mode 100755 tests/generic/901
>  create mode 100644 tests/generic/901.out
>  create mode 100755 tests/generic/902
>  create mode 100644 tests/generic/902.out
>  create mode 100755 tests/generic/903
>  create mode 100644 tests/generic/903.out
>  create mode 100755 tests/generic/904
>  create mode 100644 tests/generic/904.out
>  create mode 100755 tests/generic/905
>  create mode 100644 tests/generic/905.out
>  create mode 100755 tests/generic/906
>  create mode 100644 tests/generic/906.out
>  create mode 100755 tests/generic/907
>  create mode 100644 tests/generic/907.out
>  create mode 100755 tests/generic/908
>  create mode 100644 tests/generic/908.out
>  create mode 100755 tests/generic/909
>  create mode 100644 tests/generic/909.out
>  create mode 100755 tests/generic/910
>  create mode 100644 tests/generic/910.out
>  create mode 100755 tests/generic/911
>  create mode 100644 tests/generic/911.out
>  create mode 100755 tests/generic/912
>  create mode 100644 tests/generic/912.out
>  create mode 100755 tests/generic/913
>  create mode 100644 tests/generic/913.out
>  create mode 100755 tests/generic/914
>  create mode 100644 tests/generic/914.out
>  create mode 100755 tests/generic/915
>  create mode 100644 tests/generic/915.out
> 
> -- 
> 2.34.1
> 
> 

Re: [PATCH v3 12/17] generic: add buffered-io CoW test for mixed&source extents

[Eryu Guan]

On Tue, Dec 14, 2021 at 04:19:09PM +0800, Shiyang Ruan wrote:
> Ensuring that copy on write in buffered mode works when the CoW
> range originally covers multiple extents, mixed with reflinked, unwritten,
> hole, regular and delalloc blocks.
> 
> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        | 51 +++++++++++++++++++++++++++++++++
>  tests/generic/910     | 65 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/910.out | 12 ++++++++
>  3 files changed, 128 insertions(+)
>  create mode 100755 tests/generic/910
>  create mode 100644 tests/generic/910.out
> 
> diff --git a/common/reflink b/common/reflink
> index ee60398e..68dbdedd 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -426,3 +426,54 @@ _sweave_reflink_holes_delalloc() {
>  		_pwrite_byte 0x64 $((blksz * i)) $blksz $sfile.chk
>  	done
>  }
> +
> +# Create a file of interleaved holes, unwritten blocks, regular blocks, and
> +# reflinked blocks
> +_sweave_reflink_rainbow() {
> +	blksz=$1
> +	nr=$2
> +	sfile=$3
> +	dfile=$4

These variables should be declared as local.

> +
> +	$XFS_IO_PROG -f -c "truncate $((blksz * nr))" $sfile
> +	_pwrite_byte 0x00 0 $((blksz * nr)) $sfile.chk
> +	_pwrite_byte 0x61 0 $((blksz * nr)) $dfile
> +	seq 0 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 0 blocks are reflinked
> +	seq 0 5 $((nr - 1)) | while read i; do
> +		_reflink_range $sfile $((blksz * i)) $dfile $((blksz * i)) $blksz
> +		_pwrite_byte 0x61 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 1 blocks are unwritten
> +	seq 1 5 $((nr - 1)) | while read i; do
> +		$XFS_IO_PROG -f -c "falloc $((blksz * i)) $blksz" $sfile
> +		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 2 blocks are holes
> +	seq 2 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x00 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 3 blocks are regular
> +	seq 3 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x71 $((blksz * i)) $blksz $sfile.chk
> +	done
> +	# 4 blocks will be delalloc later
> +}
> +
> +# For a file created with _sweave_reflink_rainbow, fill the holes with delalloc
> +# extents
> +_sweave_reflink_rainbow_delalloc() {
> +	blksz=$1
> +	nr=$2
> +	dfile=$3

Same here.

> +
> +	# 4 blocks are delalloc (do later)
> +	seq 4 5 $((nr - 1)) | while read i; do
> +		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile
> +		_pwrite_byte 0x62 $((blksz * i)) $blksz $sfile.chk

And there's no 'sfile' in this function, should be $dfile and $dfile.chk

I've fixed them all on commit.

Thanks,
Eryu

> +	done
> +}
> diff --git a/tests/generic/910 b/tests/generic/910
> new file mode 100755
> index 00000000..32628060
> --- /dev/null
> +++ b/tests/generic/910
> @@ -0,0 +1,65 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 910
> +#
> +# Ensuring that copy on write in buffered mode works when the CoW
> +# range originally covers multiple extents, mixed with reflinked, unwritten,
> +# hole, regular and delalloc blocks.
> +#   - Create a file with the following repeating sequence of blocks:
> +#     1. reflinked
> +#     2. unwritten
> +#     3. hole
> +#     4. regular block
> +#     5. delalloc
> +#   - CoW across the halfway mark, starting with the unwritten extent.
> +#   - Check that the files are now different where we say they're different.
> +#
> +. ./common/preamble
> +_begin_fstest auto quick clone punch
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_scratch_delalloc
> +_require_xfs_io_command "falloc"
> +_require_xfs_io_command "fpunch"
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +mkdir $testdir
> +
> +echo "Create the original files"
> +blksz=65536
> +nr=64
> +filesize=$((blksz * nr))
> +_sweave_reflink_rainbow $blksz $nr $testdir/file1 $testdir/file3 >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +echo "CoW across the transition"
> +cowoff=$((filesize / 4))
> +cowsz=$((filesize / 2))
> +_sweave_reflink_rainbow_delalloc $blksz $nr $testdir/file1 >> $seqres.full
> +# now cow
> +$XFS_IO_PROG -f -c "pwrite -S 0x63 -b $cowsz $cowoff $cowsz" $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x63 $cowoff $cowsz $testdir/file1.chk >> $seqres.full
> +_scratch_cycle_mount
> +
> +echo "Compare files"
> +md5sum $testdir/file1 | _filter_scratch
> +md5sum $testdir/file3 | _filter_scratch
> +md5sum $testdir/file1.chk | _filter_scratch
> +
> +# success, all done
> +status=0
> diff --git a/tests/generic/910.out b/tests/generic/910.out
> new file mode 100644
> index 00000000..12956584
> --- /dev/null
> +++ b/tests/generic/910.out
> @@ -0,0 +1,12 @@
> +QA output created by 910
> +Format and mount
> +Create the original files
> +Compare files
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
> +6366fd359371414186688a0ef6988893  SCRATCH_MNT/test-910/file1.chk
> +CoW across the transition
> +Compare files
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1
> +bdbcf02ee0aa977795a79d25fcfdccb1  SCRATCH_MNT/test-910/file3
> +26aa3a0749b867ec58363c8539ee5471  SCRATCH_MNT/test-910/file1.chk
> -- 
> 2.34.1
> 
> 

Re: [PATCH v3 15/17] generic: add race test between reflink and mmap read

[Darrick J. Wong]

On Tue, Dec 14, 2021 at 04:19:12PM +0800, Shiyang Ruan wrote:
> Test for races or FS corruption between reflink and mmap reading the
> target file. (MMAP version of generic/164,165)

Hi, now that this test has been running for a couple of weeks, I have
observed periodic false positives from this test:

QA output created by 670
Format and mount
Initialize files
Reflink and mmap reread the files!
00001000:  61 61 61 61 61 61 61 61 62 62 62 62 62 62 62 62 aaaabbbb
Finished reflinking

I suspect that if the _mread_range of file3 races with the page cache
invalidation that FICLONERANGE performs, it is possible that the mread
dump will contain a mix of 0x61 and 0x62.  Looking at mread_f, it looks
like it does a byte-at-a-time copy of the mmap...

	if (rflag) {
		for (tmp = length - 1, c = 0; tmp >= 0; tmp--, c = 1) {
			*bp = *(((char *)mapping->addr) + dumpoffset + tmp);
			cnt++;

...which is a sufficient window for the page cache mapping to get
invalidated such that the mread will block on the page fault until the
reflink operation finishes.

I think the solution here is to adjust the egrep regexp above to find
any line that does /not/ contain a's or b's, since (in principle) the
reflink could run fast enough that every byte read hits a pgae fault.
What do you think?

--D

> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
> ---
>  common/reflink        | 11 +++++++
>  tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/913.out |  5 +++
>  3 files changed, 88 insertions(+)
>  create mode 100755 tests/generic/913
>  create mode 100644 tests/generic/913.out
> 
> diff --git a/common/reflink b/common/reflink
> index 68dbdedd..455260c6 100644
> --- a/common/reflink
> +++ b/common/reflink
> @@ -186,6 +186,17 @@ _read_range() {
>  	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>  }
>  
> +# Prints a range of a file as a hex dump
> +_mread_range() {
> +	file="$1"
> +	offset="$2"
> +	len="$3"
> +	xfs_io_args="$4"
> +
> +	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
> +		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
> +}
> +
>  # Compare ranges of two files
>  _compare_range() {
>  	file1="$1"
> diff --git a/tests/generic/913 b/tests/generic/913
> new file mode 100755
> index 00000000..f709c36c
> --- /dev/null
> +++ b/tests/generic/913
> @@ -0,0 +1,72 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 913
> +#
> +# Test for races or FS corruption between reflink and mmap reading the
> +# target file. (MMAP version of generic/164,165)
> +#
> +. ./common/preamble
> +_begin_fstest auto clone
> +
> +_register_cleanup "_cleanup" BUS
> +
> +# Import common functions.
> +. ./common/filter
> +. ./common/reflink
> +
> +# real QA test starts here
> +_require_scratch_reflink
> +_require_cp_reflink
> +
> +echo "Format and mount"
> +_scratch_mkfs > $seqres.full 2>&1
> +_scratch_mount >> $seqres.full 2>&1
> +
> +testdir=$SCRATCH_MNT/test-$seq
> +finished_file=/tmp/finished
> +rm -rf $finished_file
> +mkdir $testdir
> +
> +loops=512
> +nr_loops=$((loops - 1))
> +blksz=65536
> +
> +echo "Initialize files"
> +echo >> $seqres.full
> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
> +_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
> +_cp_reflink $testdir/file1 $testdir/file3
> +_scratch_cycle_mount
> +
> +fbytes() {
> +	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
> +}
> +
> +reader() {
> +	while [ ! -e $finished_file ]; do
> +		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
> +	done
> +}
> +
> +echo "Reflink and mmap reread the files!"
> +reader &
> +for i in `seq 1 2`; do
> +	seq $nr_loops -1 0 | while read i; do
> +		_reflink_range  $testdir/file1 $((i * blksz)) \
> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
> +		[ $? -ne 0 ] && break
> +	done
> +	seq $nr_loops -1 0 | while read i; do
> +		_reflink_range  $testdir/file2 $((i * blksz)) \
> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
> +		[ $? -ne 0 ] && break
> +	done
> +done
> +echo "Finished reflinking"
> +touch $finished_file
> +wait
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/913.out b/tests/generic/913.out
> new file mode 100644
> index 00000000..a34df6ce
> --- /dev/null
> +++ b/tests/generic/913.out
> @@ -0,0 +1,5 @@
> +QA output created by 913
> +Format and mount
> +Initialize files
> +Reflink and mmap reread the files!
> +Finished reflinking
> -- 
> 2.34.1
> 
> 
> 

Re: [PATCH v3 15/17] generic: add race test between reflink and mmap read

[Shiyang Ruan]

在 2022/1/12 2:55, Darrick J. Wong 写道:
> On Tue, Dec 14, 2021 at 04:19:12PM +0800, Shiyang Ruan wrote:
>> Test for races or FS corruption between reflink and mmap reading the
>> target file. (MMAP version of generic/164,165)
> 
> Hi, now that this test has been running for a couple of weeks, I have
> observed periodic false positives from this test:
> 
> QA output created by 670
> Format and mount
> Initialize files
> Reflink and mmap reread the files!
> 00001000:  61 61 61 61 61 61 61 61 62 62 62 62 62 62 62 62 aaaabbbb
> Finished reflinking
> 
> I suspect that if the _mread_range of file3 races with the page cache
> invalidation that FICLONERANGE performs, it is possible that the mread
> dump will contain a mix of 0x61 and 0x62.  Looking at mread_f, it looks
> like it does a byte-at-a-time copy of the mmap...
> 
> 	if (rflag) {
> 		for (tmp = length - 1, c = 0; tmp >= 0; tmp--, c = 1) {
> 			*bp = *(((char *)mapping->addr) + dumpoffset + tmp);
> 			cnt++;
> 
> ...which is a sufficient window for the page cache mapping to get
> invalidated such that the mread will block on the page fault until the
> reflink operation finishes.
> 
> I think the solution here is to adjust the egrep regexp above to find
> any line that does /not/ contain a's or b's, since (in principle) the
> reflink could run fast enough that every byte read hits a pgae fault.
> What do you think?

Reasonable.  I couldn't reproduce it in my test environment, so I didn't 
take that situation into consideration.

I'll change the regexp as below:
fbytes() {
-	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 
62 62 62 62 62 62 62 62 62 62 62 62)'
+	egrep -v '((61|62) ){15}(61|62)'
}


--
Thanks,
Ruan.

> 
> --D
> 
>> Signed-off-by: Shiyang Ruan <ruansy.fnst@fujitsu.com>
>> ---
>>   common/reflink        | 11 +++++++
>>   tests/generic/913     | 72 +++++++++++++++++++++++++++++++++++++++++++
>>   tests/generic/913.out |  5 +++
>>   3 files changed, 88 insertions(+)
>>   create mode 100755 tests/generic/913
>>   create mode 100644 tests/generic/913.out
>>
>> diff --git a/common/reflink b/common/reflink
>> index 68dbdedd..455260c6 100644
>> --- a/common/reflink
>> +++ b/common/reflink
>> @@ -186,6 +186,17 @@ _read_range() {
>>   	$XFS_IO_PROG $xfs_io_args -f -c "pread -q -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>>   }
>>   
>> +# Prints a range of a file as a hex dump
>> +_mread_range() {
>> +	file="$1"
>> +	offset="$2"
>> +	len="$3"
>> +	xfs_io_args="$4"
>> +
>> +	$XFS_IO_PROG $xfs_io_args -f -c "mmap -rw 0 $((offset + len))" \
>> +		-c "mread -v $offset $len" "$file" | cut -d ' ' -f '3-18'
>> +}
>> +
>>   # Compare ranges of two files
>>   _compare_range() {
>>   	file1="$1"
>> diff --git a/tests/generic/913 b/tests/generic/913
>> new file mode 100755
>> index 00000000..f709c36c
>> --- /dev/null
>> +++ b/tests/generic/913
>> @@ -0,0 +1,72 @@
>> +#! /bin/bash
>> +# SPDX-License-Identifier: GPL-2.0
>> +#
>> +# FS QA Test No. 913
>> +#
>> +# Test for races or FS corruption between reflink and mmap reading the
>> +# target file. (MMAP version of generic/164,165)
>> +#
>> +. ./common/preamble
>> +_begin_fstest auto clone
>> +
>> +_register_cleanup "_cleanup" BUS
>> +
>> +# Import common functions.
>> +. ./common/filter
>> +. ./common/reflink
>> +
>> +# real QA test starts here
>> +_require_scratch_reflink
>> +_require_cp_reflink
>> +
>> +echo "Format and mount"
>> +_scratch_mkfs > $seqres.full 2>&1
>> +_scratch_mount >> $seqres.full 2>&1
>> +
>> +testdir=$SCRATCH_MNT/test-$seq
>> +finished_file=/tmp/finished
>> +rm -rf $finished_file
>> +mkdir $testdir
>> +
>> +loops=512
>> +nr_loops=$((loops - 1))
>> +blksz=65536
>> +
>> +echo "Initialize files"
>> +echo >> $seqres.full
>> +_pwrite_byte 0x61 0 $((loops * blksz)) $testdir/file1 >> $seqres.full
>> +_pwrite_byte 0x62 0 $((loops * blksz)) $testdir/file2 >> $seqres.full
>> +_cp_reflink $testdir/file1 $testdir/file3
>> +_scratch_cycle_mount
>> +
>> +fbytes() {
>> +	egrep -v '(61 61 61 61 61 61 61 61 61 61 61 61 61 61 61 61|62 62 62 62 62 62 62 62 62 62 62 62 62 62 62 62)'
>> +}
>> +
>> +reader() {
>> +	while [ ! -e $finished_file ]; do
>> +		_mread_range $testdir/file3 0 $((loops * blksz)) | fbytes
>> +	done
>> +}
>> +
>> +echo "Reflink and mmap reread the files!"
>> +reader &
>> +for i in `seq 1 2`; do
>> +	seq $nr_loops -1 0 | while read i; do
>> +		_reflink_range  $testdir/file1 $((i * blksz)) \
>> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
>> +		[ $? -ne 0 ] && break
>> +	done
>> +	seq $nr_loops -1 0 | while read i; do
>> +		_reflink_range  $testdir/file2 $((i * blksz)) \
>> +				$testdir/file3 $((i * blksz)) $blksz >> $seqres.full
>> +		[ $? -ne 0 ] && break
>> +	done
>> +done
>> +echo "Finished reflinking"
>> +touch $finished_file
>> +wait
>> +
>> +# success, all done
>> +status=0
>> +exit
>> diff --git a/tests/generic/913.out b/tests/generic/913.out
>> new file mode 100644
>> index 00000000..a34df6ce
>> --- /dev/null
>> +++ b/tests/generic/913.out
>> @@ -0,0 +1,5 @@
>> +QA output created by 913
>> +Format and mount
>> +Initialize files
>> +Reflink and mmap reread the files!
>> +Finished reflinking
>> -- 
>> 2.34.1
>>
>>
>>