[PATCH 00/14] rollup of fstests fixes

[PATCH 00/14] rollup of fstests fixes

[Darrick J. Wong]

This is a rollup of all the patches I've been sending to the list,
because the previous thread is a mess of resent patches and patch
series cramming.

Patches 1-2,4,6,8 are unchanged from last week.  They've possibly even
gone in already, but I can't tell what's in Eryu's private branch. :/

Patch 3 is the same as "PATCH v3 5/6" from the last series.

Patch 5 now checks that repair can fix the corruption introduced.

Patch 7 is new, it checks for /usr/sbin/thin_check before running
generic/459 per an earlier discussion on the list.

Patches 9-14 are new; I've been trying to clean up all the minor test
failures in preparation for 4.15 madness.

--D

[PATCH 01/14] common/xfs: refactor xfs_scrub presence testing

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Move all the requirements checking for xfs_scrub into a helper function.
Make sure the helper properly detects the presence of the scrub ioctl
and situations where we can't run scrub (e.g. norecovery).

Refactor the existing three xfs_scrub call sites to use the helper to
check if it's appropriate to run scrub.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 README            |    6 +++---
 common/rc         |    2 +-
 common/xfs        |   41 ++++++++++++++++++++++++++++++++++-------
 tests/generic/453 |   11 +----------
 tests/generic/454 |   11 +----------
 5 files changed, 40 insertions(+), 31 deletions(-)


diff --git a/README b/README
index 4963d28..a9da4f0 100644
--- a/README
+++ b/README
@@ -88,9 +88,9 @@ Preparing system for tests:
                run xfs_repair -n to check the filesystem; xfs_repair to rebuild
                metadata indexes; and xfs_repair -n (a third time) to check the
                results of the rebuilding.
-             - set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
-               xfs_scrub -vd to scrub the filesystem metadata online before
-               unmounting to run the offline check.
+             - xfs_scrub, if present, will always check the test and scratch
+               filesystems if they are still online at the end of the test.
+               It is no longer necessary to set TEST_XFS_SCRUB.
              - setenv LOGWRITES_DEV to a block device to use for power fail
                testing.
 
diff --git a/common/rc b/common/rc
index 1a4d81e..83aaced 100644
--- a/common/rc
+++ b/common/rc
@@ -2091,7 +2091,7 @@ _require_xfs_io_command()
 			_notrun "xfs_io $command support is missing"
 		;;
 	"scrub"|"repair")
-		testio=`$XFS_IO_PROG -x -c "$command test 0" $TEST_DIR 2>&1`
+		testio=`$XFS_IO_PROG -x -c "$command probe 0" $TEST_DIR 2>&1`
 		echo $testio | grep -q "Inappropriate ioctl" && \
 			_notrun "xfs_io $command support is missing"
 		;;
diff --git a/common/xfs b/common/xfs
index dff8454..de3c560 100644
--- a/common/xfs
+++ b/common/xfs
@@ -298,6 +298,30 @@ _require_xfs_db_command()
 		_notrun "xfs_db $command support is missing"
 }
 
+# Does the filesystem mounted from a particular device support scrub?
+_supports_xfs_scrub()
+{
+	mountpoint="$1"
+	device="$2"
+
+	if [ ! -b "$device" ] || [ ! -e "$mountpoint" ]; then
+		echo "Usage: _supports_xfs_scrub mountpoint device"
+		exit 1
+	fi
+
+	test "$FSTYP" = "xfs" || return 1
+	test -x "$XFS_SCRUB_PROG" || return 1
+
+	# Probe for kernel support...
+	$XFS_IO_PROG -c 'help scrub' 2>&1 | grep -q 'types are:.*probe' || return 1
+	$XFS_IO_PROG -c "scrub probe 0" "$mountpoint" 2>&1 | grep -q "Inappropriate ioctl" && return 1
+
+	# Scrub can't run on norecovery mounts
+	_fs_options "$device" | grep -q "norecovery" && return 1
+
+	return 0
+}
+
 # run xfs_check and friends on a FS.
 _check_xfs_filesystem()
 {
@@ -330,14 +354,17 @@ _check_xfs_filesystem()
 	type=`_fs_type $device`
 	ok=1
 
-	if [ "$type" = "xfs" ]; then
-		if [ -n "$TEST_XFS_SCRUB" ] && [ -x "$XFS_SCRUB_PROG" ]; then
-			"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full
-			if [ $? -ne 0 ]; then
-				_log_err "filesystem on $device failed scrub"
-				ok=0
-			fi
+	# Run online scrub if we can.
+	mntpt="$(_is_mounted $device)"
+	if [ -n "$mntpt" ] && _supports_xfs_scrub "$mntpt" "$device"; then
+		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full 2>&1
+		if [ $? -ne 0 ]; then
+			_log_err "filesystem on $device failed scrub"
+			ok=0
 		fi
+	fi
+
+	if [ "$type" = "xfs" ]; then
 		# mounted ...
 		mountpoint=`_umount_or_remount_ro $device`
 	fi
diff --git a/tests/generic/453 b/tests/generic/453
index bda1112..16589d1 100755
--- a/tests/generic/453
+++ b/tests/generic/453
@@ -136,10 +136,7 @@ echo "Test XFS online scrub, if applicable"
 
 # Only run this on xfs if xfs_scrub is available and has the unicode checker
 check_xfs_scrub() {
-	# Ignore non-XFS fs or no scrub program...
-	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
-		return 1
-	fi
+	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
 
 	# We only care if xfs_scrub has unicode string support...
 	if ! type ldd > /dev/null 2>&1 || \
@@ -147,12 +144,6 @@ check_xfs_scrub() {
 		return 1
 	fi
 
-	# Does the ioctl work?
-	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
-	   grep -q "Inappropriate ioctl"; then
-		return 1
-	fi
-
 	return 0
 }
 
diff --git a/tests/generic/454 b/tests/generic/454
index d1f93b2..efac860 100755
--- a/tests/generic/454
+++ b/tests/generic/454
@@ -132,10 +132,7 @@ echo "Test XFS online scrub, if applicable"
 
 # Only run this on xfs if xfs_scrub is available and has the unicode checker
 check_xfs_scrub() {
-	# Ignore non-XFS fs or no scrub program...
-	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
-		return 1
-	fi
+	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
 
 	# We only care if xfs_scrub has unicode string support...
 	if ! type ldd > /dev/null 2>&1 || \
@@ -143,12 +140,6 @@ check_xfs_scrub() {
 		return 1
 	fi
 
-	# Does the ioctl work?
-	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
-	   grep -q "Inappropriate ioctl"; then
-		return 1
-	fi
-
 	return 0
 }
 

[PATCH 02/14] common/xfs: standardize the xfs_scrub output that gets recorded to $seqres.full

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make the xfs_scrub output that gets recorded to $seqres.full follow the
format of xfs_repair checks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/xfs |    8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)


diff --git a/common/xfs b/common/xfs
index de3c560..5643adf 100644
--- a/common/xfs
+++ b/common/xfs
@@ -357,11 +357,15 @@ _check_xfs_filesystem()
 	# Run online scrub if we can.
 	mntpt="$(_is_mounted $device)"
 	if [ -n "$mntpt" ] && _supports_xfs_scrub "$mntpt" "$device"; then
-		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full 2>&1
+		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device > $tmp.scrub 2>&1
 		if [ $? -ne 0 ]; then
-			_log_err "filesystem on $device failed scrub"
+			_log_err "_check_xfs_filesystem: filesystem on $device failed scrub"
+			echo "*** xfs_scrub $scrubflag -v -d -n output ***" >> $seqres.full
+			cat $tmp.scrub >> $seqres.full
+			echo "*** end xfs_scrub output" >> $serqres.full
 			ok=0
 		fi
+		rm -f $tmp.scrub
 	fi
 
 	if [ "$type" = "xfs" ]; then

[PATCH 03/14] misc: add module reloading helpers

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add some helper functions to require that we can reload a given module,
and add a helper to actually do that.  Refactor the existing users to
use the generics.

We need to hoist completely the behaviors of the old btrfs module helper
because we need to confirm before starting the test that we actually can
remove the module.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/btrfs    |   12 -------
 common/module   |   99 +++++++++++++++++++++++++++++++++++++++++++++++++++++++
 common/overlay  |    1 +
 common/rc       |   11 ------
 tests/btrfs/124 |    9 ++---
 tests/btrfs/125 |    9 ++---
 6 files changed, 108 insertions(+), 33 deletions(-)
 create mode 100644 common/module


diff --git a/common/btrfs b/common/btrfs
index fd762ef..c09206c 100644
--- a/common/btrfs
+++ b/common/btrfs
@@ -349,18 +349,6 @@ _btrfs_qgroup_units()
 	$BTRFS_UTIL_PROG qgroup show --help 2>&1 | grep -q -- --raw && echo "--raw"
 }
 
-_require_btrfs_loadable()
-{
-	modprobe -r btrfs || _notrun "btrfs unloadable"
-	modprobe btrfs || _notrun "Can't load btrfs"
-}
-
-_reload_btrfs_ko()
-{
-	modprobe -r btrfs || _fail "btrfs unload failed"
-	modprobe btrfs || _fail "btrfs load failed"
-}
-
 _btrfs_compression_algos()
 {
 	echo zlib
diff --git a/common/module b/common/module
new file mode 100644
index 0000000..7473d20
--- /dev/null
+++ b/common/module
@@ -0,0 +1,99 @@
+##/bin/bash
+
+# Routines for messing around with loadable kernel modules
+#
+#-----------------------------------------------------------------------
+#  Copyright (c) 2017 Oracle.  All Rights Reserved.
+#  This program is free software; you can redistribute it and/or modify
+#  it under the terms of the GNU General Public License as published by
+#  the Free Software Foundation; either version 2 of the License, or
+#  (at your option) any later version.
+#
+#  This program is distributed in the hope that it will be useful,
+#  but WITHOUT ANY WARRANTY; without even the implied warranty of
+#  MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+#  GNU General Public License for more details.
+#
+#  You should have received a copy of the GNU General Public License
+#  along with this program; if not, write to the Free Software
+#  Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307
+#  USA
+#-----------------------------------------------------------------------
+
+# Return the module name for this fs.
+_module_for_fs()
+{
+	echo "${FSTYP}"
+}
+
+# Reload a particular module.  This module MUST NOT be the module that
+# underlies the filesystem.
+_reload_module()
+{
+	module="$1"
+
+	modprobe -r "${module}" || _fail "${module} unload failed"
+	modprobe "${module}" || _fail "${module} load failed"
+}
+
+# Reload the filesystem module.
+_reload_fs_module()
+{
+	module="$1"
+
+	# Unload test fs, try to reload module, remount
+	had_testfs=""
+	had_scratchfs=""
+	_check_mounted_on TEST_DEV $TEST_DEV TEST_DIR $TEST_DIR && had_testfs="true"
+	_check_mounted_on SCRATCH_DEV $SCRATCH_DEV SCRATCH_MNT $SCRATCH_MNT && had_scratchfs="true"
+	test -n "${had_testfs}" && _test_unmount
+	test -n "${had_scratchfs}" && _scratch_unmount
+	_reload_module "${module}"
+	test -n "${had_scratchfs}" && _scratch_mount 2> /dev/null
+	test -n "${had_testfs}" && _test_mount 2> /dev/null
+}
+
+# Check that we have a module that can be loaded.  This module MUST NOT
+# be the module that underlies the filesystem.
+_require_loadable_module()
+{
+	module="$1"
+
+	modinfo "${module}" > /dev/null 2>&1 || _notrun "${module}: must be a module."
+	modprobe -r "${module}" || _notrun "${module} unload failed"
+	modprobe "${module}" || _notrun "${module} load failed"
+}
+
+# Check that the module for FSTYP can be loaded.
+_require_loadable_fs_module()
+{
+	module="$1"
+
+	modinfo "${module}" > /dev/null 2>&1 || _notrun "${module}: must be a module."
+
+	# Unload test fs, try to reload module, remount
+	had_testfs=""
+	had_scratchfs=""
+	_check_mounted_on TEST_DEV $TEST_DEV TEST_DIR $TEST_DIR && had_testfs="true"
+	_check_mounted_on SCRATCH_DEV $SCRATCH_DEV SCRATCH_MNT $SCRATCH_MNT && had_scratchfs="true"
+	test -n "${had_testfs}" && _test_unmount
+	test -n "${had_scratchfs}" && _scratch_unmount
+	unload_ok=""
+	load_ok=""
+	modprobe -r "${module}" || unload_ok=0
+	modprobe "${module}" || load_ok=0
+	test -n "${had_scratchfs}" && _scratch_mount 2> /dev/null
+	test -n "${had_testfs}" && _test_mount 2> /dev/null
+	test -z "${unload_ok}" || _notrun "${module} unload failed"
+	test -z "${load_ok}" || _notrun "${module} load failed"
+}
+
+# Print the value of a filesystem module parameter
+# at /sys/module/$FSTYP/parameters/$PARAM
+#
+# Usage example (FSTYP=overlay):
+#   _get_fs_module_param index
+_get_fs_module_param()
+{
+	cat /sys/module/${FSTYP}/parameters/${1} 2>/dev/null
+}
diff --git a/common/overlay b/common/overlay
index 79097ba..1da4ab1 100644
--- a/common/overlay
+++ b/common/overlay
@@ -1,6 +1,7 @@
 #
 # overlayfs specific common functions.
 #
+. ./common/module
 
 # helper function to do the actual overlayfs mount operation
 _overlay_mount_dirs()
diff --git a/common/rc b/common/rc
index 83aaced..c708bfa 100644
--- a/common/rc
+++ b/common/rc
@@ -3554,16 +3554,6 @@ _get_fs_sysfs_attr()
 	cat /sys/fs/${FSTYP}/${dname}/${attr}
 }
 
-# Print the value of a filesystem module parameter
-# at /sys/module/$FSTYP/parameters/$PARAM
-#
-# Usage example (FSTYP=overlay):
-#   _get_fs_module_param index
-_get_fs_module_param()
-{
-	cat /sys/module/${FSTYP}/parameters/${1} 2>/dev/null
-}
-
 # Generic test for specific filesystem feature.
 # Currently only implemented to test overlayfs features.
 _require_scratch_feature()
@@ -3580,7 +3570,6 @@ _require_scratch_feature()
 	esac
 }
 
-
 init_rc
 
 ################################################################################
diff --git a/tests/btrfs/124 b/tests/btrfs/124
index 7206094..a648627 100755
--- a/tests/btrfs/124
+++ b/tests/btrfs/124
@@ -55,6 +55,7 @@ _cleanup()
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
+. ./common/module
 
 # remove previous $seqres.full before test
 rm -f $seqres.full
@@ -64,10 +65,8 @@ rm -f $seqres.full
 _supported_fs btrfs
 _supported_os Linux
 _require_scratch_dev_pool 2
-
-# the mounted test dir prevent btrfs unload, we need to unmount
 _test_unmount
-_require_btrfs_loadable
+_require_loadable_fs_module "btrfs"
 
 _scratch_dev_pool_get 2
 
@@ -102,7 +101,7 @@ echo "clean btrfs ko" >> $seqres.full
 _scratch_unmount
 
 # un-scan the btrfs devices
-_reload_btrfs_ko
+_reload_fs_module "btrfs"
 
 echo >> $seqres.full
 echo "-----Write degraded mount fill upto $max_fs_sz bytes-----" >> $seqres.full
@@ -141,7 +140,7 @@ echo
 echo "Mount degraded with the other dev"
 _scratch_unmount
 # un-scan the btrfs devices
-_reload_btrfs_ko
+_reload_fs_module "btrfs"
 _mount -o degraded $dev2 $SCRATCH_MNT >>$seqres.full 2>&1
 _run_btrfs_util_prog filesystem show
 checkpoint3=`md5sum $SCRATCH_MNT/tf2`
diff --git a/tests/btrfs/125 b/tests/btrfs/125
index 91aa8d8..19961a1 100755
--- a/tests/btrfs/125
+++ b/tests/btrfs/125
@@ -54,6 +54,7 @@ _cleanup()
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
+. ./common/module
 
 # remove previous $seqres.full before test
 rm -f $seqres.full
@@ -63,10 +64,8 @@ rm -f $seqres.full
 _supported_fs btrfs
 _supported_os Linux
 _require_scratch_dev_pool 3
-
-# we need btrfs to unload, need test dir unmounted
 _test_unmount
-_require_btrfs_loadable
+_require_loadable_fs_module "btrfs"
 
 _scratch_dev_pool_get 3
 
@@ -118,7 +117,7 @@ echo "unmount" >> $seqres.full
 _scratch_unmount
 echo "clean btrfs ko" >> $seqres.full
 # un-scan the btrfs devices
-_reload_btrfs_ko
+_reload_fs_module "btrfs"
 _mount -o degraded,device=$dev2 $dev1 $SCRATCH_MNT >>$seqres.full 2>&1
 dd if=/dev/zero of="$SCRATCH_MNT"/tf2 bs=$bs count=$count \
 					>>$seqres.full 2>&1
@@ -154,7 +153,7 @@ echo "Mount degraded but with other dev"
 
 _scratch_unmount
 # un-scan the btrfs devices
-_reload_btrfs_ko
+_reload_fs_module "btrfs"
 
 _mount -o degraded,device=${dev2} $dev3 $SCRATCH_MNT >>$seqres.full 2>&1
 

[PATCH 04/14] xfs: test that we don't leak inodes and dquots during failed cow recovery

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Add a couple of tests to check that we don't leak inodes or dquots
if CoW recovery fails and therefore the mount fails.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/rc         |    1 
 tests/xfs/703     |  111 +++++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/703.out |    9 ++++
 tests/xfs/704     |   89 ++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/704.out |    5 ++
 tests/xfs/705     |  105 ++++++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/705.out |    9 ++++
 tests/xfs/group   |    3 +
 8 files changed, 332 insertions(+)
 create mode 100755 tests/xfs/703
 create mode 100644 tests/xfs/703.out
 create mode 100755 tests/xfs/704
 create mode 100644 tests/xfs/704.out
 create mode 100755 tests/xfs/705
 create mode 100644 tests/xfs/705.out


diff --git a/common/rc b/common/rc
index c708bfa..7e453e0 100644
--- a/common/rc
+++ b/common/rc
@@ -3324,6 +3324,7 @@ _check_dmesg()
 	     -e "(INFO|ERR): suspicious RCU usage" \
 	     -e "INFO: possible circular locking dependency detected" \
 	     -e "general protection fault:" \
+	     -e "BUG .* remaining" \
 	     $seqres.dmesg
 	if [ $? -eq 0 ]; then
 		_dump_err "_check_dmesg: something found in dmesg (see $seqres.dmesg)"
diff --git a/tests/xfs/703 b/tests/xfs/703
new file mode 100755
index 0000000..b2e37e6
--- /dev/null
+++ b/tests/xfs/703
@@ -0,0 +1,111 @@
+#! /bin/bash
+# FS QA Test No. 703
+#
+# Ensure that we don't leak quota inodes when CoW recovery fails.
+#
+# Use xfs_fsr to inject bmap redo items in the log for a linked file and
+# an unlinked file; enable quota so that we always mount with the quota
+# inodes; and then corrupt the refcount btree to ensure that the CoW
+# garbage collection (and therefore the mount) fail.
+#
+# On a subsequent mount attempt, we should be able to replay the bmap
+# items for the linked and unlinked files without prematurely truncating
+# the unlinked inode and without leaking the linked inode, and we should
+# be able to release the quota inodes when we're aborting the mount.  We
+# also should not leak dquots.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename "$0"`
+seqres="$RESULT_DIR/$seq"
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf "$tmp".*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+. ./common/reflink
+. ./common/inject
+. ./common/quota
+. ./common/module
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_loadable_fs_module "xfs"
+_require_quota
+_require_scratch_reflink
+_require_cp_reflink
+_require_command "$XFS_FSR_PROG" "xfs_fsr"
+_require_xfs_io_error_injection "bmap_finish_one"
+_require_xfs_scratch_rmapbt
+
+rm -f "$seqres.full"
+
+echo "Format and mount"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount -o noquota >> "$seqres.full" 2>&1
+
+testdir="$SCRATCH_MNT/test-$seq"
+blksz=65536
+blks=3
+mkdir "$testdir"
+
+echo "Create a many-block file"
+_pwrite_byte 0x62 0 $((blksz * blks)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 0 $blksz $testdir/file2 >> $seqres.full
+_reflink_range $testdir/file2 0 $testdir/file1 $blksz $blksz >> $seqres.full
+_scratch_cycle_mount noquota
+
+echo "Inject error"
+_scratch_inject_error "bmap_finish_one"
+
+echo "Defrag the file"
+$XFS_FSR_PROG -v -d $testdir/file1 >> $seqres.full 2>&1
+
+echo "FS should be shut down, touch will fail"
+touch $SCRATCH_MNT/badfs 2>&1 | _filter_scratch
+
+echo "Remount to replay log" | tee /dev/ttyprintk
+_scratch_unmount
+_scratch_dump_log >> $seqres.full
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c 'fuzz -d recs[1].startblock ones' >> $seqres.full
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c p >> $seqres.full
+
+# Suddenly enable quota to test if we can leak the quotacheck dquots!
+_scratch_mount -o quota >> $seqres.full 2>&1
+_scratch_unmount 2> /dev/null
+rm -f ${RESULT_DIR}/require_scratch
+
+echo "See if we leak"
+_reload_fs_module "xfs"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/703.out b/tests/xfs/703.out
new file mode 100644
index 0000000..d811880
--- /dev/null
+++ b/tests/xfs/703.out
@@ -0,0 +1,9 @@
+QA output created by 703
+Format and mount
+Create a many-block file
+Inject error
+Defrag the file
+FS should be shut down, touch will fail
+touch: cannot touch 'SCRATCH_MNT/badfs': Input/output error
+Remount to replay log
+See if we leak
diff --git a/tests/xfs/704 b/tests/xfs/704
new file mode 100755
index 0000000..3f4d704
--- /dev/null
+++ b/tests/xfs/704
@@ -0,0 +1,89 @@
+#! /bin/bash
+# FS QA Test No. 704
+#
+# Ensure that we don't leak dquots when CoW recovery fails.
+#
+# Corrupt the refcount btree to ensure that the CoW garbage collection
+# (and therefore the mount) fail.
+#
+# On a subsequent mount attempt, we should be able to release the quota
+# inodes when we're aborting the mount.  We also should not leak dquots.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename "$0"`
+seqres="$RESULT_DIR/$seq"
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf "$tmp".*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+. ./common/reflink
+. ./common/quota
+. ./common/module
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_loadable_fs_module "xfs"
+_require_quota
+_require_scratch_reflink
+_require_cp_reflink
+
+rm -f "$seqres.full"
+
+echo "Format and mount"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount -o quota >> "$seqres.full" 2>&1
+
+testdir="$SCRATCH_MNT/test-$seq"
+blksz=65536
+blks=3
+mkdir "$testdir"
+
+echo "Create a many-block file"
+_pwrite_byte 0x62 0 $((blksz * blks)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 0 $blksz $testdir/file2 >> $seqres.full
+_reflink_range $testdir/file2 0 $testdir/file1 $blksz $blksz >> $seqres.full
+
+echo "Remount to check recovery" | tee /dev/ttyprintk
+_scratch_unmount
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c 'fuzz -d recs[1].startblock ones' >> $seqres.full
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c p >> $seqres.full
+_scratch_mount -o quota >> $seqres.full 2>&1
+_scratch_unmount 2> /dev/null
+rm -f ${RESULT_DIR}/require_scratch
+
+echo "See if we leak"
+_reload_fs_module "xfs"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/704.out b/tests/xfs/704.out
new file mode 100644
index 0000000..b237b2d
--- /dev/null
+++ b/tests/xfs/704.out
@@ -0,0 +1,5 @@
+QA output created by 704
+Format and mount
+Create a many-block file
+Remount to check recovery
+See if we leak
diff --git a/tests/xfs/705 b/tests/xfs/705
new file mode 100755
index 0000000..6019a4e
--- /dev/null
+++ b/tests/xfs/705
@@ -0,0 +1,105 @@
+#! /bin/bash
+# FS QA Test No. 705
+#
+# Ensure that we don't leak inodes when CoW recovery fails.
+#
+# Use xfs_fsr to inject bmap redo items in the log for a linked file and
+# an unlinked file; and then corrupt the refcount btree to ensure that
+# the CoW garbage collection (and therefore the mount) fail.
+#
+# On a subsequent mount attempt, we should be able to replay the bmap
+# items for the linked and unlinked files without prematurely truncating
+# the unlinked inode and without leaking the linked inode, and we should
+# be able to release all the inodes when we're aborting the mount.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename "$0"`
+seqres="$RESULT_DIR/$seq"
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+	cd /
+	rm -rf "$tmp".*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+. ./common/attr
+. ./common/reflink
+. ./common/inject
+. ./common/module
+
+# real QA test starts here
+_supported_os Linux
+_supported_fs xfs
+_require_loadable_fs_module "xfs"
+_require_scratch_reflink
+_require_cp_reflink
+_require_command "$XFS_FSR_PROG" "xfs_fsr"
+_require_xfs_io_error_injection "bmap_finish_one"
+_require_xfs_scratch_rmapbt
+
+rm -f "$seqres.full"
+
+echo "Format and mount"
+_scratch_mkfs > "$seqres.full" 2>&1
+_scratch_mount -o noquota >> "$seqres.full" 2>&1
+
+testdir="$SCRATCH_MNT/test-$seq"
+blksz=65536
+blks=3
+mkdir "$testdir"
+
+echo "Create a many-block file"
+_pwrite_byte 0x62 0 $((blksz * blks)) $testdir/file1 >> $seqres.full
+_pwrite_byte 0x63 0 $blksz $testdir/file2 >> $seqres.full
+_reflink_range $testdir/file2 0 $testdir/file1 $blksz $blksz >> $seqres.full
+_scratch_cycle_mount noquota
+
+echo "Inject error"
+_scratch_inject_error "bmap_finish_one"
+
+echo "Defrag the file"
+$XFS_FSR_PROG -v -d $testdir/file1 >> $seqres.full 2>&1
+
+echo "FS should be shut down, touch will fail"
+touch $SCRATCH_MNT/badfs 2>&1 | _filter_scratch
+
+echo "Remount to replay log" | tee /dev/ttyprintk
+_scratch_unmount
+_scratch_dump_log >> $seqres.full
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c 'fuzz -d recs[1].startblock ones' >> $seqres.full
+_scratch_xfs_db -x -c 'agf 0' -c 'addr refcntroot' -c p >> $seqres.full
+_scratch_mount -o noquota >> $seqres.full 2>&1
+_scratch_unmount 2> /dev/null
+rm -f ${RESULT_DIR}/require_scratch
+
+echo "See if we leak"
+_reload_fs_module "xfs"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/705.out b/tests/xfs/705.out
new file mode 100644
index 0000000..0bcd9ab
--- /dev/null
+++ b/tests/xfs/705.out
@@ -0,0 +1,9 @@
+QA output created by 705
+Format and mount
+Create a many-block file
+Inject error
+Defrag the file
+FS should be shut down, touch will fail
+touch: cannot touch 'SCRATCH_MNT/badfs': Input/output error
+Remount to replay log
+See if we leak
diff --git a/tests/xfs/group b/tests/xfs/group
index b439842..e56666b 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -431,3 +431,6 @@
 431 auto quick dangerous
 432 auto quick dir metadata
 433 auto quick attr
+703 auto quick clone fsr
+704 auto quick clone
+705 auto quick clone fsr

[PATCH 05/14] xfs/333: fix errors with new inode pointer verifiers

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Fix test failures with new inode pointer verifiers... and also make sure
that the running xfs actually supports realtime rmap.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/333     |   20 ++++++--------------
 tests/xfs/333.out |    7 ++-----
 2 files changed, 8 insertions(+), 19 deletions(-)


diff --git a/tests/xfs/333 b/tests/xfs/333
index f7f233d..57aae09 100755
--- a/tests/xfs/333
+++ b/tests/xfs/333
@@ -51,6 +51,8 @@ unset SCRATCH_RTDEV
 
 echo "Format and mount"
 _scratch_mkfs > "$seqres.full" 2>&1
+rrmapino="$(_scratch_xfs_db -c 'sb 0' -c 'p rrmapino' 2>&1)"
+test "${rrmapino}" = "field rrmapino not found" && _notrun "realtime rmapbt not supported"
 _scratch_mount
 
 echo "Create some files"
@@ -62,21 +64,11 @@ _scratch_unmount
 
 echo "Corrupt fs"
 _scratch_xfs_db -x -c 'sb 0' -c "write rrmapino $ino" >> $seqres.full
-_scratch_mount
-
-echo "Check files"
-md5sum $SCRATCH_MNT/f1 2>&1 | _filter_scratch
-
-echo "Try to create more files"
-$XFS_IO_PROG -f -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f3 >> $seqres.full 2>&1
+_scratch_mount 2>&1 | _filter_scratch
 
-echo "Repair fs"
-_scratch_unmount 2>&1 | _filter_scratch
-_repair_scratch_fs >> $seqres.full 2>&1
-
-echo "Try to create more files (again)"
-_scratch_mount
-$XFS_IO_PROG -f -c "pwrite -S 0x68 0 9999" $SCRATCH_MNT/f4 >> $seqres.full
+echo "Repair, mount again"
+_repair_scratch_fs >> $seqres.full
+_scratch_mount 2>&1 | _filter_scratch
 
 # success, all done
 status=0
diff --git a/tests/xfs/333.out b/tests/xfs/333.out
index bee9bbc..5cac447 100644
--- a/tests/xfs/333.out
+++ b/tests/xfs/333.out
@@ -2,8 +2,5 @@ QA output created by 333
 Format and mount
 Create some files
 Corrupt fs
-Check files
-8f27047948255cb84872e2dd7c0bc56d  SCRATCH_MNT/f1
-Try to create more files
-Repair fs
-Try to create more files (again)
+mount: mount SCRATCH_DEV on SCRATCH_MNT failed: Structure needs cleaning
+Repair, mount again

[PATCH 06/14] generic/459: fix test running errors

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

If the DISCARD of the thin device somehow fails with this message:

device-mapper: thin: Data device (dm-1) discard unsupported: Disabling discard passdown.

Then we can end up with arbitrary gunk in the thin device.  This causes
mkfs to fail because it's afraid to format the device.  Don't be afraid,
just zap it.  FWIW mkfs.xfs thinks that the thinp device has an xfs
external log because sometimes the thinp device just happen to be backed
by the log of the previous test's scratch fs.

Fix this by making the _mkfs_dev helper always format the device, per
Eryu Guan's suggestion.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/rc |    5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)


diff --git a/common/rc b/common/rc
index 7e453e0..c441199 100644
--- a/common/rc
+++ b/common/rc
@@ -642,7 +642,10 @@ _mkfs_dev()
 	$MKFS_PROG -t $FSTYP -- -F $MKFS_OPTIONS $* \
 		2>$tmp.mkfserr 1>$tmp.mkfsstd
 	;;
-
+    xfs)
+	yes | $MKFS_PROG -t $FSTYP -- -f $MKFS_OPTIONS $* \
+		2>$tmp.mkfserr 1>$tmp.mkfsstd
+	;;
     *)
 	yes | $MKFS_PROG -t $FSTYP -- $MKFS_OPTIONS $* \
 		2>$tmp.mkfserr 1>$tmp.mkfsstd

[PATCH 07/14] generic/459: explicitly require thin_check

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The lvm command can invoke the thin pool utilities as part of managing
a thin volume.  It'll fail if the thin provisioning utilities are not
installed, so we need to check for its presence before running a test.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/config     |    1 +
 tests/generic/459 |    1 +
 2 files changed, 2 insertions(+)


diff --git a/common/config b/common/config
index 96503c6..eea8669 100644
--- a/common/config
+++ b/common/config
@@ -196,6 +196,7 @@ export MAN_PROG="`set_prog_path man`"
 export NFS4_SETFACL_PROG="`set_prog_path nfs4_setfacl`"
 export NFS4_GETFACL_PROG="`set_prog_path nfs4_getfacl`"
 export UBIUPDATEVOL_PROG="`set_prog_path ubiupdatevol`"
+THIN_CHECK_PROG="$(set_prog_path thin_check)"
 
 # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
 # newer systems have udevadm command but older systems like RHEL5 don't.
diff --git a/tests/generic/459 b/tests/generic/459
index 7c10c2a..d1ad372 100755
--- a/tests/generic/459
+++ b/tests/generic/459
@@ -63,6 +63,7 @@ _supported_os Linux
 _require_scratch_nocheck
 _require_dm_target thin-pool
 _require_command $LVM_PROG lvm
+_require_command "$THIN_CHECK_PROG" thin_check
 _require_freeze
 
 # remove previous $seqres.full before test

[PATCH 08/14] common/xfs: remove inode-paths cruft

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Remove the inode-paths check from _check_xfs_test_fs because we don't
support inode paths, xfsprogs doesn't have a xfs_{check,repair}_ipaths
tool, and it's broken anyway because we ignore _check_xfs_filesystem
(which tells whether or not the filesystem is even still mounted).

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/xfs |    7 -------
 1 file changed, 7 deletions(-)


diff --git a/common/xfs b/common/xfs
index 5643adf..79efb78 100644
--- a/common/xfs
+++ b/common/xfs
@@ -471,13 +471,6 @@ _check_xfs_test_fs()
 		TEST_RT="$TEST_RTDEV"
 
 	_check_xfs_filesystem $TEST_DEV $TEST_LOG $TEST_RT
-
-	# check for ipath consistency
-	if $XFS_GROWFS_PROG -n $TEST_DIR | grep -q 'inode-paths=1'; then
-		# errors go to stderr
-		xfs_check_ipaths $TEST_DIR >/dev/null
-		xfs_repair_ipaths -n $TEST_DIR >/dev/null
-	fi
 }
 
 _require_xfs_test_rmapbt()

[PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

A directory corrupted into a symlink will be caught by the upcoming
local format ifork verifiers.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/348.out |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/tests/xfs/348.out b/tests/xfs/348.out
index f4a7a71..17d9be2 100644
--- a/tests/xfs/348.out
+++ b/tests/xfs/348.out
@@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
 would have junked entry "DIR" in directory PARENT_INO
 would have junked entry "EMPTY" in directory PARENT_INO
 would have junked entry "FIFO" in directory PARENT_INO
-stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
+stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
 stat: 'SCRATCH_MNT/test/DATA' is a symbolic link
 stat: cannot stat 'SCRATCH_MNT/test/EMPTY': Structure needs cleaning
 stat: 'SCRATCH_MNT/test/SYMLINK' is a symbolic link

[PATCH 10/14] xfs/122: add inode log formats

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The inode log record formats have been standardized, so check them here.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/122.out |    2 ++
 1 file changed, 2 insertions(+)


diff --git a/tests/xfs/122.out b/tests/xfs/122.out
index 9ca30b7..ad9a055 100644
--- a/tests/xfs/122.out
+++ b/tests/xfs/122.out
@@ -83,6 +83,8 @@ sizeof(struct xfs_extent_data_info) = 32
 sizeof(struct xfs_fs_eofblocks) = 128
 sizeof(struct xfs_fsop_ag_resblks) = 64
 sizeof(struct xfs_icreate_log) = 28
+sizeof(struct xfs_inode_log_format) = 56
+sizeof(struct xfs_inode_log_format_32) = 52
 sizeof(struct xfs_log_dinode) = 176
 sizeof(struct xfs_map_extent) = 32
 sizeof(struct xfs_phys_extent) = 16

[PATCH 11/14] xfs/010: filter and record the unknown block state messages

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Record the unknown block state messages that xfs_repair produces when
we nuke the finobt.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/repair |    9 +++++++++
 tests/xfs/010 |    2 +-
 tests/xfs/030 |    2 +-
 3 files changed, 11 insertions(+), 2 deletions(-)


diff --git a/common/repair b/common/repair
index 2b1583b..b569541 100644
--- a/common/repair
+++ b/common/repair
@@ -95,9 +95,18 @@ s/(stripe unit) \(.*\) (and width) \(.*\)/\1 (SU) \2 (SW)/;
 s/(superblock) (\d+)/\1 AGNO/;
 s/(AG \#)(\d+)/\1AGNO/;
 s/(reset bad sb for ag) (\d+)/\1 AGNO/;
+s/(unknown block state, ag )(\d+)(, block )(\d+)/\1AGNO\3AGBNO/;
 	print;'
 }
 
+# Filter out unknown block state messages that appear when rmap is enabled
+# and we erase a btree root pointer (such that repair never finds the
+# tree and fails to reconcile the metadata reverse mappings against the
+# metadata).
+_filter_repair_lostblocks() {
+	_filter_repair | sed -e '/unknown block state, ag AGNO, block AGBNO/d'
+}
+
 _filter_dd()
 {
 	fgrep -v records	# lose records in/out lines
diff --git a/tests/xfs/010 b/tests/xfs/010
index b9f2891..17607d6 100755
--- a/tests/xfs/010
+++ b/tests/xfs/010
@@ -129,7 +129,7 @@ _check_scratch_fs
 # nuke the finobt root, repair will have to regenerate from the inobt
 _corrupt_finobt_root $SCRATCH_DEV
 
-_scratch_xfs_repair 2>&1 | _filter_repair
+_scratch_xfs_repair 2>&1 | _filter_repair_lostblocks
 
 status=0
 exit
diff --git a/tests/xfs/030 b/tests/xfs/030
index bc80775..fcea754 100755
--- a/tests/xfs/030
+++ b/tests/xfs/030
@@ -58,7 +58,7 @@ _check_ag()
 			    -e '/^bad agbno AGBNO for rmapbt/d' \
 			    -e '/^bad agbno AGBNO for refcntbt/d' \
 			    -e '/^Missing reverse-mapping record.*/d' \
-			    -e '/^unknown block state, ag 0, block.*/d'
+			    -e '/^unknown block state, ag AGNO, block.*/d'
 	done
 }
 

[PATCH 12/14] generic/204: break out of fs-filling loop early if we ENOSPC

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

If filling up the filesystem causes us to hit ENOSPC earlier than we
thought we would (the sizing estimates become less and less accurate as
we add more metadata) then just bail out -- we're checking that the fs
is robust enough to cut us off before we actually run out of space for
writing metadata and crash the fs.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/204 |   11 +++++++++--
 1 file changed, 9 insertions(+), 2 deletions(-)


diff --git a/tests/generic/204 b/tests/generic/204
index 4c203a2..1e2c1e1 100755
--- a/tests/generic/204
+++ b/tests/generic/204
@@ -82,8 +82,15 @@ echo files $files, resvblks $resv_blks >> $seqres.full
 _scratch_resvblks $resv_blks >> $seqres.full 2>&1
 
 for i in `seq 1 $files`; do
-    echo -n > $SCRATCH_MNT/$i
-    echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > $SCRATCH_MNT/$i
+	# Open/truncate file, write stuff.  If we run out of space early,
+	# we can bail out of the loop.
+	out="$($XFS_IO_PROG \
+		-c "open -f -t $SCRATCH_MNT/$i" \
+		-c 'close' \
+		-c "open -f -t $SCRATCH_MNT/$i" \
+		-c 'pwrite -q -S 0x58 0 36' 2>&1 | _filter_scratch)"
+	echo "${out}" | grep -q 'No space left on device' && break
+	test -n "${out}" && echo "${out}"
 done
 
 # success, all done

[PATCH 13/14] xfs/013: don't fail because cp ran out of space

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Don't fail xfs/013 just because cp -Rl runs out of space to allocate
inodes and sprays the ENOSPC messages into the golden output.  We want
to stress the finobt by using cp to push us near ENOSPC conditions,
so it's fine to let cp run out of space.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/013 |   11 ++++++++---
 1 file changed, 8 insertions(+), 3 deletions(-)


diff --git a/tests/xfs/013 b/tests/xfs/013
index 4e49e4b..80298ca 100755
--- a/tests/xfs/013
+++ b/tests/xfs/013
@@ -48,6 +48,10 @@ _cleanup()
 }
 trap "_cleanup; exit \$status" 0 1 2 3 15
 
+filter_enospc() {
+	sed -e '/^.*No space left on device.*/d'
+}
+
 _create()
 {
 	dir=$1
@@ -56,7 +60,7 @@ _create()
 	mkdir -p $dir
 	for i in $(seq 0 $count)
 	do
-		touch $dir/$i
+		touch $dir/$i 2>&1 | filter_enospc
 	done
 }
 
@@ -70,7 +74,7 @@ _rand_replace()
 	do
 		file=$((RANDOM % count))
 		rm -f $dir/$file
-		touch $dir/$file
+		touch $dir/$file 2>&1 | filter_enospc
 	done
 }
 
@@ -141,7 +145,8 @@ $FSSTRESS_PROG -d $SCRATCH_MNT/fsstress -n 9999999 -p 2 -S t \
 for i in $(seq 1 $LOOPS)
 do
 	# hard link the content of the current directory to the next
-	cp -Rl $SCRATCH_MNT/dir$i $SCRATCH_MNT/dir$((i+1))
+	cp -Rl $SCRATCH_MNT/dir$i $SCRATCH_MNT/dir$((i+1)) 2>&1 | \
+		filter_enospc
 
 	# do a random replacement of files in the new directory
 	_rand_replace $SCRATCH_MNT/dir$((i+1)) $COUNT

[PATCH 14/14] generic: test IO at maximum file offset

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure we can write to and read from the highest possible offset
that Linux will allow.  Format the filesystem with a variety of
possible blocksizes to stress the filesystem.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/705     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/705.out |    2 +
 tests/generic/group   |    1 +
 3 files changed, 100 insertions(+)
 create mode 100755 tests/generic/705
 create mode 100644 tests/generic/705.out


diff --git a/tests/generic/705 b/tests/generic/705
new file mode 100755
index 0000000..4ab97e5
--- /dev/null
+++ b/tests/generic/705
@@ -0,0 +1,97 @@
+#! /bin/bash
+# FS QA Test No. 705
+#
+# Check that high-offset reads and writes work.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch
+
+rm -f $seqres.full
+
+echo "Silence is golden"
+
+echo "Starting test" > $seqres.full
+for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
+	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
+	devsize=$(blockdev --getsize64 $SCRATCH_DEV)
+	_scratch_unmount > /dev/null 2>&1
+	# Try to format and mount with the given blocksize.  If they don't
+	# succeed, move on to the next block size.
+	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
+	   ! _scratch_mount >> $seqres.full 2>&1 ||
+	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
+		echo "+++ Format and mount failed" >> $seqres.full
+		continue
+	fi
+
+	testdir=$SCRATCH_MNT/test-$seq
+	mkdir $testdir
+
+	echo "++ Create the original files" >> $seqres.full
+	bigoff=9223372036854775806
+	len=9223372036854775807
+	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
+	if [ ! -s $testdir/file0 ]; then
+		# If we can't set a large file size then don't bother
+		# with this blocksize because the fs doesn't support it.
+		echo "+++ High offset ftruncate failed" >> $seqres.full
+		continue
+	fi
+	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
+
+	echo "++ Check file creation" >> $seqres.full
+	_scratch_cycle_mount
+
+	expected="7ffffffffffffffe:  61  a"
+	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
+	if [ "$expected" = "$actual" ]; then
+		echo "+++ Success!" >> $seqres.full
+	else
+		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
+		echo "Discrepancy @ blocksize $blocksize"
+	fi
+
+	echo "++ Check scratchfs" >> $seqres.full
+	_check_scratch_fs
+done
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/705.out b/tests/generic/705.out
new file mode 100644
index 0000000..6fcdec0
--- /dev/null
+++ b/tests/generic/705.out
@@ -0,0 +1,2 @@
+QA output created by 705
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index fbe0a7f..2c55b93 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -468,3 +468,4 @@
 463 auto quick clone dangerous
 464 auto rw
 465 auto rw quick aio
+705 auto quick rw

Re: [PATCH 00/14] rollup of fstests fixes

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:46:28PM -0700, Darrick J. Wong wrote:
> This is a rollup of all the patches I've been sending to the list,
> because the previous thread is a mess of resent patches and patch
> series cramming.
> 
> Patches 1-2,4,6,8 are unchanged from last week.  They've possibly even
> gone in already, but I can't tell what's in Eryu's private branch. :/
> 
> Patch 3 is the same as "PATCH v3 5/6" from the last series.

Yeah, I've queued above patches in my staging branch, a user-visible
change I've made on commit is in patch 8 "common/xfs: remove inode-paths
cruft", I added a "return $?" at the end of _check_xfs_test_fs(), as I
replied to you earlier this week. So I'd just leave my queued patches
untouched, that might be easier for me :)

> 
> Patch 5 now checks that repair can fix the corruption introduced.

It was in my queue too, but I'll drop it and take the new version if
the change looks sane.

> 
> Patch 7 is new, it checks for /usr/sbin/thin_check before running
> generic/459 per an earlier discussion on the list.
> 
> Patches 9-14 are new; I've been trying to clean up all the minor test
> failures in preparation for 4.15 madness.

Thanks a lot for the cleanups! I'll look at them, maybe tommorrow, I'll
be out for a few hours today..

Thanks,
Eryu

Re: [PATCH 00/14] rollup of fstests fixes

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 11:06:01AM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:46:28PM -0700, Darrick J. Wong wrote:
> > This is a rollup of all the patches I've been sending to the list,
> > because the previous thread is a mess of resent patches and patch
> > series cramming.
> > 
> > Patches 1-2,4,6,8 are unchanged from last week.  They've possibly even
> > gone in already, but I can't tell what's in Eryu's private branch. :/
> > 
> > Patch 3 is the same as "PATCH v3 5/6" from the last series.
> 
> Yeah, I've queued above patches in my staging branch, a user-visible
> change I've made on commit is in patch 8 "common/xfs: remove inode-paths
> cruft", I added a "return $?" at the end of _check_xfs_test_fs(), as I
> replied to you earlier this week. So I'd just leave my queued patches
> untouched, that might be easier for me :)

Ok, works for me!

> > 
> > Patch 5 now checks that repair can fix the corruption introduced.
> 
> It was in my queue too, but I'll drop it and take the new version if
> the change looks sane.
> 
> > 
> > Patch 7 is new, it checks for /usr/sbin/thin_check before running
> > generic/459 per an earlier discussion on the list.
> > 
> > Patches 9-14 are new; I've been trying to clean up all the minor test
> > failures in preparation for 4.15 madness.
> 
> Thanks a lot for the cleanups! I'll look at them, maybe tommorrow, I'll
> be out for a few hours today..

<nod>

--D

> Thanks,
> Eryu
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/14] generic/459: explicitly require thin_check

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:47:11PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> The lvm command can invoke the thin pool utilities as part of managing
> a thin volume.  It'll fail if the thin provisioning utilities are not
> installed, so we need to check for its presence before running a test.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  common/config     |    1 +
>  tests/generic/459 |    1 +
>  2 files changed, 2 insertions(+)
> 
> 
> diff --git a/common/config b/common/config
> index 96503c6..eea8669 100644
> --- a/common/config
> +++ b/common/config
> @@ -196,6 +196,7 @@ export MAN_PROG="`set_prog_path man`"
>  export NFS4_SETFACL_PROG="`set_prog_path nfs4_setfacl`"
>  export NFS4_GETFACL_PROG="`set_prog_path nfs4_getfacl`"
>  export UBIUPDATEVOL_PROG="`set_prog_path ubiupdatevol`"
> +THIN_CHECK_PROG="$(set_prog_path thin_check)"

I added an 'export' here.

Thanks,
Eryu

>  
>  # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
>  # newer systems have udevadm command but older systems like RHEL5 don't.
> diff --git a/tests/generic/459 b/tests/generic/459
> index 7c10c2a..d1ad372 100755
> --- a/tests/generic/459
> +++ b/tests/generic/459
> @@ -63,6 +63,7 @@ _supported_os Linux
>  _require_scratch_nocheck
>  _require_dm_target thin-pool
>  _require_command $LVM_PROG lvm
> +_require_command "$THIN_CHECK_PROG" thin_check
>  _require_freeze
>  
>  # remove previous $seqres.full before test
> 

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> A directory corrupted into a symlink will be caught by the upcoming
> local format ifork verifiers.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/xfs/348.out |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> 
> diff --git a/tests/xfs/348.out b/tests/xfs/348.out
> index f4a7a71..17d9be2 100644
> --- a/tests/xfs/348.out
> +++ b/tests/xfs/348.out
> @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
>  would have junked entry "DIR" in directory PARENT_INO
>  would have junked entry "EMPTY" in directory PARENT_INO
>  would have junked entry "FIFO" in directory PARENT_INO
> -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
> +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning

But this breaks tests on old kernels. Or with the new ifork verifiers,
old kernels can be considered as buggy?

Thanks,
Eryu

>  stat: 'SCRATCH_MNT/test/DATA' is a symbolic link
>  stat: cannot stat 'SCRATCH_MNT/test/EMPTY': Structure needs cleaning
>  stat: 'SCRATCH_MNT/test/SYMLINK' is a symbolic link
> 

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Amir Goldstein]

On Thu, Nov 2, 2017 at 2:42 PM, Eryu Guan <eguan@redhat.com> wrote:
> On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
>> From: Darrick J. Wong <darrick.wong@oracle.com>
>>
>> A directory corrupted into a symlink will be caught by the upcoming
>> local format ifork verifiers.
>>
>> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>> ---
>>  tests/xfs/348.out |    2 +-
>>  1 file changed, 1 insertion(+), 1 deletion(-)
>>
>>
>> diff --git a/tests/xfs/348.out b/tests/xfs/348.out
>> index f4a7a71..17d9be2 100644
>> --- a/tests/xfs/348.out
>> +++ b/tests/xfs/348.out
>> @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
>>  would have junked entry "DIR" in directory PARENT_INO
>>  would have junked entry "EMPTY" in directory PARENT_INO
>>  would have junked entry "FIFO" in directory PARENT_INO
>> -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
>> +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
>
> But this breaks tests on old kernels. Or with the new ifork verifiers,
> old kernels can be considered as buggy?
>

Can filter new and old error into canonicalized string.
Dodgy, bug we've done it before.
See:
git grep sed | grep 'Operation not permitted\|Permission denied'

Amir.

Re: [PATCH 12/14] generic/204: break out of fs-filling loop early if we ENOSPC

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:47:42PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> If filling up the filesystem causes us to hit ENOSPC earlier than we
> thought we would (the sizing estimates become less and less accurate as
> we add more metadata) then just bail out -- we're checking that the fs
> is robust enough to cut us off before we actually run out of space for
> writing metadata and crash the fs.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/generic/204 |   11 +++++++++--
>  1 file changed, 9 insertions(+), 2 deletions(-)
> 
> 
> diff --git a/tests/generic/204 b/tests/generic/204
> index 4c203a2..1e2c1e1 100755
> --- a/tests/generic/204
> +++ b/tests/generic/204
> @@ -82,8 +82,15 @@ echo files $files, resvblks $resv_blks >> $seqres.full
>  _scratch_resvblks $resv_blks >> $seqres.full 2>&1
>  
>  for i in `seq 1 $files`; do
> -    echo -n > $SCRATCH_MNT/$i
> -    echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > $SCRATCH_MNT/$i
> +	# Open/truncate file, write stuff.  If we run out of space early,
> +	# we can bail out of the loop.
> +	out="$($XFS_IO_PROG \
> +		-c "open -f -t $SCRATCH_MNT/$i" \
> +		-c 'close' \
> +		-c "open -f -t $SCRATCH_MNT/$i" \
> +		-c 'pwrite -q -S 0x58 0 36' 2>&1 | _filter_scratch)"
> +	echo "${out}" | grep -q 'No space left on device' && break

This doesn't look correct to me. This test is meant to catch spurious
ENOSPC, it's designed to be a "delayed allocation ENOSPC test"[1] by
"writing lots of single block files" and catch early ENOSPC. IMHO, this
change ignores ENOSPC, which defeats the test purpose. And I don't quite
understand the purpose of truncate open/close/truncate open/write
sequence..

Also this significantly increases the test time, 4s -> ~120s for me.

Thanks,
Eryu

[1] commit 143368a047ea ("xfstests: add test 204, a simple delayed
allocation ENOSPC test")

> +	test -n "${out}" && echo "${out}"
>  done
>  
>  # success, all done
> 

Re: [PATCH 14/14] generic: test IO at maximum file offset

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:47:54PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Make sure we can write to and read from the highest possible offset
> that Linux will allow.  Format the filesystem with a variety of
> possible blocksizes to stress the filesystem.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/generic/705     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/705.out |    2 +
>  tests/generic/group   |    1 +
>  3 files changed, 100 insertions(+)
>  create mode 100755 tests/generic/705
>  create mode 100644 tests/generic/705.out
> 
> 
> diff --git a/tests/generic/705 b/tests/generic/705
> new file mode 100755
> index 0000000..4ab97e5
> --- /dev/null
> +++ b/tests/generic/705
> @@ -0,0 +1,97 @@
> +#! /bin/bash
> +# FS QA Test No. 705
> +#
> +# Check that high-offset reads and writes work.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1    # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +    cd /
> +    rm -rf $tmp.*
> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_os Linux
> +_require_scratch

As we do _check_scratch_fs after each loop, _require_scratch_nocheck
should be OK here.

> +
> +rm -f $seqres.full
> +
> +echo "Silence is golden"
> +
> +echo "Starting test" > $seqres.full
> +for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
> +	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
> +	devsize=$(blockdev --getsize64 $SCRATCH_DEV)
> +	_scratch_unmount > /dev/null 2>&1
> +	# Try to format and mount with the given blocksize.  If they don't
> +	# succeed, move on to the next block size.
> +	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
> +	   ! _scratch_mount >> $seqres.full 2>&1 ||
> +	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
> +		echo "+++ Format and mount failed" >> $seqres.full
> +		continue

We need to unset MKFS_OPTIONS before this loop, because
_scratch_mkfs_sized takes the block size specified in MKFS_OPTIONS as
the default size and ignores the second argument, the provided block
size is only used if no block size specified in MKFS_OPTIONS.

> +	fi
> +
> +	testdir=$SCRATCH_MNT/test-$seq
> +	mkdir $testdir
> +
> +	echo "++ Create the original files" >> $seqres.full
> +	bigoff=9223372036854775806
> +	len=9223372036854775807

Calculating the numbers might be easier to read, e.g.

	len=`echo "2^63-1" | $BC_PROG`
	bigoff=`echo "$len-1" | $BC_PROG`

Thanks,
Eryu

> +	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
> +	if [ ! -s $testdir/file0 ]; then
> +		# If we can't set a large file size then don't bother
> +		# with this blocksize because the fs doesn't support it.
> +		echo "+++ High offset ftruncate failed" >> $seqres.full
> +		continue
> +	fi
> +	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
> +
> +	echo "++ Check file creation" >> $seqres.full
> +	_scratch_cycle_mount
> +
> +	expected="7ffffffffffffffe:  61  a"
> +	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
> +	if [ "$expected" = "$actual" ]; then
> +		echo "+++ Success!" >> $seqres.full
> +	else
> +		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
> +		echo "Discrepancy @ blocksize $blocksize"
> +	fi
> +
> +	echo "++ Check scratchfs" >> $seqres.full
> +	_check_scratch_fs
> +done
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/705.out b/tests/generic/705.out
> new file mode 100644
> index 0000000..6fcdec0
> --- /dev/null
> +++ b/tests/generic/705.out
> @@ -0,0 +1,2 @@
> +QA output created by 705
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index fbe0a7f..2c55b93 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -468,3 +468,4 @@
>  463 auto quick clone dangerous
>  464 auto rw
>  465 auto rw quick aio
> +705 auto quick rw
> 

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 08:42:14PM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > A directory corrupted into a symlink will be caught by the upcoming
> > local format ifork verifiers.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/xfs/348.out |    2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > 
> > diff --git a/tests/xfs/348.out b/tests/xfs/348.out
> > index f4a7a71..17d9be2 100644
> > --- a/tests/xfs/348.out
> > +++ b/tests/xfs/348.out
> > @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
> >  would have junked entry "DIR" in directory PARENT_INO
> >  would have junked entry "EMPTY" in directory PARENT_INO
> >  would have junked entry "FIFO" in directory PARENT_INO
> > -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
> > +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
> 
> But this breaks tests on old kernels. Or with the new ifork verifiers,
> old kernels can be considered as buggy?

Yes, they're buggy since we shouldn't be interpreting directory entries
as a link target given that the zero bytes in the "link target" will
screw things up.

--D

> Thanks,
> Eryu
> 
> >  stat: 'SCRATCH_MNT/test/DATA' is a symbolic link
> >  stat: cannot stat 'SCRATCH_MNT/test/EMPTY': Structure needs cleaning
> >  stat: 'SCRATCH_MNT/test/SYMLINK' is a symbolic link
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 03:14:45PM +0200, Amir Goldstein wrote:
> On Thu, Nov 2, 2017 at 2:42 PM, Eryu Guan <eguan@redhat.com> wrote:
> > On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
> >> From: Darrick J. Wong <darrick.wong@oracle.com>
> >>
> >> A directory corrupted into a symlink will be caught by the upcoming
> >> local format ifork verifiers.
> >>
> >> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> >> ---
> >>  tests/xfs/348.out |    2 +-
> >>  1 file changed, 1 insertion(+), 1 deletion(-)
> >>
> >>
> >> diff --git a/tests/xfs/348.out b/tests/xfs/348.out
> >> index f4a7a71..17d9be2 100644
> >> --- a/tests/xfs/348.out
> >> +++ b/tests/xfs/348.out
> >> @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
> >>  would have junked entry "DIR" in directory PARENT_INO
> >>  would have junked entry "EMPTY" in directory PARENT_INO
> >>  would have junked entry "FIFO" in directory PARENT_INO
> >> -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
> >> +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
> >
> > But this breaks tests on old kernels. Or with the new ifork verifiers,
> > old kernels can be considered as buggy?
> >
> 
> Can filter new and old error into canonicalized string.
> Dodgy, bug we've done it before.
> See:
> git grep sed | grep 'Operation not permitted\|Permission denied'

I tried that, but the "stat: XXX is a symbolic link" message comes from
a different path in the test file code.

Unless... you're suggesting that I should sed the error output to make
it look like the previous (broken) output?

--D

> 
> Amir.
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 14/14] generic: test IO at maximum file offset

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 11:16:27PM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:47:54PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Make sure we can write to and read from the highest possible offset
> > that Linux will allow.  Format the filesystem with a variety of
> > possible blocksizes to stress the filesystem.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/generic/705     |   97 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/705.out |    2 +
> >  tests/generic/group   |    1 +
> >  3 files changed, 100 insertions(+)
> >  create mode 100755 tests/generic/705
> >  create mode 100644 tests/generic/705.out
> > 
> > 
> > diff --git a/tests/generic/705 b/tests/generic/705
> > new file mode 100755
> > index 0000000..4ab97e5
> > --- /dev/null
> > +++ b/tests/generic/705
> > @@ -0,0 +1,97 @@
> > +#! /bin/bash
> > +# FS QA Test No. 705
> > +#
> > +# Check that high-offset reads and writes work.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1    # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +    cd /
> > +    rm -rf $tmp.*
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> > +_require_scratch
> 
> As we do _check_scratch_fs after each loop, _require_scratch_nocheck
> should be OK here.

Ok.

> > +
> > +rm -f $seqres.full
> > +
> > +echo "Silence is golden"
> > +
> > +echo "Starting test" > $seqres.full
> > +for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
> > +	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
> > +	devsize=$(blockdev --getsize64 $SCRATCH_DEV)
> > +	_scratch_unmount > /dev/null 2>&1
> > +	# Try to format and mount with the given blocksize.  If they don't
> > +	# succeed, move on to the next block size.
> > +	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
> > +	   ! _scratch_mount >> $seqres.full 2>&1 ||
> > +	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
> > +		echo "+++ Format and mount failed" >> $seqres.full
> > +		continue
> 
> We need to unset MKFS_OPTIONS before this loop, because
> _scratch_mkfs_sized takes the block size specified in MKFS_OPTIONS as
> the default size and ignores the second argument, the provided block
> size is only used if no block size specified in MKFS_OPTIONS.

Ok.  I've occasionally wondered if it would be useful to have a helper
that filters out the block size options from MKFS_OPTIONS so that we can
test whatever other mkfs options the testrunner specified, but with
various block sizes?

> > +	fi
> > +
> > +	testdir=$SCRATCH_MNT/test-$seq
> > +	mkdir $testdir
> > +
> > +	echo "++ Create the original files" >> $seqres.full
> > +	bigoff=9223372036854775806
> > +	len=9223372036854775807
> 
> Calculating the numbers might be easier to read, e.g.
> 
> 	len=`echo "2^63-1" | $BC_PROG`
> 	bigoff=`echo "$len-1" | $BC_PROG`

Ok.

(FWIW I copypasta'd this straight out of generic/303.)

> Thanks,
> Eryu
> 
> > +	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
> > +	if [ ! -s $testdir/file0 ]; then
> > +		# If we can't set a large file size then don't bother
> > +		# with this blocksize because the fs doesn't support it.
> > +		echo "+++ High offset ftruncate failed" >> $seqres.full
> > +		continue
> > +	fi
> > +	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
> > +
> > +	echo "++ Check file creation" >> $seqres.full
> > +	_scratch_cycle_mount
> > +
> > +	expected="7ffffffffffffffe:  61  a"
> > +	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
> > +	if [ "$expected" = "$actual" ]; then
> > +		echo "+++ Success!" >> $seqres.full
> > +	else
> > +		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
> > +		echo "Discrepancy @ blocksize $blocksize"
> > +	fi
> > +
> > +	echo "++ Check scratchfs" >> $seqres.full
> > +	_check_scratch_fs
> > +done
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/generic/705.out b/tests/generic/705.out
> > new file mode 100644
> > index 0000000..6fcdec0
> > --- /dev/null
> > +++ b/tests/generic/705.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 705
> > +Silence is golden
> > diff --git a/tests/generic/group b/tests/generic/group
> > index fbe0a7f..2c55b93 100644
> > --- a/tests/generic/group
> > +++ b/tests/generic/group
> > @@ -468,3 +468,4 @@
> >  463 auto quick clone dangerous
> >  464 auto rw
> >  465 auto rw quick aio
> > +705 auto quick rw
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 07/14] generic/459: explicitly require thin_check

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 08:25:14PM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:47:11PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > The lvm command can invoke the thin pool utilities as part of managing
> > a thin volume.  It'll fail if the thin provisioning utilities are not
> > installed, so we need to check for its presence before running a test.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  common/config     |    1 +
> >  tests/generic/459 |    1 +
> >  2 files changed, 2 insertions(+)
> > 
> > 
> > diff --git a/common/config b/common/config
> > index 96503c6..eea8669 100644
> > --- a/common/config
> > +++ b/common/config
> > @@ -196,6 +196,7 @@ export MAN_PROG="`set_prog_path man`"
> >  export NFS4_SETFACL_PROG="`set_prog_path nfs4_setfacl`"
> >  export NFS4_GETFACL_PROG="`set_prog_path nfs4_getfacl`"
> >  export UBIUPDATEVOL_PROG="`set_prog_path ubiupdatevol`"
> > +THIN_CHECK_PROG="$(set_prog_path thin_check)"
> 
> I added an 'export' here.

Oops, thanks for catching this.

--D

> 
> Thanks,
> Eryu
> 
> >  
> >  # use 'udevadm settle' or 'udevsettle' to wait for lv to be settled.
> >  # newer systems have udevadm command but older systems like RHEL5 don't.
> > diff --git a/tests/generic/459 b/tests/generic/459
> > index 7c10c2a..d1ad372 100755
> > --- a/tests/generic/459
> > +++ b/tests/generic/459
> > @@ -63,6 +63,7 @@ _supported_os Linux
> >  _require_scratch_nocheck
> >  _require_dm_target thin-pool
> >  _require_command $LVM_PROG lvm
> > +_require_command "$THIN_CHECK_PROG" thin_check
> >  _require_freeze
> >  
> >  # remove previous $seqres.full before test
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Amir Goldstein]

On Thu, Nov 2, 2017 at 6:39 PM, Darrick J. Wong <darrick.wong@oracle.com> wrote:
> On Thu, Nov 02, 2017 at 03:14:45PM +0200, Amir Goldstein wrote:
>> On Thu, Nov 2, 2017 at 2:42 PM, Eryu Guan <eguan@redhat.com> wrote:
>> > On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
>> >> From: Darrick J. Wong <darrick.wong@oracle.com>
>> >>
>> >> A directory corrupted into a symlink will be caught by the upcoming
>> >> local format ifork verifiers.
>> >>
>> >> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
>> >> ---
>> >>  tests/xfs/348.out |    2 +-
>> >>  1 file changed, 1 insertion(+), 1 deletion(-)
>> >>
>> >>
>> >> diff --git a/tests/xfs/348.out b/tests/xfs/348.out
>> >> index f4a7a71..17d9be2 100644
>> >> --- a/tests/xfs/348.out
>> >> +++ b/tests/xfs/348.out
>> >> @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
>> >>  would have junked entry "DIR" in directory PARENT_INO
>> >>  would have junked entry "EMPTY" in directory PARENT_INO
>> >>  would have junked entry "FIFO" in directory PARENT_INO
>> >> -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
>> >> +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
>> >
>> > But this breaks tests on old kernels. Or with the new ifork verifiers,
>> > old kernels can be considered as buggy?
>> >
>>
>> Can filter new and old error into canonicalized string.
>> Dodgy, bug we've done it before.
>> See:
>> git grep sed | grep 'Operation not permitted\|Permission denied'
>
> I tried that, but the "stat: XXX is a symbolic link" message comes from
> a different path in the test file code.
>
> Unless... you're suggesting that I should sed the error output to make
> it look like the previous (broken) output?
>

Let's say I am not *suggesting* this, but I have seen tests that do that
See generic/160 and other tests that show in this grep output:

git grep sed | grep 'Operation not permitted\|Permission denied'

Amir.

Re: [PATCH 12/14] generic/204: break out of fs-filling loop early if we ENOSPC

[Darrick J. Wong]

On Thu, Nov 02, 2017 at 10:23:48PM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:47:42PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > If filling up the filesystem causes us to hit ENOSPC earlier than we
> > thought we would (the sizing estimates become less and less accurate as
> > we add more metadata) then just bail out -- we're checking that the fs
> > is robust enough to cut us off before we actually run out of space for
> > writing metadata and crash the fs.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  tests/generic/204 |   11 +++++++++--
> >  1 file changed, 9 insertions(+), 2 deletions(-)
> > 
> > 
> > diff --git a/tests/generic/204 b/tests/generic/204
> > index 4c203a2..1e2c1e1 100755
> > --- a/tests/generic/204
> > +++ b/tests/generic/204
> > @@ -82,8 +82,15 @@ echo files $files, resvblks $resv_blks >> $seqres.full
> >  _scratch_resvblks $resv_blks >> $seqres.full 2>&1
> >  
> >  for i in `seq 1 $files`; do
> > -    echo -n > $SCRATCH_MNT/$i
> > -    echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > $SCRATCH_MNT/$i
> > +	# Open/truncate file, write stuff.  If we run out of space early,
> > +	# we can bail out of the loop.
> > +	out="$($XFS_IO_PROG \
> > +		-c "open -f -t $SCRATCH_MNT/$i" \
> > +		-c 'close' \
> > +		-c "open -f -t $SCRATCH_MNT/$i" \
> > +		-c 'pwrite -q -S 0x58 0 36' 2>&1 | _filter_scratch)"
> > +	echo "${out}" | grep -q 'No space left on device' && break
> 
> This doesn't look correct to me. This test is meant to catch spurious
> ENOSPC, it's designed to be a "delayed allocation ENOSPC test"[1] by
> "writing lots of single block files" and catch early ENOSPC. IMHO, this
> change ignores ENOSPC, which defeats the test purpose. And I don't quite
> understand the purpose of truncate open/close/truncate open/write
> sequence..
> 
> Also this significantly increases the test time, 4s -> ~120s for me.

Hmm, you're right, this one /is/ trying to test ENOSPC.  It would seem
that a better fix for this test would be to set space= based on the
available blocks count instead of hardcoding 97920000.

--D

> 
> Thanks,
> Eryu
> 
> [1] commit 143368a047ea ("xfstests: add test 204, a simple delayed
> allocation ENOSPC test")
> 
> > +	test -n "${out}" && echo "${out}"
> >  done
> >  
> >  # success, all done
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

[PATCH v2 12/14] generic/204: use available blocks to determine the number of files to create

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use the available block count to compute the number of files we think
we can create, rather than hardcoding a particular size.  This fixes
the ENOSPC failures for xfs filesystems with rmap/reflink support.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/generic/204 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/tests/generic/204 b/tests/generic/204
index 4c203a2..ac417a7 100755
--- a/tests/generic/204
+++ b/tests/generic/204
@@ -69,7 +69,7 @@ _scratch_mount
 # work out correctly. Space usages is based 22500 files and 1024 reserved blocks
 # on a 4k block size 256 byte inode size filesystem.
 resv_blks=1024
-space=97920000
+space=$(stat -f -c '%f * %S' $SCRATCH_MNT | $BC_PROG)
 
 # decrease files for inode size.
 #	22500 * (256 + 4k) = ~97MB

[PATCH v2 14/14] generic: test IO at maximum file offset

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure we can write to and read from the highest possible offset
that Linux will allow.  Format the filesystem with a variety of
possible blocksizes to stress the filesystem.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: fixes suggested by eryu
---
 tests/generic/705     |   98 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/705.out |    2 +
 tests/generic/group   |    1 +
 3 files changed, 101 insertions(+)
 create mode 100755 tests/generic/705
 create mode 100644 tests/generic/705.out

diff --git a/tests/generic/705 b/tests/generic/705
new file mode 100755
index 0000000..18916c3
--- /dev/null
+++ b/tests/generic/705
@@ -0,0 +1,98 @@
+#! /bin/bash
+# FS QA Test No. 705
+#
+# Check that high-offset reads and writes work.
+#
+#-----------------------------------------------------------------------
+# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
+#
+# This program is free software; you can redistribute it and/or
+# modify it under the terms of the GNU General Public License as
+# published by the Free Software Foundation.
+#
+# This program is distributed in the hope that it would be useful,
+# but WITHOUT ANY WARRANTY; without even the implied warranty of
+# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+# GNU General Public License for more details.
+#
+# You should have received a copy of the GNU General Public License
+# along with this program; if not, write the Free Software Foundation,
+# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
+#-----------------------------------------------------------------------
+
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "_cleanup; exit \$status" 0 1 2 3 15
+
+_cleanup()
+{
+    cd /
+    rm -rf $tmp.*
+}
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+# real QA test starts here
+_supported_os Linux
+_require_scratch_nocheck
+
+rm -f $seqres.full
+
+echo "Silence is golden"
+
+echo "Starting test" > $seqres.full
+devsize=$(blockdev --getsize64 $SCRATCH_DEV)
+for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
+	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
+	_scratch_unmount > /dev/null 2>&1
+	# Try to format and mount with the given blocksize.  If they don't
+	# succeed, move on to the next block size.
+	unset MKFS_OPTIONS
+	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
+	   ! _scratch_mount >> $seqres.full 2>&1 ||
+	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
+		echo "+++ Format and mount failed" >> $seqres.full
+		continue
+	fi
+
+	testdir=$SCRATCH_MNT/test-$seq
+	mkdir $testdir
+
+	echo "++ Create the original files" >> $seqres.full
+	bigoff=$(echo "2^63 - 2" | $BC_PROG)
+	len=$(echo "2^63 - 1" | $BC_PROG)
+	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
+	if [ ! -s $testdir/file0 ]; then
+		# If we can't set a large file size then don't bother
+		# with this blocksize because the fs doesn't support it.
+		echo "+++ High offset ftruncate failed" >> $seqres.full
+		continue
+	fi
+	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
+
+	echo "++ Check file creation" >> $seqres.full
+	_scratch_cycle_mount
+
+	expected="7ffffffffffffffe:  61  a"
+	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
+	if [ "$expected" = "$actual" ]; then
+		echo "+++ Success!" >> $seqres.full
+	else
+		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
+		echo "Discrepancy @ blocksize $blocksize"
+	fi
+
+	echo "++ Check scratchfs" >> $seqres.full
+	_check_scratch_fs
+done
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/705.out b/tests/generic/705.out
new file mode 100644
index 0000000..6fcdec0
--- /dev/null
+++ b/tests/generic/705.out
@@ -0,0 +1,2 @@
+QA output created by 705
+Silence is golden
diff --git a/tests/generic/group b/tests/generic/group
index fbe0a7f..2c55b93 100644
--- a/tests/generic/group
+++ b/tests/generic/group
@@ -468,3 +468,4 @@
 463 auto quick clone dangerous
 464 auto rw
 465 auto rw quick aio
+705 auto quick rw

[PATCH 15/14] xfs/020: check that we have enough space to write out a huge fs

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Make sure that we have enough free space on the test fs to create a
60t sparse filesystem.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/020 |    4 ++++
 1 file changed, 4 insertions(+)

diff --git a/tests/xfs/020 b/tests/xfs/020
index 120492d..b037369 100755
--- a/tests/xfs/020
+++ b/tests/xfs/020
@@ -49,6 +49,10 @@ _supported_fs xfs
 _supported_os Linux
 _require_test
 
+# Writing a 60t fs requires about 2GB of space, so make sure
+# we have plenty of space to do that.
+_require_fs_space $TEST_DIR 2500000
+
 echo "Silence is golden"
 
 fsfile=$TEST_DIR/fsfile.$seq

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Eryu Guan]

On Thu, Nov 02, 2017 at 09:37:54AM -0700, Darrick J. Wong wrote:
> On Thu, Nov 02, 2017 at 08:42:14PM +0800, Eryu Guan wrote:
> > On Wed, Nov 01, 2017 at 02:47:23PM -0700, Darrick J. Wong wrote:
> > > From: Darrick J. Wong <darrick.wong@oracle.com>
> > > 
> > > A directory corrupted into a symlink will be caught by the upcoming
> > > local format ifork verifiers.
> > > 
> > > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > > ---
> > >  tests/xfs/348.out |    2 +-
> > >  1 file changed, 1 insertion(+), 1 deletion(-)
> > > 
> > > 
> > > diff --git a/tests/xfs/348.out b/tests/xfs/348.out
> > > index f4a7a71..17d9be2 100644
> > > --- a/tests/xfs/348.out
> > > +++ b/tests/xfs/348.out
> > > @@ -239,7 +239,7 @@ would have junked entry "DATA" in directory PARENT_INO
> > >  would have junked entry "DIR" in directory PARENT_INO
> > >  would have junked entry "EMPTY" in directory PARENT_INO
> > >  would have junked entry "FIFO" in directory PARENT_INO
> > > -stat: 'SCRATCH_MNT/test/DIR' is a symbolic link
> > > +stat: cannot stat 'SCRATCH_MNT/test/DIR': Structure needs cleaning
> > 
> > But this breaks tests on old kernels. Or with the new ifork verifiers,
> > old kernels can be considered as buggy?
> 
> Yes, they're buggy since we shouldn't be interpreting directory entries
> as a link target given that the zero bytes in the "link target" will
> screw things up.

Then I'm fine with taking it, and probably will edit the commit log a
bit to reflect that we were missing a error case previously. Thanks!

Eryu

Re: [PATCH v2 14/14] generic: test IO at maximum file offset

[Eryu Guan]

On Thu, Nov 02, 2017 at 09:27:17PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Make sure we can write to and read from the highest possible offset
> that Linux will allow.  Format the filesystem with a variety of
> possible blocksizes to stress the filesystem.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
> v2: fixes suggested by eryu
> ---
>  tests/generic/705     |   98 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/705.out |    2 +
>  tests/generic/group   |    1 +
>  3 files changed, 101 insertions(+)
>  create mode 100755 tests/generic/705
>  create mode 100644 tests/generic/705.out
> 
> diff --git a/tests/generic/705 b/tests/generic/705
> new file mode 100755
> index 0000000..18916c3
> --- /dev/null
> +++ b/tests/generic/705
> @@ -0,0 +1,98 @@
> +#! /bin/bash
> +# FS QA Test No. 705
> +#
> +# Check that high-offset reads and writes work.
> +#
> +#-----------------------------------------------------------------------
> +# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
> +#
> +# This program is free software; you can redistribute it and/or
> +# modify it under the terms of the GNU General Public License as
> +# published by the Free Software Foundation.
> +#
> +# This program is distributed in the hope that it would be useful,
> +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> +# GNU General Public License for more details.
> +#
> +# You should have received a copy of the GNU General Public License
> +# along with this program; if not, write the Free Software Foundation,
> +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> +#-----------------------------------------------------------------------
> +
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1    # failure is the default!
> +trap "_cleanup; exit \$status" 0 1 2 3 15
> +
> +_cleanup()
> +{
> +    cd /
> +    rm -rf $tmp.*

I did some minor edits, removed 'r' from rm.

> +}
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +# real QA test starts here
> +_supported_os Linux

Added _supported_fs generic

> +_require_scratch_nocheck
> +
> +rm -f $seqres.full
> +
> +echo "Silence is golden"
> +
> +echo "Starting test" > $seqres.full
> +devsize=$(blockdev --getsize64 $SCRATCH_DEV)
> +for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
> +	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
> +	_scratch_unmount > /dev/null 2>&1
> +	# Try to format and mount with the given blocksize.  If they don't
> +	# succeed, move on to the next block size.
> +	unset MKFS_OPTIONS

Moved this unset out of the for loop.

Thanks,
Eryu

> +	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
> +	   ! _scratch_mount >> $seqres.full 2>&1 ||
> +	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
> +		echo "+++ Format and mount failed" >> $seqres.full
> +		continue
> +	fi
> +
> +	testdir=$SCRATCH_MNT/test-$seq
> +	mkdir $testdir
> +
> +	echo "++ Create the original files" >> $seqres.full
> +	bigoff=$(echo "2^63 - 2" | $BC_PROG)
> +	len=$(echo "2^63 - 1" | $BC_PROG)
> +	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
> +	if [ ! -s $testdir/file0 ]; then
> +		# If we can't set a large file size then don't bother
> +		# with this blocksize because the fs doesn't support it.
> +		echo "+++ High offset ftruncate failed" >> $seqres.full
> +		continue
> +	fi
> +	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
> +
> +	echo "++ Check file creation" >> $seqres.full
> +	_scratch_cycle_mount
> +
> +	expected="7ffffffffffffffe:  61  a"
> +	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
> +	if [ "$expected" = "$actual" ]; then
> +		echo "+++ Success!" >> $seqres.full
> +	else
> +		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
> +		echo "Discrepancy @ blocksize $blocksize"
> +	fi
> +
> +	echo "++ Check scratchfs" >> $seqres.full
> +	_check_scratch_fs
> +done
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/705.out b/tests/generic/705.out
> new file mode 100644
> index 0000000..6fcdec0
> --- /dev/null
> +++ b/tests/generic/705.out
> @@ -0,0 +1,2 @@
> +QA output created by 705
> +Silence is golden
> diff --git a/tests/generic/group b/tests/generic/group
> index fbe0a7f..2c55b93 100644
> --- a/tests/generic/group
> +++ b/tests/generic/group
> @@ -468,3 +468,4 @@
>  463 auto quick clone dangerous
>  464 auto rw
>  465 auto rw quick aio
> +705 auto quick rw

Re: [PATCH 01/14] common/xfs: refactor xfs_scrub presence testing

[Eryu Guan]

On Wed, Nov 01, 2017 at 02:46:34PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Move all the requirements checking for xfs_scrub into a helper function.
> Make sure the helper properly detects the presence of the scrub ioctl
> and situations where we can't run scrub (e.g. norecovery).
> 
> Refactor the existing three xfs_scrub call sites to use the helper to
> check if it's appropriate to run scrub.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  README            |    6 +++---
>  common/rc         |    2 +-
>  common/xfs        |   41 ++++++++++++++++++++++++++++++++++-------
>  tests/generic/453 |   11 +----------
>  tests/generic/454 |   11 +----------
>  5 files changed, 40 insertions(+), 31 deletions(-)
> 
> 
> diff --git a/README b/README
> index 4963d28..a9da4f0 100644
> --- a/README
> +++ b/README
> @@ -88,9 +88,9 @@ Preparing system for tests:
>                 run xfs_repair -n to check the filesystem; xfs_repair to rebuild
>                 metadata indexes; and xfs_repair -n (a third time) to check the
>                 results of the rebuilding.
> -             - set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
> -               xfs_scrub -vd to scrub the filesystem metadata online before
> -               unmounting to run the offline check.
> +             - xfs_scrub, if present, will always check the test and scratch
> +               filesystems if they are still online at the end of the test.
> +               It is no longer necessary to set TEST_XFS_SCRUB.
>               - setenv LOGWRITES_DEV to a block device to use for power fail
>                 testing.
>  
> diff --git a/common/rc b/common/rc
> index 1a4d81e..83aaced 100644
> --- a/common/rc
> +++ b/common/rc
> @@ -2091,7 +2091,7 @@ _require_xfs_io_command()
>  			_notrun "xfs_io $command support is missing"
>  		;;
>  	"scrub"|"repair")
> -		testio=`$XFS_IO_PROG -x -c "$command test 0" $TEST_DIR 2>&1`
> +		testio=`$XFS_IO_PROG -x -c "$command probe 0" $TEST_DIR 2>&1`
>  		echo $testio | grep -q "Inappropriate ioctl" && \
>  			_notrun "xfs_io $command support is missing"
>  		;;
> diff --git a/common/xfs b/common/xfs
> index dff8454..de3c560 100644
> --- a/common/xfs
> +++ b/common/xfs
> @@ -298,6 +298,30 @@ _require_xfs_db_command()
>  		_notrun "xfs_db $command support is missing"
>  }
>  
> +# Does the filesystem mounted from a particular device support scrub?
> +_supports_xfs_scrub()
> +{
> +	mountpoint="$1"
> +	device="$2"
> +
> +	if [ ! -b "$device" ] || [ ! -e "$mountpoint" ]; then
> +		echo "Usage: _supports_xfs_scrub mountpoint device"
> +		exit 1
> +	fi
> +
> +	test "$FSTYP" = "xfs" || return 1
> +	test -x "$XFS_SCRUB_PROG" || return 1
> +
> +	# Probe for kernel support...
> +	$XFS_IO_PROG -c 'help scrub' 2>&1 | grep -q 'types are:.*probe' || return 1
> +	$XFS_IO_PROG -c "scrub probe 0" "$mountpoint" 2>&1 | grep -q "Inappropriate ioctl" && return 1
> +
> +	# Scrub can't run on norecovery mounts
> +	_fs_options "$device" | grep -q "norecovery" && return 1
> +
> +	return 0
> +}
> +
>  # run xfs_check and friends on a FS.
>  _check_xfs_filesystem()
>  {
> @@ -330,14 +354,17 @@ _check_xfs_filesystem()
>  	type=`_fs_type $device`
>  	ok=1
>  
> -	if [ "$type" = "xfs" ]; then
> -		if [ -n "$TEST_XFS_SCRUB" ] && [ -x "$XFS_SCRUB_PROG" ]; then
> -			"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full
> -			if [ $? -ne 0 ]; then
> -				_log_err "filesystem on $device failed scrub"
> -				ok=0
> -			fi
> +	# Run online scrub if we can.
> +	mntpt="$(_is_mounted $device)"
> +	if [ -n "$mntpt" ] && _supports_xfs_scrub "$mntpt" "$device"; then
> +		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full 2>&1
> +		if [ $? -ne 0 ]; then
> +			_log_err "filesystem on $device failed scrub"
> +			ok=0
>  		fi
> +	fi
> +
> +	if [ "$type" = "xfs" ]; then
>  		# mounted ...
>  		mountpoint=`_umount_or_remount_ro $device`
>  	fi
> diff --git a/tests/generic/453 b/tests/generic/453
> index bda1112..16589d1 100755
> --- a/tests/generic/453
> +++ b/tests/generic/453
> @@ -136,10 +136,7 @@ echo "Test XFS online scrub, if applicable"
>  
>  # Only run this on xfs if xfs_scrub is available and has the unicode checker
>  check_xfs_scrub() {
> -	# Ignore non-XFS fs or no scrub program...
> -	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
> -		return 1
> -	fi

I added the $FSTYP check back in check_xfs_scrub here and in
generic/454, because _supports_xfs_scrub is in common/xfs and it's not
sourced if we're testing non-xfs, and test fails due to missing
_supports_xfs_scrub command. This was caught in my release testing.

Thanks,
Eryu

> +	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
>  
>  	# We only care if xfs_scrub has unicode string support...
>  	if ! type ldd > /dev/null 2>&1 || \
> @@ -147,12 +144,6 @@ check_xfs_scrub() {
>  		return 1
>  	fi
>  
> -	# Does the ioctl work?
> -	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
> -	   grep -q "Inappropriate ioctl"; then
> -		return 1
> -	fi
> -
>  	return 0
>  }
>  
> diff --git a/tests/generic/454 b/tests/generic/454
> index d1f93b2..efac860 100755
> --- a/tests/generic/454
> +++ b/tests/generic/454
> @@ -132,10 +132,7 @@ echo "Test XFS online scrub, if applicable"
>  
>  # Only run this on xfs if xfs_scrub is available and has the unicode checker
>  check_xfs_scrub() {
> -	# Ignore non-XFS fs or no scrub program...
> -	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
> -		return 1
> -	fi
> +	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
>  
>  	# We only care if xfs_scrub has unicode string support...
>  	if ! type ldd > /dev/null 2>&1 || \
> @@ -143,12 +140,6 @@ check_xfs_scrub() {
>  		return 1
>  	fi
>  
> -	# Does the ioctl work?
> -	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
> -	   grep -q "Inappropriate ioctl"; then
> -		return 1
> -	fi
> -
>  	return 0
>  }
>  
> 

Re: [PATCH v2 14/14] generic: test IO at maximum file offset

[Darrick J. Wong]

On Fri, Nov 03, 2017 at 01:33:07PM +0800, Eryu Guan wrote:
> On Thu, Nov 02, 2017 at 09:27:17PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Make sure we can write to and read from the highest possible offset
> > that Linux will allow.  Format the filesystem with a variety of
> > possible blocksizes to stress the filesystem.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> > v2: fixes suggested by eryu
> > ---
> >  tests/generic/705     |   98 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/705.out |    2 +
> >  tests/generic/group   |    1 +
> >  3 files changed, 101 insertions(+)
> >  create mode 100755 tests/generic/705
> >  create mode 100644 tests/generic/705.out
> > 
> > diff --git a/tests/generic/705 b/tests/generic/705
> > new file mode 100755
> > index 0000000..18916c3
> > --- /dev/null
> > +++ b/tests/generic/705
> > @@ -0,0 +1,98 @@
> > +#! /bin/bash
> > +# FS QA Test No. 705
> > +#
> > +# Check that high-offset reads and writes work.
> > +#
> > +#-----------------------------------------------------------------------
> > +# Copyright (c) 2017, Oracle and/or its affiliates.  All Rights Reserved.
> > +#
> > +# This program is free software; you can redistribute it and/or
> > +# modify it under the terms of the GNU General Public License as
> > +# published by the Free Software Foundation.
> > +#
> > +# This program is distributed in the hope that it would be useful,
> > +# but WITHOUT ANY WARRANTY; without even the implied warranty of
> > +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> > +# GNU General Public License for more details.
> > +#
> > +# You should have received a copy of the GNU General Public License
> > +# along with this program; if not, write the Free Software Foundation,
> > +# Inc.,  51 Franklin St, Fifth Floor, Boston, MA  02110-1301  USA
> > +#-----------------------------------------------------------------------
> > +
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1    # failure is the default!
> > +trap "_cleanup; exit \$status" 0 1 2 3 15
> > +
> > +_cleanup()
> > +{
> > +    cd /
> > +    rm -rf $tmp.*
> 
> I did some minor edits, removed 'r' from rm.
> 
> > +}
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +# real QA test starts here
> > +_supported_os Linux
> 
> Added _supported_fs generic
> 
> > +_require_scratch_nocheck
> > +
> > +rm -f $seqres.full
> > +
> > +echo "Silence is golden"
> > +
> > +echo "Starting test" > $seqres.full
> > +devsize=$(blockdev --getsize64 $SCRATCH_DEV)
> > +for blocksize in 512 1024 2048 4096 8192 16384 32768 65536; do
> > +	echo "+ Format blocksize $blocksize and mount" >> $seqres.full
> > +	_scratch_unmount > /dev/null 2>&1
> > +	# Try to format and mount with the given blocksize.  If they don't
> > +	# succeed, move on to the next block size.
> > +	unset MKFS_OPTIONS
> 
> Moved this unset out of the for loop.

Ok, thank you!

--D

> 
> Thanks,
> Eryu
> 
> > +	if ! _scratch_mkfs_sized $devsize $blocksize >> $seqres.full 2>&1 ||
> > +	   ! _scratch_mount >> $seqres.full 2>&1 ||
> > +	   test "$(stat -f -c '%S' $SCRATCH_MNT)" -ne "$blocksize"; then
> > +		echo "+++ Format and mount failed" >> $seqres.full
> > +		continue
> > +	fi
> > +
> > +	testdir=$SCRATCH_MNT/test-$seq
> > +	mkdir $testdir
> > +
> > +	echo "++ Create the original files" >> $seqres.full
> > +	bigoff=$(echo "2^63 - 2" | $BC_PROG)
> > +	len=$(echo "2^63 - 1" | $BC_PROG)
> > +	$XFS_IO_PROG -f -c "truncate $len" $testdir/file0 >> $seqres.full 2>&1
> > +	if [ ! -s $testdir/file0 ]; then
> > +		# If we can't set a large file size then don't bother
> > +		# with this blocksize because the fs doesn't support it.
> > +		echo "+++ High offset ftruncate failed" >> $seqres.full
> > +		continue
> > +	fi
> > +	_pwrite_byte 0x61 $bigoff 1 $testdir/file1 >> $seqres.full
> > +
> > +	echo "++ Check file creation" >> $seqres.full
> > +	_scratch_cycle_mount
> > +
> > +	expected="7ffffffffffffffe:  61  a"
> > +	actual="$($XFS_IO_PROG -c "pread -v -q $bigoff 1" $testdir/file1)"
> > +	if [ "$expected" = "$actual" ]; then
> > +		echo "+++ Success!" >> $seqres.full
> > +	else
> > +		echo "+++ Discrepancy @ blocksize $blocksize" >> $seqres.full
> > +		echo "Discrepancy @ blocksize $blocksize"
> > +	fi
> > +
> > +	echo "++ Check scratchfs" >> $seqres.full
> > +	_check_scratch_fs
> > +done
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/generic/705.out b/tests/generic/705.out
> > new file mode 100644
> > index 0000000..6fcdec0
> > --- /dev/null
> > +++ b/tests/generic/705.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 705
> > +Silence is golden
> > diff --git a/tests/generic/group b/tests/generic/group
> > index fbe0a7f..2c55b93 100644
> > --- a/tests/generic/group
> > +++ b/tests/generic/group
> > @@ -468,3 +468,4 @@
> >  463 auto quick clone dangerous
> >  464 auto rw
> >  465 auto rw quick aio
> > +705 auto quick rw
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Theodore Ts'o]

On Fri, Nov 03, 2017 at 12:30:22PM +0800, Eryu Guan wrote:
> > Yes, they're buggy since we shouldn't be interpreting directory entries
> > as a link target given that the zero bytes in the "link target" will
> > screw things up.
> 
> Then I'm fine with taking it, and probably will edit the commit log a
> bit to reflect that we were missing a error case previously. Thanks!

Darrick, can we add a note indicating what commit needs to be
backported to make the failure go away?

					- Ted

Re: [PATCH 01/14] common/xfs: refactor xfs_scrub presence testing

[Darrick J. Wong]

On Fri, Nov 03, 2017 at 07:10:05PM +0800, Eryu Guan wrote:
> On Wed, Nov 01, 2017 at 02:46:34PM -0700, Darrick J. Wong wrote:
> > From: Darrick J. Wong <darrick.wong@oracle.com>
> > 
> > Move all the requirements checking for xfs_scrub into a helper function.
> > Make sure the helper properly detects the presence of the scrub ioctl
> > and situations where we can't run scrub (e.g. norecovery).
> > 
> > Refactor the existing three xfs_scrub call sites to use the helper to
> > check if it's appropriate to run scrub.
> > 
> > Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> > ---
> >  README            |    6 +++---
> >  common/rc         |    2 +-
> >  common/xfs        |   41 ++++++++++++++++++++++++++++++++++-------
> >  tests/generic/453 |   11 +----------
> >  tests/generic/454 |   11 +----------
> >  5 files changed, 40 insertions(+), 31 deletions(-)
> > 
> > 
> > diff --git a/README b/README
> > index 4963d28..a9da4f0 100644
> > --- a/README
> > +++ b/README
> > @@ -88,9 +88,9 @@ Preparing system for tests:
> >                 run xfs_repair -n to check the filesystem; xfs_repair to rebuild
> >                 metadata indexes; and xfs_repair -n (a third time) to check the
> >                 results of the rebuilding.
> > -             - set TEST_XFS_SCRUB=1 to have _check_xfs_filesystem run
> > -               xfs_scrub -vd to scrub the filesystem metadata online before
> > -               unmounting to run the offline check.
> > +             - xfs_scrub, if present, will always check the test and scratch
> > +               filesystems if they are still online at the end of the test.
> > +               It is no longer necessary to set TEST_XFS_SCRUB.
> >               - setenv LOGWRITES_DEV to a block device to use for power fail
> >                 testing.
> >  
> > diff --git a/common/rc b/common/rc
> > index 1a4d81e..83aaced 100644
> > --- a/common/rc
> > +++ b/common/rc
> > @@ -2091,7 +2091,7 @@ _require_xfs_io_command()
> >  			_notrun "xfs_io $command support is missing"
> >  		;;
> >  	"scrub"|"repair")
> > -		testio=`$XFS_IO_PROG -x -c "$command test 0" $TEST_DIR 2>&1`
> > +		testio=`$XFS_IO_PROG -x -c "$command probe 0" $TEST_DIR 2>&1`
> >  		echo $testio | grep -q "Inappropriate ioctl" && \
> >  			_notrun "xfs_io $command support is missing"
> >  		;;
> > diff --git a/common/xfs b/common/xfs
> > index dff8454..de3c560 100644
> > --- a/common/xfs
> > +++ b/common/xfs
> > @@ -298,6 +298,30 @@ _require_xfs_db_command()
> >  		_notrun "xfs_db $command support is missing"
> >  }
> >  
> > +# Does the filesystem mounted from a particular device support scrub?
> > +_supports_xfs_scrub()
> > +{
> > +	mountpoint="$1"
> > +	device="$2"
> > +
> > +	if [ ! -b "$device" ] || [ ! -e "$mountpoint" ]; then
> > +		echo "Usage: _supports_xfs_scrub mountpoint device"
> > +		exit 1
> > +	fi
> > +
> > +	test "$FSTYP" = "xfs" || return 1
> > +	test -x "$XFS_SCRUB_PROG" || return 1
> > +
> > +	# Probe for kernel support...
> > +	$XFS_IO_PROG -c 'help scrub' 2>&1 | grep -q 'types are:.*probe' || return 1
> > +	$XFS_IO_PROG -c "scrub probe 0" "$mountpoint" 2>&1 | grep -q "Inappropriate ioctl" && return 1
> > +
> > +	# Scrub can't run on norecovery mounts
> > +	_fs_options "$device" | grep -q "norecovery" && return 1
> > +
> > +	return 0
> > +}
> > +
> >  # run xfs_check and friends on a FS.
> >  _check_xfs_filesystem()
> >  {
> > @@ -330,14 +354,17 @@ _check_xfs_filesystem()
> >  	type=`_fs_type $device`
> >  	ok=1
> >  
> > -	if [ "$type" = "xfs" ]; then
> > -		if [ -n "$TEST_XFS_SCRUB" ] && [ -x "$XFS_SCRUB_PROG" ]; then
> > -			"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full
> > -			if [ $? -ne 0 ]; then
> > -				_log_err "filesystem on $device failed scrub"
> > -				ok=0
> > -			fi
> > +	# Run online scrub if we can.
> > +	mntpt="$(_is_mounted $device)"
> > +	if [ -n "$mntpt" ] && _supports_xfs_scrub "$mntpt" "$device"; then
> > +		"$XFS_SCRUB_PROG" $scrubflag -v -d -n $device >>$seqres.full 2>&1
> > +		if [ $? -ne 0 ]; then
> > +			_log_err "filesystem on $device failed scrub"
> > +			ok=0
> >  		fi
> > +	fi
> > +
> > +	if [ "$type" = "xfs" ]; then
> >  		# mounted ...
> >  		mountpoint=`_umount_or_remount_ro $device`
> >  	fi
> > diff --git a/tests/generic/453 b/tests/generic/453
> > index bda1112..16589d1 100755
> > --- a/tests/generic/453
> > +++ b/tests/generic/453
> > @@ -136,10 +136,7 @@ echo "Test XFS online scrub, if applicable"
> >  
> >  # Only run this on xfs if xfs_scrub is available and has the unicode checker
> >  check_xfs_scrub() {
> > -	# Ignore non-XFS fs or no scrub program...
> > -	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
> > -		return 1
> > -	fi
> 
> I added the $FSTYP check back in check_xfs_scrub here and in
> generic/454, because _supports_xfs_scrub is in common/xfs and it's not
> sourced if we're testing non-xfs, and test fails due to missing
> _supports_xfs_scrub command. This was caught in my release testing.

Oops.  Thank you for fixing that.

--D

> Thanks,
> Eryu
> 
> > +	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
> >  
> >  	# We only care if xfs_scrub has unicode string support...
> >  	if ! type ldd > /dev/null 2>&1 || \
> > @@ -147,12 +144,6 @@ check_xfs_scrub() {
> >  		return 1
> >  	fi
> >  
> > -	# Does the ioctl work?
> > -	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
> > -	   grep -q "Inappropriate ioctl"; then
> > -		return 1
> > -	fi
> > -
> >  	return 0
> >  }
> >  
> > diff --git a/tests/generic/454 b/tests/generic/454
> > index d1f93b2..efac860 100755
> > --- a/tests/generic/454
> > +++ b/tests/generic/454
> > @@ -132,10 +132,7 @@ echo "Test XFS online scrub, if applicable"
> >  
> >  # Only run this on xfs if xfs_scrub is available and has the unicode checker
> >  check_xfs_scrub() {
> > -	# Ignore non-XFS fs or no scrub program...
> > -	if [ "${FSTYP}" != "xfs" ] || [ ! -x "${XFS_SCRUB_PROG}" ]; then
> > -		return 1
> > -	fi
> > +	_supports_xfs_scrub "$SCRATCH_MNT" "$SCRATCH_DEV" || return 1
> >  
> >  	# We only care if xfs_scrub has unicode string support...
> >  	if ! type ldd > /dev/null 2>&1 || \
> > @@ -143,12 +140,6 @@ check_xfs_scrub() {
> >  		return 1
> >  	fi
> >  
> > -	# Does the ioctl work?
> > -	if $XFS_IO_PROG -x -c "scrub probe 0" $SCRATCH_MNT 2>&1 | \
> > -	   grep -q "Inappropriate ioctl"; then
> > -		return 1
> > -	fi
> > -
> >  	return 0
> >  }
> >  
> > 
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH 09/14] xfs/348: dir->symlink corruption must not be allowed

[Darrick J. Wong]

On Fri, Nov 03, 2017 at 12:14:48PM -0400, Theodore Ts'o wrote:
> On Fri, Nov 03, 2017 at 12:30:22PM +0800, Eryu Guan wrote:
> > > Yes, they're buggy since we shouldn't be interpreting directory entries
> > > as a link target given that the zero bytes in the "link target" will
> > > screw things up.
> > 
> > Then I'm fine with taking it, and probably will edit the commit log a
> > bit to reflect that we were missing a error case previously. Thanks!
> 
> Darrick, can we add a note indicating what commit needs to be
> backported to make the failure go away?

Will do when the time comes.  TBH we've not been consistent about
marking regression tests with the appropriate upstream commit id once
they're available.

--D

> 					- Ted
> --
> To unsubscribe from this list: send the line "unsubscribe linux-xfs" in
> the body of a message to majordomo@vger.kernel.org
> More majordomo info at  http://vger.kernel.org/majordomo-info.html

Re: [PATCH v2 12/14] generic/204: use available blocks to determine the number of files to create

[Eryu Guan]

On Thu, Nov 02, 2017 at 09:26:19PM -0700, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use the available block count to compute the number of files we think
> we can create, rather than hardcoding a particular size.  This fixes
> the ENOSPC failures for xfs filesystems with rmap/reflink support.
> 
> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
> ---
>  tests/generic/204 |    2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/tests/generic/204 b/tests/generic/204
> index 4c203a2..ac417a7 100755
> --- a/tests/generic/204
> +++ b/tests/generic/204
> @@ -69,7 +69,7 @@ _scratch_mount
>  # work out correctly. Space usages is based 22500 files and 1024 reserved blocks
>  # on a 4k block size 256 byte inode size filesystem.
>  resv_blks=1024
> -space=97920000
> +space=$(stat -f -c '%f * %S' $SCRATCH_MNT | $BC_PROG)

Hmm, this regressed test with 512B block size XFS. I'll drop it for now.

Thanks,
Eryu

>  
>  # decrease files for inode size.
>  #	22500 * (256 + 4k) = ~97MB

[PATCH v3 12/14] generic/204: break out of fs-filling loop early if we ENOSPC

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

Use the available block count to compute the number of files we think
we can create, rather than hardcoding a particular size.  This fixes
the ENOSPC failures for xfs filesystems with rmap/reflink support.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v3: factor in space used for file names
---
 tests/generic/204 |   14 ++++++++++++--
 1 file changed, 12 insertions(+), 2 deletions(-)

diff --git a/tests/generic/204 b/tests/generic/204
index 4c203a2..71bee83 100755
--- a/tests/generic/204
+++ b/tests/generic/204
@@ -69,7 +69,7 @@ _scratch_mount
 # work out correctly. Space usages is based 22500 files and 1024 reserved blocks
 # on a 4k block size 256 byte inode size filesystem.
 resv_blks=1024
-space=97920000
+space=$(stat -f -c '%f * %S' $SCRATCH_MNT | $BC_PROG)
 
 # decrease files for inode size.
 #	22500 * (256 + 4k) = ~97MB
@@ -78,10 +78,20 @@ space=97920000
 
 files=$((space / (isize + dbsize)))
 
+# Now do it again, but factor in the filename sizes too.
+# We naïvely assume 8 bytes for inode number, 1 byte for ftype,
+# and 1 more byte for namelen, then round up to the nearest 8
+# bytes.
+
+namelen="$(echo -n "$files" | wc -c)"
+direntlen="$(echo "(10 + $namelen + 7) / 8 * 8" | $BC_PROG)"
+
+files=$((space / (direntlen + isize + dbsize)))
+
 echo files $files, resvblks $resv_blks >> $seqres.full
 _scratch_resvblks $resv_blks >> $seqres.full 2>&1
 
-for i in `seq 1 $files`; do
+for i in `seq -w 1 $files`; do
     echo -n > $SCRATCH_MNT/$i
     echo XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX > $SCRATCH_MNT/$i
 done

Re: [PATCH v3 12/14] generic/204: break out of fs-filling loop early if we ENOSPC

[Christoph Hellwig]

On Mon, Nov 06, 2017 at 05:54:11PM -0800, Darrick J. Wong wrote:
> From: Darrick J. Wong <darrick.wong@oracle.com>
> 
> Use the available block count to compute the number of files we think
> we can create, rather than hardcoding a particular size.  This fixes
> the ENOSPC failures for xfs filesystems with rmap/reflink support.

This seems to work fine for me.  Note that even without 204 didn't
always fail for me on reflink filesystems, although it seems like it
failed more ofen than not:

Reviewed-by: Christoph Hellwig <hch@lst.de>