[PATCH 0/3] fstests: various fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix some problems with the xfs min log size calculation code.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 1/3] xfs: rework min log size helper

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

The recent _scratch_find_xfs_min_logblocks helper has a major thinko in
it -- it relies on feeding a too-small size to _scratch_do_mkfs so that
mkfs will tell us the minimum log size.  Unfortunately, _scratch_do_mkfs
will see that first failure and retry the mkfs without MKFS_OPTIONS,
which means that we return the minimum log size for the default mkfs
settings without MKFS_OPTIONS.

This is a problem if someone's running fstests with a set of
MKFS_OPTIONS that affects the minimum log size.  To fix this, open-code
the _scratch_do_mkfs retry behavior so that we only do the "retry
without MKFS_OPTIONS" behavior if the mkfs failed for a reason other
than the minimum log size check.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 common/xfs |   25 +++++++++++++++++++++++--
 1 file changed, 23 insertions(+), 2 deletions(-)


diff --git a/common/xfs b/common/xfs
index f8dafc6c..2b38e94b 100644
--- a/common/xfs
+++ b/common/xfs
@@ -87,16 +87,35 @@ _scratch_find_xfs_min_logblocks()
 	# minimum log size.
 	local XFS_MIN_LOG_BYTES=2097152
 
-	_scratch_do_mkfs "$mkfs_cmd" "cat" $* -N -l size=$XFS_MIN_LOG_BYTES \
+	# Try formatting the filesystem with all the options given and the
+	# minimum log size.  We hope either that this succeeds or that mkfs
+	# tells us the required minimum log size for the feature set.
+	#
+	# We cannot use _scratch_do_mkfs because it will retry /any/ failed
+	# mkfs with MKFS_OPTIONS removed even if the only "failure" was that
+	# the log was too small.
+	local extra_mkfs_options="$* -N -l size=$XFS_MIN_LOG_BYTES"
+	eval "$mkfs_cmd $MKFS_OPTIONS $extra_mkfs_options $SCRATCH_DEV" \
 		2>$tmp.mkfserr 1>$tmp.mkfsstd
 	local mkfs_status=$?
 
+	# If the format fails for a reason other than the log being too small,
+	# try again without MKFS_OPTIONS because that's what _scratch_do_mkfs
+	# will do if we pass in the log size option.
+	if [ $mkfs_status -ne 0 ] &&
+	   ! grep -q 'log size.*too small, minimum' $tmp.mkfserr; then
+		eval "$mkfs_cmd $extra_mkfs_options $SCRATCH_DEV" \
+			2>$tmp.mkfserr 1>$tmp.mkfsstd
+		mkfs_status=$?
+	fi
+
 	# mkfs suceeded, so we must pick out the log block size to do the
 	# unit conversion
 	if [ $mkfs_status -eq 0 ]; then
-		local blksz="$(grep '^log.*bsize' $tmp.mkfsstd | \
+		blksz="$(grep '^log.*bsize' $tmp.mkfsstd | \
 			sed -e 's/log.*bsize=\([0-9]*\).*$/\1/g')"
 		echo $((XFS_MIN_LOG_BYTES / blksz))
+		rm -f $tmp.mkfsstd $tmp.mkfserr
 		return
 	fi
 
@@ -104,6 +123,7 @@ _scratch_find_xfs_min_logblocks()
 	if grep -q 'minimum size is' $tmp.mkfserr; then
 		grep 'minimum size is' $tmp.mkfserr | \
 			sed -e 's/^.*minimum size is \([0-9]*\) blocks/\1/g'
+		rm -f $tmp.mkfsstd $tmp.mkfserr
 		return
 	fi
 
@@ -111,6 +131,7 @@ _scratch_find_xfs_min_logblocks()
 	echo "Cannot determine minimum log size" >&2
 	cat $tmp.mkfsstd >> $seqres.full
 	cat $tmp.mkfserr >> $seqres.full
+	rm -f $tmp.mkfsstd $tmp.mkfserr
 }
 
 _scratch_mkfs_xfs()

[PATCH 2/3] xfs/016: calculate minimum log size and end locations

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

xfs/016 looks for corruption in the log when the log wraps.  However,
it hardcodes the minimum log size and the "95%" point where it wants to
start the "nudge and check for corruption" part of the test.  New
features require larger logs, which causes the test to fail when it
can't mkfs with the smaller log size and when that 95% point doesn't put
us within 20x "_log_traffic 2"s of the end of the log.

Fix the first problem by using the new min log size helper and replace
the 95% figure with an estimate of where we need to be to guarantee that
the 20x loop wraps the log.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
 tests/xfs/016     |   66 +++++++++++++++++++++++++++++++++++++++++++----------
 tests/xfs/016.out |    1 +
 2 files changed, 54 insertions(+), 13 deletions(-)


diff --git a/tests/xfs/016 b/tests/xfs/016
index 3407a4b1..2b41127e 100755
--- a/tests/xfs/016
+++ b/tests/xfs/016
@@ -44,10 +44,21 @@ _block_filter()
 
 _init()
 {
+    echo "*** determine log size"
+    local sz_mb=50
+    local dsize="-d size=${sz_mb}m"
+    local lsize="-l size=$(_scratch_find_xfs_min_logblocks $dsize)b"
+    local force_opts="$dsize $lsize"
+    _scratch_mkfs_xfs $force_opts >> $seqres.full 2>&1
+
+    # set log_size and log_size_bb globally
+    log_size_bb=`_log_size`
+    log_size=$((log_size_bb * 512))
+    echo "log_size_bb = $log_size_bb log_size = $log_size" >> $seqres.full
+
     echo "*** reset partition"
-    $here/src/devzero -b 2048 -n 50 -v 198 $SCRATCH_DEV
+    $here/src/devzero -b 2048 -n $sz_mb -v 198 $SCRATCH_DEV # write 0xc6
     echo "*** mkfs"
-    force_opts="-dsize=50m -lsize=$log_size"
     #
     # Do not discard blocks as we check for patterns in free space.
     # 
@@ -65,6 +76,9 @@ _init()
     . $tmp.mkfs
     [ $logsunit -ne 0 ] && \
         _notrun "Cannot run this test using log MKFS_OPTIONS specified"
+
+    # quotas generate extra log traffic so force it off
+    _qmount_option noquota
 }
 
 _log_traffic()
@@ -157,6 +171,7 @@ _check_corrupt()
 # get standard environment, filters and checks
 . ./common/rc
 . ./common/filter
+. ./common/quota
 
 # real QA test starts here
 _supported_fs xfs
@@ -164,10 +179,6 @@ _supported_os Linux
 
 rm -f $seqres.full
 
-# mkfs sizes
-log_size=3493888
-log_size_bb=`expr $log_size / 512`
-
 _require_scratch
 _init
 
@@ -188,18 +199,30 @@ echo "log sunit = $lsunit"			>>$seqres.full
 [ $head -eq 2 -o $head -eq $((lsunit/512)) ] || \
     _fail "!!! unexpected initial log position $head vs. $((lsunit/512))"
 
-# find how how many blocks per op for 100 ops
-# ignore the fact that it will also include an unmount record etc...
-# this should be small overall
+# Step 1: Run 200 ops to estimate how how many log blocks are used for each op.
+# Ignore the fact that it will also include an unmount record; this should be
+# small overall.
 echo "    lots of traffic for sampling" >>$seqres.full
-sample_size_ops=100
+sample_size_ops=200
 _log_traffic $sample_size_ops
 head1=`_log_head`
 num_blocks=`expr $head1 - $head`
 blocks_per_op=`echo "scale=3; $num_blocks / $sample_size_ops" | bc`
+echo "log position = $head1; old log position: $head" >> $seqres.full
 echo "blocks_per_op = $blocks_per_op" >>$seqres.full
-num_expected_ops=`echo "$log_size_bb / $blocks_per_op" | bc`
+
+# Step 2: Quickly advance the log from wherever step 1 left us to the point
+# where the log is now 80% full on its first cycle.
+
+# Estimate the number of ops needed to get the log head close to but not past
+# near_end_min for a single mount.  We'd rather fall short and have to step our
+# way closer to the end than run past the end, so our target for this second
+# step is to fill 80% of the first cycle of the log.
+num_expected_ops=$(( 8 * $(echo "$log_size_bb / $blocks_per_op" | bc) / 10))
 echo "num_expected_ops = $num_expected_ops" >>$seqres.full
+
+# Compute the number of ops needed to get from wherever we are right now in
+# the log cycle to the 80% point.
 num_expected_to_go=`echo "$num_expected_ops - $sample_size_ops" | bc`
 echo "num_expected_to_go = $num_expected_to_go" >>$seqres.full
 
@@ -208,13 +231,30 @@ _log_traffic $num_expected_to_go
 head=`_log_head`
 echo "log position = $head"                     >>$seqres.full
 
-# e.g. 3891
-near_end_min=`echo "0.95 * $log_size_bb" | bc | sed 's/\..*//'`
+# Step 3: Gradually advance log traffic to get us from wherever step 2 left us
+# to the point where the log is within approximately 20 ops of wrapping into
+# the second cycle.
+
+# Since this is a log wrapping test, it's critical to push the log head to the
+# point where it will wrap around within twenty rounds of ops.  Compute the
+# expected value of the log head when we get to this point.  This "/ 1" piece
+# tricks bc into printing integer numbers.
+near_end_min=$(echo "$log_size_bb - (20 * $blocks_per_op / 1)" | bc)
 echo "near_end_min = $near_end_min" >>$seqres.full
 
+# Step us (in batches of 10 ops) to our goal.
+while [ $head -lt $near_end_min ]; do
+	echo "    bump traffic from $head towards $near_end_min" >> $seqres.full
+	_log_traffic 10 > /dev/null 2>&1
+	head=$(_log_head)
+done
+
 [ $head -gt $near_end_min -a $head -lt $log_size_bb ] || \
     _fail "!!! unexpected near end log position $head"
 
+# Step 4: Try to wrap the log, checking for corruption with each advance.
+# This is the functionality that we're actually trying to test.  We will try
+# 40 ops (in batches of 2) to try to wrap the log.
 for c in `seq 0 20`
 do
     echo "   little traffic"            >>$seqres.full
diff --git a/tests/xfs/016.out b/tests/xfs/016.out
index f7844cdf..f4c8f88d 100644
--- a/tests/xfs/016.out
+++ b/tests/xfs/016.out
@@ -1,4 +1,5 @@
 QA output created by 016
+*** determine log size
 *** reset partition
 Wrote 51200.00Kb (value 0xc6)
 *** mkfs

[PATCH 3/3] xfs/119: fix MKFS_OPTIONS exporting

[Darrick J. Wong]

From: Darrick J. Wong <darrick.wong@oracle.com>

This test originally exported its own MKFS_OPTIONS to force the tested
filesystem config to the mkfs defaults + test-specific log size options.
This overrides whatever the test runner might have set in MKFS_OPTIONS.

In commit 2fd273886b525 ("xfs: refactor minimum log size formatting
code") we fail to export our test-specific MKFS_OPTIONS before
calculating the minimum log size, which leads to the wrong min log size
being calculated once we fixed the helper to be smarter about mkfs options.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Allison Collins <allison.henderson@oracle.com>
---
 tests/xfs/119 |    3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)


diff --git a/tests/xfs/119 b/tests/xfs/119
index 8825a5c3..f245a0a6 100755
--- a/tests/xfs/119
+++ b/tests/xfs/119
@@ -38,7 +38,8 @@ _require_scratch
 # this may hang
 sync
 
-logblks=$(_scratch_find_xfs_min_logblocks -l version=2,su=64k)
+export MKFS_OPTIONS="-l version=2,su=64k"
+logblks=$(_scratch_find_xfs_min_logblocks)
 export MKFS_OPTIONS="-l version=2,size=${logblks}b,su=64k"
 export MOUNT_OPTIONS="-o logbsize=64k"
 _scratch_mkfs_xfs >/dev/null

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix broken cleanup in some tests.

Fix the mixed buffered/directio filtering function to pick up the new
locations of the iomap directio code, fix generic/081's broken lvm
cleanup, and reduce the time generic/561 waits for duperemove to finish.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Fix the mixed buffered/directio filtering function to pick up the new
locations of the iomap directio code, fix the bogus description in
xfs/504, and reduce the time generic/561 waits for duperemove to finish.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

The first patch reduces the likelihood that the test control process
will get OOM killed by increasing the chance that the test processes
themselves will be targeted.

The second patch teaches xfs/122 about some new structures.

The third patch converts some typedef usage.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

The first two patches fix t_open_tmpfiles to shut down the scratch
filesystem properly by reverting a broken fix and teaching xfstests to
pass the relevant handles around.  The final patch cleans up some open
coded src/godown calls against the scratch fs to use the wrapper.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes

Re: [PATCH 0/3] fstests: various fixes

[xuyang]

on 2019/05/10 16:47, Eryu Guan wrote:

> On Thu, May 09, 2019 at 10:21:12AM +0800, xuyang wrote:
>> on 2019/05/08 0:56, Darrick J. Wong wrote:
>>> Hi all,
>>>
>>> Here are three patches fixing various regressions in xfstests when
>>> mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
>>> the changes deal with the change in minimum log size requirements.  They
>>> weren't caught until now because there are a number of tests that call
>>> mkfs on a loop device or a file without using MKFS_OPTIONS.
>>>
>>> If you're going to start using this mess, you probably ought to just
>>> pull from my git trees, which are linked below.
>>>
>>> This is an extraordinary way to destroy everything.  Enjoy!
>>> Comments and questions are, as always, welcome.
>>>
>>> --D
>>>
>>> fstests git tree:
>>> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
>>>
>>>
>>>
>> Hi
>>
>> Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>
> Thanks for the testing! Just want to make sure that you tested all the
> three patches so that I can add your Tested-by tag too all of them?
>
> Thanks,
> Eryu
Hi Eryu

Yes.  I tested all the three patches.

>
>

Re: [PATCH 0/3] fstests: various fixes

[Eryu Guan]

On Thu, May 09, 2019 at 10:21:12AM +0800, xuyang wrote:
> on 2019/05/08 0:56, Darrick J. Wong wrote:
> > Hi all,
> > 
> > Here are three patches fixing various regressions in xfstests when
> > mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
> > the changes deal with the change in minimum log size requirements.  They
> > weren't caught until now because there are a number of tests that call
> > mkfs on a loop device or a file without using MKFS_OPTIONS.
> > 
> > If you're going to start using this mess, you probably ought to just
> > pull from my git trees, which are linked below.
> > 
> > This is an extraordinary way to destroy everything.  Enjoy!
> > Comments and questions are, as always, welcome.
> > 
> > --D
> > 
> > fstests git tree:
> > https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
> > 
> > 
> > 
> Hi
> 
> Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>

Thanks for the testing! Just want to make sure that you tested all the
three patches so that I can add your Tested-by tag too all of them?

Thanks,
Eryu

Re: [PATCH 0/3] fstests: various fixes

[xuyang]

on 2019/05/08 0:56, Darrick J. Wong wrote:
> Hi all,
>
> Here are three patches fixing various regressions in xfstests when
> mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
> the changes deal with the change in minimum log size requirements.  They
> weren't caught until now because there are a number of tests that call
> mkfs on a loop device or a file without using MKFS_OPTIONS.
>
> If you're going to start using this mess, you probably ought to just
> pull from my git trees, which are linked below.
>
> This is an extraordinary way to destroy everything.  Enjoy!
> Comments and questions are, as always, welcome.
>
> --D
>
> fstests git tree:
> https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes
>
>
>
Hi

Tested-by: Yang Xu<xuyang2018.jy@cn.fujitsu.com>

[PATCH 0/3] fstests: various fixes

[Darrick J. Wong]

Hi all,

Here are three patches fixing various regressions in xfstests when
mkfs.xfs defaults to enabling reflink and/or rmap by default.  Most of
the changes deal with the change in minimum log size requirements.  They
weren't caught until now because there are a number of tests that call
mkfs on a loop device or a file without using MKFS_OPTIONS.

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=random-fixes