[RFC PATCH v3 0/3] xfs: testcases for shrinking free space in the last AG

[RFC PATCH v3 0/3] xfs: testcases for shrinking free space in the last AG

[Gao Xiang]

Hi,

This version matches
kernel v8: https://lore.kernel.org/r/20210305025703.3069469-1-hsiangkao@redhat.com
xfsprogs RFC: https://lore.kernel.org/r/20201028114010.545331-1-hsiangkao@redhat.com

and mainly addresses comments for the previous version.

Thanks,
Gao Xinag

changes since RFC v2:
 - [1/3] add _require_scratch to _require_xfs_shrink() (Zorro);
 - [1/3] drop unneeded _scratch_unmount (Zorro);
 - [2/3] add _repair_scratch_fs for each test (Zorro);
 - [2/3] check new size after shrinking (Zorro);
 - [2/3] drop unneeded _scratch_unmount (Zorro);
 - [3/3] update group as "auto growfs ioctl prealloc stress" to follow xfs/104 (Zorro);
 - [3/3] use generic falloc instead of resvsp (Darrick).

Gao Xiang:
  common/xfs: add a _require_xfs_shrink helper
  xfs: basic functionality test for shrinking free space in the last AG
  xfs: stress test for shrinking free space in the last AG

 common/xfs        |  11 +++++
 tests/xfs/990     |  70 ++++++++++++++++++++++++++
 tests/xfs/990.out |  12 +++++
 tests/xfs/991     | 122 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   2 +
 6 files changed, 225 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out

-- 
2.27.0

[RFC PATCH v3 1/3] common/xfs: add a _require_xfs_shrink helper

[Gao Xiang]

In order to detect whether the current kernel supports XFS shrinking.

Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
---
 common/xfs | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/common/xfs b/common/xfs
index 2156749d..ea3b6cab 100644
--- a/common/xfs
+++ b/common/xfs
@@ -432,6 +432,17 @@ _supports_xfs_scrub()
 	return 0
 }
 
+_require_xfs_shrink()
+{
+	_require_scratch
+
+	_scratch_mkfs_xfs > /dev/null
+	_scratch_mount
+	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || \
+		_notrun "kernel does not support shrinking"
+	_scratch_unmount
+}
+
 # run xfs_check and friends on a FS.
 _check_xfs_filesystem()
 {
-- 
2.27.0

Re: [RFC PATCH v3 1/3] common/xfs: add a _require_xfs_shrink helper

[Eryu Guan]

On Mon, Mar 15, 2021 at 07:19:24PM +0800, Gao Xiang wrote:
> In order to detect whether the current kernel supports XFS shrinking.
> 
> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> ---
>  common/xfs | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/common/xfs b/common/xfs
> index 2156749d..ea3b6cab 100644
> --- a/common/xfs
> +++ b/common/xfs
> @@ -432,6 +432,17 @@ _supports_xfs_scrub()
>  	return 0
>  }
>  
> +_require_xfs_shrink()
> +{
> +	_require_scratch

_require_command "$XFS_GROWFS_PROG" xfs_growfs

> +
> +	_scratch_mkfs_xfs > /dev/null
> +	_scratch_mount
> +	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || \
> +		_notrun "kernel does not support shrinking"

Better to describe the behavior here to explain why EINVAL means kernel
supports shrink.

Thanks,
Eryu

> +	_scratch_unmount
> +}
> +
>  # run xfs_check and friends on a FS.
>  _check_xfs_filesystem()
>  {
> -- 
> 2.27.0

Re: [RFC PATCH v3 1/3] common/xfs: add a _require_xfs_shrink helper

[Gao Xiang]

On Mon, Mar 29, 2021 at 12:18:25AM +0800, Eryu Guan wrote:
> On Mon, Mar 15, 2021 at 07:19:24PM +0800, Gao Xiang wrote:
> > In order to detect whether the current kernel supports XFS shrinking.
> > 
> > Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> > ---
> >  common/xfs | 11 +++++++++++
> >  1 file changed, 11 insertions(+)
> > 
> > diff --git a/common/xfs b/common/xfs
> > index 2156749d..ea3b6cab 100644
> > --- a/common/xfs
> > +++ b/common/xfs
> > @@ -432,6 +432,17 @@ _supports_xfs_scrub()
> >  	return 0
> >  }
> >  
> > +_require_xfs_shrink()
> > +{
> > +	_require_scratch
> 
> _require_command "$XFS_GROWFS_PROG" xfs_growfs

will update.

> 
> > +
> > +	_scratch_mkfs_xfs > /dev/null
> > +	_scratch_mount
> > +	$XFS_GROWFS_PROG -D1 "$SCRATCH_MNT" 2>&1 | grep -q 'Invalid argument' || \
> > +		_notrun "kernel does not support shrinking"
> 
> Better to describe the behavior here to explain why EINVAL means kernel
> supports shrink.

hmmm.. I also don't think this check is correct now. Maybe try to shrink
one fs-block would be better. Will update later.

Thanks,
Gao Xiang

> 
> Thanks,
> Eryu
> 
> > +	_scratch_unmount
> > +}
> > +
> >  # run xfs_check and friends on a FS.
> >  _check_xfs_filesystem()
> >  {
> > -- 
> > 2.27.0
> 

[RFC PATCH v3 2/3] xfs: basic functionality test for shrinking free space in the last AG

[Gao Xiang]

Add basic test to make sure the functionality works as expected.

Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
---
 tests/xfs/990     | 70 +++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/990.out | 12 ++++++++
 tests/xfs/group   |  1 +
 3 files changed, 83 insertions(+)
 create mode 100755 tests/xfs/990
 create mode 100644 tests/xfs/990.out

diff --git a/tests/xfs/990 b/tests/xfs/990
new file mode 100755
index 00000000..3c79186e
--- /dev/null
+++ b/tests/xfs/990
@@ -0,0 +1,70 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 990
+#
+# XFS shrinkfs basic functionality test
+#
+# This test attempts to shrink with a small size (512K), half AG size and
+# an out-of-bound size (agsize + 1) to observe if it works as expected.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1    # failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+test_shrink()
+{
+	$XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1
+	ret=$?
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	_scratch_mount >> $seqres.full
+
+	$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+	. $tmp.growfs
+	[ $ret -eq 0 -a $1 -eq $dblocks ]
+}
+
+# real QA test starts here
+_supported_fs xfs
+_require_xfs_shrink
+
+rm -f $seqres.full
+echo "Format and mount"
+_scratch_mkfs -dsize="$((512 * 1024 * 1024))" -dagcount=3 2>&1 | \
+	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs
+t_dblocks=$dblocks
+_scratch_mount >> $seqres.full
+
+echo "Shrink fs (small size)"
+test_shrink $((t_dblocks-512*1024/dbsize)) || \
+	_fail "Shrink fs (small size) failure"
+
+echo "Shrink fs (half AG)"
+test_shrink $((t_dblocks-agsize/2)) || \
+	_fail "Shrink fs (half AG) failure"
+
+echo "Shrink fs (out-of-bound)"
+test_shrink $((t_dblocks-agsize-1)) && \
+	_fail "Shrink fs (out-of-bound) failure"
+[ $dblocks -ne $((t_dblocks-agsize/2)) ] && \
+	_fail "dblocks changed after shrinking failure"
+
+$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
+echo "*** done"
+
+# success, all done
+status=0
+exit
diff --git a/tests/xfs/990.out b/tests/xfs/990.out
new file mode 100644
index 00000000..812f89ef
--- /dev/null
+++ b/tests/xfs/990.out
@@ -0,0 +1,12 @@
+QA output created by 990
+Format and mount
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+Shrink fs (small size)
+Shrink fs (half AG)
+Shrink fs (out-of-bound)
+*** done
diff --git a/tests/xfs/group b/tests/xfs/group
index e861cec9..a7981b67 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -525,3 +525,4 @@
 525 auto quick mkfs
 526 auto quick mkfs
 527 auto quick quota
+990 auto quick growfs
-- 
2.27.0

Re: [RFC PATCH v3 2/3] xfs: basic functionality test for shrinking free space in the last AG

[Eryu Guan]

On Mon, Mar 15, 2021 at 07:19:25PM +0800, Gao Xiang wrote:
> Add basic test to make sure the functionality works as expected.
> 
> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> ---
>  tests/xfs/990     | 70 +++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/990.out | 12 ++++++++
>  tests/xfs/group   |  1 +
>  3 files changed, 83 insertions(+)
>  create mode 100755 tests/xfs/990
>  create mode 100644 tests/xfs/990.out
> 
> diff --git a/tests/xfs/990 b/tests/xfs/990
> new file mode 100755
> index 00000000..3c79186e
> --- /dev/null
> +++ b/tests/xfs/990
> @@ -0,0 +1,70 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
> +#
> +# FS QA Test 990
> +#
> +# XFS shrinkfs basic functionality test
> +#
> +# This test attempts to shrink with a small size (512K), half AG size and
> +# an out-of-bound size (agsize + 1) to observe if it works as expected.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1    # failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +test_shrink()
> +{
> +	$XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1
> +	ret=$?

I'm not sure what's the output of xfs_growfs when shrinking filesystem,
if it's easy to do filter, but it'd be good if we could just dump the
output and let .out file match & check the test result.

> +
> +	_scratch_unmount
> +	_repair_scratch_fs >> $seqres.full

_repair_scratch_fs will fix corruption if there's any, and always return 0 if
completes without problems. Is _check_scratch_fs() what you want?

> +	_scratch_mount >> $seqres.full
> +
> +	$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> +	. $tmp.growfs
> +	[ $ret -eq 0 -a $1 -eq $dblocks ]

Just dump the expected size after shrink if possible.

> +}
> +
> +# real QA test starts here
> +_supported_fs xfs

Still missing _require_scratch

> +_require_xfs_shrink
> +
> +rm -f $seqres.full
> +echo "Format and mount"
> +_scratch_mkfs -dsize="$((512 * 1024 * 1024))" -dagcount=3 2>&1 | \
> +	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs

Better to explain why mkfs with agcount=3

> +. $tmp.mkfs
> +t_dblocks=$dblocks
> +_scratch_mount >> $seqres.full
> +
> +echo "Shrink fs (small size)"
> +test_shrink $((t_dblocks-512*1024/dbsize)) || \
> +	_fail "Shrink fs (small size) failure"

If it's possible to turn test_shrink to .out file matching way, _fail is
not needed and below.

> +
> +echo "Shrink fs (half AG)"
> +test_shrink $((t_dblocks-agsize/2)) || \
> +	_fail "Shrink fs (half AG) failure"
> +
> +echo "Shrink fs (out-of-bound)"
> +test_shrink $((t_dblocks-agsize-1)) && \
> +	_fail "Shrink fs (out-of-bound) failure"
> +[ $dblocks -ne $((t_dblocks-agsize/2)) ] && \
> +	_fail "dblocks changed after shrinking failure"
> +
> +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
> +echo "*** done"
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/xfs/990.out b/tests/xfs/990.out
> new file mode 100644
> index 00000000..812f89ef
> --- /dev/null
> +++ b/tests/xfs/990.out
> @@ -0,0 +1,12 @@
> +QA output created by 990
> +Format and mount
> +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> +         = sunit=XXX swidth=XXX, unwritten=X
> +naming   =VERN bsize=XXX
> +log      =LDEV bsize=XXX blocks=XXX
> +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> +Shrink fs (small size)
> +Shrink fs (half AG)
> +Shrink fs (out-of-bound)
> +*** done
> diff --git a/tests/xfs/group b/tests/xfs/group
> index e861cec9..a7981b67 100644
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -525,3 +525,4 @@
>  525 auto quick mkfs
>  526 auto quick mkfs
>  527 auto quick quota
> +990 auto quick growfs

Maybe it's time to add a new 'shrinkfs' group?

Thanks,
Eryu

Re: [RFC PATCH v3 2/3] xfs: basic functionality test for shrinking free space in the last AG

[Gao Xiang]

On Mon, Mar 29, 2021 at 12:32:39AM +0800, Eryu Guan wrote:
> On Mon, Mar 15, 2021 at 07:19:25PM +0800, Gao Xiang wrote:
> > Add basic test to make sure the functionality works as expected.
> > 
> > Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> > ---
> >  tests/xfs/990     | 70 +++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/990.out | 12 ++++++++
> >  tests/xfs/group   |  1 +
> >  3 files changed, 83 insertions(+)
> >  create mode 100755 tests/xfs/990
> >  create mode 100644 tests/xfs/990.out
> > 
> > diff --git a/tests/xfs/990 b/tests/xfs/990
> > new file mode 100755
> > index 00000000..3c79186e
> > --- /dev/null
> > +++ b/tests/xfs/990
> > @@ -0,0 +1,70 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
> > +#
> > +# FS QA Test 990
> > +#
> > +# XFS shrinkfs basic functionality test
> > +#
> > +# This test attempts to shrink with a small size (512K), half AG size and
> > +# an out-of-bound size (agsize + 1) to observe if it works as expected.
> > +#
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1    # failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +test_shrink()
> > +{
> > +	$XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1
> > +	ret=$?
> 
> I'm not sure what's the output of xfs_growfs when shrinking filesystem,
> if it's easy to do filter, but it'd be good if we could just dump the
> output and let .out file match & check the test result.

Not quite sure if it's of some use (e.g. also need to update expected output
when output changed), since I think just make sure the shrinked size is as
expected and xfs_repair won't argue anything on the new size, that would be
enough.

> 
> > +
> > +	_scratch_unmount
> > +	_repair_scratch_fs >> $seqres.full
> 
> _repair_scratch_fs will fix corruption if there's any, and always return 0 if
> completes without problems. Is _check_scratch_fs() what you want?

I didn't notice this, will update.

> 
> > +	_scratch_mount >> $seqres.full
> > +
> > +	$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> > +	. $tmp.growfs
> > +	[ $ret -eq 0 -a $1 -eq $dblocks ]
> 
> Just dump the expected size after shrink if possible.

er... my idea is that I don't want to continue the test case
if any fails... if dump the expected size to output, I need to
fix blocksize as well, not sure if it's necessary.

> 
> > +}
> > +
> > +# real QA test starts here
> > +_supported_fs xfs
> 
> Still missing _require_scratch

No, _require_xfs_shrink will include that, just as _require_xfs_scratch_rmapbt.

> 
> > +_require_xfs_shrink
> > +
> > +rm -f $seqres.full
> > +echo "Format and mount"
> > +_scratch_mkfs -dsize="$((512 * 1024 * 1024))" -dagcount=3 2>&1 | \
> > +	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
> 
> Better to explain why mkfs with agcount=3

just because agcount = 1 is forbidden on purpose, and I need to make sure
shrinking to 2 AG is not possible yet. So agcount = 3 is the minimum number.

It's just a functionality test, so agcount = 3 also makes the result
determinatively.

> 
> > +. $tmp.mkfs
> > +t_dblocks=$dblocks
> > +_scratch_mount >> $seqres.full
> > +
> > +echo "Shrink fs (small size)"
> > +test_shrink $((t_dblocks-512*1024/dbsize)) || \
> > +	_fail "Shrink fs (small size) failure"
> 
> If it's possible to turn test_shrink to .out file matching way, _fail is
> not needed and below.

The same as above. I could try to fix blocksize if the opinion is strong.

> 
> > +
> > +echo "Shrink fs (half AG)"
> > +test_shrink $((t_dblocks-agsize/2)) || \
> > +	_fail "Shrink fs (half AG) failure"
> > +
> > +echo "Shrink fs (out-of-bound)"
> > +test_shrink $((t_dblocks-agsize-1)) && \
> > +	_fail "Shrink fs (out-of-bound) failure"
> > +[ $dblocks -ne $((t_dblocks-agsize/2)) ] && \
> > +	_fail "dblocks changed after shrinking failure"
> > +
> > +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
> > +echo "*** done"
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/xfs/990.out b/tests/xfs/990.out
> > new file mode 100644
> > index 00000000..812f89ef
> > --- /dev/null
> > +++ b/tests/xfs/990.out
> > @@ -0,0 +1,12 @@
> > +QA output created by 990
> > +Format and mount
> > +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> > +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> > +         = sunit=XXX swidth=XXX, unwritten=X
> > +naming   =VERN bsize=XXX
> > +log      =LDEV bsize=XXX blocks=XXX
> > +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> > +Shrink fs (small size)
> > +Shrink fs (half AG)
> > +Shrink fs (out-of-bound)
> > +*** done
> > diff --git a/tests/xfs/group b/tests/xfs/group
> > index e861cec9..a7981b67 100644
> > --- a/tests/xfs/group
> > +++ b/tests/xfs/group
> > @@ -525,3 +525,4 @@
> >  525 auto quick mkfs
> >  526 auto quick mkfs
> >  527 auto quick quota
> > +990 auto quick growfs
> 
> Maybe it's time to add a new 'shrinkfs' group?

not sure if it needs, since shrinkfs reuses growfs ioctl.

Thanks,
Gao Xiang

> 
> Thanks,
> Eryu
> 

Re: [RFC PATCH v3 2/3] xfs: basic functionality test for shrinking free space in the last AG

[Eryu Guan]

On Mon, Mar 29, 2021 at 04:06:03AM +0800, Gao Xiang wrote:
> On Mon, Mar 29, 2021 at 12:32:39AM +0800, Eryu Guan wrote:
> > On Mon, Mar 15, 2021 at 07:19:25PM +0800, Gao Xiang wrote:
> > > Add basic test to make sure the functionality works as expected.
> > > 
> > > Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> > > ---
> > >  tests/xfs/990     | 70 +++++++++++++++++++++++++++++++++++++++++++++++
> > >  tests/xfs/990.out | 12 ++++++++
> > >  tests/xfs/group   |  1 +
> > >  3 files changed, 83 insertions(+)
> > >  create mode 100755 tests/xfs/990
> > >  create mode 100644 tests/xfs/990.out
> > > 
> > > diff --git a/tests/xfs/990 b/tests/xfs/990
> > > new file mode 100755
> > > index 00000000..3c79186e
> > > --- /dev/null
> > > +++ b/tests/xfs/990
> > > @@ -0,0 +1,70 @@
> > > +#! /bin/bash
> > > +# SPDX-License-Identifier: GPL-2.0
> > > +# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
> > > +#
> > > +# FS QA Test 990
> > > +#
> > > +# XFS shrinkfs basic functionality test
> > > +#
> > > +# This test attempts to shrink with a small size (512K), half AG size and
> > > +# an out-of-bound size (agsize + 1) to observe if it works as expected.
> > > +#
> > > +seq=`basename $0`
> > > +seqres=$RESULT_DIR/$seq
> > > +echo "QA output created by $seq"
> > > +
> > > +here=`pwd`
> > > +tmp=/tmp/$$
> > > +status=1    # failure is the default!
> > > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > > +
> > > +# get standard environment, filters and checks
> > > +. ./common/rc
> > > +. ./common/filter
> > > +
> > > +test_shrink()
> > > +{
> > > +	$XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1
> > > +	ret=$?
> > 
> > I'm not sure what's the output of xfs_growfs when shrinking filesystem,
> > if it's easy to do filter, but it'd be good if we could just dump the
> > output and let .out file match & check the test result.
> 
> Not quite sure if it's of some use (e.g. also need to update expected output
> when output changed), since I think just make sure the shrinked size is as
> expected and xfs_repair won't argue anything on the new size, that would be
> enough.
> 
> > 
> > > +
> > > +	_scratch_unmount
> > > +	_repair_scratch_fs >> $seqres.full
> > 
> > _repair_scratch_fs will fix corruption if there's any, and always return 0 if
> > completes without problems. Is _check_scratch_fs() what you want?
> 
> I didn't notice this, will update.
> 
> > 
> > > +	_scratch_mount >> $seqres.full
> > > +
> > > +	$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> > > +	. $tmp.growfs
> > > +	[ $ret -eq 0 -a $1 -eq $dblocks ]
> > 
> > Just dump the expected size after shrink if possible.
> 
> er... my idea is that I don't want to continue the test case
> if any fails... if dump the expected size to output, I need to
> fix blocksize as well, not sure if it's necessary.
> 
> > 
> > > +}
> > > +
> > > +# real QA test starts here
> > > +_supported_fs xfs
> > 
> > Still missing _require_scratch
> 
> No, _require_xfs_shrink will include that, just as _require_xfs_scratch_rmapbt.

Then include the "scratch" part in the helper name, so it's clear that
it requires the SCRATCH_DEV, i.e. _require_xfs_scratch_shrink

> 
> > 
> > > +_require_xfs_shrink
> > > +
> > > +rm -f $seqres.full
> > > +echo "Format and mount"
> > > +_scratch_mkfs -dsize="$((512 * 1024 * 1024))" -dagcount=3 2>&1 | \
> > > +	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
> > 
> > Better to explain why mkfs with agcount=3
> 
> just because agcount = 1 is forbidden on purpose, and I need to make sure
> shrinking to 2 AG is not possible yet. So agcount = 3 is the minimum number.
> 
> It's just a functionality test, so agcount = 3 also makes the result
> determinatively.

Yeah, I'd guess the reason, but it'd be great to make it clear in
comments.

> 
> > 
> > > +. $tmp.mkfs
> > > +t_dblocks=$dblocks
> > > +_scratch_mount >> $seqres.full
> > > +
> > > +echo "Shrink fs (small size)"
> > > +test_shrink $((t_dblocks-512*1024/dbsize)) || \
> > > +	_fail "Shrink fs (small size) failure"
> > 
> > If it's possible to turn test_shrink to .out file matching way, _fail is
> > not needed and below.
> 
> The same as above. I could try to fix blocksize if the opinion is strong.
> 
> > 
> > > +
> > > +echo "Shrink fs (half AG)"
> > > +test_shrink $((t_dblocks-agsize/2)) || \
> > > +	_fail "Shrink fs (half AG) failure"
> > > +
> > > +echo "Shrink fs (out-of-bound)"
> > > +test_shrink $((t_dblocks-agsize-1)) && \
> > > +	_fail "Shrink fs (out-of-bound) failure"
> > > +[ $dblocks -ne $((t_dblocks-agsize/2)) ] && \
> > > +	_fail "dblocks changed after shrinking failure"
> > > +
> > > +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
> > > +echo "*** done"
> > > +
> > > +# success, all done
> > > +status=0
> > > +exit
> > > diff --git a/tests/xfs/990.out b/tests/xfs/990.out
> > > new file mode 100644
> > > index 00000000..812f89ef
> > > --- /dev/null
> > > +++ b/tests/xfs/990.out
> > > @@ -0,0 +1,12 @@
> > > +QA output created by 990
> > > +Format and mount
> > > +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> > > +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> > > +         = sunit=XXX swidth=XXX, unwritten=X
> > > +naming   =VERN bsize=XXX
> > > +log      =LDEV bsize=XXX blocks=XXX
> > > +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> > > +Shrink fs (small size)
> > > +Shrink fs (half AG)
> > > +Shrink fs (out-of-bound)
> > > +*** done
> > > diff --git a/tests/xfs/group b/tests/xfs/group
> > > index e861cec9..a7981b67 100644
> > > --- a/tests/xfs/group
> > > +++ b/tests/xfs/group
> > > @@ -525,3 +525,4 @@
> > >  525 auto quick mkfs
> > >  526 auto quick mkfs
> > >  527 auto quick quota
> > > +990 auto quick growfs
> > 
> > Maybe it's time to add a new 'shrinkfs' group?
> 
> not sure if it needs, since shrinkfs reuses growfs ioctl.

Ok, I have no strong preference on this, just think it's a bit easier to
run only shrink tests by ./check -g shrinkfs.

Thanks,
Eryu

Re: [RFC PATCH v3 2/3] xfs: basic functionality test for shrinking free space in the last AG

[Gao Xiang]

Hi Eryu,

On Mon, Mar 29, 2021 at 10:14:26AM +0800, Eryu Guan wrote:
> On Mon, Mar 29, 2021 at 04:06:03AM +0800, Gao Xiang wrote:
> > On Mon, Mar 29, 2021 at 12:32:39AM +0800, Eryu Guan wrote:
> > > On Mon, Mar 15, 2021 at 07:19:25PM +0800, Gao Xiang wrote:
> > > > Add basic test to make sure the functionality works as expected.
> > > > 
> > > > Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> > > > ---
> > > >  tests/xfs/990     | 70 +++++++++++++++++++++++++++++++++++++++++++++++
> > > >  tests/xfs/990.out | 12 ++++++++
> > > >  tests/xfs/group   |  1 +
> > > >  3 files changed, 83 insertions(+)
> > > >  create mode 100755 tests/xfs/990
> > > >  create mode 100644 tests/xfs/990.out
> > > > 
> > > > diff --git a/tests/xfs/990 b/tests/xfs/990
> > > > new file mode 100755
> > > > index 00000000..3c79186e
> > > > --- /dev/null
> > > > +++ b/tests/xfs/990
> > > > @@ -0,0 +1,70 @@
> > > > +#! /bin/bash
> > > > +# SPDX-License-Identifier: GPL-2.0
> > > > +# Copyright (c) 2021 Red Hat, Inc.  All Rights Reserved.
> > > > +#
> > > > +# FS QA Test 990
> > > > +#
> > > > +# XFS shrinkfs basic functionality test
> > > > +#
> > > > +# This test attempts to shrink with a small size (512K), half AG size and
> > > > +# an out-of-bound size (agsize + 1) to observe if it works as expected.
> > > > +#
> > > > +seq=`basename $0`
> > > > +seqres=$RESULT_DIR/$seq
> > > > +echo "QA output created by $seq"
> > > > +
> > > > +here=`pwd`
> > > > +tmp=/tmp/$$
> > > > +status=1    # failure is the default!
> > > > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > > > +
> > > > +# get standard environment, filters and checks
> > > > +. ./common/rc
> > > > +. ./common/filter
> > > > +
> > > > +test_shrink()
> > > > +{
> > > > +	$XFS_GROWFS_PROG -D"$1" $SCRATCH_MNT >> $seqres.full 2>&1
> > > > +	ret=$?
> > > 
> > > I'm not sure what's the output of xfs_growfs when shrinking filesystem,
> > > if it's easy to do filter, but it'd be good if we could just dump the
> > > output and let .out file match & check the test result.
> > 
> > Not quite sure if it's of some use (e.g. also need to update expected output
> > when output changed), since I think just make sure the shrinked size is as
> > expected and xfs_repair won't argue anything on the new size, that would be
> > enough.
> > 
> > > 
> > > > +
> > > > +	_scratch_unmount
> > > > +	_repair_scratch_fs >> $seqres.full
> > > 
> > > _repair_scratch_fs will fix corruption if there's any, and always return 0 if
> > > completes without problems. Is _check_scratch_fs() what you want?
> > 
> > I didn't notice this, will update.
> > 
> > > 
> > > > +	_scratch_mount >> $seqres.full
> > > > +
> > > > +	$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> > > > +	. $tmp.growfs
> > > > +	[ $ret -eq 0 -a $1 -eq $dblocks ]
> > > 
> > > Just dump the expected size after shrink if possible.
> > 
> > er... my idea is that I don't want to continue the test case
> > if any fails... if dump the expected size to output, I need to
> > fix blocksize as well, not sure if it's necessary.
> > 
> > > 
> > > > +}
> > > > +
> > > > +# real QA test starts here
> > > > +_supported_fs xfs
> > > 
> > > Still missing _require_scratch
> > 
> > No, _require_xfs_shrink will include that, just as _require_xfs_scratch_rmapbt.
> 
> Then include the "scratch" part in the helper name, so it's clear that
> it requires the SCRATCH_DEV, i.e. _require_xfs_scratch_shrink

ok, will rename this later. I'm working on the other stuff now.

> 
> > 
> > > 
> > > > +_require_xfs_shrink
> > > > +
> > > > +rm -f $seqres.full
> > > > +echo "Format and mount"
> > > > +_scratch_mkfs -dsize="$((512 * 1024 * 1024))" -dagcount=3 2>&1 | \
> > > > +	tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
> > > 
> > > Better to explain why mkfs with agcount=3
> > 
> > just because agcount = 1 is forbidden on purpose, and I need to make sure
> > shrinking to 2 AG is not possible yet. So agcount = 3 is the minimum number.
> > 
> > It's just a functionality test, so agcount = 3 also makes the result
> > determinatively.
> 
> Yeah, I'd guess the reason, but it'd be great to make it clear in
> comments.

ok, will try to add some description.

> 
> > 
> > > 
> > > > +. $tmp.mkfs
> > > > +t_dblocks=$dblocks
> > > > +_scratch_mount >> $seqres.full
> > > > +
> > > > +echo "Shrink fs (small size)"
> > > > +test_shrink $((t_dblocks-512*1024/dbsize)) || \
> > > > +	_fail "Shrink fs (small size) failure"
> > > 
> > > If it's possible to turn test_shrink to .out file matching way, _fail is
> > > not needed and below.
> > 
> > The same as above. I could try to fix blocksize if the opinion is strong.
> > 
> > > 
> > > > +
> > > > +echo "Shrink fs (half AG)"
> > > > +test_shrink $((t_dblocks-agsize/2)) || \
> > > > +	_fail "Shrink fs (half AG) failure"
> > > > +
> > > > +echo "Shrink fs (out-of-bound)"
> > > > +test_shrink $((t_dblocks-agsize-1)) && \
> > > > +	_fail "Shrink fs (out-of-bound) failure"
> > > > +[ $dblocks -ne $((t_dblocks-agsize/2)) ] && \
> > > > +	_fail "dblocks changed after shrinking failure"
> > > > +
> > > > +$XFS_INFO_PROG $SCRATCH_MNT >> $seqres.full
> > > > +echo "*** done"
> > > > +
> > > > +# success, all done
> > > > +status=0
> > > > +exit
> > > > diff --git a/tests/xfs/990.out b/tests/xfs/990.out
> > > > new file mode 100644
> > > > index 00000000..812f89ef
> > > > --- /dev/null
> > > > +++ b/tests/xfs/990.out
> > > > @@ -0,0 +1,12 @@
> > > > +QA output created by 990
> > > > +Format and mount
> > > > +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> > > > +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> > > > +         = sunit=XXX swidth=XXX, unwritten=X
> > > > +naming   =VERN bsize=XXX
> > > > +log      =LDEV bsize=XXX blocks=XXX
> > > > +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> > > > +Shrink fs (small size)
> > > > +Shrink fs (half AG)
> > > > +Shrink fs (out-of-bound)
> > > > +*** done
> > > > diff --git a/tests/xfs/group b/tests/xfs/group
> > > > index e861cec9..a7981b67 100644
> > > > --- a/tests/xfs/group
> > > > +++ b/tests/xfs/group
> > > > @@ -525,3 +525,4 @@
> > > >  525 auto quick mkfs
> > > >  526 auto quick mkfs
> > > >  527 auto quick quota
> > > > +990 auto quick growfs
> > > 
> > > Maybe it's time to add a new 'shrinkfs' group?
> > 
> > not sure if it needs, since shrinkfs reuses growfs ioctl.
> 
> Ok, I have no strong preference on this, just think it's a bit easier to
> run only shrink tests by ./check -g shrinkfs.

Currently only 2 test cases are available. ok, if that is needed, I could
add a new shrinkfs group.

(Also noted that xfsprogs patch haven't been merged yet. So I think the
 testcase needs to be formally merged after xfsprogs side is settled.)

Thanks,
Gao Xiang

> 
> Thanks,
> Eryu
> 

[RFC PATCH v3 3/3] xfs: stress test for shrinking free space in the last AG

[Gao Xiang]

This adds a stress testcase to shrink free space as much as
possible in the last AG with background fsstress workload.

The expectation is that no crash happens with expected output.

Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
---
 tests/xfs/991     | 122 ++++++++++++++++++++++++++++++++++++++++++++++
 tests/xfs/991.out |   8 +++
 tests/xfs/group   |   1 +
 3 files changed, 131 insertions(+)
 create mode 100755 tests/xfs/991
 create mode 100644 tests/xfs/991.out

diff --git a/tests/xfs/991 b/tests/xfs/991
new file mode 100755
index 00000000..7e7d318e
--- /dev/null
+++ b/tests/xfs/991
@@ -0,0 +1,122 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
+#
+# FS QA Test 991
+#
+# XFS online shrinkfs stress test
+#
+# This test attempts to shrink unused space as much as possible with
+# background fsstress workload. It will decrease the shrink size if
+# larger size fails. And totally repeat 2 * TIME_FACTOR times.
+#
+seq=`basename $0`
+seqres=$RESULT_DIR/$seq
+echo "QA output created by $seq"
+
+here=`pwd`
+tmp=/tmp/$$
+status=1	# failure is the default!
+trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
+
+# get standard environment, filters and checks
+. ./common/rc
+. ./common/filter
+
+create_scratch()
+{
+	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
+		_filter_mkfs 2>$tmp.mkfs >/dev/null
+	. $tmp.mkfs
+
+	if ! _try_scratch_mount 2>/dev/null; then
+		echo "failed to mount $SCRATCH_DEV"
+		exit 1
+	fi
+
+	# fix the reserve block pool to a known size so that the enospc
+	# calculations work out correctly.
+	_scratch_resvblks 1024 > /dev/null 2>&1
+}
+
+fill_scratch()
+{
+	$XFS_IO_PROG -f -c "falloc -k 0 $1" $SCRATCH_MNT/resvfile
+}
+
+stress_scratch()
+{
+	procs=3
+	nops=$((1000 * LOAD_FACTOR))
+	# -w ensures that the only ops are ones which cause write I/O
+	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
+	    -n $nops $FSSTRESS_AVOID`
+	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
+}
+
+# real QA test starts here
+_supported_fs xfs
+_require_xfs_shrink
+_require_xfs_io_command "falloc"
+
+rm -f $seqres.full
+_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
+. $tmp.mkfs	# extract blocksize and data size for scratch device
+
+decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
+endsize=`expr 125 \* 1048576`	# stop after shrinking this big
+[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
+
+nags=2
+totalcount=$((2 * TIME_FACTOR))
+
+while [ $totalcount -gt 0 ]; do
+	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
+	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
+
+	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
+
+	for i in `seq 125 -1 90`; do
+		fillsize=`expr $i \* 1048576`
+		out="$(fill_scratch $fillsize 2>&1)"
+		echo "$out" | grep -q 'No space left on device' && continue
+		test -n "${out}" && echo "$out"
+		break
+	done
+
+	while [ $size -gt $endsize ]; do
+		stress_scratch
+		sleep 1
+
+		decb=`expr $decsize / $dbsize`    # in data blocks
+		while [ $decb -gt 0 ]; do
+			sizeb=`expr $size / $dbsize - $decb`
+
+			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
+				>> $seqres.full 2>&1 && break
+
+			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
+			decb=`expr $decb / 2`
+		done
+
+		wait
+		[ $decb -eq 0 ] && break
+
+		# get latest dblocks
+		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
+		. $tmp.growfs
+
+		size=`expr $dblocks \* $dbsize`
+		_scratch_unmount
+		_repair_scratch_fs >> $seqres.full
+		_scratch_mount
+	done
+
+	_scratch_unmount
+	_repair_scratch_fs >> $seqres.full
+	totalcount=`expr $totalcount - 1`
+done
+
+echo "*** done"
+status=0
+exit
diff --git a/tests/xfs/991.out b/tests/xfs/991.out
new file mode 100644
index 00000000..e8209672
--- /dev/null
+++ b/tests/xfs/991.out
@@ -0,0 +1,8 @@
+QA output created by 991
+meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
+data     = bsize=XXX blocks=XXX, imaxpct=PCT
+         = sunit=XXX swidth=XXX, unwritten=X
+naming   =VERN bsize=XXX
+log      =LDEV bsize=XXX blocks=XXX
+realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
+*** done
diff --git a/tests/xfs/group b/tests/xfs/group
index a7981b67..cf190b59 100644
--- a/tests/xfs/group
+++ b/tests/xfs/group
@@ -526,3 +526,4 @@
 526 auto quick mkfs
 527 auto quick quota
 990 auto quick growfs
+991 auto growfs ioctl prealloc stress
-- 
2.27.0

Re: [RFC PATCH v3 3/3] xfs: stress test for shrinking free space in the last AG

[Eryu Guan]

On Mon, Mar 15, 2021 at 07:19:26PM +0800, Gao Xiang wrote:
> This adds a stress testcase to shrink free space as much as
> possible in the last AG with background fsstress workload.
> 
> The expectation is that no crash happens with expected output.
> 
> Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> ---
>  tests/xfs/991     | 122 ++++++++++++++++++++++++++++++++++++++++++++++
>  tests/xfs/991.out |   8 +++
>  tests/xfs/group   |   1 +
>  3 files changed, 131 insertions(+)
>  create mode 100755 tests/xfs/991
>  create mode 100644 tests/xfs/991.out
> 
> diff --git a/tests/xfs/991 b/tests/xfs/991
> new file mode 100755
> index 00000000..7e7d318e
> --- /dev/null
> +++ b/tests/xfs/991
> @@ -0,0 +1,122 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
> +#
> +# FS QA Test 991
> +#
> +# XFS online shrinkfs stress test
> +#
> +# This test attempts to shrink unused space as much as possible with
> +# background fsstress workload. It will decrease the shrink size if
> +# larger size fails. And totally repeat 2 * TIME_FACTOR times.
> +#
> +seq=`basename $0`
> +seqres=$RESULT_DIR/$seq
> +echo "QA output created by $seq"
> +
> +here=`pwd`
> +tmp=/tmp/$$
> +status=1	# failure is the default!
> +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> +
> +# get standard environment, filters and checks
> +. ./common/rc
> +. ./common/filter
> +
> +create_scratch()
> +{
> +	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
> +		_filter_mkfs 2>$tmp.mkfs >/dev/null
> +	. $tmp.mkfs
> +
> +	if ! _try_scratch_mount 2>/dev/null; then
> +		echo "failed to mount $SCRATCH_DEV"
> +		exit 1
> +	fi

_scratch_mount will do the check and _fail the test on mount failure.

> +
> +	# fix the reserve block pool to a known size so that the enospc
> +	# calculations work out correctly.
> +	_scratch_resvblks 1024 > /dev/null 2>&1
> +}
> +
> +fill_scratch()
> +{
> +	$XFS_IO_PROG -f -c "falloc -k 0 $1" $SCRATCH_MNT/resvfile
> +}
> +
> +stress_scratch()
> +{
> +	procs=3
> +	nops=$((1000 * LOAD_FACTOR))

Declare procs and nops as local.

> +	# -w ensures that the only ops are ones which cause write I/O
> +	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
> +	    -n $nops $FSSTRESS_AVOID`
> +	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &

I think it's more explicit to run run stress_scratch in background,
instead run fsstress in background implicit.

> +}
> +
> +# real QA test starts here
> +_supported_fs xfs

_require_scratch

> +_require_xfs_shrink
> +_require_xfs_io_command "falloc"
> +
> +rm -f $seqres.full
> +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
> +. $tmp.mkfs	# extract blocksize and data size for scratch device
> +
> +decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
> +endsize=`expr 125 \* 1048576`	# stop after shrinking this big
> +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
> +
> +nags=2
> +totalcount=$((2 * TIME_FACTOR))
> +
> +while [ $totalcount -gt 0 ]; do
> +	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
> +	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
> +
> +	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
> +
> +	for i in `seq 125 -1 90`; do
> +		fillsize=`expr $i \* 1048576`
> +		out="$(fill_scratch $fillsize 2>&1)"
> +		echo "$out" | grep -q 'No space left on device' && continue
> +		test -n "${out}" && echo "$out"
> +		break
> +	done
> +
> +	while [ $size -gt $endsize ]; do
> +		stress_scratch

So just call

stress_scratch &

here? So it's clear that we put stress_scratch in background, and the
'wait' below is waiting for it.

Thanks,
Eryu

> +		sleep 1
> +
> +		decb=`expr $decsize / $dbsize`    # in data blocks
> +		while [ $decb -gt 0 ]; do
> +			sizeb=`expr $size / $dbsize - $decb`
> +
> +			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
> +				>> $seqres.full 2>&1 && break
> +
> +			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
> +			decb=`expr $decb / 2`
> +		done
> +
> +		wait
> +		[ $decb -eq 0 ] && break
> +
> +		# get latest dblocks
> +		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> +		. $tmp.growfs
> +
> +		size=`expr $dblocks \* $dbsize`
> +		_scratch_unmount
> +		_repair_scratch_fs >> $seqres.full
> +		_scratch_mount
> +	done
> +
> +	_scratch_unmount
> +	_repair_scratch_fs >> $seqres.full
> +	totalcount=`expr $totalcount - 1`
> +done
> +
> +echo "*** done"
> +status=0
> +exit
> diff --git a/tests/xfs/991.out b/tests/xfs/991.out
> new file mode 100644
> index 00000000..e8209672
> --- /dev/null
> +++ b/tests/xfs/991.out
> @@ -0,0 +1,8 @@
> +QA output created by 991
> +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> +         = sunit=XXX swidth=XXX, unwritten=X
> +naming   =VERN bsize=XXX
> +log      =LDEV bsize=XXX blocks=XXX
> +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> +*** done
> diff --git a/tests/xfs/group b/tests/xfs/group
> index a7981b67..cf190b59 100644
> --- a/tests/xfs/group
> +++ b/tests/xfs/group
> @@ -526,3 +526,4 @@
>  526 auto quick mkfs
>  527 auto quick quota
>  990 auto quick growfs
> +991 auto growfs ioctl prealloc stress
> -- 
> 2.27.0

Re: [RFC PATCH v3 3/3] xfs: stress test for shrinking free space in the last AG

[Gao Xiang]

On Mon, Mar 29, 2021 at 12:46:29AM +0800, Eryu Guan wrote:
> On Mon, Mar 15, 2021 at 07:19:26PM +0800, Gao Xiang wrote:
> > This adds a stress testcase to shrink free space as much as
> > possible in the last AG with background fsstress workload.
> > 
> > The expectation is that no crash happens with expected output.
> > 
> > Signed-off-by: Gao Xiang <hsiangkao@redhat.com>
> > ---
> >  tests/xfs/991     | 122 ++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/xfs/991.out |   8 +++
> >  tests/xfs/group   |   1 +
> >  3 files changed, 131 insertions(+)
> >  create mode 100755 tests/xfs/991
> >  create mode 100644 tests/xfs/991.out
> > 
> > diff --git a/tests/xfs/991 b/tests/xfs/991
> > new file mode 100755
> > index 00000000..7e7d318e
> > --- /dev/null
> > +++ b/tests/xfs/991
> > @@ -0,0 +1,122 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2020-2021 Red Hat, Inc.  All Rights Reserved.
> > +#
> > +# FS QA Test 991
> > +#
> > +# XFS online shrinkfs stress test
> > +#
> > +# This test attempts to shrink unused space as much as possible with
> > +# background fsstress workload. It will decrease the shrink size if
> > +# larger size fails. And totally repeat 2 * TIME_FACTOR times.
> > +#
> > +seq=`basename $0`
> > +seqres=$RESULT_DIR/$seq
> > +echo "QA output created by $seq"
> > +
> > +here=`pwd`
> > +tmp=/tmp/$$
> > +status=1	# failure is the default!
> > +trap "rm -f $tmp.*; exit \$status" 0 1 2 3 15
> > +
> > +# get standard environment, filters and checks
> > +. ./common/rc
> > +. ./common/filter
> > +
> > +create_scratch()
> > +{
> > +	_scratch_mkfs_xfs $@ | tee -a $seqres.full | \
> > +		_filter_mkfs 2>$tmp.mkfs >/dev/null
> > +	. $tmp.mkfs
> > +
> > +	if ! _try_scratch_mount 2>/dev/null; then
> > +		echo "failed to mount $SCRATCH_DEV"
> > +		exit 1
> > +	fi
> 
> _scratch_mount will do the check and _fail the test on mount failure.

It was just copied & pasted from somewhere else rather than wrote from
scratch.

> 
> > +
> > +	# fix the reserve block pool to a known size so that the enospc
> > +	# calculations work out correctly.
> > +	_scratch_resvblks 1024 > /dev/null 2>&1
> > +}
> > +
> > +fill_scratch()
> > +{
> > +	$XFS_IO_PROG -f -c "falloc -k 0 $1" $SCRATCH_MNT/resvfile
> > +}
> > +
> > +stress_scratch()
> > +{
> > +	procs=3
> > +	nops=$((1000 * LOAD_FACTOR))
> 
> Declare procs and nops as local.

Will update, thanks for the suggestion.

> 
> > +	# -w ensures that the only ops are ones which cause write I/O
> > +	FSSTRESS_ARGS=`_scale_fsstress_args -d $SCRATCH_MNT -w -p $procs \
> > +	    -n $nops $FSSTRESS_AVOID`
> > +	$FSSTRESS_PROG $FSSTRESS_ARGS >> $seqres.full 2>&1 &
> 
> I think it's more explicit to run run stress_scratch in background,
> instead run fsstress in background implicit.

Will update as well.

> 
> > +}
> > +
> > +# real QA test starts here
> > +_supported_fs xfs
> 
> _require_scratch

No, _require_xfs_shrink also checks this.

> 
> > +_require_xfs_shrink
> > +_require_xfs_io_command "falloc"
> > +
> > +rm -f $seqres.full
> > +_scratch_mkfs_xfs | tee -a $seqres.full | _filter_mkfs 2>$tmp.mkfs
> > +. $tmp.mkfs	# extract blocksize and data size for scratch device
> > +
> > +decsize=`expr  42 \* 1048576`	# shrink in chunks of this size at most
> > +endsize=`expr 125 \* 1048576`	# stop after shrinking this big
> > +[ `expr $endsize / $dbsize` -lt $dblocks ] || _notrun "Scratch device too small"
> > +
> > +nags=2
> > +totalcount=$((2 * TIME_FACTOR))
> > +
> > +while [ $totalcount -gt 0 ]; do
> > +	size=`expr 1010 \* 1048576`	# 1010 megabytes initially
> > +	logblks=$(_scratch_find_xfs_min_logblocks -dsize=${size} -dagcount=${nags})
> > +
> > +	create_scratch -lsize=${logblks}b -dsize=${size} -dagcount=${nags}
> > +
> > +	for i in `seq 125 -1 90`; do
> > +		fillsize=`expr $i \* 1048576`
> > +		out="$(fill_scratch $fillsize 2>&1)"
> > +		echo "$out" | grep -q 'No space left on device' && continue
> > +		test -n "${out}" && echo "$out"
> > +		break
> > +	done
> > +
> > +	while [ $size -gt $endsize ]; do
> > +		stress_scratch
> 
> So just call
> 
> stress_scratch &
> 
> here? So it's clear that we put stress_scratch in background, and the
> 'wait' below is waiting for it.

Will update.

Thanks,
Gao Xiang

> 
> Thanks,
> Eryu
> 
> > +		sleep 1
> > +
> > +		decb=`expr $decsize / $dbsize`    # in data blocks
> > +		while [ $decb -gt 0 ]; do
> > +			sizeb=`expr $size / $dbsize - $decb`
> > +
> > +			$XFS_GROWFS_PROG -D ${sizeb} $SCRATCH_MNT \
> > +				>> $seqres.full 2>&1 && break
> > +
> > +			[ $decb -gt 100 ] && decb=`expr $decb + $RANDOM % 10`
> > +			decb=`expr $decb / 2`
> > +		done
> > +
> > +		wait
> > +		[ $decb -eq 0 ] && break
> > +
> > +		# get latest dblocks
> > +		$XFS_INFO_PROG $SCRATCH_MNT 2>&1 | _filter_mkfs 2>$tmp.growfs >/dev/null
> > +		. $tmp.growfs
> > +
> > +		size=`expr $dblocks \* $dbsize`
> > +		_scratch_unmount
> > +		_repair_scratch_fs >> $seqres.full
> > +		_scratch_mount
> > +	done
> > +
> > +	_scratch_unmount
> > +	_repair_scratch_fs >> $seqres.full
> > +	totalcount=`expr $totalcount - 1`
> > +done
> > +
> > +echo "*** done"
> > +status=0
> > +exit
> > diff --git a/tests/xfs/991.out b/tests/xfs/991.out
> > new file mode 100644
> > index 00000000..e8209672
> > --- /dev/null
> > +++ b/tests/xfs/991.out
> > @@ -0,0 +1,8 @@
> > +QA output created by 991
> > +meta-data=DDEV isize=XXX agcount=N, agsize=XXX blks
> > +data     = bsize=XXX blocks=XXX, imaxpct=PCT
> > +         = sunit=XXX swidth=XXX, unwritten=X
> > +naming   =VERN bsize=XXX
> > +log      =LDEV bsize=XXX blocks=XXX
> > +realtime =RDEV extsz=XXX blocks=XXX, rtextents=XXX
> > +*** done
> > diff --git a/tests/xfs/group b/tests/xfs/group
> > index a7981b67..cf190b59 100644
> > --- a/tests/xfs/group
> > +++ b/tests/xfs/group
> > @@ -526,3 +526,4 @@
> >  526 auto quick mkfs
> >  527 auto quick quota
> >  990 auto quick growfs
> > +991 auto growfs ioctl prealloc stress
> > -- 
> > 2.27.0
>