[PATCHSET v3 0/1] fstests: exercise code refactored in 5.14

[PATCHSET v3 0/1] fstests: exercise code refactored in 5.14

[Darrick J. Wong]

Hi all,

Add new tests to exercise code that got refactored in 5.14.  The
nested shutdown test simulates the process of recovering after a VM host
filesystem goes down and the guests have to recover.

v2: fix some bugs pointed out by the maintainer, add cpu offlining stress test
v3: run against the test fs, and limit the io thread count

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=new-tests-for-5.14
---
 tests/generic/726     |   74 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/726.out |    2 +
 2 files changed, 76 insertions(+)
 create mode 100755 tests/generic/726
 create mode 100644 tests/generic/726.out

[PATCH 1/1] generic: fsstress with cpu offlining

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Exercise filesystem operations when we're taking CPUs online and offline
throughout the test.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 tests/generic/726     |   74 +++++++++++++++++++++++++++++++++++++++++++++++++
 tests/generic/726.out |    2 +
 2 files changed, 76 insertions(+)
 create mode 100755 tests/generic/726
 create mode 100644 tests/generic/726.out


diff --git a/tests/generic/726 b/tests/generic/726
new file mode 100755
index 00000000..1a3f2fad
--- /dev/null
+++ b/tests/generic/726
@@ -0,0 +1,74 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Oracle, Inc.  All Rights Reserved.
+#
+# FS QA Test No. 726
+#
+# Run an all-writes fsstress run with multiple threads while exercising CPU
+# hotplugging to shake out bugs in the write path.
+#
+. ./common/preamble
+_begin_fstest auto rw stress
+
+# Override the default cleanup function.
+_cleanup()
+{
+	cd /
+	rm -f $tmp.*
+	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
+	wait	# for exercise_cpu_hotplug subprocess
+	for i in "$sysfs_cpu_dir/"cpu*/online; do
+		echo 1 > "$i" 2>/dev/null
+	done
+	test -n "$stress_dir" && rm -r -f "$stress_dir"
+}
+
+exercise_cpu_hotplug()
+{
+	while [ -e $sentinel_file ]; do
+		local idx=$(( RANDOM % nr_hotplug_cpus ))
+		local cpu="${hotplug_cpus[idx]}"
+		local action=$(( RANDOM % 2 ))
+
+		echo "$action" > "$sysfs_cpu_dir/cpu$cpu/online" 2>/dev/null
+		sleep 0.5
+	done
+}
+
+_supported_fs generic
+_require_test
+_require_command "$KILLALL_PROG" "killall"
+
+sysfs_cpu_dir="/sys/devices/system/cpu"
+
+# Figure out which CPU(s) support hotplug.
+nrcpus=$(getconf _NPROCESSORS_CONF)
+hotplug_cpus=()
+for ((i = 0; i < nrcpus; i++ )); do
+	test -e "$sysfs_cpu_dir/cpu$i/online" && hotplug_cpus+=("$i")
+done
+nr_hotplug_cpus="${#hotplug_cpus[@]}"
+test "$nr_hotplug_cpus" -gt 0 || _notrun "CPU hotplugging not supported"
+
+stress_dir="$TEST_DIR/$seq"
+rm -r -f "$stress_dir"
+mkdir -p "$stress_dir"
+
+echo "Silence is golden."
+
+sentinel_file=$tmp.hotplug
+touch $sentinel_file
+exercise_cpu_hotplug &
+
+# Cap the number of fsstress threads at one per hotpluggable CPU if we exceed
+# 1024 IO threads, per maintainer request.
+nr_cpus=$((LOAD_FACTOR * nr_hotplug_cpus))
+test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
+
+nr_ops=$((25000 * TIME_FACTOR))
+$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $stress_dir -n $nr_ops -p $nr_cpus >> $seqres.full
+rm -f $sentinel_file
+
+# success, all done
+status=0
+exit
diff --git a/tests/generic/726.out b/tests/generic/726.out
new file mode 100644
index 00000000..6839f8ce
--- /dev/null
+++ b/tests/generic/726.out
@@ -0,0 +1,2 @@
+QA output created by 726
+Silence is golden.

Re: [PATCH 1/1] generic: fsstress with cpu offlining

[riteshh]

On 21/09/15 04:42PM, Darrick J. Wong wrote:
> From: Darrick J. Wong <djwong@kernel.org>
>
> Exercise filesystem operations when we're taking CPUs online and offline
> throughout the test.

Nice test coverage. Btw, I may have missed older versions, but could you point
to the link which points to the bugs which this test uncovered?
I guess it will be good to add tha in the comment section of test description
too.

This also made me think whether doing memory online/offline while running
fsstress, makes any sense?

>
> Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> ---
>  tests/generic/726     |   74 +++++++++++++++++++++++++++++++++++++++++++++++++
>  tests/generic/726.out |    2 +
>  2 files changed, 76 insertions(+)
>  create mode 100755 tests/generic/726
>  create mode 100644 tests/generic/726.out
>
>
> diff --git a/tests/generic/726 b/tests/generic/726
> new file mode 100755
> index 00000000..1a3f2fad
> --- /dev/null
> +++ b/tests/generic/726
> @@ -0,0 +1,74 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Oracle, Inc.  All Rights Reserved.
> +#
> +# FS QA Test No. 726
> +#
> +# Run an all-writes fsstress run with multiple threads while exercising CPU
> +# hotplugging to shake out bugs in the write path.
> +#
> +. ./common/preamble
> +_begin_fstest auto rw stress

Does it qualify for auto? This definitely is taking a longer time compared to
other auto tests on my qemu setup.

> +
> +# Override the default cleanup function.
> +_cleanup()
> +{
> +	cd /
> +	rm -f $tmp.*
> +	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
> +	wait	# for exercise_cpu_hotplug subprocess
> +	for i in "$sysfs_cpu_dir/"cpu*/online; do
> +		echo 1 > "$i" 2>/dev/null
> +	done
> +	test -n "$stress_dir" && rm -r -f "$stress_dir"
> +}
> +
> +exercise_cpu_hotplug()
> +{
> +	while [ -e $sentinel_file ]; do
> +		local idx=$(( RANDOM % nr_hotplug_cpus ))
> +		local cpu="${hotplug_cpus[idx]}"
> +		local action=$(( RANDOM % 2 ))
> +
> +		echo "$action" > "$sysfs_cpu_dir/cpu$cpu/online" 2>/dev/null
> +		sleep 0.5
> +	done
> +}
> +
> +_supported_fs generic
> +_require_test
> +_require_command "$KILLALL_PROG" "killall"
> +
> +sysfs_cpu_dir="/sys/devices/system/cpu"
> +
> +# Figure out which CPU(s) support hotplug.
> +nrcpus=$(getconf _NPROCESSORS_CONF)
> +hotplug_cpus=()
> +for ((i = 0; i < nrcpus; i++ )); do
> +	test -e "$sysfs_cpu_dir/cpu$i/online" && hotplug_cpus+=("$i")
> +done
> +nr_hotplug_cpus="${#hotplug_cpus[@]}"
> +test "$nr_hotplug_cpus" -gt 0 || _notrun "CPU hotplugging not supported"
> +
> +stress_dir="$TEST_DIR/$seq"
> +rm -r -f "$stress_dir"
> +mkdir -p "$stress_dir"
> +
> +echo "Silence is golden."
> +
> +sentinel_file=$tmp.hotplug
> +touch $sentinel_file
> +exercise_cpu_hotplug &
> +
> +# Cap the number of fsstress threads at one per hotpluggable CPU if we exceed
> +# 1024 IO threads, per maintainer request.
> +nr_cpus=$((LOAD_FACTOR * nr_hotplug_cpus))
> +test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
> +
> +nr_ops=$((25000 * TIME_FACTOR))
> +$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $stress_dir -n $nr_ops -p $nr_cpus >> $seqres.full
> +rm -f $sentinel_file
> +
> +# success, all done
> +status=0
> +exit
> diff --git a/tests/generic/726.out b/tests/generic/726.out
> new file mode 100644
> index 00000000..6839f8ce
> --- /dev/null
> +++ b/tests/generic/726.out
> @@ -0,0 +1,2 @@
> +QA output created by 726
> +Silence is golden.
>

Re: [PATCH 1/1] generic: fsstress with cpu offlining

[Darrick J. Wong]

On Sat, Sep 18, 2021 at 12:04:49AM +0530, riteshh wrote:
> On 21/09/15 04:42PM, Darrick J. Wong wrote:
> > From: Darrick J. Wong <djwong@kernel.org>
> >
> > Exercise filesystem operations when we're taking CPUs online and offline
> > throughout the test.
> 
> Nice test coverage. Btw, I may have missed older versions, but could you point
> to the link which points to the bugs which this test uncovered?
> I guess it will be good to add tha in the comment section of test description
> too.

I didn't uncover any bugs with 5.14, fortunately.  Dave Chinner was
messing around with per-CPU lists in XFS, so I figured I had better
write something to exercise the cpu-dead handlers to try to make sure
there weren't any obvious bugs in the code, and this is the result.

> This also made me think whether doing memory online/offline while
> running
> fsstress, makes any sense?

Heh, perhaps.  I hadn't gotten /that/ far... :)

--D

> 
> >
> > Signed-off-by: Darrick J. Wong <djwong@kernel.org>
> > ---
> >  tests/generic/726     |   74 +++++++++++++++++++++++++++++++++++++++++++++++++
> >  tests/generic/726.out |    2 +
> >  2 files changed, 76 insertions(+)
> >  create mode 100755 tests/generic/726
> >  create mode 100644 tests/generic/726.out
> >
> >
> > diff --git a/tests/generic/726 b/tests/generic/726
> > new file mode 100755
> > index 00000000..1a3f2fad
> > --- /dev/null
> > +++ b/tests/generic/726
> > @@ -0,0 +1,74 @@
> > +#! /bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2021 Oracle, Inc.  All Rights Reserved.
> > +#
> > +# FS QA Test No. 726
> > +#
> > +# Run an all-writes fsstress run with multiple threads while exercising CPU
> > +# hotplugging to shake out bugs in the write path.
> > +#
> > +. ./common/preamble
> > +_begin_fstest auto rw stress
> 
> Does it qualify for auto? This definitely is taking a longer time compared to
> other auto tests on my qemu setup.
> 
> > +
> > +# Override the default cleanup function.
> > +_cleanup()
> > +{
> > +	cd /
> > +	rm -f $tmp.*
> > +	$KILLALL_PROG -9 fsstress > /dev/null 2>&1
> > +	wait	# for exercise_cpu_hotplug subprocess
> > +	for i in "$sysfs_cpu_dir/"cpu*/online; do
> > +		echo 1 > "$i" 2>/dev/null
> > +	done
> > +	test -n "$stress_dir" && rm -r -f "$stress_dir"
> > +}
> > +
> > +exercise_cpu_hotplug()
> > +{
> > +	while [ -e $sentinel_file ]; do
> > +		local idx=$(( RANDOM % nr_hotplug_cpus ))
> > +		local cpu="${hotplug_cpus[idx]}"
> > +		local action=$(( RANDOM % 2 ))
> > +
> > +		echo "$action" > "$sysfs_cpu_dir/cpu$cpu/online" 2>/dev/null
> > +		sleep 0.5
> > +	done
> > +}
> > +
> > +_supported_fs generic
> > +_require_test
> > +_require_command "$KILLALL_PROG" "killall"
> > +
> > +sysfs_cpu_dir="/sys/devices/system/cpu"
> > +
> > +# Figure out which CPU(s) support hotplug.
> > +nrcpus=$(getconf _NPROCESSORS_CONF)
> > +hotplug_cpus=()
> > +for ((i = 0; i < nrcpus; i++ )); do
> > +	test -e "$sysfs_cpu_dir/cpu$i/online" && hotplug_cpus+=("$i")
> > +done
> > +nr_hotplug_cpus="${#hotplug_cpus[@]}"
> > +test "$nr_hotplug_cpus" -gt 0 || _notrun "CPU hotplugging not supported"
> > +
> > +stress_dir="$TEST_DIR/$seq"
> > +rm -r -f "$stress_dir"
> > +mkdir -p "$stress_dir"
> > +
> > +echo "Silence is golden."
> > +
> > +sentinel_file=$tmp.hotplug
> > +touch $sentinel_file
> > +exercise_cpu_hotplug &
> > +
> > +# Cap the number of fsstress threads at one per hotpluggable CPU if we exceed
> > +# 1024 IO threads, per maintainer request.
> > +nr_cpus=$((LOAD_FACTOR * nr_hotplug_cpus))
> > +test "$nr_cpus" -gt 1024 && nr_cpus="$nr_hotplug_cpus"
> > +
> > +nr_ops=$((25000 * TIME_FACTOR))
> > +$FSSTRESS_PROG $FSSTRESS_AVOID -w -d $stress_dir -n $nr_ops -p $nr_cpus >> $seqres.full
> > +rm -f $sentinel_file
> > +
> > +# success, all done
> > +status=0
> > +exit
> > diff --git a/tests/generic/726.out b/tests/generic/726.out
> > new file mode 100644
> > index 00000000..6839f8ce
> > --- /dev/null
> > +++ b/tests/generic/726.out
> > @@ -0,0 +1,2 @@
> > +QA output created by 726
> > +Silence is golden.
> >