[PATCH tests v2 0/8] tests: add some regression tests

[PATCH tests v2 0/8] tests: add some regression tests

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

Total 7 regression test is added:

 - 23rdev-lifetime
 - 25raid456-recovery-while-reshape
 - 25raid456-reshape-corrupt-data
 - 25raid456-reshape-deadlock
fixed patches is applied for above tests.

 - 25raid456-reshape-while-recovery
 - 24raid10deadlock
 - 24raid456deadlock
fixed patches is not merged yet, and they're still in maillist.

Changes in v2:
 - use new way to judge if test failed for patch 1,34
 - add four new tests, patch 5-8

Yu Kuai (8):
  tests: add a new test for rdev lifetime
  tests: support to skip checking dmesg
  tests: add a regression test for raid10 deadlock
  tests: add a regression test for raid456 deadlock
  tests: add a regression test that raid456 can't assemble
  tests: add a regression test that raid456 can't assemble again
  tests: add a regression test that reshape can corrupt data
  tests: add a regression test for raid456 deadlock again

 test                                   |  8 ++-
 tests/23rdev-lifetime                  | 34 ++++++++++
 tests/24raid10deadlock                 | 88 ++++++++++++++++++++++++++
 tests/24raid10deadlock.inject_error    |  0
 tests/24raid456deadlock                | 58 +++++++++++++++++
 tests/25raid456-recovery-while-reshape | 33 ++++++++++
 tests/25raid456-reshape-corrupt-data   | 35 ++++++++++
 tests/25raid456-reshape-deadlock       | 34 ++++++++++
 tests/25raid456-reshape-while-recovery | 32 ++++++++++
 9 files changed, 320 insertions(+), 2 deletions(-)
 create mode 100644 tests/23rdev-lifetime
 create mode 100644 tests/24raid10deadlock
 create mode 100644 tests/24raid10deadlock.inject_error
 create mode 100644 tests/24raid456deadlock
 create mode 100644 tests/25raid456-recovery-while-reshape
 create mode 100644 tests/25raid456-reshape-corrupt-data
 create mode 100644 tests/25raid456-reshape-deadlock
 create mode 100644 tests/25raid456-reshape-while-recovery

-- 
2.39.2

[PATCH v2 1/8] tests: add a new test for rdev lifetime

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

This test add and remove a underlying disk to raid concurretly, verify
that the following problem is fixed:

run mdadm test 23rdev-lifetime at Fri Apr 28 03:25:30 UTC 2023
md: could not open device unknown-block(1,0).
sysfs: cannot create duplicate filename '/devices/virtual/block/md0/md/dev-ram0'
CPU: 26 PID: 10521 Comm: test Not tainted 6.3.0-rc2-00134-g7b3a8828043c #115
Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIOS 1.16.1-2.fc37 04/014
Call Trace:
 <TASK>
 dump_stack_lvl+0xe7/0x180
 dump_stack+0x18/0x30
 sysfs_warn_dup+0xa2/0xd0
 sysfs_create_dir_ns+0x119/0x140
 kobject_add_internal+0x143/0x4d0
 kobject_add_varg+0x35/0x70
 kobject_add+0x64/0xd0
 bind_rdev_to_array+0x254/0x840 [md_mod]
 new_dev_store+0x14d/0x350 [md_mod]
 md_attr_store+0xc1/0x1a0 [md_mod]
 sysfs_kf_write+0x51/0x70
 kernfs_fop_write_iter+0x188/0x270
 vfs_write+0x27e/0x460
 ksys_write+0x85/0x180
 __x64_sys_write+0x21/0x30
 do_syscall_64+0x6c/0xe0
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f26bacf5387
Code: 0d 00 f7 d8 64 89 02 48 c7 c0 ff ff ff ff eb b7 0f 1f 00 f3 0f 1e fa 64 84
RSP: 002b:00007ffe98d79e68 EFLAGS: 00000246 ORIG_RAX: 0000000000000001
RAX: ffffffffffffffda RBX: 0000000000000004 RCX: 00007f26bacf5387
RDX: 0000000000000004 RSI: 000055bd10282bf0 RDI: 0000000000000001
RBP: 000055bd10282bf0 R08: 000000000000000a R09: 00007f26bad8b4e0
R10: 00007f26bad8b3e0 R11: 0000000000000246 R12: 0000000000000004
R13: 00007f26badc8520 R14: 0000000000000004 R15: 00007f26badc8700
 </TASK>

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/23rdev-lifetime | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 tests/23rdev-lifetime

diff --git a/tests/23rdev-lifetime b/tests/23rdev-lifetime
new file mode 100644
index 00000000..1750b0db
--- /dev/null
+++ b/tests/23rdev-lifetime
@@ -0,0 +1,34 @@
+devname=${dev0##*/}
+devt=`cat /sys/block/$devname/dev`
+pid=""
+runtime=2
+
+clean_up_test() {
+	pill -9 $pid
+	echo clear > /sys/block/md0/md/array_state
+}
+
+trap 'clean_up_test' EXIT
+
+add_by_sysfs() {
+        while true; do
+                echo $devt > /sys/block/md0/md/new_dev
+        done
+}
+
+remove_by_sysfs(){
+        while true; do
+                echo remove > /sys/block/md0/md/dev-${devname}/state
+        done
+}
+
+echo md0 > /sys/module/md_mod/parameters/new_array || die "create md0 failed"
+
+add_by_sysfs &
+pid="$pid $!"
+
+remove_by_sysfs &
+pid="$pid $!"
+
+sleep $runtime
+exit 0
-- 
2.39.2

[PATCH v2 2/8] tests: support to skip checking dmesg

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

Prepare to add a regression test for raid10 that require error injection
to trigger error path, and kernel will complain about io error, checking
dmesg for error log will make it impossible to pass this test.

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
Acked-by: Mariusz Tkaczyk <mariusz.tkaczyk@linux.intel.com>
---
 test | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/test b/test
index 61d9ee83..b244453b 100755
--- a/test
+++ b/test
@@ -107,8 +107,12 @@ do_test() {
 		echo -ne "$_script... "
 		if ( set -ex ; . $_script ) &> $targetdir/log
 		then
-			dmesg | grep -iq "error\|call trace\|segfault" &&
-				die "dmesg prints errors when testing $_basename!"
+			if [ -f "${_script}.inject_error" ]; then
+				echo "dmesg checking is skipped because test inject error"
+			else
+				dmesg | grep -iq "error\|call trace\|segfault" &&
+					die "dmesg prints errors when testing $_basename!"
+			fi
 			echo "succeeded"
 			_fail=0
 		else
-- 
2.39.2

[PATCH v2 3/8] tests: add a regression test for raid10 deadlock

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

The deadlock is described in [1], it's fixed first by [2], however,
it turns out this commit will trigger other problems[3], hence this
commit will be reverted and the deadlock is supposed to be fixed by [1].

[1] https://lore.kernel.org/linux-raid/20230322064122.2384589-5-yukuai1@huaweicloud.com/
[2] https://lore.kernel.org/linux-raid/20220621031129.24778-1-guoqing.jiang@linux.dev/
[3] https://lore.kernel.org/linux-raid/20230322064122.2384589-2-yukuai1@huaweicloud.com/

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/24raid10deadlock              | 88 +++++++++++++++++++++++++++++
 tests/24raid10deadlock.inject_error |  0
 2 files changed, 88 insertions(+)
 create mode 100644 tests/24raid10deadlock
 create mode 100644 tests/24raid10deadlock.inject_error

diff --git a/tests/24raid10deadlock b/tests/24raid10deadlock
new file mode 100644
index 00000000..ee330aa9
--- /dev/null
+++ b/tests/24raid10deadlock
@@ -0,0 +1,88 @@
+devs="$dev0 $dev1 $dev2 $dev3"
+runtime=120
+pid=""
+action_pid=""
+
+set_up_injection()
+{
+	echo -1 > /sys/kernel/debug/fail_make_request/times
+	echo 1 > /sys/kernel/debug/fail_make_request/probability
+	echo 0 > /sys/kernel/debug/fail_make_request/verbose
+	echo 1 > /sys/block/${1##*/}/make-it-fail
+}
+
+clean_up_injection()
+{
+	echo 0 > /sys/block/${1##*/}/make-it-fail
+	echo 0 > /sys/kernel/debug/fail_make_request/times
+	echo 0 > /sys/kernel/debug/fail_make_request/probability
+	echo 2 > /sys/kernel/debug/fail_make_request/verbose
+}
+
+test_rdev()
+{
+	while true; do
+		mdadm -f $md0 $1 &> /dev/null
+		mdadm -r $md0 $1 &> /dev/null
+		mdadm --zero-superblock $1 &> /dev/null
+		mdadm -a $md0 $1 &> /dev/null
+		sleep $2
+	done
+}
+
+test_write_action()
+{
+	while true; do
+		echo frozen > /sys/block/md0/md/sync_action
+		echo idle > /sys/block/md0/md/sync_action
+		sleep 0.1
+	done
+}
+
+set_up_test()
+{
+	fio -h &> /dev/null || die "fio not found"
+
+	# create a simple raid10
+	mdadm -Cv -R -n 4 -l10 $md0 $devs || die "create raid10 failed"
+}
+
+clean_up_test()
+{
+	clean_up_injection $dev0
+	pkill -9 fio
+	kill -9 $pid
+	kill -9 $action_pid
+
+	sleep 1
+
+	if ps $action_pid | tail -1 | awk '{print $3}' | grep D; then
+		die "thread that is writing sysfs is stuck in D state, deadlock is triggered"
+	fi
+	mdadm -S $md0
+}
+
+cat /sys/kernel/debug/fail_make_request/times || die "fault injection is not enabled"
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+# backgroup io pressure
+fio -filename=$md0 -rw=randwrite -direct=1 -name=test -bs=4k -numjobs=16 -iodepth=16 &
+
+# trigger add/remove device by io failure
+set_up_injection $dev0
+test_rdev $dev0 2 &
+pid="$pid $!"
+
+# add/remove device directly
+test_rdev $dev3 10 &
+pid="$pid $!"
+
+test_write_action &
+action_pid="$!"
+
+sleep $runtime
+
+exit 0
diff --git a/tests/24raid10deadlock.inject_error b/tests/24raid10deadlock.inject_error
new file mode 100644
index 00000000..e69de29b
-- 
2.39.2

[PATCH v2 4/8] tests: add a regression test for raid456 deadlock

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

The deadlock is described in [1], as the last patch described, it's
fixed first by [2], however this fix will be reverted and the deadlock
is supposed to be fixed by [3].

[1] https://lore.kernel.org/linux-raid/5ed54ffc-ce82-bf66-4eff-390cb23bc1ac@molgen.mpg.de/T/#t
[2] https://lore.kernel.org/linux-raid/20220621031129.24778-1-guoqing.jiang@linux.dev/
[3] https://lore.kernel.org/linux-raid/20230322064122.2384589-5-yukuai1@huaweicloud.com/

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/24raid456deadlock | 58 +++++++++++++++++++++++++++++++++++++++++
 1 file changed, 58 insertions(+)
 create mode 100644 tests/24raid456deadlock

diff --git a/tests/24raid456deadlock b/tests/24raid456deadlock
new file mode 100644
index 00000000..80e6e97e
--- /dev/null
+++ b/tests/24raid456deadlock
@@ -0,0 +1,58 @@
+devs="$dev0 $dev1 $dev2 $dev3 $dev4 $dev5"
+runtime=120
+pid=""
+old=`cat /proc/sys/vm/dirty_background_ratio`
+
+test_write_action()
+{
+	while true; do
+		echo check > /sys/block/md0/md/sync_action &> /dev/null
+		sleep 0.1
+		echo idle > /sys/block/md0/md/sync_action &> /dev/null
+	done
+}
+
+test_write_back()
+{
+	fio -filename=$md0 -bs=4k -rw=write -numjobs=1 -name=test \
+		-time_based -runtime=$runtime &> /dev/null
+}
+
+set_up_test()
+{
+	fio -h &> /dev/null || die "fio not found"
+
+	# create a simple raid6
+	mdadm -Cv -R -n 6 -l6 $md0 $devs --assume-clean || die "create raid6 failed"
+
+	# trigger dirty pages write back
+	echo 0 > /proc/sys/vm/dirty_background_ratio
+}
+
+clean_up_test()
+{
+	echo $old > /proc/sys/vm/dirty_background_ratio
+
+	pkill -9 fio
+	kill -9 $pid
+
+	sleep 1
+
+	if ps $pid | tail -1 | awk '{print $3}' | grep D; then
+		die "thread that is writing sysfs is stuck in D state, deadlock is triggered"
+	fi
+	mdadm -S $md0
+}
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+test_write_back &
+
+test_write_action &
+pid="$!"
+
+sleep $runtime
+
+exit 0
-- 
2.39.2

[PATCH v2 5/8] tests: add a regression test that raid456 can't assemble

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

If recovery is interrupted and reshape is started, then this array can't
assemble anymore. The problem is supposed to be fixed by [1].

[1] https://lore.kernel.org/linux-raid/20230529031045.1760883-1-yukuai1@huaweicloud.com/

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/25raid456-reshape-while-recovery | 32 ++++++++++++++++++++++++++
 1 file changed, 32 insertions(+)
 create mode 100644 tests/25raid456-reshape-while-recovery

diff --git a/tests/25raid456-reshape-while-recovery b/tests/25raid456-reshape-while-recovery
new file mode 100644
index 00000000..b9f871f2
--- /dev/null
+++ b/tests/25raid456-reshape-while-recovery
@@ -0,0 +1,32 @@
+devs="$dev0 $dev1 $dev2"
+
+set_up_test()
+{
+	mdadm -Cv -R -n 3 -l5 $md0 $devs --assume-clean --size=50M || die "create array failed"
+	mdadm -a $md0 $dev3 $dev4 || die "failed to bind new disk to array"
+	echo 1000 > /sys/block/md0/md/sync_speed_max
+}
+
+clean_up_test()
+{
+	mdadm -S $md0
+}
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+# set up replacement
+echo want_replacement > /sys/block/md0/md/rd0/state
+sleep 1
+
+# trigger reshape
+echo frozen > /sys/block/md0/md/sync_action
+mdadm --grow -l 6 $md0
+sleep 1
+
+# reassemeble array
+mdadm -S $md0 || die "can't stop array"
+mdadm --assemble $md0 $devs $dev3 $dev4 || die "can't assemble array"
+
+exit 0
-- 
2.39.2

[PATCH v2 6/8] tests: add a regression test that raid456 can't assemble again

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

This is a regression test for commit 0aecb06e2249 ("md/raid5: don't allow
replacement while reshape is in progress").

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/25raid456-recovery-while-reshape | 33 ++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)
 create mode 100644 tests/25raid456-recovery-while-reshape

diff --git a/tests/25raid456-recovery-while-reshape b/tests/25raid456-recovery-while-reshape
new file mode 100644
index 00000000..3f6251bf
--- /dev/null
+++ b/tests/25raid456-recovery-while-reshape
@@ -0,0 +1,33 @@
+devs="$dev0 $dev1 $dev2"
+
+set_up_test()
+{
+	mdadm -Cv -R -n 3 -l5 $md0 $devs --assume-clean --size=50M || die "create array failed"
+	mdadm -a $md0 $dev3 $dev4 || die "failed to bind new disk to array"
+	echo 1000 > /sys/block/md0/md/sync_speed_max
+}
+
+clean_up_test()
+{
+	mdadm -S $md0
+}
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+# trigger reshape
+mdadm --grow -l 6 $md0
+sleep 1
+
+# set up replacement
+echo frozen > /sys/block/md0/md/sync_action
+echo want_replacement > /sys/block/md0/md/rd0/state
+echo reshape > /sys/block/md0/md/sync_action
+sleep 1
+
+# reassemeble array
+mdadm -S $md0 || die "can't stop array"
+mdadm --assemble $md0 $devs $dev3 $dev4 || die "can't assemble array"
+
+exit 0
-- 
2.39.2

[PATCH v2 7/8] tests: add a regression test that reshape can corrupt data

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

This is a regression test for commit 1544e95c6dd8 ("md: fix data
corruption for raid456 when reshape restart while grow up").

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/25raid456-reshape-corrupt-data | 35 ++++++++++++++++++++++++++++
 1 file changed, 35 insertions(+)
 create mode 100644 tests/25raid456-reshape-corrupt-data

diff --git a/tests/25raid456-reshape-corrupt-data b/tests/25raid456-reshape-corrupt-data
new file mode 100644
index 00000000..fdb875fb
--- /dev/null
+++ b/tests/25raid456-reshape-corrupt-data
@@ -0,0 +1,35 @@
+devs="$dev0 $dev1 $dev2"
+
+set_up_test()
+{
+	mdadm -Cv -R -n 3 -l5 $md0 $devs --size=50M || die "create array failed"
+	mdadm -a $md0 $dev3 || die "failed to bind new disk to array"
+	mkfs.xfs -f $md0 || die "mkfs failed"
+	xfs_ncheck $md0 || die "check fs failed"
+}
+
+clean_up_test()
+{
+	mdadm -S $md0
+}
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+# trigger reshape
+echo 1000 > /sys/block/md0/md/sync_speed_max
+mdadm --grow -l 6 $md0
+sleep 1
+
+# stop and start reshape
+echo frozen > /sys/block/md0/md/sync_action
+echo system > /sys/block/md0/md/sync_speed_max
+echo reshape > /sys/block/md0/md/sync_action
+
+mdadm -W $md0
+
+# check if data is corrupted
+xfs_ncheck $md0 || die "data is corrupted after reshape"
+
+exit 0
-- 
2.39.2

[PATCH v2 8/8] tests: add a regression test for raid456 deadlock again

[Yu Kuai]

From: Yu Kuai <yukuai3@huawei.com>

This is a regression test for commit ("md/raid5: fix a deadlock in the
case that reshape is interrupted").

Signed-off-by: Yu Kuai <yukuai3@huawei.com>
---
 tests/25raid456-reshape-deadlock | 34 ++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)
 create mode 100644 tests/25raid456-reshape-deadlock

diff --git a/tests/25raid456-reshape-deadlock b/tests/25raid456-reshape-deadlock
new file mode 100644
index 00000000..bfa0cc56
--- /dev/null
+++ b/tests/25raid456-reshape-deadlock
@@ -0,0 +1,34 @@
+devs="$dev0 $dev1 $dev2"
+
+set_up_test()
+{
+	mdadm -Cv -R -n 3 -l5 $md0 $devs --size=50M || die "create array failed"
+	mdadm -a $md0 $dev3 || die "failed to bind new disk to array"
+	echo 1000 > /sys/block/md0/md/sync_speed_max
+}
+
+clean_up_test()
+{
+	echo idle > /sys/block/md0/md/sync_action
+	mdadm -S $md0
+}
+
+trap 'clean_up_test' EXIT
+
+set_up_test || die "set up test failed"
+
+# trigger reshape
+mdadm --grow -l 6 $md0
+sleep 1
+
+# stop reshape
+echo frozen > /sys/block/md0/md/sync_action
+
+# read accross reshape
+dd if=$md0 of=/dev/NULL bs=1m count=100 iflag=direct &> /dev/null &
+sleep 2
+
+# suspend array
+echo 1 > /sys/block/md0/md/suspend_lo
+
+exit 0
-- 
2.39.2

Re: [PATCH tests v2 0/8] tests: add some regression tests

[Jes Sorensen]

On 5/29/23 09:28, Yu Kuai wrote:
> From: Yu Kuai <yukuai3@huawei.com>
> 
> Total 7 regression test is added:
> 
>  - 23rdev-lifetime
>  - 25raid456-recovery-while-reshape
>  - 25raid456-reshape-corrupt-data
>  - 25raid456-reshape-deadlock
> fixed patches is applied for above tests.
> 
>  - 25raid456-reshape-while-recovery
>  - 24raid10deadlock
>  - 24raid456deadlock
> fixed patches is not merged yet, and they're still in maillist.
> 
> Changes in v2:
>  - use new way to judge if test failed for patch 1,34
>  - add four new tests, patch 5-8

All applied!

Thanks,
Jes