* [PATCH v2] selftest: add tests for debugging testing setup
@ 2023-02-14 21:54 Leah Rumancik
2023-02-14 22:02 ` Darrick J. Wong
2023-02-15 4:49 ` Zorro Lang
0 siblings, 2 replies; 8+ messages in thread
From: Leah Rumancik @ 2023-02-14 21:54 UTC (permalink / raw)
To: fstests; +Cc: zlang, djwong, Leah Rumancik
Many people have developed infrastructure around xfstests. In order to
test a setup, it would be helpful to have dummy tests that have
consistent test outcomes. Add a new test folder with the following
tests:
selftest/001 pass
selftest/002 fail from output mismatch
selftest/003 fail via _fail
selftest/004 skip
selftest/005 crash
selftest/006 hang
Also, create two new groups: 'selftest' which includes tests 001-004 and
'dangerous_selftest' which includes tests 005-006. The selftests will
run when running all tests but they are not part of the auto group.
Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
---
v1 -> v2:
- change name of folder from 'debug' to 'selftest'
- update tags to 'selftest' and 'dangerous_selftest'
- add selftest folder to SRC_GROUPS
- update copyright tag
- add test which _fails
v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
check | 2 +-
doc/group-names.txt | 2 ++
tests/selftest/001 | 14 ++++++++++++++
tests/selftest/001.out | 2 ++
tests/selftest/002 | 14 ++++++++++++++
tests/selftest/002.out | 2 ++
tests/selftest/003 | 16 ++++++++++++++++
tests/selftest/003.out | 2 ++
tests/selftest/004 | 16 ++++++++++++++++
tests/selftest/004.out | 2 ++
tests/selftest/005 | 17 +++++++++++++++++
tests/selftest/005.out | 2 ++
tests/selftest/006 | 20 ++++++++++++++++++++
tests/selftest/006.out | 2 ++
tests/selftest/Makefile | 24 ++++++++++++++++++++++++
15 files changed, 136 insertions(+), 1 deletion(-)
create mode 100755 tests/selftest/001
create mode 100644 tests/selftest/001.out
create mode 100755 tests/selftest/002
create mode 100644 tests/selftest/002.out
create mode 100755 tests/selftest/003
create mode 100644 tests/selftest/003.out
create mode 100755 tests/selftest/004
create mode 100644 tests/selftest/004.out
create mode 100755 tests/selftest/005
create mode 100644 tests/selftest/005.out
create mode 100755 tests/selftest/006
create mode 100644 tests/selftest/006.out
create mode 100644 tests/selftest/Makefile
diff --git a/check b/check
index 4827532e..c2065cdc 100755
--- a/check
+++ b/check
@@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
-SRC_GROUPS="generic shared"
+SRC_GROUPS="generic shared selftest"
export SRC_DIR="tests"
usage()
diff --git a/doc/group-names.txt b/doc/group-names.txt
index 771ce937..ace59e05 100644
--- a/doc/group-names.txt
+++ b/doc/group-names.txt
@@ -38,6 +38,7 @@ dangerous_fsstress_repair race fsstress and xfs_scrub online repair
dangerous_fsstress_scrub race fsstress and xfs_scrub checking
dangerous_repair fuzzers to evaluate xfs_repair offline repair
dangerous_scrub fuzzers to evaluate xfs_scrub checking
+dangerous_selftest selftests that crash/hang
data data loss checkers
dax direct access mode for persistent memory files
db xfs_db functional tests
@@ -111,6 +112,7 @@ samefs overlayfs when all layers are on the same fs
scrub filesystem metadata scrubbers
seed btrfs seeded filesystems
seek llseek functionality
+selftest tests with fixed results, used to validate testing setup
send btrfs send/receive
shrinkfs decreasing the size of a filesystem
shutdown FS_IOC_SHUTDOWN ioctl
diff --git a/tests/selftest/001 b/tests/selftest/001
new file mode 100755
index 00000000..f326fc83
--- /dev/null
+++ b/tests/selftest/001
@@ -0,0 +1,14 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 001
+#
+# A test that should always pass
+#
+. ./common/preamble
+_begin_fstest selftest
+
+echo "Silence is golden"
+status=0
+exit
+
diff --git a/tests/selftest/001.out b/tests/selftest/001.out
new file mode 100644
index 00000000..88678b8e
--- /dev/null
+++ b/tests/selftest/001.out
@@ -0,0 +1,2 @@
+QA output created by 001
+Silence is golden
diff --git a/tests/selftest/002 b/tests/selftest/002
new file mode 100755
index 00000000..a3e463b8
--- /dev/null
+++ b/tests/selftest/002
@@ -0,0 +1,14 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 002
+#
+# A test that should always fail due to output mismatch
+#
+. ./common/preamble
+_begin_fstest selftest
+
+echo "I am intentionally broken"
+status=0
+exit
+
diff --git a/tests/selftest/002.out b/tests/selftest/002.out
new file mode 100644
index 00000000..61705c7c
--- /dev/null
+++ b/tests/selftest/002.out
@@ -0,0 +1,2 @@
+QA output created by 002
+Silence is golden
diff --git a/tests/selftest/003 b/tests/selftest/003
new file mode 100755
index 00000000..80d9ea90
--- /dev/null
+++ b/tests/selftest/003
@@ -0,0 +1,16 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 003
+#
+# A test that _fail's
+#
+
+. ./common/preamble
+_begin_fstest selftest
+
+_fail "I have _fail'ed"
+
+status=0
+exit
+
diff --git a/tests/selftest/003.out b/tests/selftest/003.out
new file mode 100644
index 00000000..6895fc80
--- /dev/null
+++ b/tests/selftest/003.out
@@ -0,0 +1,2 @@
+QA output created by 003
+Silence is golden
diff --git a/tests/selftest/004 b/tests/selftest/004
new file mode 100755
index 00000000..eddf49af
--- /dev/null
+++ b/tests/selftest/004
@@ -0,0 +1,16 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 004
+#
+# A test that should always be skipped
+#
+. ./common/preamble
+_begin_fstest selftest
+
+_notrun "Always skip me"
+
+echo "I should be skipped"
+status=0
+exit
+
diff --git a/tests/selftest/004.out b/tests/selftest/004.out
new file mode 100644
index 00000000..af8614ae
--- /dev/null
+++ b/tests/selftest/004.out
@@ -0,0 +1,2 @@
+QA output created by 004
+Silence is golden
diff --git a/tests/selftest/005 b/tests/selftest/005
new file mode 100755
index 00000000..e8844d30
--- /dev/null
+++ b/tests/selftest/005
@@ -0,0 +1,17 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 005
+#
+# A test that crashes
+#
+. ./common/preamble
+_begin_fstest dangerous_selftest
+
+echo 1 > /proc/sys/kernel/sysrq
+echo c > /proc/sysrq-trigger
+
+echo "I should have crashed by now"
+status=0
+exit
+
diff --git a/tests/selftest/005.out b/tests/selftest/005.out
new file mode 100644
index 00000000..a5027f12
--- /dev/null
+++ b/tests/selftest/005.out
@@ -0,0 +1,2 @@
+QA output created by 005
+Silence is golden
diff --git a/tests/selftest/006 b/tests/selftest/006
new file mode 100755
index 00000000..c6fa0dc4
--- /dev/null
+++ b/tests/selftest/006
@@ -0,0 +1,20 @@
+#! /bin/bash
+# SPDX-License-Identifier: GPL-2.0
+#
+# FS QA Test No. 006
+#
+# A test that hangs
+#
+
+. ./common/preamble
+_begin_fstest dangerous_selftest
+
+while :
+do
+ sleep 1d
+done
+
+echo "I should still be sleeping"
+status=0
+exit
+
diff --git a/tests/selftest/006.out b/tests/selftest/006.out
new file mode 100644
index 00000000..675c1b7c
--- /dev/null
+++ b/tests/selftest/006.out
@@ -0,0 +1,2 @@
+QA output created by 006
+Silence is golden
diff --git a/tests/selftest/Makefile b/tests/selftest/Makefile
new file mode 100644
index 00000000..3ddfca37
--- /dev/null
+++ b/tests/selftest/Makefile
@@ -0,0 +1,24 @@
+#
+# Copyright (c) 2023 Google, Inc. All Rights Reserved.
+#
+
+TOPDIR = ../..
+include $(TOPDIR)/include/builddefs
+include $(TOPDIR)/include/buildgrouplist
+
+SELFTEST_DIR = selftest
+TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(SELFTEST_DIR)
+DIRT = group.list
+
+default: $(DIRT)
+
+include $(BUILDRULES)
+
+install:
+ $(INSTALL) -m 755 -d $(TARGET_DIR)
+ $(INSTALL) -m 755 $(TESTS) $(TARGET_DIR)
+ $(INSTALL) -m 644 group.list $(TARGET_DIR)
+ $(INSTALL) -m 644 $(OUTFILES) $(TARGET_DIR)
+
+# Nothing.
+install-dev install-lib:
--
2.39.1.581.gbfd45094c4-goog
^ permalink raw reply related [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-14 21:54 [PATCH v2] selftest: add tests for debugging testing setup Leah Rumancik
@ 2023-02-14 22:02 ` Darrick J. Wong
2023-02-15 4:49 ` Zorro Lang
1 sibling, 0 replies; 8+ messages in thread
From: Darrick J. Wong @ 2023-02-14 22:02 UTC (permalink / raw)
To: Leah Rumancik; +Cc: fstests, zlang
On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> Many people have developed infrastructure around xfstests. In order to
> test a setup, it would be helpful to have dummy tests that have
> consistent test outcomes. Add a new test folder with the following
> tests:
>
> selftest/001 pass
> selftest/002 fail from output mismatch
> selftest/003 fail via _fail
> selftest/004 skip
> selftest/005 crash
> selftest/006 hang
>
> Also, create two new groups: 'selftest' which includes tests 001-004 and
> 'dangerous_selftest' which includes tests 005-006. The selftests will
> run when running all tests but they are not part of the auto group.
>
> Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
Yay, thank you!
Reviewed-by: Darrick J. Wong <djwong@kernel.org>
--D
> ---
> v1 -> v2:
>
> - change name of folder from 'debug' to 'selftest'
> - update tags to 'selftest' and 'dangerous_selftest'
> - add selftest folder to SRC_GROUPS
> - update copyright tag
> - add test which _fails
>
> v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
>
> check | 2 +-
> doc/group-names.txt | 2 ++
> tests/selftest/001 | 14 ++++++++++++++
> tests/selftest/001.out | 2 ++
> tests/selftest/002 | 14 ++++++++++++++
> tests/selftest/002.out | 2 ++
> tests/selftest/003 | 16 ++++++++++++++++
> tests/selftest/003.out | 2 ++
> tests/selftest/004 | 16 ++++++++++++++++
> tests/selftest/004.out | 2 ++
> tests/selftest/005 | 17 +++++++++++++++++
> tests/selftest/005.out | 2 ++
> tests/selftest/006 | 20 ++++++++++++++++++++
> tests/selftest/006.out | 2 ++
> tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> 15 files changed, 136 insertions(+), 1 deletion(-)
> create mode 100755 tests/selftest/001
> create mode 100644 tests/selftest/001.out
> create mode 100755 tests/selftest/002
> create mode 100644 tests/selftest/002.out
> create mode 100755 tests/selftest/003
> create mode 100644 tests/selftest/003.out
> create mode 100755 tests/selftest/004
> create mode 100644 tests/selftest/004.out
> create mode 100755 tests/selftest/005
> create mode 100644 tests/selftest/005.out
> create mode 100755 tests/selftest/006
> create mode 100644 tests/selftest/006.out
> create mode 100644 tests/selftest/Makefile
>
> diff --git a/check b/check
> index 4827532e..c2065cdc 100755
> --- a/check
> +++ b/check
> @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
>
> rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
>
> -SRC_GROUPS="generic shared"
> +SRC_GROUPS="generic shared selftest"
> export SRC_DIR="tests"
>
> usage()
> diff --git a/doc/group-names.txt b/doc/group-names.txt
> index 771ce937..ace59e05 100644
> --- a/doc/group-names.txt
> +++ b/doc/group-names.txt
> @@ -38,6 +38,7 @@ dangerous_fsstress_repair race fsstress and xfs_scrub online repair
> dangerous_fsstress_scrub race fsstress and xfs_scrub checking
> dangerous_repair fuzzers to evaluate xfs_repair offline repair
> dangerous_scrub fuzzers to evaluate xfs_scrub checking
> +dangerous_selftest selftests that crash/hang
> data data loss checkers
> dax direct access mode for persistent memory files
> db xfs_db functional tests
> @@ -111,6 +112,7 @@ samefs overlayfs when all layers are on the same fs
> scrub filesystem metadata scrubbers
> seed btrfs seeded filesystems
> seek llseek functionality
> +selftest tests with fixed results, used to validate testing setup
> send btrfs send/receive
> shrinkfs decreasing the size of a filesystem
> shutdown FS_IOC_SHUTDOWN ioctl
> diff --git a/tests/selftest/001 b/tests/selftest/001
> new file mode 100755
> index 00000000..f326fc83
> --- /dev/null
> +++ b/tests/selftest/001
> @@ -0,0 +1,14 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 001
> +#
> +# A test that should always pass
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +echo "Silence is golden"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/001.out b/tests/selftest/001.out
> new file mode 100644
> index 00000000..88678b8e
> --- /dev/null
> +++ b/tests/selftest/001.out
> @@ -0,0 +1,2 @@
> +QA output created by 001
> +Silence is golden
> diff --git a/tests/selftest/002 b/tests/selftest/002
> new file mode 100755
> index 00000000..a3e463b8
> --- /dev/null
> +++ b/tests/selftest/002
> @@ -0,0 +1,14 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 002
> +#
> +# A test that should always fail due to output mismatch
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +echo "I am intentionally broken"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/002.out b/tests/selftest/002.out
> new file mode 100644
> index 00000000..61705c7c
> --- /dev/null
> +++ b/tests/selftest/002.out
> @@ -0,0 +1,2 @@
> +QA output created by 002
> +Silence is golden
> diff --git a/tests/selftest/003 b/tests/selftest/003
> new file mode 100755
> index 00000000..80d9ea90
> --- /dev/null
> +++ b/tests/selftest/003
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 003
> +#
> +# A test that _fail's
> +#
> +
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +_fail "I have _fail'ed"
> +
> +status=0
> +exit
> +
> diff --git a/tests/selftest/003.out b/tests/selftest/003.out
> new file mode 100644
> index 00000000..6895fc80
> --- /dev/null
> +++ b/tests/selftest/003.out
> @@ -0,0 +1,2 @@
> +QA output created by 003
> +Silence is golden
> diff --git a/tests/selftest/004 b/tests/selftest/004
> new file mode 100755
> index 00000000..eddf49af
> --- /dev/null
> +++ b/tests/selftest/004
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 004
> +#
> +# A test that should always be skipped
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +_notrun "Always skip me"
> +
> +echo "I should be skipped"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/004.out b/tests/selftest/004.out
> new file mode 100644
> index 00000000..af8614ae
> --- /dev/null
> +++ b/tests/selftest/004.out
> @@ -0,0 +1,2 @@
> +QA output created by 004
> +Silence is golden
> diff --git a/tests/selftest/005 b/tests/selftest/005
> new file mode 100755
> index 00000000..e8844d30
> --- /dev/null
> +++ b/tests/selftest/005
> @@ -0,0 +1,17 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 005
> +#
> +# A test that crashes
> +#
> +. ./common/preamble
> +_begin_fstest dangerous_selftest
> +
> +echo 1 > /proc/sys/kernel/sysrq
> +echo c > /proc/sysrq-trigger
> +
> +echo "I should have crashed by now"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/005.out b/tests/selftest/005.out
> new file mode 100644
> index 00000000..a5027f12
> --- /dev/null
> +++ b/tests/selftest/005.out
> @@ -0,0 +1,2 @@
> +QA output created by 005
> +Silence is golden
> diff --git a/tests/selftest/006 b/tests/selftest/006
> new file mode 100755
> index 00000000..c6fa0dc4
> --- /dev/null
> +++ b/tests/selftest/006
> @@ -0,0 +1,20 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 006
> +#
> +# A test that hangs
> +#
> +
> +. ./common/preamble
> +_begin_fstest dangerous_selftest
> +
> +while :
> +do
> + sleep 1d
> +done
> +
> +echo "I should still be sleeping"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/006.out b/tests/selftest/006.out
> new file mode 100644
> index 00000000..675c1b7c
> --- /dev/null
> +++ b/tests/selftest/006.out
> @@ -0,0 +1,2 @@
> +QA output created by 006
> +Silence is golden
> diff --git a/tests/selftest/Makefile b/tests/selftest/Makefile
> new file mode 100644
> index 00000000..3ddfca37
> --- /dev/null
> +++ b/tests/selftest/Makefile
> @@ -0,0 +1,24 @@
> +#
> +# Copyright (c) 2023 Google, Inc. All Rights Reserved.
> +#
> +
> +TOPDIR = ../..
> +include $(TOPDIR)/include/builddefs
> +include $(TOPDIR)/include/buildgrouplist
> +
> +SELFTEST_DIR = selftest
> +TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(SELFTEST_DIR)
> +DIRT = group.list
> +
> +default: $(DIRT)
> +
> +include $(BUILDRULES)
> +
> +install:
> + $(INSTALL) -m 755 -d $(TARGET_DIR)
> + $(INSTALL) -m 755 $(TESTS) $(TARGET_DIR)
> + $(INSTALL) -m 644 group.list $(TARGET_DIR)
> + $(INSTALL) -m 644 $(OUTFILES) $(TARGET_DIR)
> +
> +# Nothing.
> +install-dev install-lib:
> --
> 2.39.1.581.gbfd45094c4-goog
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-14 21:54 [PATCH v2] selftest: add tests for debugging testing setup Leah Rumancik
2023-02-14 22:02 ` Darrick J. Wong
@ 2023-02-15 4:49 ` Zorro Lang
2023-02-15 5:57 ` Amir Goldstein
1 sibling, 1 reply; 8+ messages in thread
From: Zorro Lang @ 2023-02-15 4:49 UTC (permalink / raw)
To: Leah Rumancik; +Cc: fstests
On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> Many people have developed infrastructure around xfstests. In order to
> test a setup, it would be helpful to have dummy tests that have
> consistent test outcomes. Add a new test folder with the following
> tests:
>
> selftest/001 pass
> selftest/002 fail from output mismatch
> selftest/003 fail via _fail
> selftest/004 skip
> selftest/005 crash
> selftest/006 hang
>
> Also, create two new groups: 'selftest' which includes tests 001-004 and
> 'dangerous_selftest' which includes tests 005-006. The selftests will
> run when running all tests but they are not part of the auto group.
>
> Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> ---
> v1 -> v2:
>
> - change name of folder from 'debug' to 'selftest'
> - update tags to 'selftest' and 'dangerous_selftest'
> - add selftest folder to SRC_GROUPS
> - update copyright tag
> - add test which _fails
Thanks, this version looks good to me, just each case has a blank line at EOF.
I'll help to remove those blank lines when I merge this patch.
Reviewed-by: Zorro Lang <zlang@redhat.com>
Thanks,
Zorro
>
> v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
>
> check | 2 +-
> doc/group-names.txt | 2 ++
> tests/selftest/001 | 14 ++++++++++++++
> tests/selftest/001.out | 2 ++
> tests/selftest/002 | 14 ++++++++++++++
> tests/selftest/002.out | 2 ++
> tests/selftest/003 | 16 ++++++++++++++++
> tests/selftest/003.out | 2 ++
> tests/selftest/004 | 16 ++++++++++++++++
> tests/selftest/004.out | 2 ++
> tests/selftest/005 | 17 +++++++++++++++++
> tests/selftest/005.out | 2 ++
> tests/selftest/006 | 20 ++++++++++++++++++++
> tests/selftest/006.out | 2 ++
> tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> 15 files changed, 136 insertions(+), 1 deletion(-)
> create mode 100755 tests/selftest/001
> create mode 100644 tests/selftest/001.out
> create mode 100755 tests/selftest/002
> create mode 100644 tests/selftest/002.out
> create mode 100755 tests/selftest/003
> create mode 100644 tests/selftest/003.out
> create mode 100755 tests/selftest/004
> create mode 100644 tests/selftest/004.out
> create mode 100755 tests/selftest/005
> create mode 100644 tests/selftest/005.out
> create mode 100755 tests/selftest/006
> create mode 100644 tests/selftest/006.out
> create mode 100644 tests/selftest/Makefile
>
> diff --git a/check b/check
> index 4827532e..c2065cdc 100755
> --- a/check
> +++ b/check
> @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
>
> rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
>
> -SRC_GROUPS="generic shared"
> +SRC_GROUPS="generic shared selftest"
> export SRC_DIR="tests"
>
> usage()
> diff --git a/doc/group-names.txt b/doc/group-names.txt
> index 771ce937..ace59e05 100644
> --- a/doc/group-names.txt
> +++ b/doc/group-names.txt
> @@ -38,6 +38,7 @@ dangerous_fsstress_repair race fsstress and xfs_scrub online repair
> dangerous_fsstress_scrub race fsstress and xfs_scrub checking
> dangerous_repair fuzzers to evaluate xfs_repair offline repair
> dangerous_scrub fuzzers to evaluate xfs_scrub checking
> +dangerous_selftest selftests that crash/hang
> data data loss checkers
> dax direct access mode for persistent memory files
> db xfs_db functional tests
> @@ -111,6 +112,7 @@ samefs overlayfs when all layers are on the same fs
> scrub filesystem metadata scrubbers
> seed btrfs seeded filesystems
> seek llseek functionality
> +selftest tests with fixed results, used to validate testing setup
> send btrfs send/receive
> shrinkfs decreasing the size of a filesystem
> shutdown FS_IOC_SHUTDOWN ioctl
> diff --git a/tests/selftest/001 b/tests/selftest/001
> new file mode 100755
> index 00000000..f326fc83
> --- /dev/null
> +++ b/tests/selftest/001
> @@ -0,0 +1,14 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 001
> +#
> +# A test that should always pass
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +echo "Silence is golden"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/001.out b/tests/selftest/001.out
> new file mode 100644
> index 00000000..88678b8e
> --- /dev/null
> +++ b/tests/selftest/001.out
> @@ -0,0 +1,2 @@
> +QA output created by 001
> +Silence is golden
> diff --git a/tests/selftest/002 b/tests/selftest/002
> new file mode 100755
> index 00000000..a3e463b8
> --- /dev/null
> +++ b/tests/selftest/002
> @@ -0,0 +1,14 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 002
> +#
> +# A test that should always fail due to output mismatch
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +echo "I am intentionally broken"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/002.out b/tests/selftest/002.out
> new file mode 100644
> index 00000000..61705c7c
> --- /dev/null
> +++ b/tests/selftest/002.out
> @@ -0,0 +1,2 @@
> +QA output created by 002
> +Silence is golden
> diff --git a/tests/selftest/003 b/tests/selftest/003
> new file mode 100755
> index 00000000..80d9ea90
> --- /dev/null
> +++ b/tests/selftest/003
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 003
> +#
> +# A test that _fail's
> +#
> +
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +_fail "I have _fail'ed"
> +
> +status=0
> +exit
> +
> diff --git a/tests/selftest/003.out b/tests/selftest/003.out
> new file mode 100644
> index 00000000..6895fc80
> --- /dev/null
> +++ b/tests/selftest/003.out
> @@ -0,0 +1,2 @@
> +QA output created by 003
> +Silence is golden
> diff --git a/tests/selftest/004 b/tests/selftest/004
> new file mode 100755
> index 00000000..eddf49af
> --- /dev/null
> +++ b/tests/selftest/004
> @@ -0,0 +1,16 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 004
> +#
> +# A test that should always be skipped
> +#
> +. ./common/preamble
> +_begin_fstest selftest
> +
> +_notrun "Always skip me"
> +
> +echo "I should be skipped"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/004.out b/tests/selftest/004.out
> new file mode 100644
> index 00000000..af8614ae
> --- /dev/null
> +++ b/tests/selftest/004.out
> @@ -0,0 +1,2 @@
> +QA output created by 004
> +Silence is golden
> diff --git a/tests/selftest/005 b/tests/selftest/005
> new file mode 100755
> index 00000000..e8844d30
> --- /dev/null
> +++ b/tests/selftest/005
> @@ -0,0 +1,17 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 005
> +#
> +# A test that crashes
> +#
> +. ./common/preamble
> +_begin_fstest dangerous_selftest
> +
> +echo 1 > /proc/sys/kernel/sysrq
> +echo c > /proc/sysrq-trigger
> +
> +echo "I should have crashed by now"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/005.out b/tests/selftest/005.out
> new file mode 100644
> index 00000000..a5027f12
> --- /dev/null
> +++ b/tests/selftest/005.out
> @@ -0,0 +1,2 @@
> +QA output created by 005
> +Silence is golden
> diff --git a/tests/selftest/006 b/tests/selftest/006
> new file mode 100755
> index 00000000..c6fa0dc4
> --- /dev/null
> +++ b/tests/selftest/006
> @@ -0,0 +1,20 @@
> +#! /bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +#
> +# FS QA Test No. 006
> +#
> +# A test that hangs
> +#
> +
> +. ./common/preamble
> +_begin_fstest dangerous_selftest
> +
> +while :
> +do
> + sleep 1d
> +done
> +
> +echo "I should still be sleeping"
> +status=0
> +exit
> +
> diff --git a/tests/selftest/006.out b/tests/selftest/006.out
> new file mode 100644
> index 00000000..675c1b7c
> --- /dev/null
> +++ b/tests/selftest/006.out
> @@ -0,0 +1,2 @@
> +QA output created by 006
> +Silence is golden
> diff --git a/tests/selftest/Makefile b/tests/selftest/Makefile
> new file mode 100644
> index 00000000..3ddfca37
> --- /dev/null
> +++ b/tests/selftest/Makefile
> @@ -0,0 +1,24 @@
> +#
> +# Copyright (c) 2023 Google, Inc. All Rights Reserved.
> +#
> +
> +TOPDIR = ../..
> +include $(TOPDIR)/include/builddefs
> +include $(TOPDIR)/include/buildgrouplist
> +
> +SELFTEST_DIR = selftest
> +TARGET_DIR = $(PKG_LIB_DIR)/$(TESTS_DIR)/$(SELFTEST_DIR)
> +DIRT = group.list
> +
> +default: $(DIRT)
> +
> +include $(BUILDRULES)
> +
> +install:
> + $(INSTALL) -m 755 -d $(TARGET_DIR)
> + $(INSTALL) -m 755 $(TESTS) $(TARGET_DIR)
> + $(INSTALL) -m 644 group.list $(TARGET_DIR)
> + $(INSTALL) -m 644 $(OUTFILES) $(TARGET_DIR)
> +
> +# Nothing.
> +install-dev install-lib:
> --
> 2.39.1.581.gbfd45094c4-goog
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-15 4:49 ` Zorro Lang
@ 2023-02-15 5:57 ` Amir Goldstein
2023-02-16 0:59 ` Leah Rumancik
0 siblings, 1 reply; 8+ messages in thread
From: Amir Goldstein @ 2023-02-15 5:57 UTC (permalink / raw)
To: Zorro Lang, Leah Rumancik; +Cc: fstests
On Wed, Feb 15, 2023 at 6:57 AM Zorro Lang <zlang@redhat.com> wrote:
>
> On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> > Many people have developed infrastructure around xfstests. In order to
> > test a setup, it would be helpful to have dummy tests that have
> > consistent test outcomes. Add a new test folder with the following
> > tests:
> >
> > selftest/001 pass
> > selftest/002 fail from output mismatch
> > selftest/003 fail via _fail
> > selftest/004 skip
> > selftest/005 crash
> > selftest/006 hang
> >
> > Also, create two new groups: 'selftest' which includes tests 001-004 and
> > 'dangerous_selftest' which includes tests 005-006. The selftests will
> > run when running all tests but they are not part of the auto group.
> >
> > Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> > ---
> > v1 -> v2:
> >
> > - change name of folder from 'debug' to 'selftest'
> > - update tags to 'selftest' and 'dangerous_selftest'
> > - add selftest folder to SRC_GROUPS
> > - update copyright tag
> > - add test which _fails
>
> Thanks, this version looks good to me, just each case has a blank line at EOF.
> I'll help to remove those blank lines when I merge this patch.
>
> Reviewed-by: Zorro Lang <zlang@redhat.com>
>
> Thanks,
> Zorro
>
>
> >
> > v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
> >
> > check | 2 +-
> > doc/group-names.txt | 2 ++
> > tests/selftest/001 | 14 ++++++++++++++
> > tests/selftest/001.out | 2 ++
> > tests/selftest/002 | 14 ++++++++++++++
> > tests/selftest/002.out | 2 ++
> > tests/selftest/003 | 16 ++++++++++++++++
> > tests/selftest/003.out | 2 ++
> > tests/selftest/004 | 16 ++++++++++++++++
> > tests/selftest/004.out | 2 ++
> > tests/selftest/005 | 17 +++++++++++++++++
> > tests/selftest/005.out | 2 ++
> > tests/selftest/006 | 20 ++++++++++++++++++++
> > tests/selftest/006.out | 2 ++
> > tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> > 15 files changed, 136 insertions(+), 1 deletion(-)
> > create mode 100755 tests/selftest/001
> > create mode 100644 tests/selftest/001.out
> > create mode 100755 tests/selftest/002
> > create mode 100644 tests/selftest/002.out
> > create mode 100755 tests/selftest/003
> > create mode 100644 tests/selftest/003.out
> > create mode 100755 tests/selftest/004
> > create mode 100644 tests/selftest/004.out
> > create mode 100755 tests/selftest/005
> > create mode 100644 tests/selftest/005.out
> > create mode 100755 tests/selftest/006
> > create mode 100644 tests/selftest/006.out
> > create mode 100644 tests/selftest/Makefile
> >
> > diff --git a/check b/check
> > index 4827532e..c2065cdc 100755
> > --- a/check
> > +++ b/check
> > @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
> >
> > rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
> >
> > -SRC_GROUPS="generic shared"
> > +SRC_GROUPS="generic shared selftest"
I know this is a bit of bikeshedding,
but is it really needed to add selftest/* to SRC_GROUPS and then filter it out
by group?
Seems to me that it would be simpler to not add any new groups and use
./check -g selftest/quick -g selftest/dangerous
to run the tests in the selftests subdir.
Just to be clear, the syntax above already works, nothing new to implement.
I do not object to adding the selftest groups if others like it better this way.
It would also be nice to drop a line in README about running selftests.
Thanks,
Amir.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-15 5:57 ` Amir Goldstein
@ 2023-02-16 0:59 ` Leah Rumancik
2023-02-16 5:41 ` Zorro Lang
0 siblings, 1 reply; 8+ messages in thread
From: Leah Rumancik @ 2023-02-16 0:59 UTC (permalink / raw)
To: Amir Goldstein; +Cc: Zorro Lang, fstests
On Wed, Feb 15, 2023 at 07:57:55AM +0200, Amir Goldstein wrote:
> On Wed, Feb 15, 2023 at 6:57 AM Zorro Lang <zlang@redhat.com> wrote:
> >
> > On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> > > Many people have developed infrastructure around xfstests. In order to
> > > test a setup, it would be helpful to have dummy tests that have
> > > consistent test outcomes. Add a new test folder with the following
> > > tests:
> > >
> > > selftest/001 pass
> > > selftest/002 fail from output mismatch
> > > selftest/003 fail via _fail
> > > selftest/004 skip
> > > selftest/005 crash
> > > selftest/006 hang
> > >
> > > Also, create two new groups: 'selftest' which includes tests 001-004 and
> > > 'dangerous_selftest' which includes tests 005-006. The selftests will
> > > run when running all tests but they are not part of the auto group.
> > >
> > > Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> > > ---
> > > v1 -> v2:
> > >
> > > - change name of folder from 'debug' to 'selftest'
> > > - update tags to 'selftest' and 'dangerous_selftest'
> > > - add selftest folder to SRC_GROUPS
> > > - update copyright tag
> > > - add test which _fails
> >
> > Thanks, this version looks good to me, just each case has a blank line at EOF.
> > I'll help to remove those blank lines when I merge this patch.
> >
> > Reviewed-by: Zorro Lang <zlang@redhat.com>
> >
> > Thanks,
> > Zorro
> >
> >
> > >
> > > v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
> > >
> > > check | 2 +-
> > > doc/group-names.txt | 2 ++
> > > tests/selftest/001 | 14 ++++++++++++++
> > > tests/selftest/001.out | 2 ++
> > > tests/selftest/002 | 14 ++++++++++++++
> > > tests/selftest/002.out | 2 ++
> > > tests/selftest/003 | 16 ++++++++++++++++
> > > tests/selftest/003.out | 2 ++
> > > tests/selftest/004 | 16 ++++++++++++++++
> > > tests/selftest/004.out | 2 ++
> > > tests/selftest/005 | 17 +++++++++++++++++
> > > tests/selftest/005.out | 2 ++
> > > tests/selftest/006 | 20 ++++++++++++++++++++
> > > tests/selftest/006.out | 2 ++
> > > tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> > > 15 files changed, 136 insertions(+), 1 deletion(-)
> > > create mode 100755 tests/selftest/001
> > > create mode 100644 tests/selftest/001.out
> > > create mode 100755 tests/selftest/002
> > > create mode 100644 tests/selftest/002.out
> > > create mode 100755 tests/selftest/003
> > > create mode 100644 tests/selftest/003.out
> > > create mode 100755 tests/selftest/004
> > > create mode 100644 tests/selftest/004.out
> > > create mode 100755 tests/selftest/005
> > > create mode 100644 tests/selftest/005.out
> > > create mode 100755 tests/selftest/006
> > > create mode 100644 tests/selftest/006.out
> > > create mode 100644 tests/selftest/Makefile
> > >
> > > diff --git a/check b/check
> > > index 4827532e..c2065cdc 100755
> > > --- a/check
> > > +++ b/check
> > > @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
> > >
> > > rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
> > >
> > > -SRC_GROUPS="generic shared"
> > > +SRC_GROUPS="generic shared selftest"
>
> I know this is a bit of bikeshedding,
> but is it really needed to add selftest/* to SRC_GROUPS and then filter it out
> by group?
>
> Seems to me that it would be simpler to not add any new groups and use
> ./check -g selftest/quick -g selftest/dangerous
> to run the tests in the selftests subdir.
>
> Just to be clear, the syntax above already works, nothing new to implement.
>
> I do not object to adding the selftest groups if others like it better this way.
I think it behaves more as expected when it is a part of SRC_GROUPS, but
I will probably run these by specifying specific tests - so I don't
care much about which groups are used / if it is in SRC_GROUPS or not,
and I am happy to change it if others agree.
>
> It would also be nice to drop a line in README about running selftests.
Sure, I can do this.
>
> Thanks,
> Amir.
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-16 0:59 ` Leah Rumancik
@ 2023-02-16 5:41 ` Zorro Lang
2023-02-17 18:41 ` Leah Rumancik
0 siblings, 1 reply; 8+ messages in thread
From: Zorro Lang @ 2023-02-16 5:41 UTC (permalink / raw)
To: Leah Rumancik; +Cc: Amir Goldstein, fstests
On Wed, Feb 15, 2023 at 04:59:04PM -0800, Leah Rumancik wrote:
> On Wed, Feb 15, 2023 at 07:57:55AM +0200, Amir Goldstein wrote:
> > On Wed, Feb 15, 2023 at 6:57 AM Zorro Lang <zlang@redhat.com> wrote:
> > >
> > > On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> > > > Many people have developed infrastructure around xfstests. In order to
> > > > test a setup, it would be helpful to have dummy tests that have
> > > > consistent test outcomes. Add a new test folder with the following
> > > > tests:
> > > >
> > > > selftest/001 pass
> > > > selftest/002 fail from output mismatch
> > > > selftest/003 fail via _fail
> > > > selftest/004 skip
> > > > selftest/005 crash
> > > > selftest/006 hang
> > > >
> > > > Also, create two new groups: 'selftest' which includes tests 001-004 and
> > > > 'dangerous_selftest' which includes tests 005-006. The selftests will
> > > > run when running all tests but they are not part of the auto group.
> > > >
> > > > Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> > > > ---
> > > > v1 -> v2:
> > > >
> > > > - change name of folder from 'debug' to 'selftest'
> > > > - update tags to 'selftest' and 'dangerous_selftest'
> > > > - add selftest folder to SRC_GROUPS
> > > > - update copyright tag
> > > > - add test which _fails
> > >
> > > Thanks, this version looks good to me, just each case has a blank line at EOF.
> > > I'll help to remove those blank lines when I merge this patch.
> > >
> > > Reviewed-by: Zorro Lang <zlang@redhat.com>
> > >
> > > Thanks,
> > > Zorro
> > >
> > >
> > > >
> > > > v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
> > > >
> > > > check | 2 +-
> > > > doc/group-names.txt | 2 ++
> > > > tests/selftest/001 | 14 ++++++++++++++
> > > > tests/selftest/001.out | 2 ++
> > > > tests/selftest/002 | 14 ++++++++++++++
> > > > tests/selftest/002.out | 2 ++
> > > > tests/selftest/003 | 16 ++++++++++++++++
> > > > tests/selftest/003.out | 2 ++
> > > > tests/selftest/004 | 16 ++++++++++++++++
> > > > tests/selftest/004.out | 2 ++
> > > > tests/selftest/005 | 17 +++++++++++++++++
> > > > tests/selftest/005.out | 2 ++
> > > > tests/selftest/006 | 20 ++++++++++++++++++++
> > > > tests/selftest/006.out | 2 ++
> > > > tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> > > > 15 files changed, 136 insertions(+), 1 deletion(-)
> > > > create mode 100755 tests/selftest/001
> > > > create mode 100644 tests/selftest/001.out
> > > > create mode 100755 tests/selftest/002
> > > > create mode 100644 tests/selftest/002.out
> > > > create mode 100755 tests/selftest/003
> > > > create mode 100644 tests/selftest/003.out
> > > > create mode 100755 tests/selftest/004
> > > > create mode 100644 tests/selftest/004.out
> > > > create mode 100755 tests/selftest/005
> > > > create mode 100644 tests/selftest/005.out
> > > > create mode 100755 tests/selftest/006
> > > > create mode 100644 tests/selftest/006.out
> > > > create mode 100644 tests/selftest/Makefile
> > > >
> > > > diff --git a/check b/check
> > > > index 4827532e..c2065cdc 100755
> > > > --- a/check
> > > > +++ b/check
> > > > @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
> > > >
> > > > rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
> > > >
> > > > -SRC_GROUPS="generic shared"
> > > > +SRC_GROUPS="generic shared selftest"
> >
> > I know this is a bit of bikeshedding,
> > but is it really needed to add selftest/* to SRC_GROUPS and then filter it out
> > by group?
> >
> > Seems to me that it would be simpler to not add any new groups and use
> > ./check -g selftest/quick -g selftest/dangerous
> > to run the tests in the selftests subdir.
> >
> > Just to be clear, the syntax above already works, nothing new to implement.
> >
> > I do not object to adding the selftest groups if others like it better this way.
>
> I think it behaves more as expected when it is a part of SRC_GROUPS, but
> I will probably run these by specifying specific tests - so I don't
> care much about which groups are used / if it is in SRC_GROUPS or not,
> and I am happy to change it if others agree.
I agree with the suggestion from Amir. The "selftest" should be an independent
test group only be run when a tester intend to run it.
Thanks,
Zorro
>
> >
> > It would also be nice to drop a line in README about running selftests.
>
> Sure, I can do this.
>
> >
> > Thanks,
> > Amir.
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-16 5:41 ` Zorro Lang
@ 2023-02-17 18:41 ` Leah Rumancik
2023-02-18 5:47 ` Zorro Lang
0 siblings, 1 reply; 8+ messages in thread
From: Leah Rumancik @ 2023-02-17 18:41 UTC (permalink / raw)
To: Zorro Lang; +Cc: Amir Goldstein, fstests
On Thu, Feb 16, 2023 at 01:41:07PM +0800, Zorro Lang wrote:
> On Wed, Feb 15, 2023 at 04:59:04PM -0800, Leah Rumancik wrote:
> > On Wed, Feb 15, 2023 at 07:57:55AM +0200, Amir Goldstein wrote:
> > > On Wed, Feb 15, 2023 at 6:57 AM Zorro Lang <zlang@redhat.com> wrote:
> > > >
> > > > On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> > > > > Many people have developed infrastructure around xfstests. In order to
> > > > > test a setup, it would be helpful to have dummy tests that have
> > > > > consistent test outcomes. Add a new test folder with the following
> > > > > tests:
> > > > >
> > > > > selftest/001 pass
> > > > > selftest/002 fail from output mismatch
> > > > > selftest/003 fail via _fail
> > > > > selftest/004 skip
> > > > > selftest/005 crash
> > > > > selftest/006 hang
> > > > >
> > > > > Also, create two new groups: 'selftest' which includes tests 001-004 and
> > > > > 'dangerous_selftest' which includes tests 005-006. The selftests will
> > > > > run when running all tests but they are not part of the auto group.
> > > > >
> > > > > Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> > > > > ---
> > > > > v1 -> v2:
> > > > >
> > > > > - change name of folder from 'debug' to 'selftest'
> > > > > - update tags to 'selftest' and 'dangerous_selftest'
> > > > > - add selftest folder to SRC_GROUPS
> > > > > - update copyright tag
> > > > > - add test which _fails
> > > >
> > > > Thanks, this version looks good to me, just each case has a blank line at EOF.
> > > > I'll help to remove those blank lines when I merge this patch.
> > > >
> > > > Reviewed-by: Zorro Lang <zlang@redhat.com>
> > > >
> > > > Thanks,
> > > > Zorro
> > > >
> > > >
> > > > >
> > > > > v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
> > > > >
> > > > > check | 2 +-
> > > > > doc/group-names.txt | 2 ++
> > > > > tests/selftest/001 | 14 ++++++++++++++
> > > > > tests/selftest/001.out | 2 ++
> > > > > tests/selftest/002 | 14 ++++++++++++++
> > > > > tests/selftest/002.out | 2 ++
> > > > > tests/selftest/003 | 16 ++++++++++++++++
> > > > > tests/selftest/003.out | 2 ++
> > > > > tests/selftest/004 | 16 ++++++++++++++++
> > > > > tests/selftest/004.out | 2 ++
> > > > > tests/selftest/005 | 17 +++++++++++++++++
> > > > > tests/selftest/005.out | 2 ++
> > > > > tests/selftest/006 | 20 ++++++++++++++++++++
> > > > > tests/selftest/006.out | 2 ++
> > > > > tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> > > > > 15 files changed, 136 insertions(+), 1 deletion(-)
> > > > > create mode 100755 tests/selftest/001
> > > > > create mode 100644 tests/selftest/001.out
> > > > > create mode 100755 tests/selftest/002
> > > > > create mode 100644 tests/selftest/002.out
> > > > > create mode 100755 tests/selftest/003
> > > > > create mode 100644 tests/selftest/003.out
> > > > > create mode 100755 tests/selftest/004
> > > > > create mode 100644 tests/selftest/004.out
> > > > > create mode 100755 tests/selftest/005
> > > > > create mode 100644 tests/selftest/005.out
> > > > > create mode 100755 tests/selftest/006
> > > > > create mode 100644 tests/selftest/006.out
> > > > > create mode 100644 tests/selftest/Makefile
> > > > >
> > > > > diff --git a/check b/check
> > > > > index 4827532e..c2065cdc 100755
> > > > > --- a/check
> > > > > +++ b/check
> > > > > @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
> > > > >
> > > > > rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
> > > > >
> > > > > -SRC_GROUPS="generic shared"
> > > > > +SRC_GROUPS="generic shared selftest"
> > >
> > > I know this is a bit of bikeshedding,
> > > but is it really needed to add selftest/* to SRC_GROUPS and then filter it out
> > > by group?
> > >
> > > Seems to me that it would be simpler to not add any new groups and use
> > > ./check -g selftest/quick -g selftest/dangerous
> > > to run the tests in the selftests subdir.
> > >
> > > Just to be clear, the syntax above already works, nothing new to implement.
> > >
> > > I do not object to adding the selftest groups if others like it better this way.
> >
> > I think it behaves more as expected when it is a part of SRC_GROUPS, but
> > I will probably run these by specifying specific tests - so I don't
> > care much about which groups are used / if it is in SRC_GROUPS or not,
> > and I am happy to change it if others agree.
>
> I agree with the suggestion from Amir. The "selftest" should be an independent
> test group only be run when a tester intend to run it.
When you say "selftest" should be an independent test group, do you mean
we should keep the "selftest" and "dangerous_selftest" groups but just
remove the "selftest" folder from SRC_GROUPS? I think Amir was suggesting
to completely remove the test groups "selftest" and
"dangerous_selftest" as well as removing the folder from SRC_GROUPS.
- Leah
>
> Thanks,
> Zorro
>
> >
> > >
> > > It would also be nice to drop a line in README about running selftests.
> >
> > Sure, I can do this.
> >
> > >
> > > Thanks,
> > > Amir.
> >
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: [PATCH v2] selftest: add tests for debugging testing setup
2023-02-17 18:41 ` Leah Rumancik
@ 2023-02-18 5:47 ` Zorro Lang
0 siblings, 0 replies; 8+ messages in thread
From: Zorro Lang @ 2023-02-18 5:47 UTC (permalink / raw)
To: Leah Rumancik; +Cc: Zorro Lang, Amir Goldstein, fstests
On Fri, Feb 17, 2023 at 10:41:23AM -0800, Leah Rumancik wrote:
> On Thu, Feb 16, 2023 at 01:41:07PM +0800, Zorro Lang wrote:
> > On Wed, Feb 15, 2023 at 04:59:04PM -0800, Leah Rumancik wrote:
> > > On Wed, Feb 15, 2023 at 07:57:55AM +0200, Amir Goldstein wrote:
> > > > On Wed, Feb 15, 2023 at 6:57 AM Zorro Lang <zlang@redhat.com> wrote:
> > > > >
> > > > > On Tue, Feb 14, 2023 at 01:54:31PM -0800, Leah Rumancik wrote:
> > > > > > Many people have developed infrastructure around xfstests. In order to
> > > > > > test a setup, it would be helpful to have dummy tests that have
> > > > > > consistent test outcomes. Add a new test folder with the following
> > > > > > tests:
> > > > > >
> > > > > > selftest/001 pass
> > > > > > selftest/002 fail from output mismatch
> > > > > > selftest/003 fail via _fail
> > > > > > selftest/004 skip
> > > > > > selftest/005 crash
> > > > > > selftest/006 hang
> > > > > >
> > > > > > Also, create two new groups: 'selftest' which includes tests 001-004 and
> > > > > > 'dangerous_selftest' which includes tests 005-006. The selftests will
> > > > > > run when running all tests but they are not part of the auto group.
> > > > > >
> > > > > > Signed-off-by: Leah Rumancik <leah.rumancik@gmail.com>
> > > > > > ---
> > > > > > v1 -> v2:
> > > > > >
> > > > > > - change name of folder from 'debug' to 'selftest'
> > > > > > - update tags to 'selftest' and 'dangerous_selftest'
> > > > > > - add selftest folder to SRC_GROUPS
> > > > > > - update copyright tag
> > > > > > - add test which _fails
> > > > >
> > > > > Thanks, this version looks good to me, just each case has a blank line at EOF.
> > > > > I'll help to remove those blank lines when I merge this patch.
> > > > >
> > > > > Reviewed-by: Zorro Lang <zlang@redhat.com>
> > > > >
> > > > > Thanks,
> > > > > Zorro
> > > > >
> > > > >
> > > > > >
> > > > > > v1: https://lore.kernel.org/fstests/20230209013143.2586104-1-leah.rumancik@gmail.com/
> > > > > >
> > > > > > check | 2 +-
> > > > > > doc/group-names.txt | 2 ++
> > > > > > tests/selftest/001 | 14 ++++++++++++++
> > > > > > tests/selftest/001.out | 2 ++
> > > > > > tests/selftest/002 | 14 ++++++++++++++
> > > > > > tests/selftest/002.out | 2 ++
> > > > > > tests/selftest/003 | 16 ++++++++++++++++
> > > > > > tests/selftest/003.out | 2 ++
> > > > > > tests/selftest/004 | 16 ++++++++++++++++
> > > > > > tests/selftest/004.out | 2 ++
> > > > > > tests/selftest/005 | 17 +++++++++++++++++
> > > > > > tests/selftest/005.out | 2 ++
> > > > > > tests/selftest/006 | 20 ++++++++++++++++++++
> > > > > > tests/selftest/006.out | 2 ++
> > > > > > tests/selftest/Makefile | 24 ++++++++++++++++++++++++
> > > > > > 15 files changed, 136 insertions(+), 1 deletion(-)
> > > > > > create mode 100755 tests/selftest/001
> > > > > > create mode 100644 tests/selftest/001.out
> > > > > > create mode 100755 tests/selftest/002
> > > > > > create mode 100644 tests/selftest/002.out
> > > > > > create mode 100755 tests/selftest/003
> > > > > > create mode 100644 tests/selftest/003.out
> > > > > > create mode 100755 tests/selftest/004
> > > > > > create mode 100644 tests/selftest/004.out
> > > > > > create mode 100755 tests/selftest/005
> > > > > > create mode 100644 tests/selftest/005.out
> > > > > > create mode 100755 tests/selftest/006
> > > > > > create mode 100644 tests/selftest/006.out
> > > > > > create mode 100644 tests/selftest/Makefile
> > > > > >
> > > > > > diff --git a/check b/check
> > > > > > index 4827532e..c2065cdc 100755
> > > > > > --- a/check
> > > > > > +++ b/check
> > > > > > @@ -48,7 +48,7 @@ timestamp=${TIMESTAMP:=false}
> > > > > >
> > > > > > rm -f $tmp.list $tmp.tmp $tmp.grep $here/$iam.out $tmp.xlist $tmp.report.* $tmp.arglist
> > > > > >
> > > > > > -SRC_GROUPS="generic shared"
> > > > > > +SRC_GROUPS="generic shared selftest"
> > > >
> > > > I know this is a bit of bikeshedding,
> > > > but is it really needed to add selftest/* to SRC_GROUPS and then filter it out
> > > > by group?
> > > >
> > > > Seems to me that it would be simpler to not add any new groups and use
> > > > ./check -g selftest/quick -g selftest/dangerous
> > > > to run the tests in the selftests subdir.
> > > >
> > > > Just to be clear, the syntax above already works, nothing new to implement.
> > > >
> > > > I do not object to adding the selftest groups if others like it better this way.
> > >
> > > I think it behaves more as expected when it is a part of SRC_GROUPS, but
> > > I will probably run these by specifying specific tests - so I don't
> > > care much about which groups are used / if it is in SRC_GROUPS or not,
> > > and I am happy to change it if others agree.
> >
> > I agree with the suggestion from Amir. The "selftest" should be an independent
> > test group only be run when a tester intend to run it.
>
>
> When you say "selftest" should be an independent test group, do you mean
> we should keep the "selftest" and "dangerous_selftest" groups but just
> remove the "selftest" folder from SRC_GROUPS? I think Amir was suggesting
> to completely remove the test groups "selftest" and
> "dangerous_selftest" as well as removing the folder from SRC_GROUPS.
Oh, sorry for the confusion. I think generally no one run selftest cases with
other cases together (correct me if someone has this requirement ;), so we can
remove the selftest from SRC_GROUPS.
About the new groups (selftest and dangerous_selftest), I'm both OK with or
without them. (So let's keep them if no more objection:).
Besides that, we'd better to update the documentation, to shows we have
selftest in tests/selftest directory, and how to add and run them. (in
README or add a README.selftest).
Thanks,
Zorro
>
> - Leah
>
> >
> > Thanks,
> > Zorro
> >
> > >
> > > >
> > > > It would also be nice to drop a line in README about running selftests.
> > >
> > > Sure, I can do this.
> > >
> > > >
> > > > Thanks,
> > > > Amir.
> > >
> >
>
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2023-02-18 5:47 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-02-14 21:54 [PATCH v2] selftest: add tests for debugging testing setup Leah Rumancik
2023-02-14 22:02 ` Darrick J. Wong
2023-02-15 4:49 ` Zorro Lang
2023-02-15 5:57 ` Amir Goldstein
2023-02-16 0:59 ` Leah Rumancik
2023-02-16 5:41 ` Zorro Lang
2023-02-17 18:41 ` Leah Rumancik
2023-02-18 5:47 ` Zorro Lang
This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.