[PATCHSET v4 0/8] fstests: document all test groups

[PATCHSET v4 0/8] fstests: document all test groups

[Darrick J. Wong]

Hi all,

I've noticed that fstests doesn't document what the test groups mean.  I
attempted to document what each group seems to symbolize, to reduce the
amount of confusion for new contributors.  While doing so, I noticed
that there were a handful of tests that aren't in the right group.  The
first three patches dix that problem.  The next test adds a build-time
check to ensure that all groups are listed in the documentation file so
that we don't lose track of things.  The last patch updates ./new to use
the documentation file instead of having to build group.list files.

v2: fix the 'subvolume' group tests, tweak some of the wordings of the
    group description file, add missing license declarations, remove the
    one-off overlay group
v2.1: move the group name documentation checks into the preamble code so
      we don't have to have a second script
v3: cleanup cleanups to mkgroupfile and don't allow new 'other' group
    tests
v4: split the ./new script cleanups into a separate series; add review
    tags; no other changes

If you're going to start using this mess, you probably ought to just
pull from my git trees, which are linked below.

This is an extraordinary way to destroy everything.  Enjoy!
Comments and questions are, as always, welcome.

--D

fstests git tree:
https://git.kernel.org/cgit/linux/kernel/git/djwong/xfstests-dev.git/log/?h=document-test-groups
---
 common/preamble     |   21 ++++++++
 doc/group-names.txt |  135 +++++++++++++++++++++++++++++++++++++++++++++++++++
 new                 |   24 ++++-----
 tests/btrfs/233     |    2 -
 tests/btrfs/245     |    2 -
 tests/ceph/001      |    2 -
 tests/ceph/002      |    2 -
 tests/ceph/003      |    2 -
 tests/generic/631   |    2 -
 tests/xfs/491       |    2 -
 tests/xfs/492       |    2 -
 tests/xfs/493       |    2 -
 tests/xfs/519       |    2 -
 tests/xfs/520       |    2 -
 tests/xfs/535       |    2 -
 tests/xfs/536       |    2 -
 tools/mkgroupfile   |   37 +++++++++++---
 tools/mvtest        |    5 ++
 tools/nextid        |    4 +-
 19 files changed, 215 insertions(+), 37 deletions(-)
 create mode 100644 doc/group-names.txt

[PATCH 1/8] ceph: re-tag copy_file_range as being in the copy_range group

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

copy_range is the group name for copy_file_range tests, so reclassify
these tests.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/ceph/001 |    2 +-
 tests/ceph/002 |    2 +-
 tests/ceph/003 |    2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/tests/ceph/001 b/tests/ceph/001
index aca77168..c00de308 100755
--- a/tests/ceph/001
+++ b/tests/ceph/001
@@ -11,7 +11,7 @@
 # file and 3) the middle of the dst file.
 #
 . ./common/preamble
-_begin_fstest auto quick copy
+_begin_fstest auto quick copy_range
 
 # get standard environment
 . common/filter
diff --git a/tests/ceph/002 b/tests/ceph/002
index 428f23a9..9bc728fd 100755
--- a/tests/ceph/002
+++ b/tests/ceph/002
@@ -20,7 +20,7 @@
 #   linux kernel: 78beb0ff2fec ("ceph: use copy-from2 op in copy_file_range")
 #
 . ./common/preamble
-_begin_fstest auto quick copy
+_begin_fstest auto quick copy_range
 
 # get standard environment
 . common/filter
diff --git a/tests/ceph/003 b/tests/ceph/003
index 9f8c6068..faedb48c 100755
--- a/tests/ceph/003
+++ b/tests/ceph/003
@@ -7,7 +7,7 @@
 # Test copy_file_range with infile = outfile
 #
 . ./common/preamble
-_begin_fstest auto quick copy
+_begin_fstest auto quick copy_range
 
 # get standard environment
 . common/filter

[PATCH 2/8] xfs: move reflink tests into the clone group

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

"clone" is the group for tests that exercise FICLONERANGE, so move these
tests.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/xfs/519 |    2 +-
 tests/xfs/520 |    2 +-
 tests/xfs/535 |    2 +-
 tests/xfs/536 |    2 +-
 4 files changed, 4 insertions(+), 4 deletions(-)


diff --git a/tests/xfs/519 b/tests/xfs/519
index 675ec07e..49c62b56 100755
--- a/tests/xfs/519
+++ b/tests/xfs/519
@@ -9,7 +9,7 @@
 # flushing the log and then remounting to check file contents.
 
 . ./common/preamble
-_begin_fstest auto quick reflink
+_begin_fstest auto quick clone
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/xfs/520 b/tests/xfs/520
index 8410f2ba..2fceb07c 100755
--- a/tests/xfs/520
+++ b/tests/xfs/520
@@ -12,7 +12,7 @@
 # is included in the current kernel.
 #
 . ./common/preamble
-_begin_fstest auto quick reflink
+_begin_fstest auto quick clone
 
 # Override the default cleanup function.
 _cleanup()
diff --git a/tests/xfs/535 b/tests/xfs/535
index 4c883675..1a5da61b 100755
--- a/tests/xfs/535
+++ b/tests/xfs/535
@@ -7,7 +7,7 @@
 # Verify that XFS does not cause inode fork's extent count to overflow when
 # writing to a shared extent.
 . ./common/preamble
-_begin_fstest auto quick reflink
+_begin_fstest auto quick clone
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/xfs/536 b/tests/xfs/536
index e5f904f5..64fa4fbf 100755
--- a/tests/xfs/536
+++ b/tests/xfs/536
@@ -7,7 +7,7 @@
 # Verify that XFS does not cause inode fork's extent count to overflow when
 # remapping extents from one file's inode fork to another.
 . ./common/preamble
-_begin_fstest auto quick reflink
+_begin_fstest auto quick clone
 
 # Import common functions.
 . ./common/filter

[PATCH 3/8] xfs: fix incorrect fuzz test group name

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

The group name for fuzz tests is 'fuzzers'.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/xfs/491 |    2 +-
 tests/xfs/492 |    2 +-
 tests/xfs/493 |    2 +-
 3 files changed, 3 insertions(+), 3 deletions(-)


diff --git a/tests/xfs/491 b/tests/xfs/491
index 5c7c5d1f..7402b09a 100755
--- a/tests/xfs/491
+++ b/tests/xfs/491
@@ -7,7 +7,7 @@
 # Test detection & fixing of bad summary block counts at mount time.
 #
 . ./common/preamble
-_begin_fstest auto quick fuzz
+_begin_fstest auto quick fuzzers
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/xfs/492 b/tests/xfs/492
index 8258e5d8..514ac1e4 100755
--- a/tests/xfs/492
+++ b/tests/xfs/492
@@ -7,7 +7,7 @@
 # Test detection & fixing of bad summary inode counts at mount time.
 #
 . ./common/preamble
-_begin_fstest auto quick fuzz
+_begin_fstest auto quick fuzzers
 
 # Import common functions.
 . ./common/filter
diff --git a/tests/xfs/493 b/tests/xfs/493
index 58fd9c99..58091ad7 100755
--- a/tests/xfs/493
+++ b/tests/xfs/493
@@ -8,7 +8,7 @@
 # Corrupt the AGFs to test mount failure when mount-fixing fails.
 #
 . ./common/preamble
-_begin_fstest auto quick fuzz
+_begin_fstest auto quick fuzzers
 
 # Import common functions.
 . ./common/filter

[PATCH 4/8] btrfs: fix incorrect subvolume test group name

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

The group for testing subvolume functionality is 'subvol', not
'subvolume'.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/btrfs/233 |    2 +-
 tests/btrfs/245 |    2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)


diff --git a/tests/btrfs/233 b/tests/btrfs/233
index f3e3762c..6a414443 100755
--- a/tests/btrfs/233
+++ b/tests/btrfs/233
@@ -9,7 +9,7 @@
 # performed.
 #
 . ./common/preamble
-_begin_fstest auto quick subvolume
+_begin_fstest auto quick subvol
 
 # Override the default cleanup function.
 _cleanup()
diff --git a/tests/btrfs/245 b/tests/btrfs/245
index 2b9c63c7..f3380ac2 100755
--- a/tests/btrfs/245
+++ b/tests/btrfs/245
@@ -8,7 +8,7 @@
 # as subvolume and snapshot creation and deletion.
 #
 . ./common/preamble
-_begin_fstest auto quick idmapped subvolume
+_begin_fstest auto quick idmapped subvol
 
 # get standard environment, filters and checks
 . ./common/rc

[PATCH 5/8] generic/631: change this test to use the 'whiteout' group

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

This test isn't really an overlay test; it's a regression test for a bug
that someone found in xfs handling of whiteout files.  Since the
'overlay' group has one member, let's move it to 'whiteout'.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tests/generic/631 |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)


diff --git a/tests/generic/631 b/tests/generic/631
index a1acdedb..aae181dd 100755
--- a/tests/generic/631
+++ b/tests/generic/631
@@ -22,7 +22,7 @@
 # in xfs_rename").
 
 . ./common/preamble
-_begin_fstest auto rw overlay rename
+_begin_fstest auto rw whiteout rename
 
 # Override the default cleanup function.
 _cleanup()

[PATCH 6/8] tools: make sure that test groups are described in the documentation

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Create a file to document the purpose of each test group that is
currently defined in fstests, and change mkgroupfile to check that every
group mentioned in the tests is also mentioned in the documentation.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 common/preamble     |   21 ++++++++
 doc/group-names.txt |  135 +++++++++++++++++++++++++++++++++++++++++++++++++++
 tools/mkgroupfile   |   33 +++++++++---
 3 files changed, 181 insertions(+), 8 deletions(-)
 create mode 100644 doc/group-names.txt


diff --git a/common/preamble b/common/preamble
index 66b0ed05..64d79385 100644
--- a/common/preamble
+++ b/common/preamble
@@ -23,6 +23,26 @@ _register_cleanup()
 	trap "${cleanup}exit \$status" EXIT HUP INT QUIT TERM $*
 }
 
+# Make sure each group is in the documentation file.
+_check_groups() {
+	test -n "$GROUPNAME_DOC_FILE" || return 0
+
+	local testname="$(echo "$0" | sed -e 's/^.*tests\///g')"
+	declare -a missing=()
+
+	for group in "$@"; do
+		if ! grep -q "^${group}[[:space:]]" "$GROUPNAME_DOC_FILE"; then
+			missing+=("\"${group}\"")
+		fi
+	done
+	test "${#missing}" -eq 0 && return 0
+
+	local suffix=
+	test "${#missing}" -gt 1 && suffix="s"
+	echo "$testname: group$suffix ${missing[@]} not mentioned in documentation." 1>&2
+	return 1
+}
+
 # Prepare to run a fstest by initializing the required global variables to
 # their defaults, sourcing common functions, registering a cleanup function,
 # and removing the $seqres.full file.
@@ -42,6 +62,7 @@ _begin_fstest()
 	# If we're only running the test to generate a group.list file,
 	# spit out the group data and exit.
 	if [ -n "$GENERATE_GROUPS" ]; then
+		_check_groups "$@" || exit 1
 		echo "$seq $@"
 		exit 0
 	fi
diff --git a/doc/group-names.txt b/doc/group-names.txt
new file mode 100644
index 00000000..e8e3477e
--- /dev/null
+++ b/doc/group-names.txt
@@ -0,0 +1,135 @@
+======================= =======================================================
+Group Name:		Description:
+======================= =======================================================
+all			All known tests, automatically generated by ./check at
+			runtime
+auto			Tests that should be run automatically.  These should
+			not require more than ~5 minutes to run.
+quick			Tests that should run in under 30 seconds.
+deprecated		Old tests that should not be run.
+
+acl			Access Control Lists
+admin			xfs_admin functionality
+aio			general libaio async io tests
+atime			file access time
+attr			extended attributes
+attr2			xfs v2 extended aributes
+balance			btrfs tree rebalance
+bigtime			timestamps beyond the year 2038
+blockdev		block device functionality
+broken			broken tests
+cap			Linux capabilities
+casefold		directory name casefolding
+ci			ASCII case-insensitive directory name lookups
+clone			FICLONE/FICLONERANGE ioctls
+clone_stress		stress testing FICLONE/FICLONERANGE
+collapse		fallocate FALLOC_FL_COLLAPSE_RANGE
+compress		file compression
+convert			btrfs ext[34] conversion tool
+copy			xfs_copy functionality
+copy_range		copy_file_range syscall
+copyup			overlayfs copyup
+dangerous		dangerous test that can crash the system
+dangerous_bothrepair	fuzzers to evaluate xfs_scrub + xfs_repair repair
+dangerous_fuzzers	fuzzers that can crash your computer
+dangerous_norepair	fuzzers to evaluate kernel metadata verifiers
+dangerous_online_repair	fuzzers to evaluate xfs_scrub online repair
+dangerous_repair	fuzzers to evaluate xfs_repair offline repair
+dangerous_scrub		fuzzers to evaluate xfs_scrub checking
+data			data loss checkers
+dax			direct access mode for persistent memory files
+db			xfs_db functional tests
+dedupe			FIEDEDUPERANGE ioctl
+defrag			filesystem defragmenters
+dir			directory test functions
+dump			dump and restore utilities
+eio			IO error reporting
+encrypt			encrypted file contents
+enospc			ENOSPC error reporting
+exportfs		file handles
+filestreams		XFS filestreams allocator
+freeze			filesystem freeze tests
+fsck			general fsck tests
+fsmap			FS_IOC_GETFSMAP ioctl
+fsr			XFS free space reorganizer
+fuzzers			filesystem fuzz tests
+growfs			increasing the size of a filesystem
+hardlink		hardlinks
+health			XFS health reporting
+idmapped		idmapped mount functionality
+inobtcount		XFS inode btree count tests
+insert			fallocate FALLOC_FL_INSERT_RANGE
+ioctl			general ioctl tests
+io_uring		general io_uring async io tests
+label			filesystem labelling
+limit			resource limits
+locks			file locking
+log			metadata logging
+logprint		xfs_logprint functional tests
+long_rw			long-soak read write IO path exercisers
+metacopy		overlayfs metadata-only copy-up
+metadata		filesystem metadata update exercisers
+metadump		xfs_metadump/xfs_mdrestore functionality
+mkfs			filesystem formatting tools
+mount			mount option and functionality checks
+nested			nested overlayfs instances
+nfs4_acl		NFSv4 access control lists
+nonsamefs		overlayfs layers on different filesystems
+online_repair		online repair functionality tests
+other			dumping ground, do not add more tests to this group
+pattern			specific IO pattern tests
+perms			access control and permission checking
+pipe			pipe functionality
+pnfs			PNFS
+posix			POSIX behavior conformance
+prealloc		fallocate for preallocating unwritten space
+preallocrw		fallocate, then read and write
+punch			fallocate FALLOC_FL_PUNCH_HOLE
+qgroup			btrfs qgroup feature
+quota			filesystem usage quotas
+raid			btrfs RAID
+realtime		XFS realtime volumes
+recoveryloop		crash recovery loops
+redirect		overlayfs redirect_dir feature
+remote			dump and restore with a remote tape
+remount			remounting filesystems
+rename			rename system call
+repair			xfs_repair functional tests
+replace			btrfs device replace
+replay			dm-logwrites replays
+resize			resize2fs functionality tests
+richacl			rich ACL feature
+rmap			XFS reverse mapping exercisers
+rotate			overlayfs upper layer rotate tests from the unionmount
+			test suite
+rw			read/write IO tests
+samefs			overlayfs when all layers are on the same fs
+scrub			filesystem metadata scrubbers
+seed			btrfs seeded filesystems
+seek			llseek functionality
+send			btrfs send/receive
+shrinkfs		decreasing the size of a filesystem
+shutdown		FS_IOC_SHUTDOWN ioctl
+snapshot		btrfs snapshots
+soak			long running soak tests of any kind
+spaceman		xfs_spaceman functional tests
+splice			splice system call
+stress			fsstress filesystem exerciser
+subvol			btrfs subvolumes
+swap			swap files
+symlink			symbolic links
+tape			dump and restore with a tape
+thin			thin provisioning
+trim			FITRIM ioctl
+udf			UDF functionality tests
+union			tests from the unionmount test suite
+unlink			O_TMPFILE unlinked files
+unshare			fallocate FALLOC_FL_UNSHARE_RANGE
+v2log			XFS v2 log format tests
+verity			fsverity
+volume			btrfs volume management
+whiteout		overlayfs whiteout functionality
+xino			overlayfs xino feature
+zero			fallocate FALLOC_FL_ZERO_RANGE
+zone			zoned (SMR) device support
+======================= =======================================================
diff --git a/tools/mkgroupfile b/tools/mkgroupfile
index 0681e5d2..e4244507 100755
--- a/tools/mkgroupfile
+++ b/tools/mkgroupfile
@@ -9,12 +9,28 @@ fi
 
 test_dir="$PWD"
 groupfile="$1"
+GROUPNAME_DOC_FILE="$(readlink -m ../../doc/group-names.txt)"
+export GROUPNAME_DOC_FILE
 
 if [ ! -x ../../check ]; then
 	echo "$0: Run this from tests/XXX/."
 	exit 1
 fi
 
+cleanup() {
+	test -z "$groupfile" && return
+	test -z "$ngroupfile" && return
+
+	if [ $ret -eq 0 ]; then
+		mv "$ngroupfile" "$groupfile"
+	else
+		rm -f "$ngroupfile"
+	fi
+}
+
+ret=1	# trigger cleanup of temporary files unless we succeed
+trap 'cleanup; exit $ret' EXIT INT TERM QUIT
+
 generate_groupfile() {
 	cat << ENDL
 # QA groups control file, automatically generated.
@@ -24,19 +40,20 @@ ENDL
 	cd ../../
 	export GENERATE_GROUPS=yes
 	grep -R -l "^_begin_fstest" "$test_dir/" 2>/dev/null | while read testfile; do
-		test -x "$testfile" && "$testfile"
+		test -x "$testfile" && "$testfile" || return 1
 	done | sort -g
+	ret="${PIPESTATUS[1]}"
 	cd "$test_dir"
 }
 
 if [ -z "$groupfile" ] || [ "$groupfile" = "-" ]; then
 	# Dump the group file to stdout and exit
+	unset groupfile
 	generate_groupfile
-	exit 0
+else
+	# Otherwise, write the group file to disk somewhere.
+	ngroupfile="${groupfile}.new"
+	rm -f "$ngroupfile"
+	generate_groupfile >> "$ngroupfile"
+	# let cleanup rename or delete ngroupfile
 fi
-
-# Otherwise, write the group file to disk somewhere.
-ngroupfile="${groupfile}.new"
-rm -f "$ngroupfile"
-generate_groupfile >> "$ngroupfile"
-mv "$ngroupfile" "$groupfile"

[PATCH 7/8] tools: add missing license tags to my scripts

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

I forgot to add spdx license tags and copyright statements to some of
the tools that I've contributed to fstests.  Fix this to be explicit.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 tools/mkgroupfile |    4 +++-
 tools/mvtest      |    5 ++++-
 tools/nextid      |    4 +++-
 3 files changed, 10 insertions(+), 3 deletions(-)


diff --git a/tools/mkgroupfile b/tools/mkgroupfile
index e4244507..634ec92c 100755
--- a/tools/mkgroupfile
+++ b/tools/mkgroupfile
@@ -1,5 +1,7 @@
 #!/bin/bash
-
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Oracle.  All Rights Reserved.
+#
 # Generate a group file from the _begin_fstest call in each test.
 
 if [ "$1" = "--help" ]; then
diff --git a/tools/mvtest b/tools/mvtest
index 5088b45f..99b15414 100755
--- a/tools/mvtest
+++ b/tools/mvtest
@@ -1,6 +1,9 @@
 #!/bin/sh
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2015 Oracle.  All Rights Reserved.
+#
+# Move a test and update the golden output file.
 
-# Renumber a test
 dir="$(dirname "$0")"
 
 if [ -z "$1" ] || [ "$1" = "--help" ]; then
diff --git a/tools/nextid b/tools/nextid
index 9507de29..9e31718c 100755
--- a/tools/nextid
+++ b/tools/nextid
@@ -1,5 +1,7 @@
 #!/bin/bash
-
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2015 Oracle.  All Rights Reserved.
+#
 # Compute the next available test id in a given test directory.
 
 if [ $# != 1 ] || [ "$1" = "--help" ] || [ ! -d "tests/$1/" ]; then

[PATCH 8/8] new: only allow documented test group names

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Now that we require all group names to be listed in doc/group-names.txt,
we can use that (instead of running mkgroupfile) to check if the group
name(s) supplied by the user actually exist.  This has the secondary
effect of being a second nudge towards keeping the description of groups
up to date.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Amir Goldstein <amir73il@gmail.com>
---
 new |   24 +++++++++++-------------
 1 file changed, 11 insertions(+), 13 deletions(-)


diff --git a/new b/new
index 2097a883..6b7dc5d4 100755
--- a/new
+++ b/new
@@ -83,6 +83,14 @@ then
     exit 1
 fi
 
+# Extract group names from the documentation.
+group_names() {
+	awk '/^[[:lower:][:digit:]_]/ {
+		if ($1 != "" && $1 != "Group" && $2 != "Name:" && $1 != "all")
+			printf("%s\n", $1);
+	}' doc/group-names.txt
+}
+
 if [ $# -eq 0 ]
 then
 
@@ -93,16 +101,7 @@ then
 	[ -z "$ans" ] && ans=other
 	if [ "X$ans" = "X?" ]
 	then
-	    for d in $SRC_GROUPS; do
-		(cd "tests/$d/" ; ../../tools/mkgroupfile "$tmpfile")
-		l=$(sed -n < "$tmpfile" \
-		    -e 's/#.*//' \
-		    -e 's/$/ /' \
-		    -e 's;\(^[0-9][0-9][0-9]\)\(.*$\);\2;p')
-		grpl="$grpl $l"
-	    done
-	    lst=`for word in $grpl; do echo $word; done | sort| uniq `
-	    echo $lst
+	    echo $(group_names)
 	else
 	    # only allow lower cases, spaces, digits and underscore in group
 	    inval=`echo $ans | tr -d '[:lower:][:space:][:digit:]_'`
@@ -120,11 +119,10 @@ then
 else
     # expert mode, groups are on the command line
     #
-    (cd "$tdir" ; ../../tools/mkgroupfile "$tmpfile")
     for g in $*
     do
-	if ! grep -q "[[:space:]]$g" "$tmpfile"; then
-	    echo "Warning: group \"$g\" not defined in $tdir tests"
+	if ! grep -q "^$g" doc/group-names.txt; then
+	    echo "Warning: group \"$g\" not defined in documentation"
 	fi
     done
     ans="$*"

Re: [PATCH 6/8] tools: make sure that test groups are described in the documentation

[Darrick J. Wong]

On Fri, Sep 03, 2021 at 06:38:38AM +0300, Amir Goldstein wrote:
> > diff --git a/include/buildgrouplist b/include/buildgrouplist
> > index d898efa3..489de965 100644
> > --- a/include/buildgrouplist
> > +++ b/include/buildgrouplist
> > @@ -6,3 +6,4 @@
> >  group.list:
> >         @echo " [GROUP] $$PWD/$@"
> >         $(Q)$(TOPDIR)/tools/mkgroupfile $@
> > +       $(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@
> 
> I would like to argue against checking groups post mkgroupfile
> and for checking groups during mkgroupfile

Done.

> > diff --git a/tools/check-groups b/tools/check-groups
> > new file mode 100755
> > index 00000000..0d193615
> > --- /dev/null
> > +++ b/tools/check-groups
> > @@ -0,0 +1,35 @@
> > +#!/bin/bash
> > +# SPDX-License-Identifier: GPL-2.0
> > +# Copyright (c) 2021 Oracle.  All Rights Reserved.
> > +#
> > +# Make sure that all groups listed in a group.list file are mentioned in the
> > +# group description file.
> > +
> > +if [ -z "$1" ] || [ "$1" = "--help" ]; then
> > +       echo "Usage: $0 path_to_group_names [group.list files...]"
> > +       exit 1
> > +fi
> > +
> > +groups_doc_file="$1"
> > +shift
> > +
> > +get_group_list() {
> > +       for file in "$@"; do
> > +               while read testname groups; do
> > +                       test -z "${testname}" && continue
> > +                       test "${testname:0:1}" = "#" && continue
> > +
> > +                       echo "${groups}" | tr ' ' '\n'
> > +               done < "${file}"
> > +       done | sort | uniq
> > +}
> > +
> > +ret=0
> > +while read group; do
> > +       if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
> > +               echo "${group}: group not mentioned in documentation." 1>&2
> 
> This message would have been more informative with the offending
> test file.

Hm.  This becomes much easier if I make the _begin_fstest helper do the
checking of the group names.

> Now after you crunched all the test files into group.list files and
> all the group.list files into a unique group set, this is too late.
> But this same check during generate_groupfile() would have
> been trivial and would allow reporting the offending test.
> 
> While we are on the subject of generate_groupfile(), can you please
> explain the rationale behind the method of extracting the test file
> groups by executing the test with GENERATE_GROUPS=yes?
> As opposed to just getting the list of groups on the stop from the file
> using grep?

Well... now that you point that out, it's so that we can put in custom
logic like checking group names. ;)

--D

> 
> Thanks,
> Amir.

Re: [PATCH 6/8] tools: make sure that test groups are described in the documentation

[Amir Goldstein]

> diff --git a/include/buildgrouplist b/include/buildgrouplist
> index d898efa3..489de965 100644
> --- a/include/buildgrouplist
> +++ b/include/buildgrouplist
> @@ -6,3 +6,4 @@
>  group.list:
>         @echo " [GROUP] $$PWD/$@"
>         $(Q)$(TOPDIR)/tools/mkgroupfile $@
> +       $(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@

I would like to argue against checking groups post mkgroupfile
and for checking groups during mkgroupfile

> diff --git a/tools/check-groups b/tools/check-groups
> new file mode 100755
> index 00000000..0d193615
> --- /dev/null
> +++ b/tools/check-groups
> @@ -0,0 +1,35 @@
> +#!/bin/bash
> +# SPDX-License-Identifier: GPL-2.0
> +# Copyright (c) 2021 Oracle.  All Rights Reserved.
> +#
> +# Make sure that all groups listed in a group.list file are mentioned in the
> +# group description file.
> +
> +if [ -z "$1" ] || [ "$1" = "--help" ]; then
> +       echo "Usage: $0 path_to_group_names [group.list files...]"
> +       exit 1
> +fi
> +
> +groups_doc_file="$1"
> +shift
> +
> +get_group_list() {
> +       for file in "$@"; do
> +               while read testname groups; do
> +                       test -z "${testname}" && continue
> +                       test "${testname:0:1}" = "#" && continue
> +
> +                       echo "${groups}" | tr ' ' '\n'
> +               done < "${file}"
> +       done | sort | uniq
> +}
> +
> +ret=0
> +while read group; do
> +       if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
> +               echo "${group}: group not mentioned in documentation." 1>&2

This message would have been more informative with the offending
test file.

Now after you crunched all the test files into group.list files and
all the group.list files into a unique group set, this is too late.
But this same check during generate_groupfile() would have
been trivial and would allow reporting the offending test.

While we are on the subject of generate_groupfile(), can you please
explain the rationale behind the method of extracting the test file
groups by executing the test with GENERATE_GROUPS=yes?
As opposed to just getting the list of groups on the stop from the file
using grep?

Thanks,
Amir.

[PATCH 6/8] tools: make sure that test groups are described in the documentation

[Darrick J. Wong]

From: Darrick J. Wong <djwong@kernel.org>

Create a file to document the purpose of each test group that is
currently defined in fstests, and add a build script to check that every
group mentioned in the tests is also mentioned in the documentation.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>
---
 doc/group-names.txt    |  135 ++++++++++++++++++++++++++++++++++++++++++++++++
 include/buildgrouplist |    1 
 tools/check-groups     |   35 ++++++++++++
 3 files changed, 171 insertions(+)
 create mode 100644 doc/group-names.txt
 create mode 100755 tools/check-groups


diff --git a/doc/group-names.txt b/doc/group-names.txt
new file mode 100644
index 00000000..e8e3477e
--- /dev/null
+++ b/doc/group-names.txt
@@ -0,0 +1,135 @@
+======================= =======================================================
+Group Name:		Description:
+======================= =======================================================
+all			All known tests, automatically generated by ./check at
+			runtime
+auto			Tests that should be run automatically.  These should
+			not require more than ~5 minutes to run.
+quick			Tests that should run in under 30 seconds.
+deprecated		Old tests that should not be run.
+
+acl			Access Control Lists
+admin			xfs_admin functionality
+aio			general libaio async io tests
+atime			file access time
+attr			extended attributes
+attr2			xfs v2 extended aributes
+balance			btrfs tree rebalance
+bigtime			timestamps beyond the year 2038
+blockdev		block device functionality
+broken			broken tests
+cap			Linux capabilities
+casefold		directory name casefolding
+ci			ASCII case-insensitive directory name lookups
+clone			FICLONE/FICLONERANGE ioctls
+clone_stress		stress testing FICLONE/FICLONERANGE
+collapse		fallocate FALLOC_FL_COLLAPSE_RANGE
+compress		file compression
+convert			btrfs ext[34] conversion tool
+copy			xfs_copy functionality
+copy_range		copy_file_range syscall
+copyup			overlayfs copyup
+dangerous		dangerous test that can crash the system
+dangerous_bothrepair	fuzzers to evaluate xfs_scrub + xfs_repair repair
+dangerous_fuzzers	fuzzers that can crash your computer
+dangerous_norepair	fuzzers to evaluate kernel metadata verifiers
+dangerous_online_repair	fuzzers to evaluate xfs_scrub online repair
+dangerous_repair	fuzzers to evaluate xfs_repair offline repair
+dangerous_scrub		fuzzers to evaluate xfs_scrub checking
+data			data loss checkers
+dax			direct access mode for persistent memory files
+db			xfs_db functional tests
+dedupe			FIEDEDUPERANGE ioctl
+defrag			filesystem defragmenters
+dir			directory test functions
+dump			dump and restore utilities
+eio			IO error reporting
+encrypt			encrypted file contents
+enospc			ENOSPC error reporting
+exportfs		file handles
+filestreams		XFS filestreams allocator
+freeze			filesystem freeze tests
+fsck			general fsck tests
+fsmap			FS_IOC_GETFSMAP ioctl
+fsr			XFS free space reorganizer
+fuzzers			filesystem fuzz tests
+growfs			increasing the size of a filesystem
+hardlink		hardlinks
+health			XFS health reporting
+idmapped		idmapped mount functionality
+inobtcount		XFS inode btree count tests
+insert			fallocate FALLOC_FL_INSERT_RANGE
+ioctl			general ioctl tests
+io_uring		general io_uring async io tests
+label			filesystem labelling
+limit			resource limits
+locks			file locking
+log			metadata logging
+logprint		xfs_logprint functional tests
+long_rw			long-soak read write IO path exercisers
+metacopy		overlayfs metadata-only copy-up
+metadata		filesystem metadata update exercisers
+metadump		xfs_metadump/xfs_mdrestore functionality
+mkfs			filesystem formatting tools
+mount			mount option and functionality checks
+nested			nested overlayfs instances
+nfs4_acl		NFSv4 access control lists
+nonsamefs		overlayfs layers on different filesystems
+online_repair		online repair functionality tests
+other			dumping ground, do not add more tests to this group
+pattern			specific IO pattern tests
+perms			access control and permission checking
+pipe			pipe functionality
+pnfs			PNFS
+posix			POSIX behavior conformance
+prealloc		fallocate for preallocating unwritten space
+preallocrw		fallocate, then read and write
+punch			fallocate FALLOC_FL_PUNCH_HOLE
+qgroup			btrfs qgroup feature
+quota			filesystem usage quotas
+raid			btrfs RAID
+realtime		XFS realtime volumes
+recoveryloop		crash recovery loops
+redirect		overlayfs redirect_dir feature
+remote			dump and restore with a remote tape
+remount			remounting filesystems
+rename			rename system call
+repair			xfs_repair functional tests
+replace			btrfs device replace
+replay			dm-logwrites replays
+resize			resize2fs functionality tests
+richacl			rich ACL feature
+rmap			XFS reverse mapping exercisers
+rotate			overlayfs upper layer rotate tests from the unionmount
+			test suite
+rw			read/write IO tests
+samefs			overlayfs when all layers are on the same fs
+scrub			filesystem metadata scrubbers
+seed			btrfs seeded filesystems
+seek			llseek functionality
+send			btrfs send/receive
+shrinkfs		decreasing the size of a filesystem
+shutdown		FS_IOC_SHUTDOWN ioctl
+snapshot		btrfs snapshots
+soak			long running soak tests of any kind
+spaceman		xfs_spaceman functional tests
+splice			splice system call
+stress			fsstress filesystem exerciser
+subvol			btrfs subvolumes
+swap			swap files
+symlink			symbolic links
+tape			dump and restore with a tape
+thin			thin provisioning
+trim			FITRIM ioctl
+udf			UDF functionality tests
+union			tests from the unionmount test suite
+unlink			O_TMPFILE unlinked files
+unshare			fallocate FALLOC_FL_UNSHARE_RANGE
+v2log			XFS v2 log format tests
+verity			fsverity
+volume			btrfs volume management
+whiteout		overlayfs whiteout functionality
+xino			overlayfs xino feature
+zero			fallocate FALLOC_FL_ZERO_RANGE
+zone			zoned (SMR) device support
+======================= =======================================================
diff --git a/include/buildgrouplist b/include/buildgrouplist
index d898efa3..489de965 100644
--- a/include/buildgrouplist
+++ b/include/buildgrouplist
@@ -6,3 +6,4 @@
 group.list:
 	@echo " [GROUP] $$PWD/$@"
 	$(Q)$(TOPDIR)/tools/mkgroupfile $@
+	$(Q)$(TOPDIR)/tools/check-groups $(TOPDIR)/doc/group-names.txt $@
diff --git a/tools/check-groups b/tools/check-groups
new file mode 100755
index 00000000..0d193615
--- /dev/null
+++ b/tools/check-groups
@@ -0,0 +1,35 @@
+#!/bin/bash
+# SPDX-License-Identifier: GPL-2.0
+# Copyright (c) 2021 Oracle.  All Rights Reserved.
+#
+# Make sure that all groups listed in a group.list file are mentioned in the
+# group description file.
+
+if [ -z "$1" ] || [ "$1" = "--help" ]; then
+	echo "Usage: $0 path_to_group_names [group.list files...]"
+	exit 1
+fi
+
+groups_doc_file="$1"
+shift
+
+get_group_list() {
+	for file in "$@"; do
+		while read testname groups; do
+			test -z "${testname}" && continue
+			test "${testname:0:1}" = "#" && continue
+
+			echo "${groups}" | tr ' ' '\n'
+		done < "${file}"
+	done | sort | uniq
+}
+
+ret=0
+while read group; do
+	if ! grep -q "^${group}[[:space:]]" "${groups_doc_file}"; then
+		echo "${group}: group not mentioned in documentation." 1>&2
+		ret=1
+	fi
+done < <(get_group_list "$@")
+
+exit "${ret}"