Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
From: "Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Phillip Wood <phillip.wood123@gmail.com>,
	Taylor Blau <me@ttaylorr.com>, Elijah Newren <newren@gmail.com>,
	Jeff King <peff@peff.net>, Junio C Hamano <gitster@pobox.com>,
	Elijah Newren <newren@gmail.com>,
	Elijah Newren <newren@gmail.com>
Subject: [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run
Date: Wed, 14 Oct 2020 21:13:43 +0000
Message-ID: <9c8b6a1a943261635fa09bed22ae36e368686f15.1602710025.git.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com>

From: Elijah Newren <newren@gmail.com>

Many of our test scripts have several "setup" tests.  It's a lot easier
to say

   ./t0050-filesystem.sh --run=setup,9

in order to run all the setup tests as well as test #9, than it is to
track down what all the setup tests are and enter all their numbers in
the list.  Also, I often find myself wanting to run just one or a couple
tests from the test file, but I don't know the numbering of any of the
tests -- to get it I either have to first run the whole test file (or
start counting by hand or figure out some other clever but non-obvious
tricks).  It's really convenient to be able to just look at the test
description(s) and then run

   ./t6416-recursive-corner-cases.sh --run=symlink

or

   ./t6402-merge-rename.sh --run='setup,unnecessary update'

Add such an ability to test selection which relies on merely matching
against the test description.

Signed-off-by: Elijah Newren <newren@gmail.com>
---
 t/README         | 29 +++++++++++++++++++----------
 t/t0000-basic.sh | 41 +++++++++++++++++++++++++----------------
 t/test-lib.sh    | 16 ++++++++++------
 3 files changed, 54 insertions(+), 32 deletions(-)

diff --git a/t/README b/t/README
index 2adaf7c2d2..0a8b60b5c7 100644
--- a/t/README
+++ b/t/README
@@ -258,16 +258,13 @@ For an individual test suite --run could be used to specify that
 only some tests should be run or that some tests should be
 excluded from a run.
 
-The argument for --run is a list of individual test numbers or
-ranges with an optional negation prefix that define what tests in
-a test suite to include in the run.  A range is two numbers
-separated with a dash and matches a range of tests with both ends
-been included.  You may omit the first or the second number to
-mean "from the first test" or "up to the very last test"
-respectively.
-
-Optional prefix of '!' means that the test or a range of tests
-should be excluded from the run.
+The argument for --run, <test-selector>, is a list of description
+substrings or globs or individual test numbers or ranges with an
+optional negation prefix (of '!') that define what tests in a test
+suite to include (or exclude, if negated) in the run.  A range is two
+numbers separated with a dash and matches a range of tests with both
+ends been included.  You may omit the first or the second number to
+mean "from the first test" or "up to the very last test" respectively.
 
 If --run starts with an unprefixed number or range the initial
 set of tests to run is empty. If the first item starts with '!'
@@ -317,6 +314,18 @@ test in the test suite except from 7 up to 11:
 
     $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
 
+Sometimes there may be multiple tests with e.g. "setup" in their name
+that are needed and rather than figuring out the number for all of them
+we can just use "setup" as a substring/glob to match against the test
+description:
+
+    $ sh ./t0050-filesystem.sh --run=setup,9-11
+
+or one could select both the setup tests and the rename ones (assuming all
+relevant tests had those words in their descriptions):
+
+    $ sh ./t0050-filesystem.sh --run=setup,rename
+
 Some tests in a test suite rely on the previous tests performing
 certain actions, specifically some tests are designated as
 "setup" test, so you cannot _arbitrarily_ disable one test and
diff --git a/t/t0000-basic.sh b/t/t0000-basic.sh
index 923281af93..07105b2078 100755
--- a/t/t0000-basic.sh
+++ b/t/t0000-basic.sh
@@ -705,7 +705,31 @@ test_expect_success '--run empty selectors' "
 	EOF
 "
 
-test_expect_success '--run invalid range start' "
+test_expect_success '--run substring selector' "
+	run_sub_test_lib_test run-substring-selector \
+		'--run empty selectors' \
+		--run='relevant' <<-\\EOF &&
+	test_expect_success \"relevant test\" 'true'
+	for i in 1 2 3 4 5 6
+	do
+		test_expect_success \"other test #\$i\" 'true'
+	done
+	test_done
+	EOF
+	check_sub_test_lib_test run-substring-selector <<-\\EOF
+	> ok 1 - relevant test
+	> ok 2 # skip other test #1 (--run)
+	> ok 3 # skip other test #2 (--run)
+	> ok 4 # skip other test #3 (--run)
+	> ok 5 # skip other test #4 (--run)
+	> ok 6 # skip other test #5 (--run)
+	> ok 7 # skip other test #6 (--run)
+	> # passed all 7 test(s)
+	> 1..7
+	EOF
+"
+
+test_expect_success '--run keyword selection' "
 	run_sub_test_lib_test_err run-inv-range-start \
 		'--run invalid range start' \
 		--run='a-5' <<-\\EOF &&
@@ -735,21 +759,6 @@ test_expect_success '--run invalid range end' "
 	EOF_ERR
 "
 
-test_expect_success '--run invalid selector' "
-	run_sub_test_lib_test_err run-inv-selector \
-		'--run invalid selector' \
-		--run='1?' <<-\\EOF &&
-	test_expect_success \"passing test #1\" 'true'
-	test_done
-	EOF
-	check_sub_test_lib_test_err run-inv-selector \
-		<<-\\EOF_OUT 3<<-\\EOF_ERR
-	> FATAL: Unexpected exit with code 1
-	EOF_OUT
-	> error: --run: invalid non-numeric in test selector: '1?'
-	EOF_ERR
-"
-
 
 test_set_prereq HAVEIT
 haveit=no
diff --git a/t/test-lib.sh b/t/test-lib.sh
index ef31f40037..a040d54a76 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -769,6 +769,8 @@ match_pattern_list () {
 }
 
 match_test_selector_list () {
+	operation="$1"
+	shift
 	title="$1"
 	shift
 	arg="$1"
@@ -805,13 +807,13 @@ match_test_selector_list () {
 			*-*)
 				if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
 				then
-					echo "error: $title: invalid non-numeric in range" \
+					echo "error: $operation: invalid non-numeric in range" \
 						"start: '$orig_selector'" >&2
 					exit 1
 				fi
 				if expr "z${selector#*-}" : "z[0-9]*[^0-9]" >/dev/null
 				then
-					echo "error: $title: invalid non-numeric in range" \
+					echo "error: $operation: invalid non-numeric in range" \
 						"end: '$orig_selector'" >&2
 					exit 1
 				fi
@@ -819,9 +821,11 @@ match_test_selector_list () {
 			*)
 				if expr "z$selector" : "z[0-9]*[^0-9]" >/dev/null
 				then
-					echo "error: $title: invalid non-numeric in test" \
-						"selector: '$orig_selector'" >&2
-					exit 1
+					case "$title" in *${selector}*)
+						include=$positive
+						;;
+					esac
+					continue
 				fi
 		esac
 
@@ -1031,7 +1035,7 @@ test_skip () {
 		skipped_reason="GIT_SKIP_TESTS"
 	fi
 	if test -z "$to_skip" && test -n "$run_list" &&
-	   ! match_test_selector_list '--run' $test_count "$run_list"
+	   ! match_test_selector_list '--run' "$1" $test_count "$run_list"
 	then
 		to_skip=t
 		skipped_reason="--run"
-- 
gitgitgadget


  reply index

Thread overview: 71+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-12 23:26 [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-13 15:39   ` Taylor Blau
2020-10-13 17:23     ` Elijah Newren
2020-10-13 17:28       ` Taylor Blau
2020-10-13 18:21         ` Elijah Newren
2020-10-12 23:26 ` [PATCH 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-13 15:44   ` Taylor Blau
2020-10-13 17:56     ` Elijah Newren
2020-10-13 19:27       ` Junio C Hamano
2020-10-13 13:08 ` [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Phillip Wood
2020-10-15 11:48   ` Johannes Schindelin
2020-10-15 13:42     ` Phillip Wood
2020-10-13 19:19 ` [PATCH v2 " Elijah Newren via GitGitGadget
2020-10-13 19:19   ` [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-14 17:04     ` Jeff King
2020-10-14 17:46       ` Junio C Hamano
2020-10-14 18:07         ` Jeff King
2020-10-14 19:24           ` Junio C Hamano
2020-10-14 19:41             ` Elijah Newren
2020-10-14 19:49               ` Jeff King
2020-10-15 16:09                 ` Junio C Hamano
2020-10-15 15:59               ` Junio C Hamano
2020-10-14 17:50       ` Elijah Newren
2020-10-14 17:57         ` Junio C Hamano
2020-10-14 19:12           ` Elijah Newren
2020-10-15 16:10             ` Junio C Hamano
2020-10-14 18:09         ` Jeff King
2020-10-14 19:02           ` Elijah Newren
2020-10-15 16:20       ` Junio C Hamano
2020-10-15 19:46         ` Jeff King
2020-10-15 20:08           ` Junio C Hamano
2020-10-16  0:38             ` Jeff King
2020-10-16 16:16               ` Junio C Hamano
2020-10-16 16:30                 ` Junio C Hamano
2020-10-16 20:06                 ` Jeff King
2020-10-16 20:22                   ` Junio C Hamano
2020-10-13 19:19   ` [PATCH v2 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-13 19:19   ` [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-14 16:53     ` Jeff King
2020-10-14 17:39       ` Elijah Newren
2020-10-14 17:55         ` Jeff King
2020-10-14 17:57           ` Elijah Newren
2020-10-13 19:49   ` [PATCH v2 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Taylor Blau
2020-10-14 21:13   ` [PATCH v3 " Elijah Newren via GitGitGadget
2020-10-14 21:13     ` Elijah Newren via GitGitGadget [this message]
2020-10-16 11:40       ` [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run Phillip Wood
2020-10-16 17:27         ` Elijah Newren
2020-10-16 18:24           ` Phillip Wood
2020-10-16 20:02           ` Junio C Hamano
2020-10-16 20:07             ` Jeff King
2020-10-14 21:13     ` [PATCH v3 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-14 21:13     ` [PATCH v3 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-15 19:48     ` [PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Jeff King
2020-10-16 19:28     ` [PATCH v4 " Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-16 20:25       ` [PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Junio C Hamano
2020-10-16 20:56         ` Elijah Newren
2020-10-16 22:50       ` [PATCH v5 " Elijah Newren via GitGitGadget
2020-10-16 22:50         ` [PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 23:25           ` Andrei Rybak
2020-10-17 23:43             ` Elijah Newren
2020-10-16 22:50         ` [PATCH v5 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 22:50         ` [PATCH v5 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-18  0:23         ` [PATCH v6 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=9c8b6a1a943261635fa09bed22ae36e368686f15.1602710025.git.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=me@ttaylorr.com \
    --cc=newren@gmail.com \
    --cc=peff@peff.net \
    --cc=phillip.wood123@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git