From: "Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Phillip Wood <phillip.wood123@gmail.com>,
Taylor Blau <me@ttaylorr.com>, Elijah Newren <newren@gmail.com>,
Jeff King <peff@peff.net>, Junio C Hamano <gitster@pobox.com>,
Johannes Schindelin <Johannes.Schindelin@gmx.de>,
Elijah Newren <newren@gmail.com>
Subject: [PATCH v5 0/3] Make test selection easier by specifying description substrings instead of just numeric counters
Date: Fri, 16 Oct 2020 22:50:49 +0000 [thread overview]
Message-ID: <pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.878.v4.git.git.1602876532.gitgitgadget@gmail.com>
This patch series enables me to change
./t9999-my-test.sh --run=1-3,5,17,19
into
./t9999-my-test.sh --run=setup,rename,symlink
and have it pick out tests with "setup", "rename", or "symlink" in their
description and run those. Saves me a lot of time, especially since numbers
for tests aren't readily accessible. The easiest way for me to get the
numbers corresponding to the tests I want to run, is to run all the tests
and look at the output to match up the descriptions with their numbers --
thus defeating the point of selecting just a subset of the tests to run in
the first place.
Changes since v4:
* Changed to only split tests on commas
Elijah Newren (3):
test-lib: allow selecting tests by substring/glob with --run
t6006, t6012: adjust tests to use 'setup' instead of synonyms
test-lib: reduce verbosity of skipped tests
t/README | 48 ++++++++++++++++++++------------
t/t0000-basic.sh | 53 +++++++++++++++++++++---------------
t/t6006-rev-list-format.sh | 2 +-
t/t6012-rev-list-simplify.sh | 2 +-
t/test-lib.sh | 19 +++++++------
5 files changed, 75 insertions(+), 49 deletions(-)
base-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v5
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v5
Pull-Request: https://github.com/git/git/pull/878
Range-diff vs v4:
1: 8f5d4ad318 ! 1: 389c2c171a test-lib: allow selecting tests by substring/glob with --run
@@ t/README: For an individual test suite --run could be used to specify that
+ends been included. You may omit the first or the second number to
+mean "from the first test" or "up to the very last test" respectively.
+
-+The argument to --run is split on commas and whitespace into separate
-+strings, numbers, and ranges, and picks all tests that match any of
-+the individual selection criteria. If the substring of the
-+description text that you want to match includes a comma or space, use
-+the glob character '?' instead. For example --run='unnecessary?update
-+timing' would match on all tests that match either the glob
-+*unnecessary?update* or the glob *timing*.
++The argument to --run is split on commas into separate strings,
++numbers, and ranges, and picks all tests that match any of the
++individual selection criteria. If the substring of the description
++text that you want to match includes a comma, use the glob character
++'?' instead. For example --run='rebase,merge?cherry-pick' would match
++on all tests that match either the glob *rebase* or the glob
++*merge?cherry-pick*.
If --run starts with an unprefixed number or range the initial
set of tests to run is empty. If the first item starts with '!'
+@@ t/README: all the tests are added to the initial set. After initial set is
+ determined every test number or range is added or excluded from
+ the set one by one, from left to right.
+
+-Individual numbers or ranges could be separated either by a space
+-or a comma.
+-
+ For example, to run only tests up to a specific test (21), one
+ could do this:
+
+@@ t/README: or this:
+ Common case is to run several setup tests (1, 2, 3) and then a
+ specific test (21) that relies on that setup:
+
+- $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
++ $ sh ./t9200-git-cvsexport-commit.sh --run='1,2,3,21'
+
+ or:
+
+@@ t/README: or:
+
+ or:
+
+- $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
++ $ sh ./t9200-git-cvsexport-commit.sh --run='-3,21'
+
+ As noted above, the test set is built by going through the items
+ from left to right, so this:
+
+- $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
++ $ sh ./t9200-git-cvsexport-commit.sh --run='1-4,!3'
+
+ will run tests 1, 2, and 4. Items that come later have higher
+ precedence. It means that this:
+
+- $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
++ $ sh ./t9200-git-cvsexport-commit.sh --run='!3,1-4'
+
+ would just run tests from 1 to 4, including 3.
+
@@ t/README: test in the test suite except from 7 up to 11:
$ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
@@ t/README: test in the test suite except from 7 up to 11:
"setup" test, so you cannot _arbitrarily_ disable one test and
## t/t0000-basic.sh ##
+@@ t/t0000-basic.sh: test_expect_success 'GIT_SKIP_TESTS does not skip unmatched suite' "
+
+ test_expect_success '--run basic' "
+ run_sub_test_lib_test run-basic \
+- '--run basic' --run='1 3 5' <<-\\EOF &&
++ '--run basic' --run='1,3,5' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
+@@ t/t0000-basic.sh: test_expect_success '--run with a range' "
+
+ test_expect_success '--run with two ranges' "
+ run_sub_test_lib_test run-two-ranges \
+- '--run with two ranges' --run='1-2 5-6' <<-\\EOF &&
++ '--run with two ranges' --run='1-2,5-6' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
+@@ t/t0000-basic.sh: test_expect_success '--run with basic negation' "
+
+ test_expect_success '--run with two negations' "
+ run_sub_test_lib_test run-two-neg \
+- '--run with two negations' --run='"'!3 !6'"' <<-\\EOF &&
++ '--run with two negations' --run='"'!3,!6'"' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
+@@ t/t0000-basic.sh: test_expect_success '--run with two negations' "
+
+ test_expect_success '--run a range and negation' "
+ run_sub_test_lib_test run-range-and-neg \
+- '--run a range and negation' --run='"'-4 !2'"' <<-\\EOF &&
++ '--run a range and negation' --run='"'-4,!2'"' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
+@@ t/t0000-basic.sh: test_expect_success '--run range negation' "
+ test_expect_success '--run include, exclude and include' "
+ run_sub_test_lib_test run-inc-neg-inc \
+ '--run include, exclude and include' \
+- --run='"'1-5 !1-3 2'"' <<-\\EOF &&
++ --run='"'1-5,!1-3,2'"' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
+@@ t/t0000-basic.sh: test_expect_success '--run include, exclude and include, comma separated' "
+ test_expect_success '--run exclude and include' "
+ run_sub_test_lib_test run-neg-inc \
+ '--run exclude and include' \
+- --run='"'!3- 5'"' <<-\\EOF &&
++ --run='"'!3-,5'"' <<-\\EOF &&
+ for i in 1 2 3 4 5 6
+ do
+ test_expect_success \"passing test #\$i\" 'true'
@@ t/t0000-basic.sh: test_expect_success '--run empty selectors' "
EOF
"
@@ t/test-lib.sh: match_pattern_list () {
title="$1"
shift
arg="$1"
+@@ t/test-lib.sh: match_test_selector_list () {
+
+ # Both commas and whitespace are accepted as separators.
+ OLDIFS=$IFS
+- IFS=' ,'
++ IFS=','
+ set -- $1
+ IFS=$OLDIFS
+
@@ t/test-lib.sh: match_test_selector_list () {
*-*)
if expr "z${selector%%-*}" : "z[0-9]*[^0-9]" >/dev/null
2: bbae029cc8 = 2: 7c7a3d9a34 t6006, t6012: adjust tests to use 'setup' instead of synonyms
3: 0e1a1125f7 = 3: 01494bc0ba test-lib: reduce verbosity of skipped tests
--
gitgitgadget
next prev parent reply other threads:[~2020-10-16 22:51 UTC|newest]
Thread overview: 71+ messages / expand[flat|nested] mbox.gz Atom feed top
2020-10-12 23:26 [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-13 15:39 ` Taylor Blau
2020-10-13 17:23 ` Elijah Newren
2020-10-13 17:28 ` Taylor Blau
2020-10-13 18:21 ` Elijah Newren
2020-10-12 23:26 ` [PATCH 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-13 15:44 ` Taylor Blau
2020-10-13 17:56 ` Elijah Newren
2020-10-13 19:27 ` Junio C Hamano
2020-10-13 13:08 ` [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Phillip Wood
2020-10-15 11:48 ` Johannes Schindelin
2020-10-15 13:42 ` Phillip Wood
2020-10-13 19:19 ` [PATCH v2 " Elijah Newren via GitGitGadget
2020-10-13 19:19 ` [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-14 17:04 ` Jeff King
2020-10-14 17:46 ` Junio C Hamano
2020-10-14 18:07 ` Jeff King
2020-10-14 19:24 ` Junio C Hamano
2020-10-14 19:41 ` Elijah Newren
2020-10-14 19:49 ` Jeff King
2020-10-15 16:09 ` Junio C Hamano
2020-10-15 15:59 ` Junio C Hamano
2020-10-14 17:50 ` Elijah Newren
2020-10-14 17:57 ` Junio C Hamano
2020-10-14 19:12 ` Elijah Newren
2020-10-15 16:10 ` Junio C Hamano
2020-10-14 18:09 ` Jeff King
2020-10-14 19:02 ` Elijah Newren
2020-10-15 16:20 ` Junio C Hamano
2020-10-15 19:46 ` Jeff King
2020-10-15 20:08 ` Junio C Hamano
2020-10-16 0:38 ` Jeff King
2020-10-16 16:16 ` Junio C Hamano
2020-10-16 16:30 ` Junio C Hamano
2020-10-16 20:06 ` Jeff King
2020-10-16 20:22 ` Junio C Hamano
2020-10-13 19:19 ` [PATCH v2 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-13 19:19 ` [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-14 16:53 ` Jeff King
2020-10-14 17:39 ` Elijah Newren
2020-10-14 17:55 ` Jeff King
2020-10-14 17:57 ` Elijah Newren
2020-10-13 19:49 ` [PATCH v2 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Taylor Blau
2020-10-14 21:13 ` [PATCH v3 " Elijah Newren via GitGitGadget
2020-10-14 21:13 ` [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 11:40 ` Phillip Wood
2020-10-16 17:27 ` Elijah Newren
2020-10-16 18:24 ` Phillip Wood
2020-10-16 20:02 ` Junio C Hamano
2020-10-16 20:07 ` Jeff King
2020-10-14 21:13 ` [PATCH v3 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-14 21:13 ` [PATCH v3 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-15 19:48 ` [PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Jeff King
2020-10-16 19:28 ` [PATCH v4 " Elijah Newren via GitGitGadget
2020-10-16 19:28 ` [PATCH v4 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 19:28 ` [PATCH v4 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 19:28 ` [PATCH v4 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-16 20:25 ` [PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Junio C Hamano
2020-10-16 20:56 ` Elijah Newren
2020-10-16 22:50 ` Elijah Newren via GitGitGadget [this message]
2020-10-16 22:50 ` [PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 23:25 ` Andrei Rybak
2020-10-17 23:43 ` Elijah Newren
2020-10-16 22:50 ` [PATCH v5 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 22:50 ` [PATCH v5 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-18 0:23 ` [PATCH v6 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Elijah Newren via GitGitGadget
2020-10-18 0:23 ` [PATCH v6 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-18 0:23 ` [PATCH v6 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-18 0:23 ` [PATCH v6 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=pull.878.v5.git.git.1602888652.gitgitgadget@gmail.com \
--to=gitgitgadget@gmail.com \
--cc=Johannes.Schindelin@gmx.de \
--cc=git@vger.kernel.org \
--cc=gitster@pobox.com \
--cc=me@ttaylorr.com \
--cc=newren@gmail.com \
--cc=peff@peff.net \
--cc=phillip.wood123@gmail.com \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).