Git Mailing List Archive on lore.kernel.org
 help / color / Atom feed
From: "Elijah Newren via GitGitGadget" <gitgitgadget@gmail.com>
To: git@vger.kernel.org
Cc: Phillip Wood <phillip.wood123@gmail.com>,
	Taylor Blau <me@ttaylorr.com>, Elijah Newren <newren@gmail.com>,
	Jeff King <peff@peff.net>, Junio C Hamano <gitster@pobox.com>,
	Elijah Newren <newren@gmail.com>
Subject: [PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters
Date: Wed, 14 Oct 2020 21:13:42 +0000
Message-ID: <pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com> (raw)
In-Reply-To: <pull.878.v2.git.git.1602616786.gitgitgadget@gmail.com>

This patch series enables me to change

 ./t9999-my-test.sh --run=1-3,5,17,19

into

 ./t9999-my-test.sh --run=setup,rename,symlink

and have it pick out tests with "setup", "rename", or "symlink" in their
description and run those. Saves me a lot of time, especially since numbers
for tests aren't readily accessible. The easiest way for me to get the
numbers corresponding to the tests I want to run, is to run all the tests
and look at the output to match up the descriptions with their numbers --
thus defeating the point of selecting just a subset of the tests to run in
the first place.

Changes since v2:

 * Use a glob rather than a regex for selecting tests
 * Touched up the commit message in patch 3 to mention that the TAP output
   line is still included for skipped tests and already included the word
   "skip" to highlight the fact that the test was skipped.
 * Fixed a bug where -run="!rename" would run all tests instead of skipping
   the ones with 'rename' in the description

Elijah Newren (3):
  test-lib: allow selecting tests by substring/glob with --run
  t6006, t6012: adjust tests to use 'setup' instead of synonyms
  test-lib: reduce verbosity of skipped tests

 t/README                     | 29 ++++++++++++++++---------
 t/t0000-basic.sh             | 41 ++++++++++++++++++++++--------------
 t/t6006-rev-list-format.sh   |  2 +-
 t/t6012-rev-list-simplify.sh |  2 +-
 t/test-lib.sh                | 17 +++++++++------
 5 files changed, 56 insertions(+), 35 deletions(-)


base-commit: d98273ba77e1ab9ec755576bc86c716a97bf59d7
Published-As: https://github.com/gitgitgadget/git/releases/tag/pr-git-878%2Fnewren%2Ftest-selection-v3
Fetch-It-Via: git fetch https://github.com/gitgitgadget/git pr-git-878/newren/test-selection-v3
Pull-Request: https://github.com/git/git/pull/878

Range-diff vs v2:

 1:  41e2528e83 ! 1:  9c8b6a1a94 test-lib: allow selecting tests by substring/regex with --run
     @@ Metadata
      Author: Elijah Newren <newren@gmail.com>
      
       ## Commit message ##
     -    test-lib: allow selecting tests by substring/regex with --run
     +    test-lib: allow selecting tests by substring/glob with --run
      
          Many of our test scripts have several "setup" tests.  It's a lot easier
          to say
     @@ t/README: For an individual test suite --run could be used to specify that
      -Optional prefix of '!' means that the test or a range of tests
      -should be excluded from the run.
      +The argument for --run, <test-selector>, is a list of description
     -+substrings or regexes or individual test numbers or ranges with an
     ++substrings or globs or individual test numbers or ranges with an
      +optional negation prefix (of '!') that define what tests in a test
      +suite to include (or exclude, if negated) in the run.  A range is two
      +numbers separated with a dash and matches a range of tests with both
     @@ t/README: test in the test suite except from 7 up to 11:
       
      +Sometimes there may be multiple tests with e.g. "setup" in their name
      +that are needed and rather than figuring out the number for all of them
     -+we can just use "setup" as a substring/regex to match against the test
     ++we can just use "setup" as a substring/glob to match against the test
      +description:
      +
      +    $ sh ./t0050-filesystem.sh --run=setup,9-11
     @@ t/test-lib.sh: match_test_selector_list () {
      -					echo "error: $title: invalid non-numeric in test" \
      -						"selector: '$orig_selector'" >&2
      -					exit 1
     -+					echo "$title" | grep -q "$selector" && return
     ++					case "$title" in *${selector}*)
     ++						include=$positive
     ++						;;
     ++					esac
      +					continue
       				fi
       		esac
 2:  8f1a4420f4 = 2:  6479d73603 t6006, t6012: adjust tests to use 'setup' instead of synonyms
 3:  85a4ca164a ! 3:  24f6febde6 test-lib: reduce verbosity of skipped tests
     @@ Commit message
          When using the --run flag to run just two or three tests from a test
          file which contains several dozen tests, having every skipped test print
          out dozens of lines of output for the test code for that skipped test
     -    adds up to hundreds or thousands of lines of irrelevant output that make
     -    it very hard to fish out the relevant results you were looking for.
     -    Simplify the output for skipped tests down to just showing the one-line
     -    descriptions.
     +    (in addition to the TAP output line) adds up to hundreds or thousands of
     +    lines of irrelevant output that make it very hard to fish out the
     +    relevant results you were looking for.  Simplify the output for skipped
     +    tests to remove this extra output, leaving only the TAP output line
     +    (i.e. the line reading "ok <number> # skip <test-description>", which
     +    already mentions that the test was "skip"ped).
      
          Signed-off-by: Elijah Newren <newren@gmail.com>
      

-- 
gitgitgadget

  parent reply index

Thread overview: 71+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
2020-10-12 23:26 [PATCH " Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-13 15:39   ` Taylor Blau
2020-10-13 17:23     ` Elijah Newren
2020-10-13 17:28       ` Taylor Blau
2020-10-13 18:21         ` Elijah Newren
2020-10-12 23:26 ` [PATCH 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-12 23:26 ` [PATCH 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-13 15:44   ` Taylor Blau
2020-10-13 17:56     ` Elijah Newren
2020-10-13 19:27       ` Junio C Hamano
2020-10-13 13:08 ` [PATCH 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Phillip Wood
2020-10-15 11:48   ` Johannes Schindelin
2020-10-15 13:42     ` Phillip Wood
2020-10-13 19:19 ` [PATCH v2 " Elijah Newren via GitGitGadget
2020-10-13 19:19   ` [PATCH v2 1/3] test-lib: allow selecting tests by substring/regex with --run Elijah Newren via GitGitGadget
2020-10-14 17:04     ` Jeff King
2020-10-14 17:46       ` Junio C Hamano
2020-10-14 18:07         ` Jeff King
2020-10-14 19:24           ` Junio C Hamano
2020-10-14 19:41             ` Elijah Newren
2020-10-14 19:49               ` Jeff King
2020-10-15 16:09                 ` Junio C Hamano
2020-10-15 15:59               ` Junio C Hamano
2020-10-14 17:50       ` Elijah Newren
2020-10-14 17:57         ` Junio C Hamano
2020-10-14 19:12           ` Elijah Newren
2020-10-15 16:10             ` Junio C Hamano
2020-10-14 18:09         ` Jeff King
2020-10-14 19:02           ` Elijah Newren
2020-10-15 16:20       ` Junio C Hamano
2020-10-15 19:46         ` Jeff King
2020-10-15 20:08           ` Junio C Hamano
2020-10-16  0:38             ` Jeff King
2020-10-16 16:16               ` Junio C Hamano
2020-10-16 16:30                 ` Junio C Hamano
2020-10-16 20:06                 ` Jeff King
2020-10-16 20:22                   ` Junio C Hamano
2020-10-13 19:19   ` [PATCH v2 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-13 19:19   ` [PATCH v2 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-14 16:53     ` Jeff King
2020-10-14 17:39       ` Elijah Newren
2020-10-14 17:55         ` Jeff King
2020-10-14 17:57           ` Elijah Newren
2020-10-13 19:49   ` [PATCH v2 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Taylor Blau
2020-10-14 21:13   ` Elijah Newren via GitGitGadget [this message]
2020-10-14 21:13     ` [PATCH v3 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 11:40       ` Phillip Wood
2020-10-16 17:27         ` Elijah Newren
2020-10-16 18:24           ` Phillip Wood
2020-10-16 20:02           ` Junio C Hamano
2020-10-16 20:07             ` Jeff King
2020-10-14 21:13     ` [PATCH v3 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-14 21:13     ` [PATCH v3 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-15 19:48     ` [PATCH v3 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Jeff King
2020-10-16 19:28     ` [PATCH v4 " Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 19:28       ` [PATCH v4 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-16 20:25       ` [PATCH v4 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Junio C Hamano
2020-10-16 20:56         ` Elijah Newren
2020-10-16 22:50       ` [PATCH v5 " Elijah Newren via GitGitGadget
2020-10-16 22:50         ` [PATCH v5 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-16 23:25           ` Andrei Rybak
2020-10-17 23:43             ` Elijah Newren
2020-10-16 22:50         ` [PATCH v5 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-16 22:50         ` [PATCH v5 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget
2020-10-18  0:23         ` [PATCH v6 0/3] Make test selection easier by specifying description substrings instead of just numeric counters Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 1/3] test-lib: allow selecting tests by substring/glob with --run Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 2/3] t6006, t6012: adjust tests to use 'setup' instead of synonyms Elijah Newren via GitGitGadget
2020-10-18  0:23           ` [PATCH v6 3/3] test-lib: reduce verbosity of skipped tests Elijah Newren via GitGitGadget

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=pull.878.v3.git.git.1602710025.gitgitgadget@gmail.com \
    --to=gitgitgadget@gmail.com \
    --cc=git@vger.kernel.org \
    --cc=gitster@pobox.com \
    --cc=me@ttaylorr.com \
    --cc=newren@gmail.com \
    --cc=peff@peff.net \
    --cc=phillip.wood123@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link

Git Mailing List Archive on lore.kernel.org

Archives are clonable:
	git clone --mirror https://lore.kernel.org/git/0 git/git/0.git

	# If you have public-inbox 1.1+ installed, you may
	# initialize and index your mirror using the following commands:
	public-inbox-init -V2 git git/ https://lore.kernel.org/git \
		git@vger.kernel.org
	public-inbox-index git

Example config snippet for mirrors

Newsgroup available over NNTP:
	nntp://nntp.lore.kernel.org/org.kernel.vger.git


AGPL code for this site: git clone https://public-inbox.org/public-inbox.git