From: Junio C Hamano <gitster@pobox.com>
To: Ilya Bobyr <ilya.bobyr@gmail.com>
Cc: git@vger.kernel.org, Thomas Rast <trast@inf.ethz.ch>,
Eric Sunshine <sunshine@sunshineco.com>,
Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Subject: Re: [PATCH 3/3] test-lib: '--run' to run only specific tests
Date: Wed, 23 Apr 2014 11:40:44 -0700 [thread overview]
Message-ID: <xmqqlhuvzy6r.fsf@gitster.dls.corp.google.com> (raw)
In-Reply-To: <1398154767-1276-4-git-send-email-ilya.bobyr@gmail.com> (Ilya Bobyr's message of "Tue, 22 Apr 2014 01:19:27 -0700")
Ilya Bobyr <ilya.bobyr@gmail.com> writes:
> @@ -187,10 +192,70 @@ and either can match the "t[0-9]{4}" part to skip the whole
> test, or t[0-9]{4} followed by ".$number" to say which
> particular test to skip.
>
> -Note that some tests in the existing test suite rely on previous
> -test item, so you cannot arbitrarily disable one and expect the
> -remainder of test to check what the test originally was intended
> -to check.
> +For an individual test suite --run could be used to specify that
> +only some tests should be run or that some tests should be
> +excluded from a run.
> +
> +The argument for --run is a list of individual test numbers or
> +ranges with an optional negation prefix that define what tests in
> +a test suite to include in the run. A range is two numbers
> +separated with a dash and matches a range of tests with both ends
> +been included. You may omit the first or the second number to
> +mean "from the first test" or "up to the very last test"
> +respectively.
> +
> +Optional prefix of '!' means that the test or a range of tests
> +should be excluded from the run.
> +
> +If --run starts with an unprefixed number or range the initial
> +set of tests to run is empty. If the first item starts with '!'
> +all the tests are added to the initial set. After initial set is
> +determined every test number or range is added or excluded from
> +the set one by one, from left to right.
> +
> +Individual numbers or ranges could be separated either by a space
> +or a comma.
> +
> +For example, common case is to run several setup tests (1, 2, 3)
> +and then a specific test (21) that relies on that setup:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='1 2 3 21'
> +
> +or:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run=1,2,3,21
> +
> +or:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='-3 21'
Good and easily understandable examples.
> +To run only tests up to a specific test (21), one could do this:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='1-21'
> +
> +or this:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='-21'
These may be redundant, given that the reader would have to have
grokked the earlier "-3 21" already at this point.
> +As noted above, the test set is built going though items left to
> +right, so this:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
> +
> +will run tests 1, 2, and 4.
I do not quite understand what you mean by "left to right"; is that
implementation detail necessary for the user of the feature, or is
it talking about some limitation coming from the implementation?
e.g. perhaps "!3 1-4" would not work as people would expect "do not
run 3, but run tests from 1 thru 4 otherwise", and warning against
having such an expectation that cannot be fulfilled?
> +You may use negation with ranges. The following will run all
> +test as a test suite except from 7 upto 11:
> +
> + $ sh ./t9200-git-cvsexport-commit.sh --run='!7-11'
Hmm, that is somewhat counter-intuitive or at least ambiguous. I
first thought you would be running everything but skipping 7 thru
11, but your explanation is that it is equivalent to "-6,8-11" (that
is, to intersect set "-11" and set "!7").
The above two illustrate the reason rather well why I said it would
be better to avoid negation because it would complicate the mental
model the user needs to form when using the feature.
> +Some tests in a test suite rely on the previous tests performing
> +certain actions, specifically some tests are designated as
> +"setup" test, so you cannot _arbitrarily_ disable one test and
> +expect the rest to function correctly.
What this text (moved from the top of this hunk) tells the reader
applies to both the traditional t0123.4 and the new "--run=1-3,5-"
syntaxes, but the new placement of it make it sound as if it is only
for skipping with "--run", especially because the text before this
paragraph and also after this paragraph both apply only to "--run".
> +--run is mostly useful when you want to focus on a specific test
> +and know what you are doing. Or when you want to run up to a
> +certain test.
Likewise for "and know what you are doing" part. I'd suggest
dropping that phrase from here, and/or make it part of the "you
cannot randomly omit and expect later ones to work" that covers both
ways to skip tests.
Thanks.
next prev parent reply other threads:[~2014-04-23 18:41 UTC|newest]
Thread overview: 24+ messages / expand[flat|nested] mbox.gz Atom feed top
2014-04-22 8:19 [RFC/PATCH v3] Better control of the tests run by a test suite Ilya Bobyr
2014-04-22 8:19 ` [PATCH 1/3] test-lib: Document short options in t/README Ilya Bobyr
2014-04-23 18:24 ` Junio C Hamano
2014-04-30 9:38 ` Ilya Bobyr
2014-04-22 8:19 ` [PATCH 2/3] test-lib: tests skipped by GIT_SKIP_TESTS say so Ilya Bobyr
2014-04-22 8:19 ` [PATCH 3/3] test-lib: '--run' to run only specific tests Ilya Bobyr
2014-04-23 18:40 ` Junio C Hamano [this message]
2014-04-30 9:40 ` Ilya Bobyr
2014-04-30 14:17 ` Junio C Hamano
2014-04-23 19:51 ` Eric Sunshine
2014-04-30 9:41 ` Ilya Bobyr
2014-04-30 9:50 ` [RFC/PATCH v4] Better control of the tests run by a test suite Ilya Bobyr
2014-04-30 9:50 ` [PATCH 1/3] test-lib: Document short options in t/README Ilya Bobyr
2014-04-30 9:50 ` [PATCH 2/3] test-lib: tests skipped by GIT_SKIP_TESTS say so Ilya Bobyr
2014-04-30 9:50 ` [PATCH 3/3] test-lib: '--run' to run only specific tests Ilya Bobyr
2014-05-06 20:53 ` Junio C Hamano
2014-05-06 21:02 ` Junio C Hamano
-- strict thread matches above, loose matches on Subject: below --
2014-03-24 8:49 [RFC/PATCH] Better control of the tests run by a test suite Ilya Bobyr
2014-03-24 8:49 ` [PATCH 3/3] test-lib: '--run' to run only specific tests Ilya Bobyr
2014-03-27 10:32 ` [RFC/PATCH v2] Better control of the tests run by a test suite Ilya Bobyr
2014-03-27 10:32 ` [PATCH 3/3] test-lib: '--run' to run only specific tests Ilya Bobyr
2014-03-28 3:36 ` Eric Sunshine
2014-03-28 7:05 ` Ilya Bobyr
2014-03-30 9:41 ` Eric Sunshine
2014-03-31 17:09 ` Junio C Hamano
2014-03-31 19:35 ` David Tran
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=xmqqlhuvzy6r.fsf@gitster.dls.corp.google.com \
--to=gitster@pobox.com \
--cc=git@vger.kernel.org \
--cc=ilya.bobyr@gmail.com \
--cc=ramsay@ramsay1.demon.co.uk \
--cc=sunshine@sunshineco.com \
--cc=trast@inf.ethz.ch \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).