Re: [PATCH 3/3] test-lib: '--run' to run only specific tests

From: Junio C Hamano <gitster@pobox.com>
To: Ilya Bobyr <ilya.bobyr@gmail.com>
Cc: git@vger.kernel.org, Thomas Rast <trast@inf.ethz.ch>,
	Eric Sunshine <sunshine@sunshineco.com>,
	Ramsay Jones <ramsay@ramsay1.demon.co.uk>
Subject: Re: [PATCH 3/3] test-lib: '--run' to run only specific tests
Date: Tue, 06 May 2014 13:53:31 -0700	[thread overview]
Message-ID: <xmqqk39yeiic.fsf@gitster.dls.corp.google.com> (raw)
In-Reply-To: <1398851444-2492-4-git-send-email-ilya.bobyr@gmail.com> (Ilya Bobyr's message of "Wed, 30 Apr 2014 02:50:44 -0700")

Ilya Bobyr <ilya.bobyr@gmail.com> writes:

> Allow better control of the set of tests that will be executed for a
> single test suite.  Mostly useful while debugging or developing as it
> allows to focus on a specific test.
>
> Signed-off-by: Ilya Bobyr <ilya.bobyr@gmail.com>
> ---
> A number of minor changes according to the review comments.

I think the interaction between multiple selectors, especially when
some of them are negated, are much better explained in this version,
compared to the previous round in the README.

But I still think that the negation a feature that is unnecessary
and having it makes it harder to understand for users, especially
after reading this part:

> +If --run starts with an unprefixed number or range the initial
> +set of tests to run is empty. If the first item starts with '!'
> +all the tests are added to the initial set.  After initial set is
> +determined every test number or range is added or excluded from
> +the set one by one, from left to right.
> ...
> +As noted above, the test set is built going though items left to
> +right, so this:
> +
> +    $ sh ./t9200-git-cvsexport-commit.sh --run='1-4 !3'
> +
> +will run tests 1, 2, and 4.  Items that comes later have higher
> +precendence.  It means that this:
> +
> +    $ sh ./t9200-git-cvsexport-commit.sh --run='!3 1-4'
> +
> +would just run tests from 1 to 4, including 3.

The initial !3 means the same thing as "1-2,4-", and then 1-4 will
do what to that set?  The answer is "It is added"... wouldn't the
reader expect then that the result should be "1-", not "1-4"?  I
myself wondered what would happen to the fifth test from your
description.  Has the text told the reader that t9200 test has only
four tests?

The need to explain better with longer description will reduce the
likelyhood that the feature is understood and correctly used.  When
you can write "1-2,4-", why accept "1-4 !3" and force yourself to
explain to people why that is different from "!3 1-4"?