From: "SZEDER Gábor" <szeder@ira.uka.de>
To: Jeff King <peff@peff.net>
Cc: git@vger.kernel.org
Subject: Re: [PATCH 01/25] t/test-lib: introduce --chain-lint option
Date: Wed, 25 Mar 2015 03:53:52 +0100 [thread overview]
Message-ID: <20150325035352.Horde.wlh5PImm_EGVzNTHeSEb7g1@webmail.informatik.kit.edu> (raw)
In-Reply-To: <20150320100548.GA12543@peff.net>
Quoting Jeff King <peff@peff.net>:
> However, there are a number of places it cannot reach:
>
> - it cannot find a failure to break out of loops on error,
> like:
>
> cmd1 &&
> for i in a b c; do
> cmd2 $i
> done &&
> cmd3
>
> which will not notice failures of "cmd2 a" or "cmd b"
s/cmd b/cmd2 b/ ?
> - it cannot find a missing &&-chain inside a block or
> subfunction, like:
>
> foo () {
> cmd1
> cmd2
> }
>
> foo &&
> bar
>
> which will not notice a failure of cmd1.
And inside subshells. I think it's worth mentioning, too, because
subshells are used frequently when setting environment variables
( GIT_FOO=bar && export GIT_FOO && cmd1 && ... ) && test_cmp
or changing directory
( cd subdir && cmd1 && ... ) && test_cmp
I was wondering whether we could do better here with helper functions,
something along the lines of:
# Set and export environment variable, automatically unsetting it after
# the test is over
test_setenv () {
eval "$1=\$2" &&
export "$1" &&
# sane_unset, to allow unsetting during the test
test_when_finished "sane_unset '$1'"
}
# Change to given directory, automatically return to current working
# directory after the test is over
test_cd () {
test_when_finished "cd '$PWD'" &&
cd "$1"
}
With these the above examples would become
test_setenv GIT_FOO bar && cmd1 && ... && test_cmp
and
test_cd subdir && cmd1 && ... && test_cmp
which means increased coverage for --chain-lint.
Thanks for working on this. I looked into this after seeing Jonathan's
patch back then, got quite far but never reached a chain-lint-clean
state, and only sent patches for the two most amusing breakages
(ddeaf7ef0d, 056f34bbcd).
I'm glad it's off my TODO list and I don't have to rebase a 1.5 year old
branch to current master :)
Gábor
> - it only checks tests that you run; every platform will
> have some tests skipped due to missing prequisites,
> so it's impossible to say from one run that the test
> suite is free of broken &&-chains. However, all tests get
> run by _somebody_, so eventually we will notice problems.
>
> - it does not operate on test_when_finished or prerequisite
> blocks. It could, but these tends to be much shorter and
> less of a problem, so I punted on them in this patch.
>
> This patch was inspired by an earlier patch by Jonathan
> Nieder:
>
> http://article.gmane.org/gmane.comp.version-control.git/235913
>
> This implementation and all bugs are mine.
>
> Signed-off-by: Jeff King <peff@peff.net>
> ---
> t/README | 10 ++++++++++
> t/test-lib.sh | 16 ++++++++++++++++
> 2 files changed, 26 insertions(+)
>
> diff --git a/t/README b/t/README
> index d5bb0c9..35438bc 100644
> --- a/t/README
> +++ b/t/README
> @@ -168,6 +168,16 @@ appropriately before running "make".
> Using this option with a RAM-based filesystem (such as tmpfs)
> can massively speed up the test suite.
>
> +--chain-lint::
> +--no-chain-lint::
> + If --chain-lint is enabled, the test harness will check each
> + test to make sure that it properly "&&-chains" all commands (so
> + that a failure in the middle does not go unnoticed by the final
> + exit code of the test). This check is performed in addition to
> + running the tests themselves. You may also enable or disable
> + this feature by setting the GIT_TEST_CHAIN_LINT environment
> + variable to "1" or "0", respectively.
> +
> You can also set the GIT_TEST_INSTALLED environment variable to
> the bindir of an existing git installation to test that installation.
> You still need to have built this git sandbox, from which various
> diff --git a/t/test-lib.sh b/t/test-lib.sh
> index c096778..50b3d3f 100644
> --- a/t/test-lib.sh
> +++ b/t/test-lib.sh
> @@ -232,6 +232,12 @@ do
> --root=*)
> root=$(expr "z$1" : 'z[^=]*=\(.*\)')
> shift ;;
> + --chain-lint)
> + GIT_TEST_CHAIN_LINT=1
> + shift ;;
> + --no-chain-lint)
> + GIT_TEST_CHAIN_LINT=0
> + shift ;;
> -x)
> trace=t
> verbose=t
> @@ -524,6 +530,16 @@ test_eval_ () {
> test_run_ () {
> test_cleanup=:
> expecting_failure=$2
> +
> + if test "${GIT_TEST_CHAIN_LINT:-0}" != 0; then
> + # 117 is magic because it is unlikely to match the exit
> + # code of other programs
> + test_eval_ "(exit 117) && $1"
> + if test "$?" != 117; then
> + error "bug in the test script: broken &&-chain: $1"
> + fi
> + fi
> +
> setup_malloc_check
> test_eval_ "$1"
> eval_ret=$?
> --
> 2.3.3.520.g3cfbb5d
next prev parent reply other threads:[~2015-03-25 2:54 UTC|newest]
Thread overview: 68+ messages / expand[flat|nested] mbox.gz Atom feed top
2015-03-20 10:04 [PATCH 0/25] detecting &&-chain breakage Jeff King
2015-03-20 10:05 ` [PATCH 01/25] t/test-lib: introduce --chain-lint option Jeff King
2015-03-25 2:53 ` SZEDER Gábor [this message]
2015-03-25 3:05 ` Jeff King
2015-03-20 10:06 ` [PATCH 02/25] t: fix severe &&-chain breakage Jeff King
2015-03-20 10:06 ` [PATCH 03/25] t: fix moderate " Jeff King
2015-03-20 10:07 ` [PATCH 04/25] t: fix trivial " Jeff King
2015-03-20 10:07 ` [PATCH 05/25] t: assume test_cmp produces verbose output Jeff King
2015-03-20 10:09 ` [PATCH 06/25] t: use verbose instead of hand-rolled errors Jeff King
2015-03-20 10:09 ` [PATCH 07/25] t: use test_must_fail instead of hand-rolled blocks Jeff King
2015-03-20 10:10 ` [PATCH 08/25] t: fix &&-chaining issues around setup which might fail Jeff King
2015-03-20 10:11 ` [PATCH 09/25] t: use test_might_fail for diff and grep Jeff King
2015-03-20 10:11 ` [PATCH 10/25] t: use test_expect_code instead of hand-rolled comparison Jeff King
2015-03-20 10:12 ` [PATCH 11/25] t: wrap complicated expect_code users in a block Jeff King
2015-03-20 10:12 ` [PATCH 12/25] t: avoid using ":" for comments Jeff King
2015-03-20 10:12 ` [PATCH 13/25] t3600: fix &&-chain breakage for setup commands Jeff King
2015-03-20 10:12 ` [PATCH 14/25] t7201: fix &&-chain breakage Jeff King
2015-03-20 10:13 ` [PATCH 15/25] t9502: " Jeff King
2015-03-20 17:48 ` Johannes Sixt
2015-03-20 18:03 ` Jeff King
2015-03-20 10:13 ` [PATCH 16/25] t6030: use modern test_* helpers Jeff King
2015-03-20 10:13 ` [PATCH 17/25] t0020: " Jeff King
2015-03-25 0:23 ` SZEDER Gábor
2015-03-25 2:56 ` Jeff King
2015-03-25 5:24 ` [PATCH 0/8] more &&-chaining test fixups Jeff King
2015-03-25 5:25 ` [PATCH 1/8] perf-lib: fix ignored exit code inside loop Jeff King
2015-03-25 5:28 ` [PATCH 2/8] t0020: fix ignored exit code inside loops Jeff King
2015-03-25 5:28 ` [PATCH 3/8] t3305: fix ignored exit code inside loop Jeff King
2015-03-25 8:40 ` Johan Herland
2015-03-25 5:29 ` [PATCH 4/8] t7701: " Jeff King
2015-03-25 5:29 ` [PATCH 5/8] t: fix some trivial cases of ignored exit codes in loops Jeff King
2015-03-25 5:30 ` [PATCH 6/8] t: simplify loop exit-code status variables Jeff King
2015-03-25 17:27 ` Junio C Hamano
2015-03-25 17:43 ` Jeff King
2015-03-25 5:31 ` [PATCH 7/8] t0020: use test_* helpers instead of hand-rolled messages Jeff King
2015-03-25 5:32 ` [PATCH 8/8] t9001: drop save_confirm helper Jeff King
2015-03-25 17:29 ` [PATCH 0/8] more &&-chaining test fixups Junio C Hamano
2015-03-20 10:13 ` [PATCH 18/25] t1301: use modern test_* helpers Jeff King
2015-03-24 23:51 ` SZEDER Gábor
2015-03-25 2:45 ` Jeff King
2015-03-20 10:13 ` [PATCH 19/25] t6034: " Jeff King
2015-03-24 23:43 ` SZEDER Gábor
2015-03-20 10:13 ` [PATCH 20/25] t4117: " Jeff King
2015-03-20 10:13 ` [PATCH 21/25] t9001: use test_when_finished Jeff King
2015-03-25 2:00 ` SZEDER Gábor
2015-03-25 2:47 ` Jeff King
2015-03-20 10:13 ` [PATCH 22/25] t0050: appease --chain-lint Jeff King
2015-03-20 10:13 ` [PATCH 23/25] t7004: fix embedded single-quotes Jeff King
2015-03-20 10:13 ` [PATCH 24/25] t0005: fix broken &&-chains Jeff King
2015-03-20 10:13 ` [PATCH 25/25] t4104: drop hand-rolled error reporting Jeff King
2015-03-20 10:23 ` [PATCH 0/25] detecting &&-chain breakage Jeff King
2015-03-20 14:28 ` Michael J Gruber
2015-03-20 14:32 ` [PATCH 26/27] t/*svn*: fix moderate " Michael J Gruber
2015-03-20 14:32 ` [PATCH 27/27] t9104: fix test for following larger parents Michael J Gruber
2015-03-20 18:04 ` [PATCH 26/27] t/*svn*: fix moderate &&-chain breakage Junio C Hamano
2015-03-20 19:35 ` Junio C Hamano
2015-03-20 20:02 ` Jeff King
2015-03-20 20:13 ` Jeff King
2015-03-23 9:36 ` Michael J Gruber
2015-03-20 17:57 ` [PATCH 0/25] detecting " Jeff King
2015-03-20 17:44 ` Junio C Hamano
2015-03-20 18:00 ` Junio C Hamano
2015-03-20 18:04 ` Jeff King
2015-03-20 18:33 ` Junio C Hamano
2015-03-20 23:18 ` Eric Sunshine
2015-03-21 8:19 ` Jeff King
2015-03-21 18:01 ` Junio C Hamano
2015-03-21 22:23 ` Jeff King
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=20150325035352.Horde.wlh5PImm_EGVzNTHeSEb7g1@webmail.informatik.kit.edu \
--to=szeder@ira.uka.de \
--cc=git@vger.kernel.org \
--cc=peff@peff.net \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).