[PATCH v2 0/2] test-lib.sh: have all tests pass under "-x", remove BASH_XTRACEFD

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

I think this v2 should nicely address the outstanding comments on
v1. I.e. v1 made the trade-off of not caring about -x in t1510 for
some of its assertions.

But as noted in the new 1/2 here we can get a much better and easily
debuggable test regardless of which shell we use if we don't use the
pattern that's being worked around by test_untraceable.

Which, is the updated 2/2 notes is mostly what SZEDER was doing in
2018 when "test_untraceable" was introduced, it's just that t1510
wasn't migrated over at the time, as some other tests in the his
series discussed in 2/2 were.

Ævar Arnfjörð Bjarmason (2):
  t1510: remove need for "test_untraceable", retain coverage
  test-lib.sh: remove the now-unused "test_untraceable" facility

 t/README              |  3 --
 t/t1510-repo-setup.sh | 83 +++++++++++++++++++++----------------------
 t/test-lib.sh         | 66 ++++------------------------------
 3 files changed, 47 insertions(+), 105 deletions(-)

Range-diff against v1:
-:  ----------- > 1:  91402624777 t1510: remove need for "test_untraceable", retain coverage
1:  9f735bd0d49 ! 2:  867d18d14bd test-lib.sh: have all tests pass under "-x", remove BASH_XTRACEFD
    @@ Metadata
     Author: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## Commit message ##
    -    test-lib.sh: have all tests pass under "-x", remove BASH_XTRACEFD
    +    test-lib.sh: remove the now-unused "test_untraceable" facility
     
    -    Change the "t1510-repo-setup.sh" test to use a new
    -    "test_must_be_empty_trace" wrapper, instead of turning on
    -    "test_untraceable=UnfortunatelyYes".
    +    In the preceding commit the use of "test_untraceable=UnfortunatelyYes"
    +    was removed from "t1510-repo-setup.sh" in favor of more narrow
    +    redirections of the output of specific commands (and not entire
    +    sub-shells or functions).
     
    -    The only reason the test was incompatible with "-x" was because of
    -    these "test_must_be_empty" checks, which we can instead instead skip
    -    if we're running under "set -x".
    +    This is in line with the fixes in the series that introduced the
    +    "test_untraceable" facility. See 571e472dc43 (Merge branch
    +    'sg/test-x', 2018-03-14) for the series as a whole, and
    +    e.g. 91538d0cde9 (t5570-git-daemon: don't check the stderr of a
    +    subshell, 2018-02-24) for a commit that's in line with the changes in
    +    the preceding commit.
     
    -    Skipping the tests is preferable to not having the "-x" output at all,
    -    as it's much easier to debug the test. The result loss of test
    -    coverage is minimal. If someone is adjusting a "test_must_be_empty"
    -    test a failure might go away under "-x", but the new "say" we emit
    -    here should highlight that appropriately.
    +    We've thus solved the TODO item noted when "test_untraceable" was
    +    added to "t1510-repo-setup.sh" in 58275069288 (t1510-repo-setup: mark
    +    as untraceable with '-x', 2018-02-24).
     
    -    Since the only user of "test_untraceable" is gone, we can remove that,
    -    not only isn't it used now, but I think the rationale for using it in
    -    the future no longer applies.
    +    So let's remove the feature entirely. Not only is it currently unused,
    +    but it actively encourages an anti-pattern in our tests. We should be
    +    testing the output of specific commands, not entire subshells or
    +    functions.
     
    -    We'll be better off by using a simple wrapper like the new
    -    "test_must_be_empty_trace". See 58275069288 (t1510-repo-setup: mark as
    -    untraceable with '-x', 2018-02-24) and 5fc98e79fc0 (t: add means to
    -    disable '-x' tracing for individual test scripts, 2018-02-24) for the
    -    addition of "test_untraceable".
    -
    -    Once that's been removed we can dig deeper and see that we only have
    -    "BASH_XTRACEFD" due to an earlier attempt to work around the same
    -    issue. See d88785e424a (test-lib: set BASH_XTRACEFD automatically,
    -    2016-05-11) and the 90c8a1db9d6 (test-lib: silence "-x" cleanup under
    -    bash, 2017-12-08) follow-up.
    -
    -    I.e. we had redirection in "test_eval_" to get more relevant trace
    -    output under bash, which in turn was only needed because
    -    BASH_XTRACEFD=1 was set, which in turn was trying to work around test
    -    failures under "set -x".
    -
    -    It's better if our test suite works the same way on all shells, rather
    -    than getting a passing run under "bash", only to have it fail with
    -    "-x" under say "dash". As the deleted code shows this is much simpler
    -    to implement across our supported POSIX shells.
    +    That the "-x" output had to be disabled as a result is only one
    +    symptom, but even under bash those tests will be harder to debug as
    +    the subsequent check of the redirected file will be far removed from
    +    the command that emitted the output.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ t/README: appropriately before running "make". Short options can be bundled, i.e
      -d::
      --debug::
     
    - ## t/t1510-repo-setup.sh ##
    -@@ t/t1510-repo-setup.sh: A few rules for repo setup:
    -     prefix is NULL.
    - "
    - 
    --# This test heavily relies on the standard error of nested function calls.
    --test_untraceable=UnfortunatelyYes
    -+test_must_be_empty_trace () {
    -+	if want_trace
    -+	then
    -+		say "$TEST_NAME does not check test_must_be_empty on \"$@\" under -x"
    -+		return 0
    -+	fi
    -+	test_must_be_empty "$@"
    -+}
    - 
    - TEST_PASSES_SANITIZE_LEAK=true
    - . ./test-lib.sh
    -@@ t/t1510-repo-setup.sh: test_expect_success '#0: nonbare repo, no explicit configuration' '
    - 	try_repo 0 unset unset unset "" unset \
    - 		.git "$here/0" "$here/0" "(null)" \
    - 		.git "$here/0" "$here/0" sub/ 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#1: GIT_WORK_TREE without explicit GIT_DIR is accepted' '
    - 	try_repo 1 "$here" unset unset "" unset \
    - 		"$here/1/.git" "$here" "$here" 1/ \
    - 		"$here/1/.git" "$here" "$here" 1/sub/ 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#2: worktree defaults to cwd with explicit GIT_DIR' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#4: core.worktree without GIT_DIR set is accepted' '
    - 	try_case 4 unset unset \
    - 		.git "$here/4/sub" "$here/4" "(null)" \
    - 		"$here/4/.git" "$here/4/sub" "$here/4/sub" "(null)" 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#5: core.worktree + GIT_WORK_TREE is accepted' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#5: core.worktree + GIT_WORK_TREE is accepted' '
    - 	try_repo 5a .. unset "$here/5a" "" unset \
    - 		"$here/5a/.git" "$here" "$here" 5a/ \
    - 		"$here/5a/.git" "$here/5a" "$here/5a" sub/ &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#6: setting GIT_DIR brings core.worktree to life' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#9: GIT_WORK_TREE accepted with gitfile' '
    - 	try_repo 9 wt unset unset gitfile unset \
    - 		"$here/9.git" "$here/9/wt" "$here/9" "(null)" \
    - 		"$here/9.git" "$here/9/sub/wt" "$here/9/sub" "(null)" 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#10: GIT_DIR can point to gitfile' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#12: core.worktree with gitfile is accepted' '
    - 	try_repo 12 unset unset "$here/12" gitfile unset \
    - 		"$here/12.git" "$here/12" "$here/12" "(null)" \
    - 		"$here/12.git" "$here/12" "$here/12" sub/ 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#13: core.worktree+GIT_WORK_TREE accepted (with gitfile)' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#13: core.worktree+GIT_WORK_TREE accepted (with gitfile)' '
    - 	try_repo 13 non-existent-too unset non-existent gitfile unset \
    - 		"$here/13.git" "$here/13/non-existent-too" "$here/13" "(null)" \
    - 		"$here/13.git" "$here/13/sub/non-existent-too" "$here/13/sub" "(null)" 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - # case #14.
    -@@ t/t1510-repo-setup.sh: test_expect_success '#17: GIT_WORK_TREE without explicit GIT_DIR is accepted (ba
    - 	try_repo 17c "$here/17c" unset unset "" true \
    - 		.git "$here/17c" "$here/17c" "(null)" \
    - 		"$here/17c/.git" "$here/17c" "$here/17c" sub/ 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#18: bare .git named by GIT_DIR has no worktree' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#20a: core.worktree without GIT_DIR accepted (inside .git)'
    - 		"$here/20a/.git" "$here/20a" "$here/20a" .git/wt/ &&
    - 	try_case 20a/.git/wt/sub unset unset \
    - 		"$here/20a/.git" "$here/20a" "$here/20a" .git/wt/sub/ &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#20b/c: core.worktree and core.bare conflict' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#21: setup, core.worktree warns before overriding core.bare
    - 		export GIT_WORK_TREE &&
    - 		git status >/dev/null
    - 	) 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - 
    - '
    - run_wt_tests 21
    -@@ t/t1510-repo-setup.sh: test_expect_success '#25: GIT_WORK_TREE accepted if GIT_DIR unset (bare gitfile
    - 	try_repo 25 "$here/25" unset unset gitfile true \
    - 		"$here/25.git" "$here/25" "$here/25" "(null)"  \
    - 		"$here/25.git" "$here/25" "$here/25" "sub/" 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - 
    - test_expect_success '#26: bare repo has no worktree (GIT_DIR -> gitfile case)' '
    -@@ t/t1510-repo-setup.sh: test_expect_success '#29: setup' '
    - 		export GIT_WORK_TREE &&
    - 		git status
    - 	) 2>message &&
    --	test_must_be_empty message
    -+	test_must_be_empty_trace message
    - '
    - run_wt_tests 29 gitfile
    - 
    -
      ## t/test-lib.sh ##
     @@ t/test-lib.sh: then
      	exit
-- 
2.34.1.876.gdb91009a90c