[PATCH v2] test: fix for TEST_OUTPUT_DIRECTORY

[PATCH v2] test: fix for TEST_OUTPUT_DIRECTORY

[Felipe Contreras]

The test_atexit unit test relies on the specific location of the
generated files.

When TEST_OUTPUT_DIRECTORY is unset, _run_sub_test_lib_test_common sets
it to pwd, which is two levels under the pwd of the parent unit test,
and the parent can find the generated files just fine.

But when TEST_OUTPUT_DIRECTORY is set, it's stored in GIT-BUILD-OPTIONS,
and even though _run_sub_test_lib_test_common correctly overrides it,
when the child script is run it sources GIT-BUILD-OPTIONS and
TEST_OUTPUT_DIRECTORY is overridden.

Effectively both the parent and child scripts output to the same
directory.

  make TEST_OUTPUT_DIRECTORY=/tmp/foobar GIT-BUILD-OPTIONS &&
  make -C t t0000-basic.sh

We could try to specify --root, as 6883047071 (t0000: set
TEST_OUTPUT_DIRECTORY for sub-tests, 2013-12-28) suggested, but then the
results of subtests would leak out because TEST_RESULTS_DIR would not
be changed from the parent.

Instead, let's revert part of 2d14e13c56 (test output: respect
$TEST_OUTPUT_DIRECTORY, 2013-04-29) by removing TEST_OUTPUT_DIRECTORY
from GIT-BUILD-OPTIONS.

It's unclear how much value t/valgrind/analyze.sh provides today, but
users of that script that use TEST_OUTPUT_DIRECTORY as well can simply
call the script with that variable in the environment.

It doesn't make much sense to break t0000-basic.sh for users of
TEST_OUTPUT_DIRECTORY, just to provide a little convenience for the
users of t/valgrind/analyze.sh.

Presumably this was broken since 900721e15c (test-lib: introduce
'test_atexit', 2019-03-13).

Cc: John Keeping <john@keeping.me.uk>
Cc: Johannes Schindelin <Johannes.Schindelin@gmx.de>
Helped-by: Jeff King <peff@peff.net>
Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
---

Since v1 I completely changed the approach and instead of using --root
which leaks test results, I remove TEST_OUTPUT_DIRECTORY from
GIT-BUILD-OPTIONS.

Apparently only people who would care are the users of
t/valgrind/analyze.sh which now would need to specify that variable
themselves.

Marginal convenience for the users of an obscure script is not a good
reason to break t0000-basic.sh.

Range-diff against v1:
1:  04047359b9 < -:  ---------- test: fix for TEST_OUTPUT_DIRECTORY
-:  ---------- > 1:  d8430aee08 test: fix for TEST_OUTPUT_DIRECTORY

 Makefile              | 3 ---
 t/valgrind/analyze.sh | 3 ---
 2 files changed, 6 deletions(-)

diff --git a/Makefile b/Makefile
index c3565fc0f8..2e25489569 100644
--- a/Makefile
+++ b/Makefile
@@ -2790,9 +2790,6 @@ GIT-BUILD-OPTIONS: FORCE
 	@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+
 	@echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
 	@echo X=\'$(X)\' >>$@+
-ifdef TEST_OUTPUT_DIRECTORY
-	@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
-endif
 ifdef GIT_TEST_OPTS
 	@echo GIT_TEST_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_OPTS)))'\' >>$@+
 endif
diff --git a/t/valgrind/analyze.sh b/t/valgrind/analyze.sh
index 2ffc80f721..378d0a8daa 100755
--- a/t/valgrind/analyze.sh
+++ b/t/valgrind/analyze.sh
@@ -1,8 +1,5 @@
 #!/bin/sh
 
-# Get TEST_OUTPUT_DIRECTORY from GIT-BUILD-OPTIONS if it's there...
-. "$(dirname "$0")/../../GIT-BUILD-OPTIONS"
-# ... otherwise set it to the default value.
 : ${TEST_OUTPUT_DIRECTORY=$(dirname "$0")/..}
 
 output=
-- 
2.32.0

Re: [PATCH v2] test: fix for TEST_OUTPUT_DIRECTORY

[Ævar Arnfjörð Bjarmason]

On Tue, Jun 15 2021, Felipe Contreras wrote:

> The test_atexit unit test relies on the specific location of the
> generated files.
>
> When TEST_OUTPUT_DIRECTORY is unset, _run_sub_test_lib_test_common sets
> it to pwd, which is two levels under the pwd of the parent unit test,
> and the parent can find the generated files just fine.
>
> But when TEST_OUTPUT_DIRECTORY is set, it's stored in GIT-BUILD-OPTIONS,
> and even though _run_sub_test_lib_test_common correctly overrides it,
> when the child script is run it sources GIT-BUILD-OPTIONS and
> TEST_OUTPUT_DIRECTORY is overridden.
>
> Effectively both the parent and child scripts output to the same
> directory.
>
>   make TEST_OUTPUT_DIRECTORY=/tmp/foobar GIT-BUILD-OPTIONS &&
>   make -C t t0000-basic.sh
>
> We could try to specify --root, as 6883047071 (t0000: set
> TEST_OUTPUT_DIRECTORY for sub-tests, 2013-12-28) suggested, but then the
> results of subtests would leak out because TEST_RESULTS_DIR would not
> be changed from the parent.
>
> Instead, let's revert part of 2d14e13c56 (test output: respect
> $TEST_OUTPUT_DIRECTORY, 2013-04-29) by removing TEST_OUTPUT_DIRECTORY
> from GIT-BUILD-OPTIONS.
>
> It's unclear how much value t/valgrind/analyze.sh provides today, but
> users of that script that use TEST_OUTPUT_DIRECTORY as well can simply
> call the script with that variable in the environment.

I've only skimmed this & the patch, but this approach sounds good to me.

> It doesn't make much sense to break t0000-basic.sh for users of
> TEST_OUTPUT_DIRECTORY, just to provide a little convenience for the
> users of t/valgrind/analyze.sh.

I would also think that if anyone really cares about this
valgrind/analyze.sh script the thing to do is to make a Makefile target
for it, similar to the aggregate target(s).

We could then make that target go through some light wrapper in
test-lib.sh so it would e.g. pick up GIT_TEST_OPTS (which can have
--root=*), and otherwise find things in the same location that
test-lib.sh put them / thinks the are in.

> Presumably this was broken since 900721e15c (test-lib: introduce
> 'test_atexit', 2019-03-13).
>
> Cc: John Keeping <john@keeping.me.uk>
> Cc: Johannes Schindelin <Johannes.Schindelin@gmx.de>
> Helped-by: Jeff King <peff@peff.net>
> Signed-off-by: Felipe Contreras <felipe.contreras@gmail.com>
> ---
>
> Since v1 I completely changed the approach and instead of using --root
> which leaks test results, I remove TEST_OUTPUT_DIRECTORY from
> GIT-BUILD-OPTIONS.
>
> Apparently only people who would care are the users of
> t/valgrind/analyze.sh which now would need to specify that variable
> themselves.
>
> Marginal convenience for the users of an obscure script is not a good
> reason to break t0000-basic.sh.
>
> Range-diff against v1:
> 1:  04047359b9 < -:  ---------- test: fix for TEST_OUTPUT_DIRECTORY
> -:  ---------- > 1:  d8430aee08 test: fix for TEST_OUTPUT_DIRECTORY
>
>  Makefile              | 3 ---
>  t/valgrind/analyze.sh | 3 ---
>  2 files changed, 6 deletions(-)
>
> diff --git a/Makefile b/Makefile
> index c3565fc0f8..2e25489569 100644
> --- a/Makefile
> +++ b/Makefile
> @@ -2790,9 +2790,6 @@ GIT-BUILD-OPTIONS: FORCE
>  	@echo PAGER_ENV=\''$(subst ','\'',$(subst ','\'',$(PAGER_ENV)))'\' >>$@+
>  	@echo DC_SHA1=\''$(subst ','\'',$(subst ','\'',$(DC_SHA1)))'\' >>$@+
>  	@echo X=\'$(X)\' >>$@+
> -ifdef TEST_OUTPUT_DIRECTORY
> -	@echo TEST_OUTPUT_DIRECTORY=\''$(subst ','\'',$(subst ','\'',$(TEST_OUTPUT_DIRECTORY)))'\' >>$@+
> -endif
>  ifdef GIT_TEST_OPTS
>  	@echo GIT_TEST_OPTS=\''$(subst ','\'',$(subst ','\'',$(GIT_TEST_OPTS)))'\' >>$@+
>  endif
> diff --git a/t/valgrind/analyze.sh b/t/valgrind/analyze.sh
> index 2ffc80f721..378d0a8daa 100755
> --- a/t/valgrind/analyze.sh
> +++ b/t/valgrind/analyze.sh
> @@ -1,8 +1,5 @@
>  #!/bin/sh
>  
> -# Get TEST_OUTPUT_DIRECTORY from GIT-BUILD-OPTIONS if it's there...
> -. "$(dirname "$0")/../../GIT-BUILD-OPTIONS"
> -# ... otherwise set it to the default value.
>  : ${TEST_OUTPUT_DIRECTORY=$(dirname "$0")/..}
>  
>  output=