Re: [PATCH 5/5] cmake: avoid editing t/test-lib.sh

[Johannes Schindelin <Johannes.Schindelin@gmx.de>]

Hi Phillip,

On Thu, 11 Aug 2022, Phillip Wood wrote:

> On 10/08/2022 16:02, Johannes Schindelin via GitGitGadget wrote:
> > From: Johannes Schindelin <johannes.schindelin@gmx.de>
> >
> > In 7f5397a07c6c (cmake: support for testing git when building out of the
> > source tree, 2020-06-26), we implemented support for running Git's test
> > scripts even after building Git in a different directory than the source
> > directory.
> >
> > The way we did this was to edit the file `t/test-lib.sh` to override
> > `GIT_BUILD_DIR` to point somewhere else than the parent of the `t/`
> > directory.
> >
> > This is unideal because it always leaves a tracked file marked as
> > modified, and it is all too easy to commit that change by mistake.
> >
> > Let's change the strategy by teaching `t/test-lib.sh` to detect the
> > presence of a file called `GIT-BUILD-DIR` in the source directory. If it
> > exists, the contents are interpreted as the location to the _actual_
> > build directory. We then write this file as part of the CTest
> > definition.
>
> I think it is really good to get away from editing the test files, but one of
> the nice things about CMake's out of tree builds is that you can have several
> build directories with different build configurations and this change does not
> support that. Could we pass the build directory to the test scripts as a
> commandline option or environment variable instead? e.g.
>
>   foreach(tsh ${test_scipts})
>    	add_test(NAME ${tsh}
> -		COMMAND ${SH_EXE} ${tsh} --no-bin-wrappers --no-chain-lint -vx
> +		COMMAND ${SH_EXE} ${tsh} --no-bin-wrappers --no-chain-lint
> --build-dir=${CMAKE_BINARY_DIR} -vx
>
>   		WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/t)
>   endforeach()
>
> Doing that would avoid changing the main Makefile to remove a file which
> almost certainly does not exist every time make is invoked as well.

This is indeed tempting, but one of the things I recommend to Visual
Studio users for ages already is to run the tests in a Git Bash, i.e.
outside of Visual Studio, to allow for passing `--run=...` options and the
likes. This recommendation predates support for CTest, naturally, but it
is still valid.

And that recommended way of running tests would be broken by the suggested
change because the tests would no longer run except when using CTest.

Besides, while the semantics look tempting, the implementation details do
not. The reason is that we use `GIT_BUILD_DIR` to source the
`GIT-BUILD-OPTIONS` and to validate that Git was built, long before we
parse the command-line in `t/test-lib.sh`. I want to refrain from the
prerequisite extensive refactoring required to support this, at least in
the context of this here patch series.

Can we maybe agree that the proposed patch is a net improvement over the
status quo, and think about a better solution independently (without
blocking this here patch)?

Ciao,
Dscho