[PATCH v5 10/15] cmake: optionally be able to run tests before "ctest"

From: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
To: git@vger.kernel.org
Cc: "Junio C Hamano" <gitster@pobox.com>,
	"Johannes Schindelin" <Johannes.Schindelin@gmx.de>,
	"Phillip Wood" <phillip.wood123@gmail.com>,
	"Victoria Dye" <vdye@github.com>,
	"Eric Sunshine" <ericsunshine@gmail.com>,
	"Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Subject: [PATCH v5 10/15] cmake: optionally be able to run tests before "ctest"
Date: Fri,  2 Dec 2022 12:28:31 +0100	[thread overview]
Message-ID: <patch-v5-10.15-529e18f7f72-20221202T110947Z-avarab@gmail.com> (raw)
In-Reply-To: <cover-v5-00.15-00000000000-20221202T110947Z-avarab@gmail.com>

Since [1] the "cmake" build has had support for running the tests
manually from the "t/" directory once we've built with "cmake", but
only after a one-off run of "ctest".

I.e. we create the build recipe via "cmake" and build with
e.g. "make", but after doing so would get:

	$ (cd t && ./t0071-sort.sh)
	error: GIT-BUILD-OPTIONS missing (has Git been built?).

The recently added amendment to this of using a "GIT-BUILD-DIR"[2]
didn't change these semantics. After we did a one-off run of "ctest"
we'd be able to run the tests without "ctest":

	$ ctest --test-dir contrib/buildsystems/out -R t0071
	[...]
	100% tests passed, 0 tests failed out of 1
	[...]
	$ (cd t && ./t0071-sort.sh)
	ok 1 - DEFINE_LIST_SORT_DEBUG
	# passed all 1 test(s)
	1..1

This change optionally closes that gap, and allows for more sensible
behavior. Due to concerns about existing MS Visual Studio users
relying on some of these edge cases this is being made optional, with
the default of the new "GIT_CTEST_SETS_BUILD_DIR" being "ON" on
Windows (i.e. the old behavior), and "OFF" elsewhere.

The resulting behavior differences are the following:

	|---+----+-----|
	|   | ON | OFF |
	|---+----+-----|
	| A | N  | Y   |
	| B | N  | Y   |
	| C | Y  | Y   |
	| D | Y  | N   |
	| E | N  | Y   |
	|---+----+-----|
	A. Can manually run tests before ctest?
	B. Manually run tests point to latest cmake build?...
	C. Manually run tests point to latest "ctest"'d build?
        D. "git" picked at ctest start time?
        E. "git" picked at test start time?

On "D" and "E": Because GIT_CTEST_SETS_BUILD_DIR=ON relies on the
global "GIT-BUILD-DIR" when using "ctest" it cannot be used to run
concurrent tests for two different builds from the same source
directory.

But as noted in [3] existing users using Visual Studio may be relying
on some of these edge cases, so let's make this optional on Windows,
but switch the default to this new discover method on other platforms.

1. 7f5397a07c6 (cmake: support for testing git when building out of
   the source tree, 2020-06-26)
2. ee9e66e4e76 (cmake: avoid editing t/test-lib.sh, 2022-10-18)
3. https://lore.kernel.org/git/663b93ef-0c89-a5f6-1069-b4be97915d20@dunelm.org.uk/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 contrib/buildsystems/CMakeLists.txt | 38 ++++++++++++++++++++++++-----
 t/test-lib.sh                       |  3 ++-
 2 files changed, 34 insertions(+), 7 deletions(-)

diff --git a/contrib/buildsystems/CMakeLists.txt b/contrib/buildsystems/CMakeLists.txt
index 5b3fb93120d..866bdc1914a 100644
--- a/contrib/buildsystems/CMakeLists.txt
+++ b/contrib/buildsystems/CMakeLists.txt
@@ -1115,12 +1115,33 @@ if(USE_VCPKG)
 	file(APPEND ${CMAKE_BINARY_DIR}/GIT-BUILD-OPTIONS "PATH=\"$PATH:$TEST_DIRECTORY/../compat/vcbuild/vcpkg/installed/x64-windows/bin\"\n")
 endif()
 
-#Make the tests work when building out of the source tree
-get_filename_component(CACHE_PATH ${CMAKE_CURRENT_LIST_DIR}/../../CMakeCache.txt ABSOLUTE)
-if(NOT ${CMAKE_BINARY_DIR}/CMakeCache.txt STREQUAL ${CACHE_PATH})
-	#Setting the build directory in test-lib.sh before running tests
-	file(WRITE ${CMAKE_BINARY_DIR}/CTestCustom.cmake
-		"file(WRITE ${CMAKE_SOURCE_DIR}/GIT-BUILD-DIR \"${CMAKE_BINARY_DIR}\")")
+set(GIT_CTEST_SETS_BUILD_DIR_DOC "find cmake build dir via ctest-set GIT-BUILD-DIR?")
+if(DEFINED ENV{GIT_CTEST_SETS_BUILD_DIR})
+	set(GIT_CTEST_SETS_BUILD_DIR "$ENV{GIT_CTEST_SETS_BUILD_DIR}"
+		CACHE BOOL GIT_CTEST_SETS_BUILD_DIR_DOC)
+elseif(WIN32)
+	set(GIT_CTEST_SETS_BUILD_DIR "ON" CACHE BOOL GIT_CTEST_SETS_BUILD_DIR_DOC)
+else()
+	set(GIT_CTEST_SETS_BUILD_DIR "OFF" CACHE BOOL GIT_CTEST_SETS_BUILD_DIR_DOC)
+endif()
+if(GIT_CTEST_SETS_BUILD_DIR)
+	message(STATUS "Enabling manual runs of t/* tests once ctest writes CMAKE-BUILD-DIR")
+else()
+	message(STATUS "Enabling manual runs of t/* tests via cmake build dir discovery")
+endif()
+
+# When using GIT_CTEST_SETS_BUILD_DIR, running the tests from the "t/"
+# directory will only work once "ctest" has been run, as we write to
+# "GIT-BUILD-DIR" from the optional "ctest" invocation following the
+# "cmake"
+if(GIT_CTEST_SETS_BUILD_DIR)
+	get_filename_component(CACHE_PATH ${CMAKE_CURRENT_LIST_DIR}/../../CMakeCache.txt ABSOLUTE)
+	if(NOT ${CMAKE_BINARY_DIR}/CMakeCache.txt STREQUAL ${CACHE_PATH})
+		file(WRITE ${CMAKE_BINARY_DIR}/CTestCustom.cmake
+			"file(WRITE ${CMAKE_SOURCE_DIR}/GIT-BUILD-DIR \"${CMAKE_BINARY_DIR}\")")
+	endif()
+else()
+	file(WRITE "${CMAKE_SOURCE_DIR}/GIT-BUILD-DIR" "${CMAKE_BINARY_DIR}")
 endif()
 
 file(GLOB test_scipts "${CMAKE_SOURCE_DIR}/t/t[0-9]*.sh")
@@ -1130,6 +1151,11 @@ foreach(tsh ${test_scipts})
 	add_test(NAME ${tsh}
 		COMMAND ${SH_EXE} ${tsh} --no-bin-wrappers --no-chain-lint -vx
 		WORKING_DIRECTORY ${CMAKE_SOURCE_DIR}/t)
+if(NOT GIT_CTEST_SETS_BUILD_DIR)
+	set_property(TEST ${tsh} APPEND PROPERTY ENVIRONMENT
+		GIT_TEST_PREFER_BUILD_DIR_ENV="Yes, ignore GIT-BUILD-DIR"
+		GIT_TEST_BUILD_DIR=${CMAKE_BINARY_DIR})
+endif()
 endforeach()
 
 # This test script takes an extremely long time and is known to time out even
diff --git a/t/test-lib.sh b/t/test-lib.sh
index a1b5c7c6ce1..36a9a32db05 100644
--- a/t/test-lib.sh
+++ b/t/test-lib.sh
@@ -51,7 +51,8 @@ fi
 # For CMake the top-level source directory is different from our build
 # directory. With the top-level Makefile they're the same.
 GIT_SOURCE_DIR="$GIT_BUILD_DIR"
-if test ! -x "$GIT_BUILD_DIR/git" &&
+if test -z "$GIT_TEST_PREFER_BUILD_DIR_ENV" &&
+	test ! -x "$GIT_BUILD_DIR/git" &&
 	test -f "$GIT_BUILD_DIR/GIT-BUILD-DIR"
 then
 	GIT_BUILD_DIR="$(cat "$GIT_BUILD_DIR/GIT-BUILD-DIR")" || exit 1
-- 
2.39.0.rc1.981.gf846af54b4b

