[PATCH 00/10] ci: make it easy to run locally, part 1

[PATCH 00/10] ci: make it easy to run locally, part 1

[Ævar Arnfjörð Bjarmason]

This series extracted from the much larger series to run CI in
"steps"[1] submitted in the last cycle, and rebased on CI work that's
happened in-tree since then.

At the end of this series the "ci/run-static-analysis.sh" has been
extracted to a "make ci-static-analysis", target, which the CI then
runs. The introduction of "setenv" and some of the structures here is
larger than strictly necessary to get to that point, but as seen in
[1] is in service of the larger goal of making CI less of a special
snowflake.

The eventual goal down the line is to be able to trivially run local
tests "as in CI", to the extent that's possible. Obviously we can't
easily run the OSX CI locally on a Linux box, but it's now a needless
hassle to set up e.g. the same environment variables as various CI
jobs require before they run "make" or "make test".

This series starts out by removing some dead Travis CI code, the
removal of the Azure bits of CI code has been ejected, as Johannes is
planning to re-use it[2]. Per earlier discussions I'm not sure exactly
what parts he had in mind to keep[3], but I've made an educated guess
here.

Passing CI for this series can be seen at [4].

1. https://lore.kernel.org/git/cover-v6-00.29-00000000000-20220525T094123Z-avarab@gmail.com/
2. https://lore.kernel.org/git/pull.1288.git.1657789234416.gitgitgadget@gmail.com/
3. https://lore.kernel.org/git/220609.86r13xsj00.gmgdl@evledraar.gmail.com/
4. https://github.com/avar/git/actions/runs/2671838770

Ævar Arnfjörð Bjarmason (10):
  CI: run "set -ex" early in ci/lib.sh
  CI: remove more dead Travis CI support
  CI: remove dead "tree skipping" code
  CI: make "$jobname" explicit, remove fallback
  CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS
  CI: consistently use "export" in ci/lib.sh
  CI: export variables via a wrapper
  CI: don't have "git grep" invoke a pager in tree content check
  CI: have "static-analysis" run a "make ci-static-analysis" target
  CI: have "static-analysis" run "check-builtins", not "documentation"

 .github/workflows/main.yml           |  13 +-
 Makefile                             |  32 ++++-
 ci/check-directional-formatting.bash |   2 +-
 ci/lib.sh                            | 200 ++++++++++-----------------
 ci/print-test-failures.sh            |  39 ------
 ci/run-build-and-tests.sh            |  41 +++---
 ci/run-docker-build.sh               |  66 ---------
 ci/run-docker.sh                     |  47 -------
 ci/run-static-analysis.sh            |  34 -----
 ci/run-test-slice.sh                 |   5 -
 ci/test-documentation.sh             |   3 -
 ci/util/extract-trash-dirs.sh        |  50 -------
 shared.mak                           |   1 +
 13 files changed, 135 insertions(+), 398 deletions(-)
 delete mode 100755 ci/run-docker-build.sh
 delete mode 100755 ci/run-docker.sh
 delete mode 100755 ci/run-static-analysis.sh
 delete mode 100755 ci/util/extract-trash-dirs.sh

Range-diff:
 1:  25cb08ad06c !  1:  905a9419267 CI: run "set -ex" early in ci/lib.sh
    @@ Metadata
      ## Commit message ##
         CI: run "set -ex" early in ci/lib.sh
     
    -    Change ci/lib.sh to run "set -ex" before it does anything else, before
    -    this we were omitting a few "export" commands in our trace output that
    -    happened before we enabled tracing.
    +    Change ci/lib.sh to run "set -e" before it does anything else, this
    +    help us to assert that we have no failing commands early on in the
    +    file.
    +
    +    While we're at it let's fix a stray typo in
    +    ecaba2ad4c0 (ci/run-build-and-tests: add some structure to the GitHub
    +    workflow output, 2022-05-21), which seemingly added "#" to the middle
    +    of a line while re-wrapping a comment.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ ci/lib.sh
      
     +# Set 'exit on error' for all CI scripts to let the caller know that
     +# something went wrong.
    -+# Set tracing executed commands, primarily setting environment variables
    -+# and installing dependencies.
    -+set -ex
    ++#
    ++# We'll enable "set -x" below for tracing executed commands, that will
    ++# help to show how environment variables are set and and dependencies
    ++# are installed.
    ++set -e
     +
    - skip_branch_tip_with_tag () {
    - 	# Sometimes, a branch is pushed at the same time the tag that points
    - 	# at the same commit as the tip of the branch is pushed, and building
    -@@ ci/lib.sh: export TERM=${TERM:-dumb}
    - # Clear MAKEFLAGS that may come from the outside world.
    - export MAKEFLAGS=
    + if test true != "$GITHUB_ACTIONS"
    + then
    + 	begin_group () { :; }
    +@@ ci/lib.sh: else
    + 	begin_group "CI setup"
    + fi
      
     -# Set 'exit on error' for all CI scripts to let the caller know that
     -# something went wrong.
    --# Set tracing executed commands, primarily setting environment variables
    --# and installing dependencies.
    --set -ex
    +-#
    +-# We already enabled tracing executed commands earlier. This helps by showing
    +-# how # environment variables are set and and dependencies are installed.
    +-set -e
     -
    - if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
    - then
    - 	CI_TYPE=azure-pipelines
    + skip_branch_tip_with_tag () {
    + 	# Sometimes, a branch is pushed at the same time the tag that points
    + 	# at the same commit as the tip of the branch is pushed, and building
 3:  234e248e831 !  2:  39ef9d2ab6e CI: remove more dead Travis CI support
    @@ ci/lib.sh: then
      	DONT_SKIP_TAGS=t
     @@ ci/lib.sh: then
      	export GIT_PROVE_OPTS="--timer --jobs 10"
    - 	export GIT_TEST_OPTS="--verbose-log -x"
    + 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
      	MAKEFLAGS="$MAKEFLAGS --jobs=10"
     -	test windows != "$CI_OS_NAME" ||
     +	test Windows != "$RUNNER_OS" ||
    @@ ci/run-build-and-tests.sh
     -*) ln -s "$cache_dir/.prove" t/.prove;;
     -esac
     -
    - export MAKE_TARGETS="all test"
    + run_tests=t
      
      case "$jobname" in
     
    @@ ci/run-test-slice.sh
     -*) ln -s "$cache_dir/.prove" t/.prove;;
     -esac
     -
    - make --quiet -C t T="$(cd t &&
    + group "Run tests" make --quiet -C t T="$(cd t &&
      	./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
    - 	tr '\n' ' ')"
    + 	tr '\n' ' ')" ||
     
      ## ci/util/extract-trash-dirs.sh (deleted) ##
     @@
 4:  36a6287ee9b !  3:  10ea16766f1 CI: remove dead "tree skipping" code
    @@ Commit message
         runs for already-tested commits/trees, 2020-10-08).
     
         This code hasn't been used since 4a6e4b96026 (CI: remove Travis CI
    -    support, 2021-11-23), and before that for the retired Azure pipeline
    -    support removed in 6081d3898fe (ci: retire the Azure Pipelines
    -    definition, 2020-04-11).
    +    support, 2021-11-23). At one point it was used for the now-removed
    +    Azure Pipelines support. That support was removed in 6081d3898fe (ci:
    +    retire the Azure Pipelines definition, 2020-04-11), but before in
    +    50b206371d2 (travis: remove the hack to build the Windows job on Azure
    +    Pipelines, 2019-02-28) the "save_good_tree" function had not been
    +    called by the Azure Pipelines code.
     
    -    This change is needed because this and subsequent commits are turning
    -    "ci/lib.sh" into a dumber library that'll only be tasked with setting
    -    variables for CI jobs, or for specific steps within those jobs. Now we
    -    don't need to worry about it potentially skipping the run on its own.
    +    Therefore even for those who'd like to resurrect the
    +    azure-pipelines.yml for occasional ad-hoc use (see the thread(s)
    +    starting at [1]) this change should be OK. This code was really only
    +    used for the now-removed Travis CI.
     
         This change also removes a subtle potential logic error introduced in
         0e7696c64db (ci: disallow directional formatting, 2021-11-04). The
    @@ Commit message
         removed. It appears to have been added to appease the subsequent
         'mkdir -p "$cache_dir"', which wasn't needed in GitHub CI.
     
    -    1. https://lore.kernel.org/git/patch-1.1-eec0a8c3164-20211217T000418Z-avarab@gmail.com/
    +    1. https://lore.kernel.org/git/220609.86r13xsj00.gmgdl@evledraar.gmail.com/
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## ci/lib.sh ##
    -@@ ci/lib.sh: then
    - 	exit 1
    +@@ ci/lib.sh: else
    + 	begin_group "CI setup"
      fi
      
     -skip_branch_tip_with_tag () {
    @@ ci/lib.sh: then
     -	exit 0
     -}
     -
    - check_unignored_build_artifacts ()
    - {
    + check_unignored_build_artifacts () {
      	! git ls-files --other --exclude-standard --error-unmatch \
    + 		-- ':/*' 2>/dev/null ||
     @@ ci/lib.sh: if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
      then
      	CI_TYPE=azure-pipelines
    @@ ci/lib.sh: then
     -	CI_JOB_ID="$GITHUB_RUN_ID"
      	CC="${CC_PACKAGE:-${CC:-gcc}}"
     -	DONT_SKIP_TAGS=t
    --
    --	cache_dir="$HOME/none"
    + 	handle_failed_tests () {
    + 		mkdir -p t/failed-test-artifacts
    + 		echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
    +@@ ci/lib.sh: then
    + 		return 1
    + 	}
      
    +-	cache_dir="$HOME/none"
    +-
      	export GIT_PROVE_OPTS="--timer --jobs 10"
    - 	export GIT_TEST_OPTS="--verbose-log -x"
    + 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
    + 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
     @@ ci/lib.sh: else
      	exit 1
      fi
    @@ ci/lib.sh: else
     -skip_branch_tip_with_tag
     -skip_good_tree
     -
    - export DEVELOPER=1
    - export DEFAULT_TEST_TARGET=prove
    - export GIT_TEST_CLONE_2GB=true
    + if test -z "$jobname"
    + then
    + 	jobname="$CI_OS_NAME-$CC"
     
      ## ci/run-build-and-tests.sh ##
    -@@ ci/run-build-and-tests.sh: esac
    - # start running tests.
    - make $MAKE_TARGETS
    +@@ ci/run-build-and-tests.sh: then
    + 	handle_failed_tests
    + fi
      check_unignored_build_artifacts
     -
     -save_good_tree
     
      ## ci/run-static-analysis.sh ##
    -@@ ci/run-static-analysis.sh: fi
    - 
    - make hdr-check ||
    +@@ ci/run-static-analysis.sh: make hdr-check ||
      exit 1
    + 
    + make check-pot
     -
     -save_good_tree
     
 2:  1be7910c728 !  4:  5d1df78d0de CI: make "$jobname" explicit, remove fallback
    @@ Commit message
     
         This makes all of our CI jobs have an explicit $jobname, and removes
         the special-case of the Windows jobs implicitly setting one (that they
    -    don't currently use). In subsequent commits we'll use this new
    -    $jobname explicitly.
    +    don't currently use).
     
         This is a follow-up to my 25715419bf4 (CI: don't run "make test" twice
         in one job, 2021-11-23), since then all the jobs except the Windows
    @@ Commit message
         ci/lib.sh.
     
         Since we'll need this to run ci/lib.sh it makes sense to move the
    -    check to the top of the file. Subsequent commits will add more such
    -    assertions to it.
    +    check to the top of the file.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ .github/workflows/main.yml: jobs:
          needs: [windows-build]
          strategy:
     @@ .github/workflows/main.yml: jobs:
    -         path: ${{env.FAILED_TEST_ARTIFACTS}}
    -   vs-build:
    -     name: win+VS build
    -+    env:
    -+      jobname: vs-build
          needs: ci-config
          if: needs.ci-config.outputs.enabled == 'yes'
          env:
    ++      jobname: vs-build
    +       NO_PERL: 1
    +       GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
    +     runs-on: windows-latest
     @@ .github/workflows/main.yml: jobs:
              path: artifacts
        vs-test:
    @@ .github/workflows/main.yml: jobs:
     
      ## ci/lib.sh ##
     @@
    - # and installing dependencies.
    - set -ex
    + # are installed.
    + set -e
      
     +# Starting assertions
     +if test -z "$jobname"
    @@ ci/lib.sh
     +	exit 1
     +fi
     +
    - skip_branch_tip_with_tag () {
    - 	# Sometimes, a branch is pushed at the same time the tag that points
    - 	# at the same commit as the tip of the branch is pushed, and building
    -@@ ci/lib.sh: test -n "${DONT_SKIP_TAGS-}" ||
    - skip_branch_tip_with_tag
    - skip_good_tree
    + if test true != "$GITHUB_ACTIONS"
    + then
    + 	begin_group () { :; }
    +@@ ci/lib.sh: else
    + 	exit 1
    + fi
      
     -if test -z "$jobname"
     -then
 5:  66b0a383daf <  -:  ----------- CI: remove unused Azure ci/* code
 6:  d96333c1bd8 !  5:  ee36d4eb2a9 CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS
    @@ Commit message
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## ci/lib.sh ##
    -@@ ci/lib.sh: check_unignored_build_artifacts ()
    +@@ ci/lib.sh: handle_failed_tests () {
      # GitHub Action doesn't set TERM, which is required by tput
      export TERM=${TERM:-dumb}
      
    @@ ci/lib.sh: check_unignored_build_artifacts ()
     -export MAKEFLAGS=
     +export MAKEFLAGS=--jobs=$NPROC
      
    - if test "$GITHUB_ACTIONS" = "true"
    + if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
      then
    - 	CI_TYPE=github-actions
    - 	CC="${CC_PACKAGE:-${CC:-gcc}}"
    +@@ ci/lib.sh: then
    + 	# We are running in Azure Pipelines
    + 	CC="${CC:-gcc}"
    + 
    +-	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
    ++	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
    + 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
    +-	MAKEFLAGS="$MAKEFLAGS --jobs=10"
    + 	test Windows_NT != "$AGENT_OS" ||
    + 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    + elif test true = "$GITHUB_ACTIONS"
    +@@ ci/lib.sh: then
    + 		return 1
    + 	}
      
     -	export GIT_PROVE_OPTS="--timer --jobs 10"
     +	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
    - 	export GIT_TEST_OPTS="--verbose-log -x"
    + 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
     -	MAKEFLAGS="$MAKEFLAGS --jobs=10"
      	test Windows != "$RUNNER_OS" ||
      	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
11:  9655bae9e21 !  6:  7cbf8191105 CI: consistently use "export" in ci/lib.sh
    @@ Metadata
      ## Commit message ##
         CI: consistently use "export" in ci/lib.sh
     
    -    Change the "ci/lib.sh" script co consistently use "export", for
    +    Change the "ci/lib.sh" script to consistently use "export", for
         e.g. MAKEFLAGS we were exporting it, and then assigning to it, let's
         do it the other way around.
     
         Right now this doesn't matter, since we in
         e.g. "ci/install-dependencies.sh" source this file, and don't use
    -    something like "env(1)" to retrieve these variables. But in a
    -    subsequent commit we'll "export" these variables through a wrapper (to
    -    additionally write them to a GitHub CI-specific $GITHUB_ENV
    -    file). This change makes that subsequent change easier to read, as it
    -    won't need to do any control flow refactoring.
    +    something like "env(1)" to retrieve these variables.
    +
    +    But in a subsequent commit we'll "export" these variables through a
    +    wrapper (and to eventually write them to a GitHub CI-specific
    +    $GITHUB_ENV file). This change makes that subsequent change easier to
    +    read, as it won't need to do any control flow refactoring.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
    @@ ci/lib.sh: export TERM=${TERM:-dumb}
     -export MAKEFLAGS=--jobs=$NPROC
     +MAKEFLAGS=--jobs=$NPROC
      
    - if test "$GITHUB_ACTIONS" = "true"
    + if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
      then
     @@ ci/lib.sh: then
    - 	CC="${CC_PACKAGE:-${CC:-gcc}}"
    + 	CC="${CC:-gcc}"
    + 
    + 	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
    +-	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
    ++	GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
    + 	test Windows_NT != "$AGENT_OS" ||
    + 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    ++	export GIT_TEST_OPTS
    + elif test true = "$GITHUB_ACTIONS"
    + then
    + 	CI_TYPE=github-actions
    +@@ ci/lib.sh: then
    + 	}
      
      	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
    --	export GIT_TEST_OPTS="--verbose-log -x"
    -+	GIT_TEST_OPTS="--verbose-log -x"
    +-	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
    ++	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
      	test Windows != "$RUNNER_OS" ||
      	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
     +	export GIT_TEST_OPTS
    @@ ci/lib.sh: then
      	echo "Could not identify CI type" >&2
      	env >&2
     @@ ci/lib.sh: linux-leaks)
    - 	;;
      esac
      
    --MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    -+export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    + MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    ++export MAKEFLAGS
    + 
    + end_group
    + set -x
12:  2da0aa1d40e !  7:  e6805874eb9 CI: export variables via a wrapper
    @@ Commit message
         expose what variables we set in what CI "step", so having only the
         ones we actually use will improve readability.
     
    -    To that end call the "setenv" function with optional "--build",
    -    "--test" and "--all" parameters to indicate whether they're needed for
    -    building, testing or both.
    +    To that end call the "setenv" function with optional "--install",
    +    "--build", "--test" and "--all" parameters to indicate whether they're
    +    needed for building, testing or both.
     
         We currently ignore the "varname" parameter, but will make use of it in a
         subsequent commit. Adding it here reduces the needless churn, as we
    @@ Commit message
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## ci/lib.sh ##
    -@@ ci/lib.sh: then
    - 	exit 1
    +@@ ci/lib.sh: else
    + 	begin_group "CI setup"
      fi
      
     +# Helper functions
    @@ ci/lib.sh: then
     +	while test $# != 0
     +	do
     +		case "$1" in
    ++		--install)
    ++			;;
     +		--build)
     +			;;
     +		--test)
    @@ ci/lib.sh: then
     +	val=$2
     +	shift 2
     +
    -+	eval "export $key=\"$val\""
    ++	eval "$key=\"$val\""
    ++	eval "export $key"
     +}
     +
    - check_unignored_build_artifacts ()
    - {
    + check_unignored_build_artifacts () {
      	! git ls-files --other --exclude-standard --error-unmatch \
    -@@ ci/lib.sh: check_unignored_build_artifacts ()
    + 		-- ':/*' 2>/dev/null ||
    +@@ ci/lib.sh: handle_failed_tests () {
      }
      
      # GitHub Action doesn't set TERM, which is required by tput
     -export TERM=${TERM:-dumb}
    -+setenv TERM ${TERM:-dumb}
    ++setenv --all TERM ${TERM:-dumb}
      
      # How many jobs to run in parallel?
      NPROC=10
     @@ ci/lib.sh: then
    + 	# We are running in Azure Pipelines
    + 	CC="${CC:-gcc}"
    + 
    +-	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
    ++	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC --state=failed,slow,save"
    + 	GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
    + 	test Windows_NT != "$AGENT_OS" ||
    + 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
    +-	export GIT_TEST_OPTS
    ++	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
    + elif test true = "$GITHUB_ACTIONS"
    + then
      	CI_TYPE=github-actions
    - 	CC="${CC_PACKAGE:-${CC:-gcc}}"
    +@@ ci/lib.sh: then
    + 		return 1
    + 	}
      
     -	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
     +	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
    - 	GIT_TEST_OPTS="--verbose-log -x"
    + 	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
      	test Windows != "$RUNNER_OS" ||
      	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
     -	export GIT_TEST_OPTS
    @@ ci/lib.sh: ubuntu-latest)
      	fi
      
     -	export GIT_TEST_HTTPD=true
    ++	setenv --test GIT_TEST_HTTPD true
    + 
    + 	# The Linux build installs the defined dependency versions below.
    + 	# The OS X build installs much more recent versions, whichever
    + 	# were recorded in the Homebrew database upon creating the OS X
    + 	# image.
    + 	# Keep that in mind when you encounter a broken OS X build!
    +-	export LINUX_P4_VERSION="16.2"
    +-	export LINUX_GIT_LFS_VERSION="1.5.2"
    ++	setenv --install LINUX_P4_VERSION "16.2"
    ++	setenv --install LINUX_GIT_LFS_VERSION "1.5.2"
    + 
    +-	P4_PATH="$HOME/custom/p4"
    +-	GIT_LFS_PATH="$HOME/custom/git-lfs"
    +-	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
    ++	setenv --install P4_PATH "$HOME/custom/p4"
    ++	setenv --install GIT_LFS_PATH "$HOME/custom/git-lfs"
    ++	setenv --all PATH "$GIT_LFS_PATH:$P4_PATH:$PATH"
     +	setenv --test GIT_TEST_HTTPD true
      	;;
      macos-latest)
    @@ ci/lib.sh: linux-musl)
      	;;
      esac
      
    --export MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    -+setenv --all MAKEFLAGS "$MAKEFLAGS CC=${CC:-cc}"
    + MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
    +-export MAKEFLAGS
    ++setenv --all MAKEFLAGS "$MAKEFLAGS"
    + 
    + end_group
    + set -x
     
      ## ci/run-build-and-tests.sh ##
    -@@ ci/run-build-and-tests.sh: export MAKE_TARGETS="all test"
    +@@ ci/run-build-and-tests.sh: run_tests=t
      
      case "$jobname" in
      linux-gcc)
    @@ ci/run-build-and-tests.sh: export MAKE_TARGETS="all test"
     -	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
     -	export GIT_TEST_MULTI_PACK_INDEX=1
     -	export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
    --	export GIT_TEST_ADD_I_USE_BUILTIN=1
    +-	export GIT_TEST_ADD_I_USE_BUILTIN=0
     -	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
     -	export GIT_TEST_WRITE_REV_INDEX=1
     -	export GIT_TEST_CHECKOUT_WORKERS=2
    @@ ci/run-build-and-tests.sh: export MAKE_TARGETS="all test"
     +	setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1
     +	setenv --test GIT_TEST_MULTI_PACK_INDEX 1
     +	setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1
    -+	setenv --test GIT_TEST_ADD_I_USE_BUILTIN 1
    ++	setenv --test GIT_TEST_ADD_I_USE_BUILTIN 0
     +	setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master
     +	setenv --test GIT_TEST_WRITE_REV_INDEX 1
     +	setenv --test GIT_TEST_CHECKOUT_WORKERS 2
    @@ ci/run-build-and-tests.sh: export MAKE_TARGETS="all test"
      	# built.
     -	export DEVOPTS=pedantic
     +	setenv --build DEVOPTS pedantic
    - 	export MAKE_TARGETS=all
    + 	run_tests=
      	;;
      esac
 7:  d0780ab4a96 =  8:  8bcaeb1dd79 CI: don't have "git grep" invoke a pager in tree content check
 8:  84e368736e2 !  9:  95b9c3797c0 CI: have "static-analysis" run a "make ci-static-analysis" target
    @@ Commit message
     
         But without those improvements being able to easily run a 1=1
         equivalent of the checks we do in CI during a local build is already
    -    an improvement. Subsequent commits will expand on this ability.
    +    an improvement.
     
         This change will also make the CI check faster, since we can take
    -    advantage of parallelism. The "make coccicheck" takes a very long
    -    time, and at the tail end we might only have one job pegging 100% of
    -    one CPU core for a long time. Now any extra cores will be free to run
    -    the rest of the targets under "ci-static-analysis".
    +    advantage of parallelism across these "make" invocations. The "make
    +    coccicheck" command in particular takes a long to finish its last job,
    +    at the end we might only have one job pegging 100% of one CPU
    +    core. Now any extra cores will be free to run the rest of the targets
    +    under "ci-static-analysis".
    +
    +    Because we're now going to invoke "make" directly from the CI recipe
    +    we'll need to amend the new "setenv" wrapper to write the "MAKEFLAGS"
    +    and other variables to "$GITHUB_ENV".
    +
    +    In my testing the "static-analysis" job runs in just over 10 minutes
    +    without this change, but this change cuts just over a minute off the
    +    runtime.
     
         Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
     
      ## .github/workflows/main.yml ##
     @@ .github/workflows/main.yml: jobs:
    +     runs-on: ubuntu-18.04
          steps:
          - uses: actions/checkout@v2
    ++    - run: ci/lib.sh
    ++      env:
    ++        NO_CI_GROUPS: 1
          - run: ci/install-dependencies.sh
     -    - run: ci/run-static-analysis.sh
     -    - run: ci/check-directional-formatting.bash
    @@ Makefile: $(FUZZ_PROGRAMS): all
     +ci-static-analysis: ci-check-directional-formatting
     +ci-static-analysis: check-coccicheck
     +ci-static-analysis: hdr-check
    ++ci-static-analysis: check-pot
    +
    + ## ci/lib.sh ##
    +@@ ci/lib.sh: then
    + 	exit 1
    + fi
    + 
    +-if test true != "$GITHUB_ACTIONS"
    ++if test true != "$GITHUB_ACTIONS" || test -n "$NO_CI_GROUPS"
    + then
    + 	begin_group () { :; }
    + 	end_group () { :; }
    +@@ ci/lib.sh: setenv () {
    + 
    + 	eval "$key=\"$val\""
    + 	eval "export $key"
    ++	eval "export $key=\"$val\""
    ++	if test -n "$GITHUB_ENV"
    ++	then
    ++		echo "$key=$val" >>"$GITHUB_ENV"
    ++	fi
    + }
    + 
    + check_unignored_build_artifacts () {
     
      ## ci/run-static-analysis.sh (deleted) ##
     @@
    @@ ci/run-static-analysis.sh (deleted)
     -
     -make hdr-check ||
     -exit 1
    +-
    +-make check-pot
     
      ## shared.mak ##
     @@ shared.mak: ifndef V
    - 	QUIET_HDR      = @echo '   ' HDR $(<:hcc=h);
    - 	QUIET_RC       = @echo '   ' RC $@;
    - 	QUIET_SPATCH   = @echo '   ' SPATCH $<;
    + ## Used in "Makefile"
    + 	QUIET_CC       = @echo '   ' CC $@;
    + 	QUIET_AR       = @echo '   ' AR $@;
     +	QUIET_CHECK    = @echo '   ' CHECK $@;
    - 
    - ## Used in "Documentation/Makefile"
    - 	QUIET_ASCIIDOC	= @echo '   ' ASCIIDOC $@;
    + 	QUIET_LINK     = @echo '   ' LINK $@;
    + 	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
    + 	QUIET_LNCP     = @echo '   ' LN/CP $@;
 9:  1a9c98b909c ! 10:  5ef362505c1 CI: have "static-analysis" run "check-builtins", not "documentation"
    @@ Makefile: ci-check-directional-formatting:
     +ci-static-analysis: check-builtins
      ci-static-analysis: check-coccicheck
      ci-static-analysis: hdr-check
    + ci-static-analysis: check-pot
     
      ## ci/test-documentation.sh ##
     @@ ci/test-documentation.sh: filter_log () {
10:  e208a9ab1e2 <  -:  ----------- CI: move p4 and git-lfs variables to ci/install-dependencies.sh
13:  6e97633652e <  -:  ----------- CI: remove "run-build-and-tests.sh", run "make [test]" directly
14:  0488e29859b <  -:  ----------- CI: make ci/{lib,install-dependencies}.sh POSIX-compatible
15:  f7ac6c33044 <  -:  ----------- CI: check ignored unignored build artifacts in "win[+VS] build" too
16:  f89346f11eb <  -:  ----------- CI: invoke "make artifacts-tar" directly in windows-build
17:  7fa9c69e3ca <  -:  ----------- CI: split up and reduce "ci/test-documentation.sh"
18:  7925b2610fc <  -:  ----------- CI: combine ci/install{,-docker}-dependencies.sh
19:  b00abc07637 <  -:  ----------- CI: move "env" definitions into ci/lib.sh
20:  bb5960355fe <  -:  ----------- ci/run-test-slice.sh: replace shelling out with "echo"
21:  1eeb2e8e7f0 <  -:  ----------- CI: pre-select test slice in Windows & VS tests
22:  39979d9887d <  -:  ----------- CI: only invoke ci/lib.sh as "steps" in main.yml
23:  e2b3bf032d6 <  -:  ----------- CI: narrow down variable definitions in --build and --test
24:  c6f47d52823 <  -:  ----------- CI: add more variables to MAKEFLAGS, except under vs-build
25:  377460e8024 <  -:  ----------- CI: set CC in MAKEFLAGS directly, don't add it to the environment
26:  98e320d5e67 <  -:  ----------- CI: set SANITIZE=leak in MAKEFLAGS directly
27:  282b7c89a4f <  -:  ----------- CI: set PYTHON_PATH setting for osx-{clang,gcc} into "$jobname" case
28:  7a208927819 <  -:  ----------- CI: don't use "set -x" in "ci/lib.sh" output
29:  4d8a8f3be07 <  -:  ----------- CI: make it easy to use ci/*.sh outside of CI
-- 
2.37.1.996.g651fc6e809f

[PATCH 01/10] CI: run "set -ex" early in ci/lib.sh

[Ævar Arnfjörð Bjarmason]

Change ci/lib.sh to run "set -e" before it does anything else, this
help us to assert that we have no failing commands early on in the
file.

While we're at it let's fix a stray typo in
ecaba2ad4c0 (ci/run-build-and-tests: add some structure to the GitHub
workflow output, 2022-05-21), which seemingly added "#" to the middle
of a line while re-wrapping a comment.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh | 15 ++++++++-------
 1 file changed, 8 insertions(+), 7 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index f095519f8db..4f12bcaabb9 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -1,5 +1,13 @@
 # Library of functions shared by all CI scripts
 
+# Set 'exit on error' for all CI scripts to let the caller know that
+# something went wrong.
+#
+# We'll enable "set -x" below for tracing executed commands, that will
+# help to show how environment variables are set and and dependencies
+# are installed.
+set -e
+
 if test true != "$GITHUB_ACTIONS"
 then
 	begin_group () { :; }
@@ -44,13 +52,6 @@ else
 	begin_group "CI setup"
 fi
 
-# Set 'exit on error' for all CI scripts to let the caller know that
-# something went wrong.
-#
-# We already enabled tracing executed commands earlier. This helps by showing
-# how # environment variables are set and and dependencies are installed.
-set -e
-
 skip_branch_tip_with_tag () {
 	# Sometimes, a branch is pushed at the same time the tag that points
 	# at the same commit as the tip of the branch is pushed, and building
-- 
2.37.1.996.g651fc6e809f

[PATCH 02/10] CI: remove more dead Travis CI support

[Ævar Arnfjörð Bjarmason]

Remove code that I missed in 4a6e4b96026 (CI: remove Travis CI
support, 2021-11-23). This code was only called from or used by the
now-removed .travis.yml, or needed by the Travis CI environment.

For the symlinking in ci/run-build-and-tests.sh: Back when
3c93b829205 (travis-ci: build Git during the 'script' phase,
2018-01-08) and 4b060a4d973 (ci: use a junction on Windows instead of
a symlink, 2019-01-27) adjusted this "ln" command, the Windows build
would use ci/run-build-and-tests.sh.

As seen in 889cacb6897 (ci: configure GitHub Actions for CI/PR,
2020-04-11) the current windows build uses a different entry point
under the GitHub CI, which doesn't use this .prove caching. Namely
"ci/run-test-slice.sh".

We can be certain that it's never used in "ci/run-test-slice.sh"
because to have a ".prove" file we'd need to use "--state=save", which
we only do in the dead Azure codepath in ci/lib.sh. If it were used it
would do the wrong thing, because the different test slices would each
try to clobber the same "t/.prove" file.

If a subsequent run then used the "--state=failed,slow,save" it would
defeat the purpose of "ci/run-test-slice.sh", since all slices would
then run all tests. I.e. behavior of prove's "--state" options is to
select tests to run from the provided "--state" file, in addition to
those specified on the command-line.

For ci/run-docker{,-build}.sh: It was likewise last referenced in the
.travis.yml removed in my 4a6e4b96026. The current "dockerized" run in
".github/workflows/main.yml" calls the same entry points as the main
"regular" job.

For "ci/print-test-failures.sh" the creation of the tarball added in
aea8879a6ac (travis-ci: include the trash directories of failed tests
in the trace log, 2018-08-01) has been dead since my 4a6e4b96026,
which removed the fall-through case from the "case/esac" statement.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh                     | 10 ++----
 ci/print-test-failures.sh     | 39 ---------------------
 ci/run-build-and-tests.sh     |  5 ---
 ci/run-docker-build.sh        | 66 -----------------------------------
 ci/run-docker.sh              | 47 -------------------------
 ci/run-test-slice.sh          |  5 ---
 ci/util/extract-trash-dirs.sh | 50 --------------------------
 7 files changed, 2 insertions(+), 220 deletions(-)
 delete mode 100755 ci/run-docker-build.sh
 delete mode 100755 ci/run-docker.sh
 delete mode 100755 ci/util/extract-trash-dirs.sh

diff --git a/ci/lib.sh b/ci/lib.sh
index 4f12bcaabb9..f32c588f6ad 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -148,9 +148,6 @@ then
 	CI_COMMIT="$BUILD_SOURCEVERSION"
 	CI_JOB_ID="$BUILD_BUILDID"
 	CI_JOB_NUMBER="$BUILD_BUILDNUMBER"
-	CI_OS_NAME="$(echo "$AGENT_OS" | tr A-Z a-z)"
-	test darwin != "$CI_OS_NAME" || CI_OS_NAME=osx
-	CI_REPO_SLUG="$(expr "$BUILD_REPOSITORY_URI" : '.*/\([^/]*/[^/]*\)$')"
 	CC="${CC:-gcc}"
 
 	# use a subdirectory of the cache dir (because the file share is shared
@@ -160,16 +157,13 @@ then
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
-	test windows_nt != "$CI_OS_NAME" ||
+	test Windows_NT != "$AGENT_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 elif test true = "$GITHUB_ACTIONS"
 then
 	CI_TYPE=github-actions
 	CI_BRANCH="$GITHUB_REF"
 	CI_COMMIT="$GITHUB_SHA"
-	CI_OS_NAME="$(echo "$RUNNER_OS" | tr A-Z a-z)"
-	test macos != "$CI_OS_NAME" || CI_OS_NAME=osx
-	CI_REPO_SLUG="$GITHUB_REPOSITORY"
 	CI_JOB_ID="$GITHUB_RUN_ID"
 	CC="${CC_PACKAGE:-${CC:-gcc}}"
 	DONT_SKIP_TAGS=t
@@ -200,7 +194,7 @@ then
 	export GIT_PROVE_OPTS="--timer --jobs 10"
 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
-	test windows != "$CI_OS_NAME" ||
+	test Windows != "$RUNNER_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
 	echo "Could not identify CI type" >&2
diff --git a/ci/print-test-failures.sh b/ci/print-test-failures.sh
index 57277eefcd0..740ebb7c529 100755
--- a/ci/print-test-failures.sh
+++ b/ci/print-test-failures.sh
@@ -16,15 +16,6 @@ then
 	exit
 fi
 
-case "$jobname" in
-osx-clang|osx-gcc)
-	# base64 in OSX doesn't wrap its output at 76 columns by
-	# default, but prints a single, very long line.
-	base64_opts="-b 76"
-	;;
-esac
-
-combined_trash_size=0
 for TEST_EXIT in test-results/*.exit
 do
 	if [ "$(cat "$TEST_EXIT")" != "0" ]
@@ -42,47 +33,17 @@ do
 		azure-pipelines)
 			mkdir -p failed-test-artifacts
 			mv "$trash_dir" failed-test-artifacts
-			continue
 			;;
 		github-actions)
 			mkdir -p failed-test-artifacts
 			echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
 			cp "${TEST_EXIT%.exit}.out" failed-test-artifacts/
 			tar czf failed-test-artifacts/"$test_name".trash.tar.gz "$trash_dir"
-			continue
 			;;
 		*)
 			echo "Unhandled CI type: $CI_TYPE" >&2
 			exit 1
 			;;
 		esac
-		trash_tgz_b64="trash.$test_name.base64"
-		if [ -d "$trash_dir" ]
-		then
-			tar czp "$trash_dir" |base64 $base64_opts >"$trash_tgz_b64"
-
-			trash_size=$(wc -c <"$trash_tgz_b64")
-			if [ $trash_size -gt 1048576 ]
-			then
-				# larger than 1MB
-				echo "$(tput setaf 1)Didn't include the trash directory of '$test_name' in the trace log, it's too big$(tput sgr0)"
-				continue
-			fi
-
-			new_combined_trash_size=$(($combined_trash_size + $trash_size))
-			if [ $new_combined_trash_size -gt 1048576 ]
-			then
-				echo "$(tput setaf 1)Didn't include the trash directory of '$test_name' in the trace log, there is plenty of trash in there already.$(tput sgr0)"
-				continue
-			fi
-			combined_trash_size=$new_combined_trash_size
-
-			# DO NOT modify these two 'echo'-ed strings below
-			# without updating 'ci/util/extract-trash-dirs.sh'
-			# as well.
-			echo "$(tput setaf 1)Start of trash directory of '$test_name':$(tput sgr0)"
-			cat "$trash_tgz_b64"
-			echo "$(tput setaf 1)End of trash directory of '$test_name'$(tput sgr0)"
-		fi
 	fi
 done
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 8ebff425967..9a5032e6a77 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -5,11 +5,6 @@
 
 . ${0%/*}/lib.sh
 
-case "$CI_OS_NAME" in
-windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
-*) ln -s "$cache_dir/.prove" t/.prove;;
-esac
-
 run_tests=t
 
 case "$jobname" in
diff --git a/ci/run-docker-build.sh b/ci/run-docker-build.sh
deleted file mode 100755
index 6cd832efb9c..00000000000
--- a/ci/run-docker-build.sh
+++ /dev/null
@@ -1,66 +0,0 @@
-#!/bin/sh
-#
-# Build and test Git inside container
-#
-# Usage:
-#   run-docker-build.sh <host-user-id>
-#
-
-set -ex
-
-if test $# -ne 1 || test -z "$1"
-then
-	echo >&2 "usage: run-docker-build.sh <host-user-id>"
-	exit 1
-fi
-
-case "$jobname" in
-linux32)
-	switch_cmd="linux32 --32bit i386"
-	;;
-linux-musl)
-	switch_cmd=
-	useradd () { adduser -D "$@"; }
-	;;
-*)
-	exit 1
-	;;
-esac
-
-"${0%/*}/install-docker-dependencies.sh"
-
-# If this script runs inside a docker container, then all commands are
-# usually executed as root. Consequently, the host user might not be
-# able to access the test output files.
-# If a non 0 host user id is given, then create a user "ci" with that
-# user id to make everything accessible to the host user.
-HOST_UID=$1
-if test $HOST_UID -eq 0
-then
-	# Just in case someone does want to run the test suite as root.
-	CI_USER=root
-else
-	CI_USER=ci
-	if test "$(id -u $CI_USER 2>/dev/null)" = $HOST_UID
-	then
-		echo "user '$CI_USER' already exists with the requested ID $HOST_UID"
-	else
-		useradd -u $HOST_UID $CI_USER
-	fi
-fi
-
-# Build and test
-command $switch_cmd su -m -l $CI_USER -c "
-	set -ex
-	export DEVELOPER='$DEVELOPER'
-	export DEFAULT_TEST_TARGET='$DEFAULT_TEST_TARGET'
-	export GIT_PROVE_OPTS='$GIT_PROVE_OPTS'
-	export GIT_TEST_OPTS='$GIT_TEST_OPTS'
-	export GIT_TEST_CLONE_2GB='$GIT_TEST_CLONE_2GB'
-	export MAKEFLAGS='$MAKEFLAGS'
-	export cache_dir='$cache_dir'
-	cd /usr/src/git
-	test -n '$cache_dir' && ln -s '$cache_dir/.prove' t/.prove
-	make
-	make test
-"
diff --git a/ci/run-docker.sh b/ci/run-docker.sh
deleted file mode 100755
index af89d1624a4..00000000000
--- a/ci/run-docker.sh
+++ /dev/null
@@ -1,47 +0,0 @@
-#!/bin/sh
-#
-# Download and run Docker image to build and test Git
-#
-
-. ${0%/*}/lib.sh
-
-case "$jobname" in
-linux32)
-	CI_CONTAINER="daald/ubuntu32:xenial"
-	;;
-linux-musl)
-	CI_CONTAINER=alpine
-	;;
-*)
-	exit 1
-	;;
-esac
-
-docker pull "$CI_CONTAINER"
-
-# Use the following command to debug the docker build locally:
-# <host-user-id> must be 0 if podman is used as drop-in replacement for docker
-# $ docker run -itv "${PWD}:/usr/src/git" --entrypoint /bin/sh "$CI_CONTAINER"
-# root@container:/# export jobname=<jobname>
-# root@container:/# /usr/src/git/ci/run-docker-build.sh <host-user-id>
-
-container_cache_dir=/tmp/container-cache
-
-docker run \
-	--interactive \
-	--env DEVELOPER \
-	--env DEFAULT_TEST_TARGET \
-	--env GIT_PROVE_OPTS \
-	--env GIT_TEST_OPTS \
-	--env GIT_TEST_CLONE_2GB \
-	--env MAKEFLAGS \
-	--env jobname \
-	--env cache_dir="$container_cache_dir" \
-	--volume "${PWD}:/usr/src/git" \
-	--volume "$cache_dir:$container_cache_dir" \
-	"$CI_CONTAINER" \
-	/usr/src/git/ci/run-docker-build.sh $(id -u $USER)
-
-check_unignored_build_artifacts
-
-save_good_tree
diff --git a/ci/run-test-slice.sh b/ci/run-test-slice.sh
index a3c67956a8d..312a3ef73fe 100755
--- a/ci/run-test-slice.sh
+++ b/ci/run-test-slice.sh
@@ -5,11 +5,6 @@
 
 . ${0%/*}/lib.sh
 
-case "$CI_OS_NAME" in
-windows*) cmd //c mklink //j t\\.prove "$(cygpath -aw "$cache_dir/.prove")";;
-*) ln -s "$cache_dir/.prove" t/.prove;;
-esac
-
 group "Run tests" make --quiet -C t T="$(cd t &&
 	./helper/test-tool path-utils slice-tests "$1" "$2" t[0-9]*.sh |
 	tr '\n' ' ')" ||
diff --git a/ci/util/extract-trash-dirs.sh b/ci/util/extract-trash-dirs.sh
deleted file mode 100755
index 8e67bec21a2..00000000000
--- a/ci/util/extract-trash-dirs.sh
+++ /dev/null
@@ -1,50 +0,0 @@
-#!/bin/sh
-
-error () {
-	echo >&2 "error: $@"
-	exit 1
-}
-
-find_embedded_trash () {
-	while read -r line
-	do
-		case "$line" in
-		*Start\ of\ trash\ directory\ of\ \'t[0-9][0-9][0-9][0-9]-*\':*)
-			test_name="${line#*\'}"
-			test_name="${test_name%\'*}"
-
-			return 0
-		esac
-	done
-
-	return 1
-}
-
-extract_embedded_trash () {
-	while read -r line
-	do
-		case "$line" in
-		*End\ of\ trash\ directory\ of\ \'$test_name\'*)
-			return
-			;;
-		*)
-			printf '%s\n' "$line"
-			;;
-		esac
-	done
-
-	error "unexpected end of input"
-}
-
-# Raw logs from Linux build jobs have CRLF line endings, while OSX
-# build jobs mostly have CRCRLF, except an odd line every now and
-# then that has CRCRCRLF.  'base64 -d' from 'coreutils' doesn't like
-# CRs and complains about "invalid input", so remove all CRs at the
-# end of lines.
-sed -e 's/\r*$//' | \
-while find_embedded_trash
-do
-	echo "Extracting trash directory of '$test_name'"
-
-	extract_embedded_trash |base64 -d |tar xzp
-done
-- 
2.37.1.996.g651fc6e809f

[PATCH 03/10] CI: remove dead "tree skipping" code

[Ævar Arnfjörð Bjarmason]

Remove the code related to skipping CI runs if the tree was
successfully tested before. With the GitHub CI this is done with the
"ci-config" step, see e76eec35540 (ci: allow per-branch config for
GitHub Actions, 2020-05-07) and 7d78d5fc1a9 (ci: skip GitHub workflow
runs for already-tested commits/trees, 2020-10-08).

This code hasn't been used since 4a6e4b96026 (CI: remove Travis CI
support, 2021-11-23). At one point it was used for the now-removed
Azure Pipelines support. That support was removed in 6081d3898fe (ci:
retire the Azure Pipelines definition, 2020-04-11), but before in
50b206371d2 (travis: remove the hack to build the Windows job on Azure
Pipelines, 2019-02-28) the "save_good_tree" function had not been
called by the Azure Pipelines code.

Therefore even for those who'd like to resurrect the
azure-pipelines.yml for occasional ad-hoc use (see the thread(s)
starting at [1]) this change should be OK. This code was really only
used for the now-removed Travis CI.

This change also removes a subtle potential logic error introduced in
0e7696c64db (ci: disallow directional formatting, 2021-11-04). The
"ci/check-directional-formatting.bash" script would have been made to
run after the "save_good_tree" in invoked in
"ci/run-static-analysis.sh". So if e.g. the "make coccicheck" failed
we'd still mark the tree as good.

I.e. the addition of "ci/check-directional-formatting.bash" didn't
take into account that the various users of the "save_good_tree"
function in ci/*.sh made a hard assumption that they're the only
ci/*.sh script that's being run (and they use "set -e").

This is entirely academic since we weren't actually running this code,
but is something to be careful of if anyone ever needs to resurrect
parts of this.

Finally, the cache_dir="$HOME/none" added in a3f2eec862b (ci/lib:
allow running in GitHub Actions, 2020-04-08) for the GitHub CI can be
removed. It appears to have been added to appease the subsequent
'mkdir -p "$cache_dir"', which wasn't needed in GitHub CI.

1. https://lore.kernel.org/git/220609.86r13xsj00.gmgdl@evledraar.gmail.com/

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh                 | 91 ---------------------------------------
 ci/run-build-and-tests.sh |  2 -
 ci/run-static-analysis.sh |  2 -
 ci/test-documentation.sh  |  2 -
 4 files changed, 97 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index f32c588f6ad..fcb79c83087 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -52,75 +52,6 @@ else
 	begin_group "CI setup"
 fi
 
-skip_branch_tip_with_tag () {
-	# Sometimes, a branch is pushed at the same time the tag that points
-	# at the same commit as the tip of the branch is pushed, and building
-	# both at the same time is a waste.
-	#
-	# When the build is triggered by a push to a tag, $CI_BRANCH will
-	# have that tagname, e.g. v2.14.0.  Let's see if $CI_BRANCH is
-	# exactly at a tag, and if so, if it is different from $CI_BRANCH.
-	# That way, we can tell if we are building the tip of a branch that
-	# is tagged and we can skip the build because we won't be skipping a
-	# build of a tag.
-
-	if TAG=$(git describe --exact-match "$CI_BRANCH" 2>/dev/null) &&
-		test "$TAG" != "$CI_BRANCH"
-	then
-		echo "$(tput setaf 2)Tip of $CI_BRANCH is exactly at $TAG$(tput sgr0)"
-		exit 0
-	fi
-}
-
-# Save some info about the current commit's tree, so we can skip the build
-# job if we encounter the same tree again and can provide a useful info
-# message.
-save_good_tree () {
-	echo "$(git rev-parse $CI_COMMIT^{tree}) $CI_COMMIT $CI_JOB_NUMBER $CI_JOB_ID" >>"$good_trees_file"
-	# limit the file size
-	tail -1000 "$good_trees_file" >"$good_trees_file".tmp
-	mv "$good_trees_file".tmp "$good_trees_file"
-}
-
-# Skip the build job if the same tree has already been built and tested
-# successfully before (e.g. because the branch got rebased, changing only
-# the commit messages).
-skip_good_tree () {
-	if test true = "$GITHUB_ACTIONS"
-	then
-		return
-	fi
-
-	if ! good_tree_info="$(grep "^$(git rev-parse $CI_COMMIT^{tree}) " "$good_trees_file")"
-	then
-		# Haven't seen this tree yet, or no cached good trees file yet.
-		# Continue the build job.
-		return
-	fi
-
-	echo "$good_tree_info" | {
-		read tree prev_good_commit prev_good_job_number prev_good_job_id
-
-		if test "$CI_JOB_ID" = "$prev_good_job_id"
-		then
-			cat <<-EOF
-			$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)
-			This commit has already been built and tested successfully by this build job.
-			To force a re-build delete the branch's cache and then hit 'Restart job'.
-			EOF
-		else
-			cat <<-EOF
-			$(tput setaf 2)Skipping build job for commit $CI_COMMIT.$(tput sgr0)
-			This commit's tree has already been built and tested successfully in build job $prev_good_job_number for commit $prev_good_commit.
-			The log of that build job is available at $SYSTEM_TASKDEFINITIONSURI$SYSTEM_TEAMPROJECT/_build/results?buildId=$prev_good_job_id
-			To force a re-build delete the branch's cache and then hit 'Restart job'.
-			EOF
-		fi
-	}
-
-	exit 0
-}
-
 check_unignored_build_artifacts () {
 	! git ls-files --other --exclude-standard --error-unmatch \
 		-- ':/*' 2>/dev/null ||
@@ -144,16 +75,8 @@ if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
 	CI_TYPE=azure-pipelines
 	# We are running in Azure Pipelines
-	CI_BRANCH="$BUILD_SOURCEBRANCH"
-	CI_COMMIT="$BUILD_SOURCEVERSION"
-	CI_JOB_ID="$BUILD_BUILDID"
-	CI_JOB_NUMBER="$BUILD_BUILDNUMBER"
 	CC="${CC:-gcc}"
 
-	# use a subdirectory of the cache dir (because the file share is shared
-	# among *all* phases)
-	cache_dir="$HOME/test-cache/$SYSTEM_PHASENAME"
-
 	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
@@ -162,11 +85,7 @@ then
 elif test true = "$GITHUB_ACTIONS"
 then
 	CI_TYPE=github-actions
-	CI_BRANCH="$GITHUB_REF"
-	CI_COMMIT="$GITHUB_SHA"
-	CI_JOB_ID="$GITHUB_RUN_ID"
 	CC="${CC_PACKAGE:-${CC:-gcc}}"
-	DONT_SKIP_TAGS=t
 	handle_failed_tests () {
 		mkdir -p t/failed-test-artifacts
 		echo "FAILED_TEST_ARTIFACTS=t/failed-test-artifacts" >>$GITHUB_ENV
@@ -189,8 +108,6 @@ then
 		return 1
 	}
 
-	cache_dir="$HOME/none"
-
 	export GIT_PROVE_OPTS="--timer --jobs 10"
 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
 	MAKEFLAGS="$MAKEFLAGS --jobs=10"
@@ -202,14 +119,6 @@ else
 	exit 1
 fi
 
-good_trees_file="$cache_dir/good-trees"
-
-mkdir -p "$cache_dir"
-
-test -n "${DONT_SKIP_TAGS-}" ||
-skip_branch_tip_with_tag
-skip_good_tree
-
 if test -z "$jobname"
 then
 	jobname="$CI_OS_NAME-$CC"
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 9a5032e6a77..1a89a4967bf 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -47,5 +47,3 @@ then
 	handle_failed_tests
 fi
 check_unignored_build_artifacts
-
-save_good_tree
diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
index 0d51e5ce0e7..faae31f0078 100755
--- a/ci/run-static-analysis.sh
+++ b/ci/run-static-analysis.sh
@@ -30,5 +30,3 @@ make hdr-check ||
 exit 1
 
 make check-pot
-
-save_good_tree
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
index de41888430a..9e0652c30dd 100755
--- a/ci/test-documentation.sh
+++ b/ci/test-documentation.sh
@@ -41,5 +41,3 @@ grep '<meta name="generator" content="Asciidoctor ' Documentation/git.html
 
 rm -f stdout.log stderr.log stderr.raw
 check_unignored_build_artifacts
-
-save_good_tree
-- 
2.37.1.996.g651fc6e809f

[PATCH 04/10] CI: make "$jobname" explicit, remove fallback

[Ævar Arnfjörð Bjarmason]

Remove the $jobname fallback last adjusted in b011fabd6e0 (ci/lib.sh:
encapsulate Travis-specific things, 2019-01-27).

This makes all of our CI jobs have an explicit $jobname, and removes
the special-case of the Windows jobs implicitly setting one (that they
don't currently use).

This is a follow-up to my 25715419bf4 (CI: don't run "make test" twice
in one job, 2021-11-23), since then all the jobs except the Windows
jobs have had an explicit $jobname.

In the case of the Windows jobs we were using a jobname of windows-gcc
for both, due to picking "gcc" as a default fallback for "$CC" in
ci/lib.sh.

Since we'll need this to run ci/lib.sh it makes sense to move the
check to the top of the file.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 .github/workflows/main.yml |  7 +++++++
 ci/lib.sh                  | 12 +++++++-----
 2 files changed, 14 insertions(+), 5 deletions(-)

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index cd1f52692a5..8b7697df9fb 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -79,6 +79,8 @@ jobs:
 
   windows-build:
     name: win build
+    env:
+      jobname: windows-build
     needs: ci-config
     if: needs.ci-config.outputs.enabled == 'yes'
     runs-on: windows-latest
@@ -100,6 +102,8 @@ jobs:
         path: artifacts
   windows-test:
     name: win test
+    env:
+      jobname: windows-test
     runs-on: windows-latest
     needs: [windows-build]
     strategy:
@@ -134,6 +138,7 @@ jobs:
     needs: ci-config
     if: needs.ci-config.outputs.enabled == 'yes'
     env:
+      jobname: vs-build
       NO_PERL: 1
       GIT_CONFIG_PARAMETERS: "'user.name=CI' 'user.email=ci@git'"
     runs-on: windows-latest
@@ -183,6 +188,8 @@ jobs:
         path: artifacts
   vs-test:
     name: win+VS test
+    env:
+      jobname: vs-test
     runs-on: windows-latest
     needs: vs-build
     strategy:
diff --git a/ci/lib.sh b/ci/lib.sh
index fcb79c83087..969ba2ff0d0 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -8,6 +8,13 @@
 # are installed.
 set -e
 
+# Starting assertions
+if test -z "$jobname"
+then
+	echo "error: must set a CI jobname in the environment" >&2
+	exit 1
+fi
+
 if test true != "$GITHUB_ACTIONS"
 then
 	begin_group () { :; }
@@ -119,11 +126,6 @@ else
 	exit 1
 fi
 
-if test -z "$jobname"
-then
-	jobname="$CI_OS_NAME-$CC"
-fi
-
 export DEVELOPER=1
 export DEFAULT_TEST_TARGET=prove
 export GIT_TEST_CLONE_2GB=true
-- 
2.37.1.996.g651fc6e809f

[PATCH 05/10] CI/lib.sh: stop adding leading whitespace to $MAKEFLAGS

[Ævar Arnfjörð Bjarmason]

In a8c51f77d16 (ci: clear and mark MAKEFLAGS exported just once,
2019-02-07) the assignment of MAKEFLAGS was done up-front to
future-proof this code for the bug fixed in 406f93ae488 (ci: make sure
we build Git parallel, 2019-02-07).

But since 4a6e4b96026 (CI: remove Travis CI support, 2021-11-23) the
--jobs=10 has been an invariant here, so let's assign --jobs=$NPROC to
the MAKEFLAGS right away, this gets rid of the leading whitespace from
the $MAKEFLAGS variable, as it would previously be empty on the first
assignment.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh | 11 ++++++-----
 1 file changed, 6 insertions(+), 5 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index 969ba2ff0d0..3591919ba44 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -75,8 +75,11 @@ handle_failed_tests () {
 # GitHub Action doesn't set TERM, which is required by tput
 export TERM=${TERM:-dumb}
 
+# How many jobs to run in parallel?
+NPROC=10
+
 # Clear MAKEFLAGS that may come from the outside world.
-export MAKEFLAGS=
+export MAKEFLAGS=--jobs=$NPROC
 
 if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
@@ -84,9 +87,8 @@ then
 	# We are running in Azure Pipelines
 	CC="${CC:-gcc}"
 
-	export GIT_PROVE_OPTS="--timer --jobs 10 --state=failed,slow,save"
+	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
 	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
-	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test Windows_NT != "$AGENT_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 elif test true = "$GITHUB_ACTIONS"
@@ -115,9 +117,8 @@ then
 		return 1
 	}
 
-	export GIT_PROVE_OPTS="--timer --jobs 10"
+	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
 	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
-	MAKEFLAGS="$MAKEFLAGS --jobs=10"
 	test Windows != "$RUNNER_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
 else
-- 
2.37.1.996.g651fc6e809f

[PATCH 06/10] CI: consistently use "export" in ci/lib.sh

[Ævar Arnfjörð Bjarmason]

Change the "ci/lib.sh" script to consistently use "export", for
e.g. MAKEFLAGS we were exporting it, and then assigning to it, let's
do it the other way around.

Right now this doesn't matter, since we in
e.g. "ci/install-dependencies.sh" source this file, and don't use
something like "env(1)" to retrieve these variables.

But in a subsequent commit we'll "export" these variables through a
wrapper (and to eventually write them to a GitHub CI-specific
$GITHUB_ENV file). This change makes that subsequent change easier to
read, as it won't need to do any control flow refactoring.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh | 9 ++++++---
 1 file changed, 6 insertions(+), 3 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index 3591919ba44..44039152ebc 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -79,7 +79,7 @@ export TERM=${TERM:-dumb}
 NPROC=10
 
 # Clear MAKEFLAGS that may come from the outside world.
-export MAKEFLAGS=--jobs=$NPROC
+MAKEFLAGS=--jobs=$NPROC
 
 if test -n "$SYSTEM_COLLECTIONURI" || test -n "$SYSTEM_TASKDEFINITIONSURI"
 then
@@ -88,9 +88,10 @@ then
 	CC="${CC:-gcc}"
 
 	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
-	export GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
+	GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	test Windows_NT != "$AGENT_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
+	export GIT_TEST_OPTS
 elif test true = "$GITHUB_ACTIONS"
 then
 	CI_TYPE=github-actions
@@ -118,9 +119,10 @@ then
 	}
 
 	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
-	export GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
+	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
 	test Windows != "$RUNNER_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
+	export GIT_TEST_OPTS
 else
 	echo "Could not identify CI type" >&2
 	env >&2
@@ -187,6 +189,7 @@ linux-leaks)
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
+export MAKEFLAGS
 
 end_group
 set -x
-- 
2.37.1.996.g651fc6e809f

[PATCH 07/10] CI: export variables via a wrapper

[Ævar Arnfjörð Bjarmason]

Add and use a "setenv" wrapper to export the variables set in
ci/lib.sh.

For now this merely adds debugging information about what variables
are set, but in a subsequent commit we'll expand this to set variables
in $GITHUB_ENV instead of the environment, and thus carry variable
settings between steps in a job.

We'll also eventually be interested in what type of variable we're
setting, and when it needs to be used, because we'll prominently
expose what variables we set in what CI "step", so having only the
ones we actually use will improve readability.

To that end call the "setenv" function with optional "--install",
"--build", "--test" and "--all" parameters to indicate whether they're
needed for building, testing or both.

We currently ignore the "varname" parameter, but will make use of it in a
subsequent commit. Adding it here reduces the needless churn, as we
won't need to change all these lines again for that subsequent change.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/lib.sh                 | 69 +++++++++++++++++++++++++++++----------
 ci/run-build-and-tests.sh | 34 +++++++++----------
 2 files changed, 68 insertions(+), 35 deletions(-)

diff --git a/ci/lib.sh b/ci/lib.sh
index 44039152ebc..67b7b32a0f1 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -59,6 +59,38 @@ else
 	begin_group "CI setup"
 fi
 
+# Helper functions
+setenv () {
+	while test $# != 0
+	do
+		case "$1" in
+		--install)
+			;;
+		--build)
+			;;
+		--test)
+			;;
+		--all)
+			;;
+		-*)
+			echo "BUG: bad setenv() option '$1'" >&2
+			exit 1
+			;;
+		*)
+			break
+			;;
+		esac
+		shift
+	done
+
+	key=$1
+	val=$2
+	shift 2
+
+	eval "$key=\"$val\""
+	eval "export $key"
+}
+
 check_unignored_build_artifacts () {
 	! git ls-files --other --exclude-standard --error-unmatch \
 		-- ':/*' 2>/dev/null ||
@@ -73,7 +105,7 @@ handle_failed_tests () {
 }
 
 # GitHub Action doesn't set TERM, which is required by tput
-export TERM=${TERM:-dumb}
+setenv --all TERM ${TERM:-dumb}
 
 # How many jobs to run in parallel?
 NPROC=10
@@ -87,11 +119,11 @@ then
 	# We are running in Azure Pipelines
 	CC="${CC:-gcc}"
 
-	export GIT_PROVE_OPTS="--timer --jobs $NPROC --state=failed,slow,save"
+	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC --state=failed,slow,save"
 	GIT_TEST_OPTS="--verbose-log -x --write-junit-xml"
 	test Windows_NT != "$AGENT_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
-	export GIT_TEST_OPTS
+	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
 elif test true = "$GITHUB_ACTIONS"
 then
 	CI_TYPE=github-actions
@@ -118,21 +150,21 @@ then
 		return 1
 	}
 
-	export GIT_PROVE_OPTS="--timer --jobs $NPROC"
+	setenv --test GIT_PROVE_OPTS "--timer --jobs $NPROC"
 	GIT_TEST_OPTS="--verbose-log -x --github-workflow-markup"
 	test Windows != "$RUNNER_OS" ||
 	GIT_TEST_OPTS="--no-chain-lint --no-bin-wrappers $GIT_TEST_OPTS"
-	export GIT_TEST_OPTS
+	setenv --test GIT_TEST_OPTS "$GIT_TEST_OPTS"
 else
 	echo "Could not identify CI type" >&2
 	env >&2
 	exit 1
 fi
 
-export DEVELOPER=1
-export DEFAULT_TEST_TARGET=prove
-export GIT_TEST_CLONE_2GB=true
-export SKIP_DASHED_BUILT_INS=YesPlease
+setenv --build DEVELOPER 1
+setenv --test DEFAULT_TEST_TARGET prove
+setenv --test GIT_TEST_CLONE_2GB true
+setenv --build SKIP_DASHED_BUILT_INS YesPlease
 
 case "$runs_on_pool" in
 ubuntu-latest)
@@ -148,19 +180,20 @@ ubuntu-latest)
 		MAKEFLAGS="$MAKEFLAGS PYTHON_PATH=/usr/bin/python2"
 	fi
 
-	export GIT_TEST_HTTPD=true
+	setenv --test GIT_TEST_HTTPD true
 
 	# The Linux build installs the defined dependency versions below.
 	# The OS X build installs much more recent versions, whichever
 	# were recorded in the Homebrew database upon creating the OS X
 	# image.
 	# Keep that in mind when you encounter a broken OS X build!
-	export LINUX_P4_VERSION="16.2"
-	export LINUX_GIT_LFS_VERSION="1.5.2"
+	setenv --install LINUX_P4_VERSION "16.2"
+	setenv --install LINUX_GIT_LFS_VERSION "1.5.2"
 
-	P4_PATH="$HOME/custom/p4"
-	GIT_LFS_PATH="$HOME/custom/git-lfs"
-	export PATH="$GIT_LFS_PATH:$P4_PATH:$PATH"
+	setenv --install P4_PATH "$HOME/custom/p4"
+	setenv --install GIT_LFS_PATH "$HOME/custom/git-lfs"
+	setenv --all PATH "$GIT_LFS_PATH:$P4_PATH:$PATH"
+	setenv --test GIT_TEST_HTTPD true
 	;;
 macos-latest)
 	if [ "$jobname" = osx-gcc ]
@@ -183,13 +216,13 @@ linux-musl)
 	MAKEFLAGS="$MAKEFLAGS GIT_TEST_UTF8_LOCALE=C.UTF-8"
 	;;
 linux-leaks)
-	export SANITIZE=leak
-	export GIT_TEST_PASSING_SANITIZE_LEAK=true
+	setenv --build SANITIZE leak
+	setenv --test GIT_TEST_PASSING_SANITIZE_LEAK true
 	;;
 esac
 
 MAKEFLAGS="$MAKEFLAGS CC=${CC:-cc}"
-export MAKEFLAGS
+setenv --all MAKEFLAGS "$MAKEFLAGS"
 
 end_group
 set -x
diff --git a/ci/run-build-and-tests.sh b/ci/run-build-and-tests.sh
index 1a89a4967bf..b7774e4abcf 100755
--- a/ci/run-build-and-tests.sh
+++ b/ci/run-build-and-tests.sh
@@ -9,33 +9,33 @@ run_tests=t
 
 case "$jobname" in
 linux-gcc)
-	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=main
+	setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME main
 	;;
 linux-TEST-vars)
-	export GIT_TEST_SPLIT_INDEX=yes
-	export GIT_TEST_MERGE_ALGORITHM=recursive
-	export GIT_TEST_FULL_IN_PACK_ARRAY=true
-	export GIT_TEST_OE_SIZE=10
-	export GIT_TEST_OE_DELTA_SIZE=5
-	export GIT_TEST_COMMIT_GRAPH=1
-	export GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS=1
-	export GIT_TEST_MULTI_PACK_INDEX=1
-	export GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP=1
-	export GIT_TEST_ADD_I_USE_BUILTIN=0
-	export GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME=master
-	export GIT_TEST_WRITE_REV_INDEX=1
-	export GIT_TEST_CHECKOUT_WORKERS=2
+	setenv --test GIT_TEST_SPLIT_INDEX yes
+	setenv --test GIT_TEST_MERGE_ALGORITHM recursive
+	setenv --test GIT_TEST_FULL_IN_PACK_ARRAY true
+	setenv --test GIT_TEST_OE_SIZE 10
+	setenv --test GIT_TEST_OE_DELTA_SIZE 5
+	setenv --test GIT_TEST_COMMIT_GRAPH 1
+	setenv --test GIT_TEST_COMMIT_GRAPH_CHANGED_PATHS 1
+	setenv --test GIT_TEST_MULTI_PACK_INDEX 1
+	setenv --test GIT_TEST_MULTI_PACK_INDEX_WRITE_BITMAP 1
+	setenv --test GIT_TEST_ADD_I_USE_BUILTIN 0
+	setenv --test GIT_TEST_DEFAULT_INITIAL_BRANCH_NAME master
+	setenv --test GIT_TEST_WRITE_REV_INDEX 1
+	setenv --test GIT_TEST_CHECKOUT_WORKERS 2
 	;;
 linux-clang)
-	export GIT_TEST_DEFAULT_HASH=sha1
+	setenv --test GIT_TEST_DEFAULT_HASH sha1
 	;;
 linux-sha256)
-	export GIT_TEST_DEFAULT_HASH=sha256
+	setenv --test GIT_TEST_DEFAULT_HASH sha256
 	;;
 pedantic)
 	# Don't run the tests; we only care about whether Git can be
 	# built.
-	export DEVOPTS=pedantic
+	setenv --build DEVOPTS pedantic
 	run_tests=
 	;;
 esac
-- 
2.37.1.996.g651fc6e809f

[PATCH 08/10] CI: don't have "git grep" invoke a pager in tree content check

[Ævar Arnfjörð Bjarmason]

Fix the omission of not adding "-P" (a.k.a. "--no-pager") to the "git"
invocation that spaws "git grep", added in 0e7696c64db (ci: disallow
directional formatting, 2021-11-04).

This didn't matter for the GitHub CI, but does when running the check
locally. A subsequent commit will start invoking it via a new check
from the top-level Makefile, and without this we'd visibly spawn a
pager when doing so.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 ci/check-directional-formatting.bash | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/ci/check-directional-formatting.bash b/ci/check-directional-formatting.bash
index e6211b141aa..de8e6e486f0 100755
--- a/ci/check-directional-formatting.bash
+++ b/ci/check-directional-formatting.bash
@@ -23,5 +23,5 @@
 # U+2066..U+2069: LRI, RLI, FSI and PDI
 regex='(\u202a|\u202b|\u202c|\u202d|\u202e|\u2066|\u2067|\u2068|\u2069)'
 
-! LC_CTYPE=C git grep -El "$(LC_CTYPE=C.UTF-8 printf "$regex")" \
+! LC_CTYPE=C git -P grep -El "$(LC_CTYPE=C.UTF-8 printf "$regex")" \
 	-- ':(exclude,attr:binary)' ':(exclude)*.po'
-- 
2.37.1.996.g651fc6e809f

[PATCH 09/10] CI: have "static-analysis" run a "make ci-static-analysis" target

[Ævar Arnfjörð Bjarmason]

Make it easier to run the CI logic locally by creating
"ci-static-analysis" target.

Now the "check-coccicheck" logic added in 0860a7641ba (travis-ci: fail
if Coccinelle static analysis found something to transform,
2018-07-23) can be easily tested outside of CI, and likewise the
sanity check added in 0e7696c64db (ci: disallow directional
formatting, 2021-11-04).

These targets could be improved, the "hdr-check" target will re-do all
of its work every time it's run, and could with a change similar to my
c234e8a0ecf (Makefile: make the "sparse" target non-.PHONY,
2021-09-23). The same goes for the new
"ci-check-directional-formatting" target.

But without those improvements being able to easily run a 1=1
equivalent of the checks we do in CI during a local build is already
an improvement.

This change will also make the CI check faster, since we can take
advantage of parallelism across these "make" invocations. The "make
coccicheck" command in particular takes a long to finish its last job,
at the end we might only have one job pegging 100% of one CPU
core. Now any extra cores will be free to run the rest of the targets
under "ci-static-analysis".

Because we're now going to invoke "make" directly from the CI recipe
we'll need to amend the new "setenv" wrapper to write the "MAKEFLAGS"
and other variables to "$GITHUB_ENV".

In my testing the "static-analysis" job runs in just over 10 minutes
without this change, but this change cuts just over a minute off the
runtime.

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 .github/workflows/main.yml |  6 ++++--
 Makefile                   | 29 +++++++++++++++++++++++++++++
 ci/lib.sh                  |  7 ++++++-
 ci/run-static-analysis.sh  | 32 --------------------------------
 shared.mak                 |  1 +
 5 files changed, 40 insertions(+), 35 deletions(-)
 delete mode 100755 ci/run-static-analysis.sh

diff --git a/.github/workflows/main.yml b/.github/workflows/main.yml
index 8b7697df9fb..fa6d861c75a 100644
--- a/.github/workflows/main.yml
+++ b/.github/workflows/main.yml
@@ -319,9 +319,11 @@ jobs:
     runs-on: ubuntu-18.04
     steps:
     - uses: actions/checkout@v2
+    - run: ci/lib.sh
+      env:
+        NO_CI_GROUPS: 1
     - run: ci/install-dependencies.sh
-    - run: ci/run-static-analysis.sh
-    - run: ci/check-directional-formatting.bash
+    - run: make ci-static-analysis
   sparse:
     needs: ci-config
     if: needs.ci-config.outputs.enabled == 'yes'
diff --git a/Makefile b/Makefile
index 04d0fd1fe60..c328e190d64 100644
--- a/Makefile
+++ b/Makefile
@@ -3147,6 +3147,20 @@ coccicheck: $(addsuffix .patch,$(filter-out %.pending.cocci,$(wildcard contrib/c
 # See contrib/coccinelle/README
 coccicheck-pending: $(addsuffix .patch,$(wildcard contrib/coccinelle/*.pending.cocci))
 
+.PHONY: check-coccicheck
+check-coccicheck: coccicheck
+	$(QUIET_CHECK)for cocci_patch in contrib/coccinelle/*.patch; do \
+		if test -s "$$cocci_patch"; then \
+			echo "Coccinelle suggests the following changes in '$$cocci_patch':"; \
+			cat "$$cocci_patch"; \
+			fail=UnfortunatelyYes; \
+		fi \
+	done; \
+	if test -n "$$fail"; then \
+		echo "error: Coccinelle suggested some changes"; \
+		exit 1; \
+	fi
+
 .PHONY: coccicheck coccicheck-pending
 
 ### Installation rules
@@ -3589,3 +3603,18 @@ $(FUZZ_PROGRAMS): all
 		$(XDIFF_OBJS) $(EXTLIBS) git.o $@.o $(LIB_FUZZING_ENGINE) -o $@
 
 fuzz-all: $(FUZZ_PROGRAMS)
+
+### CI "check" targets
+#
+# These targets are run from the CI, see .github/workflows/main.yml,
+# but can also be run manually to run the same assertions locally.
+
+.PHONY: ci-check-directional-formatting
+ci-check-directional-formatting:
+	$(QUIET_CHECK)ci/check-directional-formatting.bash
+
+.PHONY: ci-static-analysis
+ci-static-analysis: ci-check-directional-formatting
+ci-static-analysis: check-coccicheck
+ci-static-analysis: hdr-check
+ci-static-analysis: check-pot
diff --git a/ci/lib.sh b/ci/lib.sh
index 67b7b32a0f1..14d0af2fa7f 100755
--- a/ci/lib.sh
+++ b/ci/lib.sh
@@ -15,7 +15,7 @@ then
 	exit 1
 fi
 
-if test true != "$GITHUB_ACTIONS"
+if test true != "$GITHUB_ACTIONS" || test -n "$NO_CI_GROUPS"
 then
 	begin_group () { :; }
 	end_group () { :; }
@@ -89,6 +89,11 @@ setenv () {
 
 	eval "$key=\"$val\""
 	eval "export $key"
+	eval "export $key=\"$val\""
+	if test -n "$GITHUB_ENV"
+	then
+		echo "$key=$val" >>"$GITHUB_ENV"
+	fi
 }
 
 check_unignored_build_artifacts () {
diff --git a/ci/run-static-analysis.sh b/ci/run-static-analysis.sh
deleted file mode 100755
index faae31f0078..00000000000
--- a/ci/run-static-analysis.sh
+++ /dev/null
@@ -1,32 +0,0 @@
-#!/bin/sh
-#
-# Perform various static code analysis checks
-#
-
-. ${0%/*}/lib.sh
-
-make coccicheck
-
-set +x
-
-fail=
-for cocci_patch in contrib/coccinelle/*.patch
-do
-	if test -s "$cocci_patch"
-	then
-		echo "$(tput setaf 1)Coccinelle suggests the following changes in '$cocci_patch':$(tput sgr0)"
-		cat "$cocci_patch"
-		fail=UnfortunatelyYes
-	fi
-done
-
-if test -n "$fail"
-then
-	echo "$(tput setaf 1)error: Coccinelle suggested some changes$(tput sgr0)"
-	exit 1
-fi
-
-make hdr-check ||
-exit 1
-
-make check-pot
diff --git a/shared.mak b/shared.mak
index 4330192e9c3..3bd846aaf9e 100644
--- a/shared.mak
+++ b/shared.mak
@@ -58,6 +58,7 @@ ifndef V
 ## Used in "Makefile"
 	QUIET_CC       = @echo '   ' CC $@;
 	QUIET_AR       = @echo '   ' AR $@;
+	QUIET_CHECK    = @echo '   ' CHECK $@;
 	QUIET_LINK     = @echo '   ' LINK $@;
 	QUIET_BUILT_IN = @echo '   ' BUILTIN $@;
 	QUIET_LNCP     = @echo '   ' LN/CP $@;
-- 
2.37.1.996.g651fc6e809f

[PATCH 10/10] CI: have "static-analysis" run "check-builtins", not "documentation"

[Ævar Arnfjörð Bjarmason]

Move the "make check-builtins" check from the "documentation" job to
the "static-analysis" job.

The "check-builtins" target added in c74390e4a1d (cherry is built-in,
do not ship git-cherry.sh, 2006-11-05) is unrelated to the
documentation, so it's odd that b98712b9aa9 (travis-ci: build
documentation, 2016-05-04) added it to the "documentation" job.

Let's just move it to the "ci-static-analysis" target, and while we're
at it improve the output with $(QUIET_CHECK).

Signed-off-by: Ævar Arnfjörð Bjarmason <avarab@gmail.com>
---
 Makefile                 | 3 ++-
 ci/test-documentation.sh | 1 -
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/Makefile b/Makefile
index c328e190d64..c2b0a728df5 100644
--- a/Makefile
+++ b/Makefile
@@ -3520,7 +3520,7 @@ check-docs::
 ### Make sure built-ins do not have dups and listed in git.c
 #
 check-builtins::
-	./check-builtins.sh
+	$(QUIET_CHECK)./check-builtins.sh
 
 ### Test suite coverage testing
 #
@@ -3615,6 +3615,7 @@ ci-check-directional-formatting:
 
 .PHONY: ci-static-analysis
 ci-static-analysis: ci-check-directional-formatting
+ci-static-analysis: check-builtins
 ci-static-analysis: check-coccicheck
 ci-static-analysis: hdr-check
 ci-static-analysis: check-pot
diff --git a/ci/test-documentation.sh b/ci/test-documentation.sh
index 9e0652c30dd..41e2b126311 100755
--- a/ci/test-documentation.sh
+++ b/ci/test-documentation.sh
@@ -14,7 +14,6 @@ filter_log () {
 	    "$1"
 }
 
-make check-builtins
 make check-docs
 
 # Build docs with AsciiDoc
-- 
2.37.1.996.g651fc6e809f