Re: [RFC PATCH v2 6/6] t/Makefile: run unit tests alongside shell tests

[Jeff King <peff@peff.net>]

On Wed, Feb 07, 2024 at 12:55:09PM -0800, Junio C Hamano wrote:

> > +# A simple wrapper to run shell tests via TEST_SHELL_PATH,
> > +# or exec unit tests directly.
> > +
> > +case "$1" in
> > +*.sh)
> > +	exec ${TEST_SHELL_PATH:-/bin/sh} "$@"
> > +	;;
> > +*)
> > +	exec "$@"
> > +	;;
> > +esac
> 
> Hmph.  This penalizes the non-unit tests by doing an extra "exec",
> once per program?

It does, but IMHO that is not likely to be a problem. It's once per
top-level script (so ~1000), and each of those scripts spawns hundreds
or thousands of sub-commands. I didn't do any measurements, though.

You can extend "prove" with extra perl modules so that it makes the
distinction internally without the extra shell invocation. But when I
tried to do it, I found it rather baroque and complicated (I can't
remember if I succeeded but found it too gross, or just gave up halfway
through trying).

> Of course we cannot run two $(PROVE) invocations serially, one for
> doing $(T) and the other for doing $(UNIT_TESTS)?

Not if they share the same command-line options. If you use something
like "--state=slow,save", then the first run will write the list of all
tests to ".prove", and then the second will run every test mentioned in
.prove (in addition to the unit-tests provided on the command-line).

You should be able to work around it by passing "--statefile". I _think_
it might be OK to just do that unconditionally. Something like:

  prove --exec $(TEST_SHELL_PATH $(GIT_PROVE_OPTS) $(T) :: $(GIT_TEST_OPTS)
  prove --statefile=.prove-unit-tests $(GIT_PROVE_OPTS) $(UNIT_TESTS) :: $(GIT_TEST_OPTS)

and then it's just a noop if GIT_PROVE_OPTS doesn't use --state. But I
haven't played with it myself.

-Peff