Re: git 2.36.0 regression: pre-commit hooks no longer have stdout/stderr as tty

[Junio C Hamano <gitster@pobox.com>]

Ævar Arnfjörð Bjarmason <avarab@gmail.com> writes:

> ... I.e. as a mechanism for
> mitigating mistakes and catching obscure edge cases just being more
> careful or having things sit in 'next' for longer has, I think, proved
> itself to not be an effective method (not just in this case, but a few
> similar cases).

I tend to agree.  Given that people do not discover possible
regression that will affect even after a change hits 'master',
cooking in 'next' alone would not be all that effective.

But that does not mean we shouldn't cook in 'next' at all.

Especially the previous cycle, I was experimenting with a tweak in
my workflow to have topics in 'next' to cook for one week and have
them graduate to 'master' unless we saw regression in a week.
Previously, I tended to keep topics on the larger side in 'next' and
we did see "oops we found this after the topic hit 'next' and here
is a fix-up" to them, which I think helped to catch bugs before it
broke 'master'.

> I'm not sure what the solution is exactly, but I'm pretty sure it
> involves more controlled exposure to the wild (e.g. shipping certain
> things as feature flags first), not deferring that exposure for long
> periods, which is what having things sit it "next" for longer amounts
> to.

To be fair, "is the hook invoked with its standard output stream
connected to the original standard output of the main process?" is
not something either of you cared while working on and reviewing the
changes, and releases based on 'next' $BIGCOMPANY have its users use
internally wouldn't have helped to catch this particular regression,
as the primary reason we didn't think of it as a problem is because
internal users of $BIGCOMPANY tend to be more monoculture than folks
in the wild.

So the fundamental solution would be to find a way to involve those
who found the regression after a release was done in the development
process at a much earlier stage.  I do not offhand know how to get
there.

It may be very hard for us to do with end-users, who typically have
only one instance of Git they use and a single valuable repository
they cannot subject to "experiments".  They may depend on certain
aspects of the behaviour of the current version, but they lack an
environment to try out our new version to see if we broke them.
They probably may not even be aware of what they are relying on,
just as we are unaware of their dependence.

But for toolsmiths who integrate their gears with Git, there may be
something we can do.  Perhaps we can start giving "works with Git"
badge out (we need to control its use with some kind of trademark
registration), to those who "maintain X that enhances Git" when they
regularly test their ware with 'next' or much earlier.  And when
they stop us before unleashing a possible regression by reporting
bugs early, give them a "star", so that their "works with Git *****"
logo can boast how much contribution in testing they are maing
upstream.

Or something like that, perhaps?