Re: [PATCH] Makefile: add and use the ".DELETE_ON_ERROR" flag

From: Jeff King <peff@peff.net>
To: "Ævar Arnfjörð Bjarmason" <avarab@gmail.com>
Cc: Taylor Blau <me@ttaylorr.com>,
	git@vger.kernel.org, Junio C Hamano <gitster@pobox.com>,
	Felipe Contreras <felipe.contreras@gmail.com>
Subject: Re: [PATCH] Makefile: add and use the ".DELETE_ON_ERROR" flag
Date: Mon, 28 Jun 2021 22:26:56 -0400	[thread overview]
Message-ID: <YNqE8BIRF6NeYQcd@coredump.intra.peff.net> (raw)
In-Reply-To: <87fsx6xn0b.fsf@evledraar.gmail.com>

On Fri, Jun 25, 2021 at 11:49:14AM +0200, Ævar Arnfjörð Bjarmason wrote:

> > Ah, right. Thanks for refreshing me.
> >
> > TBH, I don't find this that serious a problem. Your compile will fail.
> > But is rebuilding in the middle of a test run something it's important
> > to support seamlessly? It seems like a bad practice in the first place.
> 
> Yeah I think so, and I think it's good practice, it enabled development
> workflows that you and Junio clearly don't use (which is fine), but
> which are useful to others. For me it would result in more total
> testing, and earlier catching of bugs, not less.
> 
> Quoting an earlier mail of yours[1]:
> 
>     I think having the test suite loudly complain is a good way to
>     remind you that you have not in fact run the whole suite on a given
>     build.
> 
> It's useful as you're programming you save/compile, and have the tests
> run in a loop in the background, and are alerted if they start
> failing.
> 
> That's not really possible with git currently without having that loop
> continually push to another workdir, making it work in one checkout
> helps for some workflows.

I actually _do_ have a workflow like this, but as you might guess, it
involves a separate workdir. I have a loop going in one terminal waiting
for new commits, which it then checks out on a detached HEAD and tests.

So I'm definitely sympathetic to this kind of continuous testing. My
questioning was really about doing it in a hacky way where you're not
actually sure what's been tested and what hasn't. If you're not willing
to test just committed states, then it gets a lot harder (OTOH, I'd
expect a ton of spurious failures there as you have half-finished
states). But if you are, then I think making sure you've tested each
commit fully has huge value, because then you can cache the results.

I'm not sure if the "only test committed states" thing is a deal-breaker
for you or not. If not, the script I use is at:

  https://github.com/peff/git/blob/meta/ci

which is really just a glorified infinite loop with some inotify
hackery. It relies on this program to do the caching:

  https://github.com/mhagger/git-test

A nice extra thing you can do is use the same cache with "git rebase -x"
as a final check of each patch in a series before you send it out. If
the CI loop was running continuously in the background, then it's just a
noop of "yes, we already tested this".

> Yes it could allow you to run format-patch and send-email while you're
> 50% through a full loop, or not just run the full tests then, but at
> some point I think we've got to assume some basic competency from
> people. We also have CI running the full tests, and I could have just
> run tests 0000-5000, compiled, and then run 5001-9999.

Yeah, I can see the view that running the test suite as a basic sanity
check may have value, if it's backed by more careful testing later (and
certainly while I'm developing, I wouldn't hesitate to run a subset of
the test suite to see how my work is progressing).

My main point was that I don't see much reason to do work to make that
kind of continuous "make test" work with simultaneous recompiles and
test-runs, if we can encourage people to do it more robustly with a
single compile-and-test-run loop. Maybe adding in the extra workdir
there makes it too heavy-weight? (Certainly my "ci" script is overkill,
but it seems like a loop of "reset to the current branch tip, compile,
run" in a worktree would be the minimal thing).

-Peff