Re: [PATCH 00/15] tests: don't ignore "git" exit codes

["Ævar Arnfjörð Bjarmason" <avarab@gmail.com>]

On Wed, Mar 02 2022, Derrick Stolee wrote:

> On 3/2/2022 12:27 PM, Ævar Arnfjörð Bjarmason wrote:
>> This series fixes issues where we ignored the exit code of "git" due
>> to it being on the LHS of a pipe, or because we interpolated its
>> output with $() in a "test" construct, or had missing &&- chains in
>> helper functions etc.
>> 
>> This series is not made by string-replacing things in our test suite,
>> if it was it would be much larger. These are all tests I've seen real
>> hide real failures under SANITIZE=leak, either on current "master", or
>> in combination with various local leak fixes I've got unsubmitted.
>
> My first reaction was to check that Subham was on the CC line (yes)
> because they have been working in this space, too. Your focus on
> examples that break SANITIZE=leak is appreciated so there is room
> for that project.

Yeah, I'm certainly not meaning to steal this whole "hide exit code"
microproject, and there's *a lot* more to chew on there :) (and I have
no intention of doing the rest).

But hopefully this series also serves as a nice tour-de-force through
the test suite showing a wide variety of exit-code-hiding cases for any
future work in the area.

I.e. the "git on theon LHS of a pipe" is an overly narrow definition of
it.

But then again I think these should also all be covered in passing by
t/README's "don'ts" list.

>> In cases where I was starting to fix a pattern in a file I'd fix the
>> rest of the file if it was easy, but otherwise these are all cases
>> where I ran SANITIZE=leak, had a test pass, but having ASAN_OPTIONS
>> log to a file revealed that we had memory leaks within that test.
>
> Neat trick.
>
> The patches in this series clearly do the right transformations to
> expose these errors at the appropriate time. The only time I got
> hung up was patch 11 where test_expect_failure was swapped for
> test_expect_success (while also dropping a test_must_fail inside
> the test). The double-negative confused me at first, but in the end
> the patch works as-is.
>
>> As an aside we still have various potential issues with hidden
>> segfaults etc. in the test suite after this that are tricked to solve,
>> because:
>> 
>>  * Our tests will (mostly) catch segfaults and abort(), but if we
>>    invoke a command that invokes another command it needs to ferry the
>>    exit code up to us.
>> 
>>  * run-command.c notably does not do that, so for e.g. "git push"
>>    tests where we expect a failure and an underlying "git" command
>>    fails we won't ferry up the segfault or abort exit code.
>
> Perhaps run-command.c could auto-exit for certain well-known error
> codes that could only happen on certain kinds of failures (segfault,
> for example). A simple die() might be something that is expected to
> be handled by the top-level command in some cases.

Yes. I have a local WIP patch to make it do that.

Basically just set a a global "uh oh, one of our ran commands segfaulted
or abort()-ed, here's that exit status".

Then when we do the real exit() (which we always intercept due to the
trace2 logging needing to do so) an non-zero exit() of e.g. status 128
will be changed to the appropriate exit status of that segfault or
abort().

We'll thus ferry such exit codes upwards, clobbering whatever the
"desired" exit code of e.g. "git pull" would have been (but that's a
feature in this case).

One thing I got hung up on is that it's relatively straightforward for
the wait_or_whine() case in run-command.c, but I didn't look into how to
do that with the pthread_join() we do for finish_async(), which
e.g. happens if the multiplex'd dialog exits in this way.

>>  * We have gitweb.perl and some other perl code ignoring return values
>>    from close(), i.e. ignoring exit codes from "git rev-parse" et al.
>> 
>>  * We have in-tree shellscripts like "git-merge-one-file.sh" invoking
>>    git commands, and if they fail returning "1", not ferrying up the
>>    segfault or abort() exit code.
>
> These are more involved and harder to evaluate. Add them to the pile
> of projects for new contributors?

FWIW those are relatively easy, for the first:

    use v5.10.1;
    use autodie qw(close);

And for the second just go through the relatively small amount of such
remaining shell code and convert:

	if ! git ...
	then
		exit 1

To:

	git ...
	code=$?
        if test $code -ne 0
	then
		exit $code

Or whatever.