Re: [kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds

From: Thomas Huth <thuth@redhat.com>
To: "Alex Bennée" <alex.bennee@linaro.org>
Cc: kvm@vger.kernel.org, Paolo Bonzini <pbonzini@redhat.com>,
	David Hildenbrand <david@redhat.com>
Subject: Re: [kvm-unit-test PATCH 5/5] travis.yml: Expect that at least one test succeeds
Date: Wed, 13 Nov 2019 16:17:51 +0100	[thread overview]
Message-ID: <417bee9b-578b-5b3c-25a6-9998020ba514@redhat.com> (raw)
In-Reply-To: <87mucz7r3s.fsf@linaro.org>

On 13/11/2019 16.06, Alex Bennée wrote:
> 
> Thomas Huth <thuth@redhat.com> writes:
> 
>> While working on the travis.yml file, I've run into cases where
>> all tests are reported as "SKIP" by the run_test.sh script (e.g.
>> when QEMU could not be started). This should not result in a
>> successful test run, so mark it as failed if not at least one
>> test passed.
> 
> But doesn't this mean you could have everything fail except one pass and
> still report success?

The FAILs are already handled one line earlier...

>>
>> Signed-off-by: Thomas Huth <thuth@redhat.com>
>> ---
>>  .travis.yml | 1 +
>>  1 file changed, 1 insertion(+)
>>
>> diff --git a/.travis.yml b/.travis.yml
>> index 9ceb04d..aacf7d2 100644
>> --- a/.travis.yml
>> +++ b/.travis.yml
>> @@ -115,3 +115,4 @@ script:
>>    - make -j3
>>    - ACCEL="${ACCEL:-tcg}" ./run_tests.sh -v $TESTS | tee results.txt
>>    - if grep -q FAIL results.txt ; then exit 1 ; fi

... here -----------^

>> +  - if ! grep -q PASS results.txt ; then exit 1 ; fi

Maybe it would also be nicer to provide proper exit values in the
run_tests.sh script, but the logic there gives me a bad headache...
grep'ing for FAIL and PASS in the yml script is way more easy.

 Thomas