Re: RFC - kernel selftest result documentation (KTAP)

[Frank Rowand <frowand.list@gmail.com>]

On 2020-06-19 17:58, Paolo Bonzini wrote:
> On 19/06/20 20:47, Frank Rowand wrote:
>> Or if the entire test depends on the missing config then Bail out might
>> be appropriate.
> 
> No, in that case you want
> 
> 	1..0 # SKIP: unsupported configuration
> 
> The spec is not clear if "Bail out!" is an error condition or just a
> warning that only part of the test was run, but prove(1) and Automake
> both treat it as the former, for example.
> 
> For example, an ENOSPC error creating a temporary file could be turned
> into a bail-out, while an ENOSYS would be a skip.
> 
> Paolo
> 

I think that "Bail out!" needs to be more clearly defined by the spec,
and we should come up with an intent of what it is intended to mean.

The TAP 13 spec gives a "Bail out!" example of "MySQL is not running."
The spec gives this example after the general guidance of (among other
things) "missing dependencies".  A missing config _could_ be thought
of as a missing dependency.

To me, the key differentiator for "Bail out!" is "that further tests
are useless".  In other words, the detected problem means that
every subsequent part of the tests will fail for the same reason.

If _some_ subsequent parts of the tests will NOT fail for the same
reason then SKIP becomes more reasonable.  What percent of subsequent
parts of the tests is large enough to comprise "_some_" is probably
a value judgement.

For the case that I was commenting on, with the limited amount of
description provided, my preference is also SKIP, which is what I
was suggesting.