Re: [Ksummit-discuss] [CORE TOPIC] stable workflow

[Mark Brown <broonie@kernel.org>]

[-- Attachment #1: Type: text/plain, Size: 1595 bytes --]

On Fri, Jul 29, 2016 at 11:59:47AM +0300, Laurent Pinchart wrote:
> On Thursday 28 Jul 2016 20:10:10 Steven Rostedt wrote:

> > Does tools/testing/selftests/ not satisfy this?

> It does, but lacks features to support driver-related test cases. For instance 
> it doesn't (for quite obvious reasons) provide machine-readable information 
> about the hardware requirements for a particular test.

Plus in general the hardware related tests can end up requiring some
specific environment beyond that which is machine enumerable.

> I'm not sure whether kselftest could/should be extended for that purpose. Due 
> to its integration in the kernel, there is little need to standardize the test 
> case interface beyond providing a Makefile to declare the list of test 
> programs and compile them. Something slightly more formal is in my opinion 
> needed if we want to scale to device driver tests with out-of-tree test cases.

There's also the risk that we make it harder for a random user to pick
up the tests and predict what the expected results should be - one of
the things that can really hurt a testsuite is if users don't find it
consistent and stable.

> Another limitation of kselftest is the lack of standardization for logging and 
> status reporting. This would be needed to interpret the test output in a 
> consistent way and generate reports. Regardless of whether we extend kselftest 
> to cover device drivers this would in my opinion be worth fixing.

I thought that was supposed to be logging via stdout/stderr and the
return code for the result.

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 473 bytes --]