Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] & [TECH TOPIC] Improve regression tracking

[Shuah Khan <shuahkh@osg.samsung.com>]

On 07/05/2017 09:32 AM, Greg KH wrote:
> On Wed, Jul 05, 2017 at 08:16:33AM -0700, Guenter Roeck wrote:
>> If we start shaming people for not providing unit tests, all we'll accomplish is
>> that people will stop providing bug fixes.
> 
> Yes, this is the key!
> 
> Steven, just look at everything marked with a "Fixes:" or "stable@" tag
> from 4.12-rc1..4.12 and try to determine how you would write a test for
> the majority of them.
> 
> Yes, for some subsystems this can work (look at xfstests as one great
> example for filesystems, same for the i915 tests), but for the majority
> of the kernel, at this point in time, it doesn't make sense.
> 
> So take Carlos's advice, start small, do it for your subsystem if you
> don't touch hardware (easy peasy, right?), and let's see how it goes,
> and see if we have the infrastructure to do it even today.  Right now,
> kselftests is finally getting a unified output format, which is great,
> it shows that people are starting to use and rely on it.  What else will
> we need to make this more widely used, we don't know yet...
> 

Over the past couple of years, kselftests have seen improvements to run
on ARM in kernel ci rings. TAP13 will definitely make it easier to find
run to run differences. There is the effort to use ksefltests to test
stable releases (4.4 LTS for example), which will help make the tests
fail/skip gracefully when a feature isn't enabled/supported.

The work so far is two fold:

- enable them to run in test rings.
- making them easy to use

As per test development, we are constantly adding tests and I see new tests
getting added for sub-systems that aren't hardware dependent. You will see
lots of activity in mm, timers, seccomp, net, sys-calls to name a few.

I am going to be looking for TAP13 format compliance for new tests starting
4.13.

I am not sure how popular they are among developers and sub-system maintainers
though. Maybe this is one area we can try to improve usage.

thanks,
-- Shuah