Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug-introducing patches

[Guenter Roeck <linux@roeck-us.net>]

On Mon, Sep 10, 2018 at 04:32:40PM -0700, Eduardo Valentin wrote:
> On Mon, Sep 10, 2018 at 07:12:39PM -0400, Steven Rostedt wrote:
> > On Mon, 10 Sep 2018 16:01:06 -0700
> > Eduardo Valentin <edubezval@gmail.com> wrote:
> > 
> > > One thing that could be done to help is to ask from developers for
> > > some sort of selftest that can be executed by the bots and used while
> > > backporting their fixes to stable. That way the developer can have a way
> > 
> > We have that already, it's tools/testing/selftests/...
> 
> well, yes, but what I really meant was to populate that directory with a
> full set of tests that can detect regressions, on all subsystems
> 
> > 
> > There's a series of ftrace selftests there that I run before running my
> > own more complicated tests. There's still tests I need to move to that
> > selftest directory and out of my own suite, but there's some tests that
> > are too complicated for the the selftests directory.
> > 
> > 
> > > to tell how to check if the kernel did not regress and whoever wants to
> > > try out the fix can validate it. Of course, can this really fly, that is
> > > a different story. Not sure the community will end up in a place where
> > > all patches post -rc5 requires a selftest :-)
> > > 
> > > And of course, there is the other type of regression, which is the fix /
> > > backport causing issue on other parts of the kernel/subsystem. Maybe
> > > forcing each subsystem to have some sort of selftest/sanity check would
> > > be one way to improve the reliability of the results of the bots
> > > overall.
> > 
> > Heh, "forcing"? That hasn't been able to work yet ;-)
> > 
> 
> :-)
> 
> > Also, tests for others that don't have the necessary hardware is not
> > going to help much. A lot of regressions show up on hardware that we
> > don't use.
> > 
> 
> I agree. Thermal is one of those weird cases one would find most of real
> problems while putting devices inside a thermal chamber and running real
> workloads in a controlled manner. And on top of that, those are many
> times not enough, and only end users would really trigger corner cases
> that can really be seen when the device gets into a person's hand.
> 
> But still, the fact that selftests do not get all bugs does not mean it
> cannot be used to catch at least a subset of it.
> 
> Also, some CI / bots do have a rig of hardware attached (kernelCI for
> one). But yeah, I agree, hardware availability is a real issue.
> 

I think a lot of this could be resolved with qemu. That is not perfect
either, but simulating environmental conditions to trigger a system
response should be quite straightforward and much less costly than
thermal chambers.

Guenter