Re: [Ksummit-discuss] [MAINTAINERS SUMMIT] Bug-introducing patches

[Mauro Carvalho Chehab <mchehab+samsung@kernel.org>]

Em Sat, 08 Sep 2018 12:44:32 +0300
Laurent Pinchart <laurent.pinchart@ideasonboard.com> escreveu:

> On Saturday, 8 September 2018 00:06:33 EEST Mauro Carvalho Chehab wrote:
> > Em Fri, 7 Sep 2018 11:13:20 +0200 Daniel Vetter escreveu:  
> > > On Fri, Sep 7, 2018 at 6:27 AM, Theodore Y. Ts'o <tytso@mit.edu> wrote:  
> > > > On Fri, Sep 07, 2018 at 01:49:31AM +0000, Sasha Levin via
> > > > Ksummit-discuss wrote:
> > > > 
> > > > There actually is a perverse incentive to having all of the test
> > > > 'bots, which is that I suspect some people have come to rely on it to
> > > > catch problems.  I generally run a full set of regression tests before
> > > > I push an update to git.kernel.org (it only takes about 2 hours, and
> > > > 12 VM's :-); and by the time we get to the late -rc's I *always* will
> > > > do a full regression test.  
> > > 
> > > This is what imo a well-run subsystem should sound like from a testing
> > > pov. All the subsystem specific testing should be done before merging.
> > > Post-merge is only for integration testing and catching the long-tail
> > > issues that need months/years of machine time to surface.
> > > 
> > > Of course this is much harder for anything that needs physical
> > > hardware, but even for driver subsystems there's lots you can do with
> > > test-drivers, selftests and a pile of emulation, to at least catch
> > > bugs in generic code. And for reasonably sized teams like drm/i915
> > > building a proper CI is a very obvious investement that will pay off.  
> > 
> > IMHO, CI would do even a better job for smaller teams, as they won't
> > have much resources for testing, but the problem here is that those
> > teams probably lack resources and money to invest on a physical hardware
> > to setup a CI infra and to buy the myriad of different hardware to
> > do regression testing.
> > 
> > Also, some devices are harder to test: how would you check if a camera
> > microphone is working? How to check if the camera captured images
> > are ok?  
> 
> The same way you would check the display output. Cameras can be pointed at 
> known scenes with controlled lightning. TV capture cards can be fed a known 
> signal. Even for microphone testing we could put the camera in a sound-proof 
> enclosure, with an audio source. Solutions exist, whether we have the budget 
> to implement them is the real question.

Solutions exist, but they require a hole new kind of environment control.

In the case of DRM (and TV cards), display output can be tested with some
HDMI grabber card. No need for a "controlled lightning environment" or
anything like that. Once it is set, people can just place it into a
random datacenter located anywhere and forget about it.

However, in the case of hardware like cameras, microphones, speakers,
keyboards, mice, touchscreen, etc, it is a way more complex, as the
environment will require adjustments (a silent room, specific
lightning, mechanical components, etc) and a more proactive supervision,
as it would tend to produce more false positive errors if something
changes there. A normal datacenter won't fit those needs.

Thanks,
Mauro