Re: [RFC] Create test script(s?) for regression testing

[Laurent Pinchart <laurent.pinchart@ideasonboard.com>]

Hi Hans,

On Tuesday, 6 November 2018 15:56:34 EET Hans Verkuil wrote:
> On 11/06/18 14:12, Laurent Pinchart wrote:
> > On Tuesday, 6 November 2018 13:36:55 EET Sakari Ailus wrote:
> >> On Tue, Nov 06, 2018 at 09:37:07AM +0100, Hans Verkuil wrote:
> >>> Hi all,
> >>> 
> >>> After the media summit (heavy on test discussions) and the V4L2 event
> >>> regression we just found it is clear we need to do a better job with
> >>> testing.
> >>> 
> >>> All the pieces are in place, so what is needed is to combine it and
> >>> create a script that anyone of us as core developers can run to check
> >>> for regressions. The same script can be run as part of the kernelci
> >>> regression testing.
> >> 
> >> I'd say that *some* pieces are in place. Of course, the more there is,
> >> the better.
> >> 
> >> The more there are tests, the more important it would be they're
> >> automated, preferrably without the developer having to run them on his/
> >> her own machine.
> > 
> > From my experience with testing, it's important to have both a core set of
> > tests (a.k.a. smoke tests) that can easily be run on developers' machines,
> > and extended tests that can be offloaded to a shared testing
> > infrastructure (but possibly also run locally if desired).
> 
> That was my idea as well for the longer term. First step is to do the basic
> smoke tests (i.e. run compliance tests, do some (limited) streaming test).
> 
> There are more extensive (and longer running) tests that can be done, but
> that's something to look at later.
> 
> >>> We have four virtual drivers: vivid, vim2m, vimc and vicodec. The last
> >>> one is IMHO not quite good enough yet for testing: it is not fully
> >>> compliant to the upcoming stateful codec spec. Work for that is planned
> >>> as part of an Outreachy project.
> >>> 
> >>> My idea is to create a script that is maintained as part of v4l-utils
> >>> that loads the drivers and runs v4l2-compliance and possibly other tests
> >>> against the virtual drivers.
> >> 
> >> How about spending a little time to pick a suitable framework for running
> >> the tests? It could be useful to get more informative reports than just
> >> pass / fail.
> > 
> > We should keep in mind that other tests will be added later, and the test
> > framework should make that easy.
> 
> Since we want to be able to run this on kernelci.org, I think it makes sense
> to let the kernelci folks (Hi Ezequiel!) decide this.

KernelCI isn't the only test infrastructure out there, so let's not forget 
about the other ones.

> As a developer all I need is a script to run smoke tests so I can catch most
> regressions (you never catch all).
> 
> I'm happy to work with them to make any changes to compliance tools and
> scripts so they fit better into their test framework.
> 
> The one key requirement to all this is that you should be able to run these
> tests without dependencies to all sorts of external packages/libraries.

v4l-utils already has a set of dependencies, but those are largely manageable. 
For v4l2-compliance we'll install libv4l, which depends on libjpeg.

> > Regarding the test output, many formats exist (see
> > https://testanything.org/ and
> > https://chromium.googlesource.com/chromium/src/+/master/docs/testing/
> > json_test_results_format.md for instance), we should pick one of the
> > leading industry standards (what those standards are still needs to be
> > researched  :-)).
> > 
> >> Do note that for different hardware the tests would be likely different
> >> as well although there are classes of devices for which the exact same
> >> tests would be applicable.
> > 
> > See http://git.ideasonboard.com/renesas/vsp-tests.git for an example of
> > device-specific tests. I think some of that could be generalized.
> > 
> >>> It should be simple to use and require very little in the way of
> >>> dependencies. Ideally no dependencies other than what is in v4l-utils so
> >>> it can easily be run on an embedded system as well.
> >>> 
> >>> For a 64-bit kernel it should run the tests both with 32-bit and 64-bit
> >>> applications.
> >>> 
> >>> It should also test with both single and multiplanar modes where
> >>> available.
> >>> 
> >>> Since vivid emulates CEC as well, it should run CEC tests too.
> >>> 
> >>> As core developers we should have an environment where we can easily
> >>> test our patches with this script (I use a VM for that).
> >>> 
> >>> I think maintaining the script (or perhaps scripts) in v4l-utils is best
> >>> since that keeps it in sync with the latest kernel and v4l-utils
> >>> developments.
> >> 
> >> Makes sense --- and that can be always changed later on if there's a need
> >> to.
> > 
> > I wonder whether that would be best going forward, especially if we want
> > to add more tests. Wouldn't a v4l-tests project make sense ?
> 
> Let's see what happens. The more repos you have, the harder it becomes to
> keep everything in sync with the latest kernel code.

Why is that ? How would a v4l-tests repository make it more difficult ?

> My experience is that if you want to have good tests, then writing tests
> should be as easy as possible. Keep dependencies at an absolute minimum.

To make it as easy as possible we need to provide high-level APIs, so 
dependencies will be unavoidable. I found for instance that Python bindings 
were very useful to write tests for DRM/KMS (using libkmsxx), and I plan to 
have a look at Python bindings for V4L2.

> Let's be honest, we (well, mainly me) are doing these tests as a side job,
> it's not our main focus.

That's a mindset that needs to evolve :-)

> Anything that makes writing tests more painful is bad and just gets in the
> way.

I don't see any disagreement on this. What makes it easy to write tests will 
however be much more prone to arguments.

-- 
Regards,

Laurent Pinchart