[STATUS UPDATE] test-media regression script status

[STATUS UPDATE] test-media regression script status

[Hans Verkuil]

Hi all,

As you all know I have been working on creating a regression test script
to help catch regressions before they get into released kernels.

The test-media script is in v4l-utils, contrib/test. It is currently testing
with vivid, vim2m and vimc. Once vicodec is stable enough it will be added
to the test suite as well.

It is already run as part of the daily build using a kernel config with many
debug options enabled.

The daily build summary looks like this:

	virtme: OK: Final Summary: 1981, Succeeded: 1981, Failed: 0, Warnings: 14

	<snip>

	Detailed regression test results are available here:

	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media.log
	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media-dmesg.log

The daily build is currently only checking for failures in the compliance tests
and ignores warnings. It also ignores issues in the kernel log.

The main problems that I am trying to solve are kernel oopses (should be solved
once all my pending 5.2 pull requests are merged) and vivid warnings (12 of the
warnings are known and work is in progress to fix this, the remainder of the
warnings I need to look at and are likely caused by v4l2-compliance itself).

Note that there is one known remaining kernel oops when testing vimc, but resolving
that takes a lot more time (low-level media life-cycle issues) and the test has
been disabled for now in test-media.

The test can be run on a system (or VM) running the kernel you want to test with
by simply running it as root.

Usually for V4L2/MC changes it is enough to run it as:

sudo test-media mc

If you want the script to unload all media modules first, then add the -unload
option. If you want the script to scan for memory leaks (and the corresponding
debug option is enabled in the kernel), then add the -kmemleak option. To dump
the kernel log when done, add the -dmesg option.

Run test-media without options to see a help text.

Note that 'mc' is shorthand for 'vivid vim2m vimc'.

There are also cec compliance tests and the 'all' test which tests everything
(used in the daily build). The cec tests are slow (especially the cec-pwr tests),
so unless you changed something in the cec subsystem you want to avoid running
those.

The test script assumes that the very latest v4l-utils are installed.

The test can also be run using virtme. I made a virtme.sh script to run the tests.
It assumes that it is run from the top of the kernel tree. You can find the script
here: https://hverkuil.home.xs4all.nl/virtme/

The start of the script documents where to get virtme and how to install it.

This also contains a kernel config that enables various kernel debug options.

I strongly recommend that testing is done using virtme since this allows you
to enable many kernel debug options that you don't want to enable on the host.
Delayed kobject release is a particularly nasty debug option :-)

Note: running virtme when a VMWare instance is also running on the same host
does not work reliably. I don't know why exactly, but the only way I can use virtme
is from inside a VMWare linux guest. Unfortunately, I need to use VMWare Workstation,
so I'm stuck with it.

I hope that during the 5.2 development cycle we can fix the last remaining issues
and this can become part of the regular development process.

Future plans (besides adding vicodec support) is adding support for other drivers
like uvc and perhaps au0828 (for testing the upcoming Media Device Allocator API).
But those tests require real hardware to be connected, and there may be different
variants of the same hardware. We'll see how that will work out.

Regards,

	Hans

Re: [STATUS UPDATE] test-media regression script status

[Guillaume Tucker]

Hi Hans,

On 13/03/2019 08:54, Hans Verkuil wrote:
> Hi all,
> 
> As you all know I have been working on creating a regression test script
> to help catch regressions before they get into released kernels.
> 
> The test-media script is in v4l-utils, contrib/test. It is currently testing
> with vivid, vim2m and vimc. Once vicodec is stable enough it will be added
> to the test suite as well.
> 
> It is already run as part of the daily build using a kernel config with many
> debug options enabled.

Sounds great.  Out of interest, do you have a config fragment or
the list of debug configs enabled in your builds somewhere?

> The daily build summary looks like this:
> 
> 	virtme: OK: Final Summary: 1981, Succeeded: 1981, Failed: 0, Warnings: 14
> 
> 	<snip>
> 
> 	Detailed regression test results are available here:
> 
> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media.log
> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media-dmesg.log
> 
> The daily build is currently only checking for failures in the compliance tests
> and ignores warnings. It also ignores issues in the kernel log.
[...]

> The test can be run on a system (or VM) running the kernel you want to test with
> by simply running it as root.
> 
> Usually for V4L2/MC changes it is enough to run it as:
> 
> sudo test-media mc
> 
> If you want the script to unload all media modules first, then add the -unload
> option. If you want the script to scan for memory leaks (and the corresponding
> debug option is enabled in the kernel), then add the -kmemleak option. To dump
> the kernel log when done, add the -dmesg option.
> 
> Run test-media without options to see a help text.
> 
> Note that 'mc' is shorthand for 'vivid vim2m vimc'.
> 
> There are also cec compliance tests and the 'all' test which tests everything
> (used in the daily build). The cec tests are slow (especially the cec-pwr tests),
> so unless you changed something in the cec subsystem you want to avoid running
> those.
> 
> The test script assumes that the very latest v4l-utils are installed.

This sounds very similar to what the KernelCI tests are doing.
The main differences are that your script has a neater way of
picking the media devices, and that the KernelCI tests break down
the results into individual cases to spot regressions
automatically [1].

At the moment we have a shell script to parse the output of
v4l2-compliance to find each test case result.  Ultimately, it
would be better to have some machine-readable output available,
as I believe we've already agreed.  It should then enable
KernelCI to run your test-media script directly, removing extra
post-processing and making the system more robust.

If this sounds like a good idea then we can discuss how to
proceed, i.e. agree on the format and command line options to
enable it.  A popular format which would seem suitable for this
is TAP 13 [2].  As Michał now has a fix to address the last issue
with the KernelCI email reports, we should be available to work
on this next.

[...]

> Future plans (besides adding vicodec support) is adding support for other drivers
> like uvc and perhaps au0828 (for testing the upcoming Media Device Allocator API).
> But those tests require real hardware to be connected, and there may be different
> variants of the same hardware. We'll see how that will work out.

This is obviously another strength of running the tests in
KernelCI, such hardware could be added to an existing test lab.

Best wishes,
Guillaume

[1] https://people.collabora.com/~gtucker/kernelci/doc/v4l2-compliance-uvcvideo-5.1-rc6.txt
[2] https://testanything.org/tap-version-13-specification.html

Re: [STATUS UPDATE] test-media regression script status

[Guillaume Tucker]

Hi Hans,

On 13/03/2019 08:54, Hans Verkuil wrote:
> Hi all,
> 
> As you all know I have been working on creating a regression test script
> to help catch regressions before they get into released kernels.
> 
> The test-media script is in v4l-utils, contrib/test. It is currently testing
> with vivid, vim2m and vimc. Once vicodec is stable enough it will be added
> to the test suite as well.
> 
> It is already run as part of the daily build using a kernel config with many
> debug options enabled.

Sounds great.  Out of interest, do you have a config fragment or
the list of debug configs enabled in your builds somewhere?

> The daily build summary looks like this:
> 
> 	virtme: OK: Final Summary: 1981, Succeeded: 1981, Failed: 0, Warnings: 14
> 
> 	<snip>
> 
> 	Detailed regression test results are available here:
> 
> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media.log
> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media-dmesg.log
> 
> The daily build is currently only checking for failures in the compliance tests
> and ignores warnings. It also ignores issues in the kernel log.
[...]

> The test can be run on a system (or VM) running the kernel you want to test with
> by simply running it as root.
> 
> Usually for V4L2/MC changes it is enough to run it as:
> 
> sudo test-media mc
> 
> If you want the script to unload all media modules first, then add the -unload
> option. If you want the script to scan for memory leaks (and the corresponding
> debug option is enabled in the kernel), then add the -kmemleak option. To dump
> the kernel log when done, add the -dmesg option.
> 
> Run test-media without options to see a help text.
> 
> Note that 'mc' is shorthand for 'vivid vim2m vimc'.
> 
> There are also cec compliance tests and the 'all' test which tests everything
> (used in the daily build). The cec tests are slow (especially the cec-pwr tests),
> so unless you changed something in the cec subsystem you want to avoid running
> those.
> 
> The test script assumes that the very latest v4l-utils are installed.

This sounds very similar to what the KernelCI tests are doing.
The main differences are that your script has a neater way of
picking the media devices, and that the KernelCI tests break down
the results into individual cases to spot regressions
automatically [1].

At the moment we have a shell script to parse the output of
v4l2-compliance to find each test case result.  Ultimately, it
would be better to have some machine-readable output available,
as I believe we've already agreed.  It should then enable
KernelCI to run your test-media script directly, removing extra
post-processing and making the system more robust.

If this sounds like a good idea then we can discuss how to
proceed, i.e. agree on the format and command line options to
enable it.  A popular format which would seem suitable for this
is TAP 13 [2].  As Michał now has a fix to address the last issue
with the KernelCI email reports, we should be available to work
on this next.

[...]

> Future plans (besides adding vicodec support) is adding support for other drivers
> like uvc and perhaps au0828 (for testing the upcoming Media Device Allocator API).
> But those tests require real hardware to be connected, and there may be different
> variants of the same hardware. We'll see how that will work out.

This is obviously another strength of running the tests in
KernelCI, such hardware could be added to an existing test lab.

Best wishes,
Guillaume

[1] https://people.collabora.com/~gtucker/kernelci/doc/v4l2-compliance-uvcvideo-5.1-rc6.txt
[2] https://testanything.org/tap-version-13-specification.html

Re: [STATUS UPDATE] test-media regression script status

[Hans Verkuil]

On 4/26/19 9:06 AM, Guillaume Tucker wrote:
> Hi Hans,
> 
> On 13/03/2019 08:54, Hans Verkuil wrote:
>> Hi all,
>>
>> As you all know I have been working on creating a regression test script
>> to help catch regressions before they get into released kernels.
>>
>> The test-media script is in v4l-utils, contrib/test. It is currently testing
>> with vivid, vim2m and vimc. Once vicodec is stable enough it will be added
>> to the test suite as well.
>>
>> It is already run as part of the daily build using a kernel config with many
>> debug options enabled.
> 
> Sounds great.  Out of interest, do you have a config fragment or
> the list of debug configs enabled in your builds somewhere?

https://git.linuxtv.org/hverkuil/build-scripts.git/tree/configs/virtme.config

> 
>> The daily build summary looks like this:
>>
>> 	virtme: OK: Final Summary: 1981, Succeeded: 1981, Failed: 0, Warnings: 14
>>
>> 	<snip>
>>
>> 	Detailed regression test results are available here:
>>
>> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media.log
>> 	http://www.xs4all.nl/~hverkuil/logs/Wednesday-test-media-dmesg.log
>>
>> The daily build is currently only checking for failures in the compliance tests
>> and ignores warnings. It also ignores issues in the kernel log.
> [...]
> 
>> The test can be run on a system (or VM) running the kernel you want to test with
>> by simply running it as root.
>>
>> Usually for V4L2/MC changes it is enough to run it as:
>>
>> sudo test-media mc
>>
>> If you want the script to unload all media modules first, then add the -unload
>> option. If you want the script to scan for memory leaks (and the corresponding
>> debug option is enabled in the kernel), then add the -kmemleak option. To dump
>> the kernel log when done, add the -dmesg option.
>>
>> Run test-media without options to see a help text.
>>
>> Note that 'mc' is shorthand for 'vivid vim2m vimc'.
>>
>> There are also cec compliance tests and the 'all' test which tests everything
>> (used in the daily build). The cec tests are slow (especially the cec-pwr tests),
>> so unless you changed something in the cec subsystem you want to avoid running
>> those.
>>
>> The test script assumes that the very latest v4l-utils are installed.
> 
> This sounds very similar to what the KernelCI tests are doing.
> The main differences are that your script has a neater way of
> picking the media devices, and that the KernelCI tests break down
> the results into individual cases to spot regressions
> automatically [1].
> 
> At the moment we have a shell script to parse the output of
> v4l2-compliance to find each test case result.  Ultimately, it
> would be better to have some machine-readable output available,
> as I believe we've already agreed.  It should then enable
> KernelCI to run your test-media script directly, removing extra
> post-processing and making the system more robust.
> 
> If this sounds like a good idea then we can discuss how to
> proceed, i.e. agree on the format and command line options to
> enable it.  A popular format which would seem suitable for this
> is TAP 13 [2].  As Michał now has a fix to address the last issue
> with the KernelCI email reports, we should be available to work
> on this next.

Sure, not a problem. As long as the same command line option and same
format it implemented for both v4l2-compliance and cec-compliance.

It's probably best to write the machine-readable output to a file,
so an option like --tap13 <file> would be a natural choice.

Most (all?) tests are done like this:

               printf("\ttest VIDIOC_QUERYCAP: %s\n", ok(testCap(&node)));

This should probably change to:

	run_test("VIDIOC_QUERYCAP", testCap(&node));

And run_test() would then print the human readable text and, if enabled,
write the machine-readable results to a file.

There might be some corner cases where this is a bit more complex.

> 
> [...]
> 
>> Future plans (besides adding vicodec support) is adding support for other drivers
>> like uvc and perhaps au0828 (for testing the upcoming Media Device Allocator API).
>> But those tests require real hardware to be connected, and there may be different
>> variants of the same hardware. We'll see how that will work out.
> 
> This is obviously another strength of running the tests in
> KernelCI, such hardware could be added to an existing test lab.

Absolutely.

Regards,

	Hans

> 
> Best wishes,
> Guillaume
> 
> [1] https://people.collabora.com/~gtucker/kernelci/doc/v4l2-compliance-uvcvideo-5.1-rc6.txt
> [2] https://testanything.org/tap-version-13-specification.html
>