* Test results (v4l2) for media/master - v4.20-rc5-281-gd2b4387f3bdf
@ 2018-12-18 15:31 kernelci.org bot
2018-12-18 17:52 ` Mauro Carvalho Chehab
0 siblings, 1 reply; 3+ messages in thread
From: kernelci.org bot @ 2018-12-18 15:31 UTC (permalink / raw)
To: linux-media, ezequiel, guillaume.tucker, ana.guerrero
Test results for:
Tree: media
Branch: master
Kernel: v4.20-rc5-281-gd2b4387f3bdf
URL: https://git.linuxtv.org/media_tree.git
Commit: d2b4387f3bdf016e266d23cf657465f557721488
Test plans: v4l2
Summary
-------
4 test groups results
1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 6 FAIL 26 SKIP
3 | v4l2 | qemu | arm64 | 115 total: 77 PASS 7 FAIL 31 SKIP
4 | v4l2 | qemu | arm | 115 total: 77 PASS 7 FAIL 31 SKIP
---
1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
Config: defconfig
Lab Name: lab-collabora
Date: 2018-12-14 19:51:24.841000
TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.txt
HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.html
Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/arm64/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
Output-ioctls - 5 tests: 0 PASS, 0 FAIL, 5 SKIP
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
Input/Output-configuration-ioctls - 4 tests: 0 PASS, 0 FAIL, 4 SKIP
Control-ioctls-Input-0 - 6 tests: 3 PASS, 2 FAIL, 1 SKIP
* VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
* VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Debug-ioctls - 2 tests: 0 PASS, 0 FAIL, 2 SKIP
Format-ioctls-Input-0 - 10 tests: 4 PASS, 1 FAIL, 5 SKIP
* VIDIOC_G/S_PARM: FAIL
Streaming-ioctls_Test-input-0 - 4 tests: 1 PASS, 2 FAIL, 1 SKIP
* USERPTR: FAIL
* MMAP: FAIL
Input-ioctls - 6 tests: 1 PASS, 0 FAIL, 5 SKIP
Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 6 FAIL 26 SKIP
Config: multi_v7_defconfig
Lab Name: lab-collabora
Date: 2018-12-14 19:51:35.774000
TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig/lab-collabora/v4l2-rk3288-veyron-jaq.txt
HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig/lab-collabora/v4l2-rk3288-veyron-jaq.html
Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/armhf/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
Output-ioctls - 5 tests: 0 PASS, 0 FAIL, 5 SKIP
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
Input/Output-configuration-ioctls - 4 tests: 0 PASS, 0 FAIL, 4 SKIP
Control-ioctls-Input-0 - 6 tests: 3 PASS, 2 FAIL, 1 SKIP
* VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
* VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Debug-ioctls - 2 tests: 0 PASS, 0 FAIL, 2 SKIP
Format-ioctls-Input-0 - 10 tests: 4 PASS, 1 FAIL, 5 SKIP
* VIDIOC_G/S_PARM: FAIL
Streaming-ioctls_Test-input-0 - 4 tests: 1 PASS, 2 FAIL, 1 SKIP
* USERPTR: FAIL
* MMAP: FAIL
Input-ioctls - 6 tests: 1 PASS, 0 FAIL, 5 SKIP
Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
3 | v4l2 | qemu | arm64 | 115 total: 77 PASS 7 FAIL 31 SKIP
Config: defconfig+virtualvideo
Lab Name: lab-collabora
Date: 2018-12-14 20:04:54.451000
TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig+virtualvideo/lab-collabora/v4l2-qemu.txt
HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig+virtualvideo/lab-collabora/v4l2-qemu.html
Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/arm64/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
Buffer-ioctls-Input-3 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Buffer-ioctls-Input-2 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Buffer-ioctls-Input-1 - 3 tests: 1 PASS, 2 FAIL, 0 SKIP
* Requests: FAIL
* VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Control-ioctls-Input-3 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-2 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-1 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-0 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Debug-ioctls - 2 tests: 1 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-2 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-3 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-0 - 10 tests: 6 PASS, 0 FAIL, 4 SKIP
Format-ioctls-Input-1 - 10 tests: 5 PASS, 2 FAIL, 3 SKIP
* Scaling: FAIL
* VIDIOC_S_FMT: FAIL
Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
Input/Output-configuration-ioctls - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
Codec-ioctls-Input-2 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-3 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-1 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Output-ioctls - 5 tests: 1 PASS, 0 FAIL, 4 SKIP
Streaming-ioctls_Test-input-0 - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
Input-ioctls - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
4 | v4l2 | qemu | arm | 115 total: 77 PASS 7 FAIL 31 SKIP
Config: multi_v7_defconfig+virtualvideo
Lab Name: lab-collabora
Date: 2018-12-14 20:10:05.962000
TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig+virtualvideo/lab-collabora/v4l2-qemu.txt
HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig+virtualvideo/lab-collabora/v4l2-qemu.html
Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/armhf/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
Buffer-ioctls-Input-3 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Buffer-ioctls-Input-2 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Buffer-ioctls-Input-1 - 3 tests: 1 PASS, 2 FAIL, 0 SKIP
* Requests: FAIL
* VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: FAIL
Control-ioctls-Input-3 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-2 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-1 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Control-ioctls-Input-0 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Debug-ioctls - 2 tests: 1 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-2 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-3 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
Format-ioctls-Input-0 - 10 tests: 6 PASS, 0 FAIL, 4 SKIP
Format-ioctls-Input-1 - 10 tests: 5 PASS, 2 FAIL, 3 SKIP
* Scaling: FAIL
* VIDIOC_S_FMT: FAIL
Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
Input/Output-configuration-ioctls - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
Codec-ioctls-Input-2 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-3 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Codec-ioctls-Input-1 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
Output-ioctls - 5 tests: 1 PASS, 0 FAIL, 4 SKIP
Streaming-ioctls_Test-input-0 - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
Input-ioctls - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Test results (v4l2) for media/master - v4.20-rc5-281-gd2b4387f3bdf
2018-12-18 15:31 Test results (v4l2) for media/master - v4.20-rc5-281-gd2b4387f3bdf kernelci.org bot
@ 2018-12-18 17:52 ` Mauro Carvalho Chehab
2019-01-24 9:33 ` Guillaume Tucker
0 siblings, 1 reply; 3+ messages in thread
From: Mauro Carvalho Chehab @ 2018-12-18 17:52 UTC (permalink / raw)
To: ezequiel; +Cc: linux-media, guillaume.tucker, ana.guerrero
Em Tue, 18 Dec 2018 07:31:23 -0800 (PST)
"kernelci.org bot" <bot@kernelci.org> escreveu:
> Test results for:
> Tree: media
> Branch: master
> Kernel: v4.20-rc5-281-gd2b4387f3bdf
> URL: https://git.linuxtv.org/media_tree.git
> Commit: d2b4387f3bdf016e266d23cf657465f557721488
> Test plans: v4l2
>
> Summary
> -------
> 4 test groups results
>
> 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
> 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 6 FAIL 26 SKIP
> 3 | v4l2 | qemu | arm64 | 115 total: 77 PASS 7 FAIL 31 SKIP
> 4 | v4l2 | qemu | arm | 115 total: 77 PASS 7 FAIL 31 SKIP
Please add, at linuxtv.org's wiki page, what each test actually means...
>
> ---
> 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
>
> Config: defconfig
> Lab Name: lab-collabora
> Date: 2018-12-14 19:51:24.841000
> TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.txt
> HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.html
> Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/arm64/rootfs.cpio.gz
> Test Git: git://linuxtv.org/v4l-utils.git
> Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
>
>
> Output-ioctls - 5 tests: 0 PASS, 0 FAIL, 5 SKIP
>
> Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
>
> Input/Output-configuration-ioctls - 4 tests: 0 PASS, 0 FAIL, 4 SKIP
>
> Control-ioctls-Input-0 - 6 tests: 3 PASS, 2 FAIL, 1 SKIP
> * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
> * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
... for example, in this specific case, I have no idea what driver
failed. Ok, one could open the log txt file, look on it and
discover that this specific test was against the uvcvideo driver,
but it is doubtful to expect that everybody would do that.
The best would be to also c/c the developer of the specific
driver, if listed at MAINTAINERS, as he is the one that should
come up with a fix.
I also think that the patch subject should be changed to reflect the
actual problems that was got, e. g., something like:
[KernelCI] v4l2-compliance: uvcvideo: 6 failures, driver_foo: 8 failures
>
> Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Debug-ioctls - 2 tests: 0 PASS, 0 FAIL, 2 SKIP
>
> Format-ioctls-Input-0 - 10 tests: 4 PASS, 1 FAIL, 5 SKIP
> * VIDIOC_G/S_PARM: FAIL
>
> Streaming-ioctls_Test-input-0 - 4 tests: 1 PASS, 2 FAIL, 1 SKIP
> * USERPTR: FAIL
> * MMAP: FAIL
>
> Input-ioctls - 6 tests: 1 PASS, 0 FAIL, 5 SKIP
>
> Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 6 FAIL 26 SKIP
>
> Config: multi_v7_defconfig
> Lab Name: lab-collabora
> Date: 2018-12-14 19:51:35.774000
> TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig/lab-collabora/v4l2-rk3288-veyron-jaq.txt
> HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig/lab-collabora/v4l2-rk3288-veyron-jaq.html
> Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/armhf/rootfs.cpio.gz
> Test Git: git://linuxtv.org/v4l-utils.git
> Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
>
>
> Output-ioctls - 5 tests: 0 PASS, 0 FAIL, 5 SKIP
>
> Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
>
> Input/Output-configuration-ioctls - 4 tests: 0 PASS, 0 FAIL, 4 SKIP
>
> Control-ioctls-Input-0 - 6 tests: 3 PASS, 2 FAIL, 1 SKIP
> * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
> * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
>
> Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Debug-ioctls - 2 tests: 0 PASS, 0 FAIL, 2 SKIP
>
> Format-ioctls-Input-0 - 10 tests: 4 PASS, 1 FAIL, 5 SKIP
> * VIDIOC_G/S_PARM: FAIL
>
> Streaming-ioctls_Test-input-0 - 4 tests: 1 PASS, 2 FAIL, 1 SKIP
> * USERPTR: FAIL
> * MMAP: FAIL
>
> Input-ioctls - 6 tests: 1 PASS, 0 FAIL, 5 SKIP
>
> Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> 3 | v4l2 | qemu | arm64 | 115 total: 77 PASS 7 FAIL 31 SKIP
>
> Config: defconfig+virtualvideo
> Lab Name: lab-collabora
> Date: 2018-12-14 20:04:54.451000
> TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig+virtualvideo/lab-collabora/v4l2-qemu.txt
> HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig+virtualvideo/lab-collabora/v4l2-qemu.html
> Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/arm64/rootfs.cpio.gz
> Test Git: git://linuxtv.org/v4l-utils.git
> Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
>
>
> Buffer-ioctls-Input-3 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Buffer-ioctls-Input-2 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Buffer-ioctls-Input-1 - 3 tests: 1 PASS, 2 FAIL, 0 SKIP
> * Requests: FAIL
> * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
>
> Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Control-ioctls-Input-3 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-2 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-1 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-0 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Debug-ioctls - 2 tests: 1 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-2 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-3 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-0 - 10 tests: 6 PASS, 0 FAIL, 4 SKIP
>
> Format-ioctls-Input-1 - 10 tests: 5 PASS, 2 FAIL, 3 SKIP
> * Scaling: FAIL
> * VIDIOC_S_FMT: FAIL
>
> Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
>
> Input/Output-configuration-ioctls - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> Codec-ioctls-Input-2 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-3 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-1 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Output-ioctls - 5 tests: 1 PASS, 0 FAIL, 4 SKIP
>
> Streaming-ioctls_Test-input-0 - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> Input-ioctls - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> 4 | v4l2 | qemu | arm | 115 total: 77 PASS 7 FAIL 31 SKIP
>
> Config: multi_v7_defconfig+virtualvideo
> Lab Name: lab-collabora
> Date: 2018-12-14 20:10:05.962000
> TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig+virtualvideo/lab-collabora/v4l2-qemu.txt
> HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm/multi_v7_defconfig+virtualvideo/lab-collabora/v4l2-qemu.html
> Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/armhf/rootfs.cpio.gz
> Test Git: git://linuxtv.org/v4l-utils.git
> Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
>
>
> Buffer-ioctls-Input-3 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Buffer-ioctls-Input-2 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Buffer-ioctls-Input-1 - 3 tests: 1 PASS, 2 FAIL, 0 SKIP
> * Requests: FAIL
> * VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: FAIL
>
> Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
> * Requests: FAIL
>
> Control-ioctls-Input-3 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-2 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-1 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Control-ioctls-Input-0 - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Debug-ioctls - 2 tests: 1 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-2 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-3 - 10 tests: 9 PASS, 0 FAIL, 1 SKIP
>
> Format-ioctls-Input-0 - 10 tests: 6 PASS, 0 FAIL, 4 SKIP
>
> Format-ioctls-Input-1 - 10 tests: 5 PASS, 2 FAIL, 3 SKIP
> * Scaling: FAIL
> * VIDIOC_S_FMT: FAIL
>
> Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
>
> Input/Output-configuration-ioctls - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> Codec-ioctls-Input-2 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-3 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Codec-ioctls-Input-1 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>
> Output-ioctls - 5 tests: 1 PASS, 0 FAIL, 4 SKIP
>
> Streaming-ioctls_Test-input-0 - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
> Input-ioctls - 6 tests: 5 PASS, 0 FAIL, 1 SKIP
>
> Allow-for-multiple-opens - 4 tests: 4 PASS, 0 FAIL, 0 SKIP
>
Again, I would simplify the report, printing only a summary of the
failures. You can store a less summarized report at the KernelCI
storage server.
Thanks,
Mauro
^ permalink raw reply [flat|nested] 3+ messages in thread
* Re: Test results (v4l2) for media/master - v4.20-rc5-281-gd2b4387f3bdf
2018-12-18 17:52 ` Mauro Carvalho Chehab
@ 2019-01-24 9:33 ` Guillaume Tucker
0 siblings, 0 replies; 3+ messages in thread
From: Guillaume Tucker @ 2019-01-24 9:33 UTC (permalink / raw)
To: Mauro Carvalho Chehab, ezequiel; +Cc: linux-media, kernel
Hi Mauro,
Thanks for the review. We've made some progress but I would like
to check things with you before putting this in production,
please see below.
On 18/12/2018 17:52, Mauro Carvalho Chehab wrote:
> Em Tue, 18 Dec 2018 07:31:23 -0800 (PST)
> "kernelci.org bot" <bot@kernelci.org> escreveu:
>
>> Test results for:
>> Tree: media
>> Branch: master
>> Kernel: v4.20-rc5-281-gd2b4387f3bdf
>> URL: https://git.linuxtv.org/media_tree.git
>> Commit: d2b4387f3bdf016e266d23cf657465f557721488
>> Test plans: v4l2
>>
>> Summary
>> -------
>> 4 test groups results
>>
>> 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
>> 2 | v4l2 | rk3288-veyron-jaq | arm | 49 total: 17 PASS 6 FAIL 26 SKIP
>> 3 | v4l2 | qemu | arm64 | 115 total: 77 PASS 7 FAIL 31 SKIP
>> 4 | v4l2 | qemu | arm | 115 total: 77 PASS 7 FAIL 31 SKIP
>
> Please add, at linuxtv.org's wiki page, what each test actually means...
I've now added an explanation with a link to the wiki:
"""
V4L2 Compliance on the uvcvideo driver.
This test ran "v4l2-compliance -s" from v4l-utils:
https://www.linuxtv.org/wiki/index.php/V4l2-utils
See each detailed section in the report below to find out the git URL and
particular revision that was used to build the test binaries.
"""
Is this the kind of information you would expect to see here?
>> ---
>> 1 | v4l2 | rk3399-gru-kevin | arm64 | 49 total: 17 PASS 6 FAIL 26 SKIP
>>
>> Config: defconfig
>> Lab Name: lab-collabora
>> Date: 2018-12-14 19:51:24.841000
>> TXT log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.txt
>> HTML log: https://storage.kernelci.org//media/master/v4.20-rc5-281-gd2b4387f3bdf/arm64/defconfig/lab-collabora/v4l2-rk3399-gru-kevin.html
>> Rootfs: http://storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20181207.0/arm64/rootfs.cpio.gz
>> Test Git: git://linuxtv.org/v4l-utils.git
>> Test Commit: 7a118f166609c0d05c6447cc79484af37875d6fc
>>
>>
>> Output-ioctls - 5 tests: 0 PASS, 0 FAIL, 5 SKIP
>>
>> Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
>> * Requests: FAIL
>>
>> Required-ioctls - 2 tests: 2 PASS, 0 FAIL, 0 SKIP
>>
>> Input/Output-configuration-ioctls - 4 tests: 0 PASS, 0 FAIL, 4 SKIP
>>
>> Control-ioctls-Input-0 - 6 tests: 3 PASS, 2 FAIL, 1 SKIP
>> * VIDIOC_G/S/TRY_EXT_CTRLS: FAIL
>> * VIDIOC_QUERY_EXT_CTRL/QUERYMENU: FAIL
>
> ... for example, in this specific case, I have no idea what driver
> failed. Ok, one could open the log txt file, look on it and
> discover that this specific test was against the uvcvideo driver,
> but it is doubtful to expect that everybody would do that.
Each test is now run on a per-driver basis, on relevant
platforms. So there's vivid on QEMU and uvcvideo on a couple of
real hardware devices as shown in this example summary:
1 | rk3288-veyron-jaq | arm | 51 total: 17 PASS 8 FAIL 26 SKIP
2 | rk3399-gru-kevin | arm64 | 51 total: 17 PASS 8 FAIL 26 SKIP
The driver name is mentioned in the email subject and at the top
of the report. The test first looks for a /dev/video* device
corresponding to the requested driver, and then passes that on to
the v4l2-compliance test suite as an argument.
> The best would be to also c/c the developer of the specific
> driver, if listed at MAINTAINERS, as he is the one that should
> come up with a fix.
Sure, it's easy to add the maintainers to the recipients for each
type of report (i.e. Hans for vivid and Laurent for uvcvideo, and
you for both) in addition to the linux-media list.
> I also think that the patch subject should be changed to reflect the
> actual problems that was got, e. g., something like:
>
> [KernelCI] v4l2-compliance: uvcvideo: 6 failures, driver_foo: 8 failures
The email subject has been changed to look more like other
KernelCI reports and take your suggestion into account. For this
particular test report, it would now look like this:
media/master v4l2-compliance on uvcvideo: 102 tests, 0 regressions (v4.20-rc5-281-gd2b4387f3bdf)
There also would be another separate one with results for vivid.
Other mailing lists are receiving similar reports without a
[KernelCI] tag or anything, but it's easy to filter them based on
the sender as it's always "kernelci.org bot" <bot@kernelci.org>.
>> Codec-ioctls-Input-0 - 3 tests: 0 PASS, 0 FAIL, 3 SKIP
>>
>> Debug-ioctls - 2 tests: 0 PASS, 0 FAIL, 2 SKIP
>>
>> Format-ioctls-Input-0 - 10 tests: 4 PASS, 1 FAIL, 5 SKIP
>> * VIDIOC_G/S_PARM: FAIL
>>
>> Streaming-ioctls_Test-input-0 - 4 tests: 1 PASS, 2 FAIL, 1 SKIP
>> * USERPTR: FAIL
>> * MMAP: FAIL
[...]
> Again, I would simplify the report, printing only a summary of the
> failures. You can store a less summarized report at the KernelCI
> storage server.
The report now only shows failures and regression information (as
in the example below). There are links to see the full console
log of the test run, which helps but I think it would be good to
also have a separate log file with just the full test output.
Please let us know if you're happy with this or if it needs
further changes before enabling the automated reports again.
Best wishes,
Guillaume
> Thanks,
> Mauro
Sample report from a test branch with a regression caused on
purpose:
----8<--------8<----
gtucker/kernelci-local v4l2-compliance on vivid: 102 tests, 4 regressions (kernelci-local-snapshot-016-6-g6fa02d2c8766)
Test results summary
--------------------
V4L2 Compliance on the vivid driver.
This test ran "v4l2-compliance -s" from v4l-utils:
https://www.linuxtv.org/wiki/index.php/V4l2-utils
See each detailed section in the report below to find out the git URL and
particular revision that was used to build the test binaries.
Tree: gtucker
Branch: kernelci-local
Kernel: kernelci-local-snapshot-016-6-g6fa02d2c8766
URL: https://gitlab.collabora.com/gtucker/linux.git
Commit: 6fa02d2c87664d7bc19de3ea7b288bfba7d36644
1 | qemu | arm64 | 51 total: 29 PASS 4 FAIL 18 SKIP
2 | qemu | arm | 51 total: 29 PASS 4 FAIL 18 SKIP
Test failures
-------------
1 | qemu | arm64 | 51 total: 29 PASS 4 FAIL 18 SKIP
Config: defconfig+virtualvideo
Lab Name: lab-collabora-dev
Plain log: http://staging-storage.kernelci.org/gtucker/kernelci-local/kernelci-local-snapshot-016-6-g6fa02d2c8766/arm64/defconfig+virtualvideo/lab-collabora-dev/v4l2-compliance-vivid-qemu.txt
HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-local/kernelci-local-snapshot-016-6-g6fa02d2c8766/arm64/defconfig+virtualvideo/lab-collabora-dev/v4l2-compliance-vivid-qemu.html
Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20190117.0/arm64/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 6de7a0df7e8c06af36ed609d87649e704279ea2d
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: never passed
Input-ioctls - 15 tests: 3 PASS, 3 FAIL, 9 SKIP
* VIDIOC_G/S_AUDOUT: never passed
* VIDIOC_G/S/ENUMINPUT: new failure (last pass: kernelci-local-snapshot-016-5-gcec25f749cfa)
* VIDIOC_ENUMAUDIO: new failure (last pass: kernelci-local-snapshot-016-5-gcec25f749cfa)
2 | qemu | arm | 51 total: 29 PASS 4 FAIL 18 SKIP
Config: multi_v7_defconfig+virtualvideo
Lab Name: lab-collabora-dev
Plain log: http://staging-storage.kernelci.org/gtucker/kernelci-local/kernelci-local-snapshot-016-6-g6fa02d2c8766/arm/multi_v7_defconfig+virtualvideo/lab-collabora-dev/v4l2-compliance-vivid-qemu.txt
HTML log: http://staging-storage.kernelci.org/gtucker/kernelci-local/kernelci-local-snapshot-016-6-g6fa02d2c8766/arm/multi_v7_defconfig+virtualvideo/lab-collabora-dev/v4l2-compliance-vivid-qemu.html
Rootfs: http://staging-storage.kernelci.org/images/rootfs/debian/stretch-v4l2/20190117.0/armhf/rootfs.cpio.gz
Test Git: git://linuxtv.org/v4l-utils.git
Test Commit: 6de7a0df7e8c06af36ed609d87649e704279ea2d
Buffer-ioctls-Input-0 - 3 tests: 2 PASS, 1 FAIL, 0 SKIP
* Requests: never passed
Input-ioctls - 15 tests: 3 PASS, 3 FAIL, 9 SKIP
* VIDIOC_G/S_AUDOUT: never passed
* VIDIOC_G/S/ENUMINPUT: new failure (last pass: kernelci-local-snapshot-016-5-gcec25f749cfa)
* VIDIOC_ENUMAUDIO: new failure (last pass: kernelci-local-snapshot-016-5-gcec25f749cfa)
^ permalink raw reply [flat|nested] 3+ messages in thread
end of thread, other threads:[~2019-01-24 9:33 UTC | newest]
Thread overview: 3+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-12-18 15:31 Test results (v4l2) for media/master - v4.20-rc5-281-gd2b4387f3bdf kernelci.org bot
2018-12-18 17:52 ` Mauro Carvalho Chehab
2019-01-24 9:33 ` Guillaume Tucker
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).