RE: [EXT] Re: [PATCH v12 00/13] amphion video decoder/encoder driver

[Ming Qian <ming.qian@nxp.com>]

> -----Original Message-----
> From: Nicolas Dufresne [mailto:nicolas@ndufresne.ca]
> Sent: Wednesday, November 24, 2021 3:23 AM
> To: Ming Qian <ming.qian@nxp.com>; mchehab@kernel.org;
> shawnguo@kernel.org; robh+dt@kernel.org; s.hauer@pengutronix.de
> Cc: hverkuil-cisco@xs4all.nl; kernel@pengutronix.de; festevam@gmail.com;
> dl-linux-imx <linux-imx@nxp.com>; Aisheng Dong <aisheng.dong@nxp.com>;
> linux-media@vger.kernel.org; linux-kernel@vger.kernel.org;
> devicetree@vger.kernel.org; linux-arm-kernel@lists.infradead.org
> Subject: [EXT] Re: [PATCH v12 00/13] amphion video decoder/encoder driver
> 
> Caution: EXT Email
> 
> Hi Ming,
> 
> For the patchset:
> 
> Tested-By: Nicolas Dufresne <nicolas.dufresne@collabora.com>
> 
> I've validated encoding manually with GStreamer:
> 
>   gst-launch-1.0 videotestsrc num-buffers=100 ! v4l2h264enc !
> video/x-h264,profile=main,level=\(string\)2 ! h264parse ! qtmux ! filesink
> location=test.mov
> 
> I've also verify the that the number of frames was exactly 100, this is common
> issue with V4L2 encoders. Then I have validated VP8, H.264 and H.265
> decoders
> using fluster [0] with this PR [1] applied. You can find full summary at the end
> of this email. Markdown report didn't get generated, I will have to check with
> upstream fluster if there is a regression.
> 
> $> ./fluster.py run -s -so amphion-imx8qxp-conformance.md -d
> GStreamer-VP8-V4L2-Gst1.0 GStreamer-H.264-V4L2-Gst1.0
> GStreamer-H.265-V4L2-Gst1.0
> 
> VP8:   Ran 59/61 tests successfully               in 131.788 secs
> H.264: Ran 75/135 tests successfully               in 501.206 secs
> H.265: Ran 126/147 tests successfully               in 1131.933 secs
> 
> Note that in mainline, only 1 core get fired and is kept at its lowest possible
> frequency, so perhaps it may cause some of the timeout seen. The driver is
> overall functional, and I would like to thank you for this extra work. Also,
> note that this very first time I run Fluster over the stateful CODEC wrappers. I
> will need to run this on more platforms to locate the GStreamer specific fail.
> 
> VP8 note, conformance vector vp80-03-segmentation-1425 cause a hang but it
> then
> recover:
> 
> [ 8264.851841] amphion-vpu-core 2d040000.vpu_core: [0] sync session
> timeout
> [ 8264.858634] amphion-vpu-core 2d040000.vpu_core: [0] send cmd(0x2) fail
> [ 8264.867992] amphion-vpu-core 2d040000.vpu_core: [0] start fail
> [ 8264.905173] amphion-vpu-core 2d040000.vpu_core: reset hang core
> 

HI Nicolas

    There is a bug in firmware that if send a command to firmware too close after stop cmd,
The firmware may enter wfi wrong, and led to hang issue you met in vp80-03-segmentation-1425.
I'll add a workaround in driver that add a delay after send stop cmd to firmware in next version.

    Because the amphion's vpu doesn't support to output i420, so the test will convert nv12_8l128 to i420 by videoconvert, it leds to most of timeout failure.

    The FM1_BT_B.h264 can't be decoded by amphion's vpu, the vpu is keeping parse sequence header, and it led to timeout failure.

    I run the test and change the timeout to 300, then most of timeout failures are gone. Besides that, my result is almost as same as yours.

    The failures of assertion error means that the vpu's output is different from the pattern, I think it should be the vpu's limitation.

> 
> See comments about your GStreamer MR below...
> 
> [0]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Ffluendo%2Ffluster&amp;data=04%7C01%7Cming.qian%40nxp.com%7
> C45d33490bfe546aa8e1408d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c3
> 01635%7C0%7C0%7C637732921809373007%7CUnknown%7CTWFpbGZsb3d
> 8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3
> D%7C3000&amp;sdata=bBqADaZ9ZN00SbCPalQE5fN740hB2bMUxHDL4QEki
> mg%3D&amp;reserved=0
> [1]
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.
> com%2Ffluendo%2Ffluster%2Fpull%2F98&amp;data=04%7C01%7Cming.qian
> %40nxp.com%7C45d33490bfe546aa8e1408d9aeb6a822%7C686ea1d3bc2b4c
> 6fa92cd99c5c301635%7C0%7C0%7C637732921809382989%7CUnknown%7C
> TWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLC
> JXVCI6Mn0%3D%7C3000&amp;sdata=90dUC%2FNzK%2BcFNSMwY7bgQ0iD8
> ctBwU%2FMhyrLoBBEf4g%3D&amp;reserved=0
> 
> 
> [...]
> > encoder is tested with gstreamer
> > decoder is tested with gstreamer, but the following patches are required:
> 
> GStreamer is now a single repo (and using main branch instead of master), I
> have
> migrated your 3 MRs into 2 MRs.

Thank you very much.
> 
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.fr
> eedesktop.org%2Fgstreamer%2Fgst-plugins-base%2F-%2Fmerge_requests%2F
> 1252&amp;data=04%7C01%7Cming.qian%40nxp.com%7C45d33490bfe546aa
> 8e1408d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7
> C637732921809382989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sda
> ta=G4PnqxRsnPjQtV%2FUcp4dkoz9fAGxkOv%2FzKR47RaHtu0%3D&amp;reser
> ved=0
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.fr
> eedesktop.org%2Fgstreamer%2Fgst-plugins-good%2F-%2Fmerge_requests%2F
> 1098&amp;data=04%7C01%7Cming.qian%40nxp.com%7C45d33490bfe546aa
> 8e1408d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7
> C637732921809382989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sda
> ta=h6nEISAaiG54NPz9Xw0MiSNDx2QxWsNWyG7FLSAaLgI%3D&amp;reserved
> =0
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.fr
> eedesktop.org%2Fgstreamer%2Fgstreamer%2F-%2Fmerge_requests%2F1379&
> amp;data=04%7C01%7Cming.qian%40nxp.com%7C45d33490bfe546aa8e140
> 8d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6377
> 32921809382989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=PFe
> Db4v%2BmiksgvJ08KgncRjZEt7GdMO7gV8hV0DD0uY%3D&amp;reserved=0
> 
> >
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.fr
> eedesktop.org%2Fgstreamer%2Fgst-plugins-good%2F-%2Fmerge_requests%2F
> 1099&amp;data=04%7C01%7Cming.qian%40nxp.com%7C45d33490bfe546aa
> 8e1408d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7
> C637732921809382989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAw
> MDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sda
> ta=e6hSDpiP2scLN5C0f8Fb%2FoMbt8k4zC45qWPrQpG%2Fe5A%3D&amp;rese
> rved=0
> 
> https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgitlab.fr
> eedesktop.org%2Fgstreamer%2Fgstreamer%2F-%2Fmerge_requests%2F1381&
> amp;data=04%7C01%7Cming.qian%40nxp.com%7C45d33490bfe546aa8e140
> 8d9aeb6a822%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6377
> 32921809382989%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiL
> CJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&amp;sdata=xkX
> WjCsgVaLb0wZgfaXDMiBchtbJBNnwtZNhKe42k5o%3D&amp;reserved=0
> 
> 
> regards,
> Nicolas
> 
> [JVT-AVC_V1] (GStreamer-H.264-V4L2-Gst1.0) FM1_BT_B                 ...
> Timeout
The vpu can't decode this stream.

> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> AMP_D_Hisilicon_3               ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> AMP_E_Hisilicon_3               ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> AMP_F_Hisilicon_3               ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> BUMPING_A_ericsson_1            ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> CONFWIN_A_Sony_1                ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> DELTAQP_A_BRCM_4                ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> MVDL1ZERO_A_docomo_4            ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> NoOutPrior_A_Qualcomm_1         ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> NoOutPrior_B_Qualcomm_1         ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> NUT_A_ericsson_5                ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> PICSIZE_A_Bossen_1              ... Error
The size is 1056x8440, vpu doesn't support it
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> PICSIZE_B_Bossen_1              ... Error
The size is 8440x1056, vpu doesn't support it
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> PICSIZE_C_Bossen_1              ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> PICSIZE_D_Bossen_1              ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> RAP_B_Bossen_2                  ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> SAODBLK_A_MainConcept_4         ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> SAODBLK_B_MainConcept_4         ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> TILES_A_Cisco_2                 ... Timeout
Performance issue of videoconvert
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> TILES_B_Cisco_1                 ... Timeout
Performance issue of videoconvert, but assert fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> TSUNEQBD_A_MAIN10_Technicolor_2 ... Fail
> [JCT-VC-HEVC_V1] (GStreamer-H.265-V4L2-Gst1.0)
> VPSSPSPPS_A_MainConcept_1       ... Error
Gstream report error:
ERROR Error from element /GstPlayBin:playbin/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/v4l2h265dec:v4l2h265dec0: Could not read from resource.
Could not read from resource.
../sys/v4l2/gstv4l2bufferpool.c(1155): gst_v4l2_buffer_pool_poll (): /GstPlayBin:playbin/GstURIDecodeBin:uridecodebin0/GstDecodeBin:decodebin0/v4l2h265dec:v4l2h265dec0:
poll error 1: Success (0) for /mnt/fluster/resources/JCT-VC-HEVC_V1/VPSSPSPPS_A_MainConcept_1/VPSSPSPPS_A_MainConcept_1.bin
 
> [VP8-TEST-VECTORS] (GStreamer-VP8-V4L2-Gst1.0)
> vp80-03-segmentation-1425 ... Timeout
Firmware bug, after apply driver workaround, assert fail
> [VP8-TEST-VECTORS] (GStreamer-VP8-V4L2-Gst1.0)
> vp80-03-segmentation-1436 ... Fail
> 
>