Re: [PATCH 0/5] media: meson: vdec: Add VP9 decoding support

From: Neil Armstrong <narmstrong@baylibre.com>
To: Nicolas Dufresne <nicolas@ndufresne.ca>,
	mchehab@kernel.org, hans.verkuil@cisco.com
Cc: linux-media@vger.kernel.org, linux-amlogic@lists.infradead.org,
	linux-arm-kernel@lists.infradead.org,
	linux-kernel@vger.kernel.org,
	Stanimir Varbanov <stanimir.varbanov@linaro.org>
Subject: Re: [PATCH 0/5] media: meson: vdec: Add VP9 decoding support
Date: Fri, 13 Dec 2019 10:12:29 +0100	[thread overview]
Message-ID: <ba2cefa3-1ab8-6eb9-8e69-93d6a3ef68d9@baylibre.com> (raw)
In-Reply-To: <1a570c6e07fcd0d0ddc401a3640dfca07f22304d.camel@ndufresne.ca>

[-- Attachment #1.1: Type: text/plain, Size: 9174 bytes --]

On 05/12/2019 19:46, Nicolas Dufresne wrote:
> Le jeudi 05 décembre 2019 à 16:49 +0100, Neil Armstrong a écrit :
>> On 05/12/2019 16:42, Nicolas Dufresne wrote:
>>> Hi Neil,
>>>
>>> Le jeudi 05 décembre 2019 à 10:24 +0100, Neil Armstrong a écrit :
>>>> Hello,
>>>>
>>>> This patchset aims to bring VP9 decoding support to Amlogic GXL, G12A & SM1
>>>> platforms for the amlogic stateful video decoder driver.
>>>>
>>>> With this, it passes v4l2-compliance with streaming on Amlogic G12A and
>>>> Amlogic SM1 SoCs successfully using the stream at [1] with a fixed
>>>> pyv4l2compliance script for VP9 at [2].
>>>>
>>>> The original script kept the IVF headers in the stream, confusing the
>>>> decoder. The fixed script only extracts the payload from the IVF container.
>>>>
>>>> The decoder has been tested using the Google CTS TestVectorsIttiam VP9 yuv420
>>>> samples, passing 82 resolutions test streams, with 13 fails by pixel
>>>> differences and 3 timeouts.
>>>
>>> How do you handle resolution changes on delta frames ? It's a bit of a
>>> challenge since the reference frames are not at the same resolution as
>>> the frames to be decoded. This breaks the assumption for the resolution
>>> changes mechanism as described in the spec.
>>
>> I don't have a lot of experience on the subject, but in the vendor implementation,
>> they store the resolution along the reference frames and when loading all the
>> reference frames to the HW, the original resolution is also loaded.
>> But we don't handle it.
>>
>>> On stateless side, Boris is introducing DESTROY_BUFS, so we can free
>>> the references when they are not used anymore. But the reference are
>>> managed by userspace and are not queued. While on stateful side so far,
>>> it was assumed that references are queued, and the semantic of S_FMT is
>>> that it must apply to the entire set of queued buffer.
>>
>> yes
>>
>>> I think most streams will work and won't use this feature, but I'm
>>> worried that writing a compliant VP9 decoder is currently not possible.
>>
>> Indeed, but I don't have a clear enough view on the subject, and it doesn't seem
>> I have any test stream with such feature.
> 
> https://www.webmproject.org/vp9/levels/
> 
> I believe in the performance suite, the FRM_RESIZE sample is what
> should exercise this.

Thanks for pointing this test suite... all samples were failing :-p

Now we added the probability handling and all samples works (except the unsupported level).
The ref handling was tricky aswell, anyway it's all fixed except FRM_RESIZE which will
be a hell of hacking to tackle.

The hack would be to keep the refs buffers we still need after frame resizing on drain,
then use the max(old_frame_size, new_frame_size) as capture buffer size, then copy back
the old refs buffers into the newly queued buffers and continue.

It's a ugly hack, but won't break the drain + S_FMT semantic.
But I will fix this _after_ the v2 of this serie is merged.

Neil

> 
>>
>> Neil
>>
>>>> This patchset depends on :
>>>> - G12A enablement at [3]
>>>> - SM1 enablement at [4]
>>>> - H.264 and compliance at [5]
>>>>
>>>> [1] https://github.com/superna9999/pyv4l2compliance/raw/tests/output/Jellyfish_1080_10s_5MB.vp9.hdr
>>>> [2] https://github.com/superna9999/pyv4l2compliance
>>>> [3] https://lore.kernel.org/linux-media/20191120111430.29552-1-narmstrong@baylibre.com
>>>> [4] https://lore.kernel.org/linux-media/20191121101429.23831-1-narmstrong@baylibre.com
>>>> [5] https://lore.kernel.org/linux-media/20191126093733.32404-1-narmstrong@baylibre.com
>>>>
>>>> The compliance log is:
>>>> # v4l2-compliance --stream-from-hdr Jellyfish_1080_10s_5MB.vp9.hdr -s 200
>>>> v4l2-compliance SHA: 7ead0e1856b89f2e19369af452bb03fd0cd16793, 64 bits
>>>>
>>>> Compliance test for meson-vdec device /dev/video0:
>>>>
>>>> Driver Info:
>>>> 	Driver name      : meson-vdec
>>>> 	Card type        : Amlogic Video Decoder
>>>> 	Bus info         : platform:meson-vdec
>>>> 	Driver version   : 5.4.0
>>>> 	Capabilities     : 0x84204000
>>>> 		Video Memory-to-Memory Multiplanar
>>>> 		Streaming
>>>> 		Extended Pix Format
>>>> 		Device Capabilities
>>>> 	Device Caps      : 0x04204000
>>>> 		Video Memory-to-Memory Multiplanar
>>>> 		Streaming
>>>> 		Extended Pix Format
>>>> 	Detected Stateful Decoder
>>>>
>>>> Required ioctls:
>>>> 	test VIDIOC_QUERYCAP: OK
>>>>
>>>> Allow for multiple opens:
>>>> 	test second /dev/video0 open: OK
>>>> 	test VIDIOC_QUERYCAP: OK
>>>> 	test VIDIOC_G/S_PRIORITY: OK
>>>> 	test for unlimited opens: OK
>>>>
>>>> Debug ioctls:
>>>> 	test VIDIOC_DBG_G/S_REGISTER: OK (Not Supported)
>>>> 	test VIDIOC_LOG_STATUS: OK (Not Supported)
>>>>
>>>> Input ioctls:
>>>> 	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
>>>> 	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>>> 	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
>>>> 	test VIDIOC_ENUMAUDIO: OK (Not Supported)
>>>> 	test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
>>>> 	test VIDIOC_G/S_AUDIO: OK (Not Supported)
>>>> 	Inputs: 0 Audio Inputs: 0 Tuners: 0
>>>>
>>>> Output ioctls:
>>>> 	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
>>>> 	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
>>>> 	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
>>>> 	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
>>>> 	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
>>>> 	Outputs: 0 Audio Outputs: 0 Modulators: 0
>>>>
>>>> Input/Output configuration ioctls:
>>>> 	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
>>>> 	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
>>>> 	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
>>>> 	test VIDIOC_G/S_EDID: OK (Not Supported)
>>>>
>>>> Control ioctls:
>>>> 	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
>>>> 	test VIDIOC_QUERYCTRL: OK
>>>> 	test VIDIOC_G/S_CTRL: OK
>>>> 	test VIDIOC_G/S/TRY_EXT_CTRLS: OK
>>>> 	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
>>>> 	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
>>>> 	Standard Controls: 2 Private Controls: 0
>>>>
>>>> Format ioctls:
>>>> 	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
>>>> 	test VIDIOC_G/S_PARM: OK (Not Supported)
>>>> 	test VIDIOC_G_FBUF: OK (Not Supported)
>>>> 	test VIDIOC_G_FMT: OK
>>>> 	test VIDIOC_TRY_FMT: OK
>>>> 	test VIDIOC_S_FMT: OK
>>>> 	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
>>>> 	test Cropping: OK (Not Supported)
>>>> 	test Composing: OK (Not Supported)
>>>> 	test Scaling: OK (Not Supported)
>>>>
>>>> Codec ioctls:
>>>> 	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
>>>> 	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
>>>> 	test VIDIOC_(TRY_)DECODER_CMD: OK
>>>>
>>>> Buffer ioctls:
>>>> 	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
>>>> 	test VIDIOC_EXPBUF: OK
>>>> 	test Requests: OK (Not Supported)
>>>>
>>>> Test input 0:
>>>>
>>>> Streaming ioctls:
>>>> 	test read/write: OK (Not Supported)
>>>> 	test blocking wait: OK
>>>> 	Video Capture Multiplanar: Captured 200 buffers   
>>>> 	test MMAP (select): OK
>>>> 	Video Capture Multiplanar: Captured 200 buffers   
>>>> 	test MMAP (epoll): OK
>>>> 	test USERPTR (select): OK (Not Supported)
>>>> 	test DMABUF: Cannot test, specify --expbuf-device
>>>>
>>>> Total for meson-vdec device /dev/video0: 49, Succeeded: 49, Failed: 0, Warnings: 0
>>>>
>>>> Maxime Jourdan (4):
>>>>   media: meson: vdec: add helpers for lossless framebuffer compression
>>>>     buffers
>>>>   media: meson: vdec: add common HEVC decoder support
>>>>   media: meson: vdec: add VP9 input support
>>>>   media: meson: vdec: add VP9 decoder support
>>>>
>>>> Neil Armstrong (1):
>>>>   media: meson: vdec: align stride on 32 bytes
>>>>
>>>>  drivers/staging/media/meson/vdec/Makefile     |    4 +-
>>>>  .../media/meson/vdec/codec_hevc_common.c      |  285 ++++
>>>>  .../media/meson/vdec/codec_hevc_common.h      |   77 ++
>>>>  drivers/staging/media/meson/vdec/codec_vp9.c  | 1192 +++++++++++++++++
>>>>  drivers/staging/media/meson/vdec/codec_vp9.h  |   13 +
>>>>  drivers/staging/media/meson/vdec/esparser.c   |  142 +-
>>>>  drivers/staging/media/meson/vdec/hevc_regs.h  |  218 +++
>>>>  drivers/staging/media/meson/vdec/vdec.c       |   10 +-
>>>>  .../staging/media/meson/vdec/vdec_helpers.c   |   31 +-
>>>>  .../staging/media/meson/vdec/vdec_helpers.h   |    4 +
>>>>  drivers/staging/media/meson/vdec/vdec_hevc.c  |  231 ++++
>>>>  drivers/staging/media/meson/vdec/vdec_hevc.h  |   13 +
>>>>  .../staging/media/meson/vdec/vdec_platform.c  |   38 +
>>>>  13 files changed, 2245 insertions(+), 13 deletions(-)
>>>>  create mode 100644 drivers/staging/media/meson/vdec/codec_hevc_common.c
>>>>  create mode 100644 drivers/staging/media/meson/vdec/codec_hevc_common.h
>>>>  create mode 100644 drivers/staging/media/meson/vdec/codec_vp9.c
>>>>  create mode 100644 drivers/staging/media/meson/vdec/codec_vp9.h
>>>>  create mode 100644 drivers/staging/media/meson/vdec/hevc_regs.h
>>>>  create mode 100644 drivers/staging/media/meson/vdec/vdec_hevc.c
>>>>  create mode 100644 drivers/staging/media/meson/vdec/vdec_hevc.h
>>>>
>>
>>

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]