Re: video decoding on imx8mq vpu via v4l2

From: Philipp Zabel <p.zabel@pengutronix.de>
To: Martin Kepplinger <martin.kepplinger@puri.sm>,
	ezequiel@collabora.com, mchehab@kernel.org,
	Shawn Guo <shawnguo@kernel.org>,
	s.hauer@pengutronix.de,
	Nicolas Dufresne <nicolas.dufresne@collabora.com>
Cc: linux-media@vger.kernel.org
Subject: Re: video decoding on imx8mq vpu via v4l2
Date: Fri, 14 Feb 2020 14:50:04 +0100	[thread overview]
Message-ID: <b30ab10a43c80e9096e45a6d52d125c1e297f021.camel@pengutronix.de> (raw)
In-Reply-To: <ce102c01-74b1-628d-2192-9d5cb7baa616@puri.sm>

Hi Martin,

On Fri, 2020-02-14 at 12:46 +0100, Martin Kepplinger wrote:
> Hi,
> 
> I'd like to test decoding an mpeg2 or h264 video file on imx8mq, using
> its vpu:
> 
> I use Philipp's changes for imx8m from
> https://git.pengutronix.de/cgit/pza/linux/log/?h=hantro/imx8m-wip in my
> tree:
> 
> https://source.puri.sm/martin.kepplinger/linux-next/commits/5.6-rc1/librem5__integration
> 
> and play around with gstreamer, taking notes at
> https://source.puri.sm/Librem5/linux-next/issues/74
> 
> The driver probes fine and v4l2 appearently sees the MEM2MEM decoder:
> 
> $ v4l2-ctl --list-devices
> nxp,imx8mq-vpu-dec (platform: hantro-vpu):
> 	/dev/video0
> 
> 
> AFAIK gstreamer should provide a "v4l2videodec" element which it doesn't:

The GStreamer V4L2 decoder elements are only implemented for the
stateful codec API [1] so far.

[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-decoder.html

There are plans to add direct support for stateless codecs [2] to
GStreamer, but that will still be a bit of work as far as I understand.

[added Nicolas to Cc:]

[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-stateless-decoder.html

In the interim, I have plugged together a GStreamer VA-API stack based
on Bootlin's libva-v4l2-request backend. See below.

> $ gst-inspect-1.0|grep -i v4l
> video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
> video4linux2:  v4l2radio: Radio (video4linux2) Tuner
> video4linux2:  v4l2sink: Video (video4linux2) Sink
> video4linux2:  v4l2src: Video (video4linux2) Source
> 
> First: Philipp, do you plan to continue working on supporting hantro for
> imx8m upstream?

Yes. We need to sort out the i.MX8MM power domain bindings / drivers,
and I have to test that the Hantro G1 kernel patches work on i.MX8MM as
well, to make sure we got the DT bindings right. I'll then resend the
series for both i.MX8MQ and i.MX8MM.

> Then: What codec would be appropriate to test decoding now? It seems
> like h264 is supposed to be implemented. How do you test?

There is a patched FFmpeg floating around that can drive the Hantro
driver. I'm using a patched libva-v4l2-request [3] / libva [4] /
GStreamer VA-API [5,6] stack to test.

[3] https://github.com/bootlin/libva-v4l2-request/pull/29
[4] https://github.com/intel/libva/pull/332
[5] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/merge_requests/729
[6] https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/merge_requests/171

> And finally: What could be missing here? I use debian's gstreamer
> package, but in this case it can't really be a config/build issue in
> gstreamer, right?

None of this was merged in 1.16.2. The GStreamer VA-API changes depend
on API changes in libva, which currently add Hantro-specifics to a
generic API, and all this is based on a still unstable kernel API.

regards
Philipp