video decoding on imx8mq vpu via v4l2

video decoding on imx8mq vpu via v4l2

[Martin Kepplinger]

Hi,

I'd like to test decoding an mpeg2 or h264 video file on imx8mq, using
its vpu:

I use Philipp's changes for imx8m from
https://git.pengutronix.de/cgit/pza/linux/log/?h=hantro/imx8m-wip in my
tree:

https://source.puri.sm/martin.kepplinger/linux-next/commits/5.6-rc1/librem5__integration

and play around with gstreamer, taking notes at
https://source.puri.sm/Librem5/linux-next/issues/74

The driver probes fine and v4l2 appearently sees the MEM2MEM decoder:

$ v4l2-ctl --list-devices
nxp,imx8mq-vpu-dec (platform: hantro-vpu):
	/dev/video0


AFAIK gstreamer should provide a "v4l2videodec" element which it doesn't:

$ gst-inspect-1.0|grep -i v4l
video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
video4linux2:  v4l2radio: Radio (video4linux2) Tuner
video4linux2:  v4l2sink: Video (video4linux2) Sink
video4linux2:  v4l2src: Video (video4linux2) Source


First: Philipp, do you plan to continue working on supporting hantro for
imx8m upstream?

Then: What codec would be appropriate to test decoding now? It seems
like h264 is supposed to be implemented. How do you test?

And finally: What could be missing here? I use debian's gstreamer
package, but in this case it can't really be a config/build issue in
gstreamer, right?

thanks,

                                martin

Re: video decoding on imx8mq vpu via v4l2

[Philipp Zabel]

Hi Martin,

On Fri, 2020-02-14 at 12:46 +0100, Martin Kepplinger wrote:
> Hi,
> 
> I'd like to test decoding an mpeg2 or h264 video file on imx8mq, using
> its vpu:
> 
> I use Philipp's changes for imx8m from
> https://git.pengutronix.de/cgit/pza/linux/log/?h=hantro/imx8m-wip in my
> tree:
> 
> https://source.puri.sm/martin.kepplinger/linux-next/commits/5.6-rc1/librem5__integration
> 
> and play around with gstreamer, taking notes at
> https://source.puri.sm/Librem5/linux-next/issues/74
> 
> The driver probes fine and v4l2 appearently sees the MEM2MEM decoder:
> 
> $ v4l2-ctl --list-devices
> nxp,imx8mq-vpu-dec (platform: hantro-vpu):
> 	/dev/video0
> 
> 
> AFAIK gstreamer should provide a "v4l2videodec" element which it doesn't:

The GStreamer V4L2 decoder elements are only implemented for the
stateful codec API [1] so far.

[1] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-decoder.html

There are plans to add direct support for stateless codecs [2] to
GStreamer, but that will still be a bit of work as far as I understand.

[added Nicolas to Cc:]

[2] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-stateless-decoder.html

In the interim, I have plugged together a GStreamer VA-API stack based
on Bootlin's libva-v4l2-request backend. See below.

> $ gst-inspect-1.0|grep -i v4l
> video4linux2:  v4l2deviceprovider (GstDeviceProviderFactory)
> video4linux2:  v4l2radio: Radio (video4linux2) Tuner
> video4linux2:  v4l2sink: Video (video4linux2) Sink
> video4linux2:  v4l2src: Video (video4linux2) Source
> 
> First: Philipp, do you plan to continue working on supporting hantro for
> imx8m upstream?

Yes. We need to sort out the i.MX8MM power domain bindings / drivers,
and I have to test that the Hantro G1 kernel patches work on i.MX8MM as
well, to make sure we got the DT bindings right. I'll then resend the
series for both i.MX8MQ and i.MX8MM.

> Then: What codec would be appropriate to test decoding now? It seems
> like h264 is supposed to be implemented. How do you test?

There is a patched FFmpeg floating around that can drive the Hantro
driver. I'm using a patched libva-v4l2-request [3] / libva [4] /
GStreamer VA-API [5,6] stack to test.

[3] https://github.com/bootlin/libva-v4l2-request/pull/29
[4] https://github.com/intel/libva/pull/332
[5] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/merge_requests/729
[6] https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/merge_requests/171

> And finally: What could be missing here? I use debian's gstreamer
> package, but in this case it can't really be a config/build issue in
> gstreamer, right?

None of this was merged in 1.16.2. The GStreamer VA-API changes depend
on API changes in libva, which currently add Hantro-specifics to a
generic API, and all this is based on a still unstable kernel API.

regards
Philipp

Re: video decoding on imx8mq vpu via v4l2

[+1 514-466-1652]

14 févr. 2020 08:50:16 Philipp Zabel :

> Hi Martin,
>
> On Fri, 2020-02-14 at 12:46 +0100, Martin Kepplinger wrote:
>
> > Hi,
> >
> > I'd like to test decoding an mpeg2 or h264 video file on imx8mq, using
> > its vpu:
> >
> > I use Philipp's changes for imx8m from
> > https://git.pengutronix.de/cgit/pza/linux/log/?h=hantro/imx8m-wip in my
> > tree:
> >
> > https://source.puri.sm/martin.kepplinger/linux-next/commits/5.6-rc1/librem5__integration
> >
> > and play around with gstreamer, taking notes at
> > https://source.puri.sm/Librem5/linux-next/issues/74
> >
> > The driver probes fine and v4l2 appearently sees the MEM2MEM decoder:
> >
> > $ v4l2-ctl --list-devices
> > nxp,imx8mq-vpu-dec (platform: hantro-vpu):
> > /dev/video0
> >
> >
> > AFAIK gstreamer should provide a "v4l2videodec" element which it doesn't:
> >
>
> The GStreamer V4L2 decoder elements are only implemented for the
> stateful codec API [1] so far.
>
> [1] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-decoder.html
>
> There are plans to add direct support for stateless codecs [2] to
> GStreamer, but that will still be a bit of work as far as I understand.
>
> [added Nicolas to Cc:]

That is correct. I have a WIP branch here, than decode some H264, forward, assuming no queues after the decoder. So still need to be polished. It is also limited to Hantro driver, as I didn't fully implemented the slice data and the reference lists needed by Cedrus on Allwinner.

https://gitlab.freedesktop.org/ndufresne/gst-plugins-bad/commits/v4l2codecs-hantro



>
> [2] https://linuxtv.org/downloads/v4l-dvb-apis-new/uapi/v4l/dev-stateless-decoder.html
>
> In the interim, I have plugged together a GStreamer VA-API stack based
> on Bootlin's libva-v4l2-request backend. See below.
>
>
> > $ gst-inspect-1.0|grep -i v4l
> > video4linux2: v4l2deviceprovider (GstDeviceProviderFactory)
> > video4linux2: v4l2radio: Radio (video4linux2) Tuner
> > video4linux2: v4l2sink: Video (video4linux2) Sink
> > video4linux2: v4l2src: Video (video4linux2) Source
> >
> > First: Philipp, do you plan to continue working on supporting hantro for
> > imx8m upstream?
> >
>
> Yes. We need to sort out the i.MX8MM power domain bindings / drivers,
> and I have to test that the Hantro G1 kernel patches work on i.MX8MM as
> well, to make sure we got the DT bindings right. I'll then resend the
> series for both i.MX8MQ and i.MX8MM.
>
>
> > Then: What codec would be appropriate to test decoding now? It seems
> > like h264 is supposed to be implemented. How do you test?
> >
>
> There is a patched FFmpeg floating around that can drive the Hantro
> driver. I'm using a patched libva-v4l2-request [3] / libva [4] /
> GStreamer VA-API [5,6] stack to test.
>
> [3] https://github.com/bootlin/libva-v4l2-request/pull/29
> [4] https://github.com/intel/libva/pull/332
> [5] https://gitlab.freedesktop.org/gstreamer/gst-plugins-bad/merge_requests/729
> [6] https://gitlab.freedesktop.org/gstreamer/gstreamer-vaapi/merge_requests/171
>
>
> > And finally: What could be missing here? I use debian's gstreamer
> > package, but in this case it can't really be a config/build issue in
> > gstreamer, right?
> >
>
> None of this was merged in 1.16.2. The GStreamer VA-API changes depend
> on API changes in libva, which currently add Hantro-specifics to a
> generic API, and all this is based on a still unstable kernel API.

I've tested Philipp patch against VA driver, and that worked well for me. Though it requires patches on so many components that it's quite challenging to package. Notably you need to backport some parser work and its comes with a new VA API. I think it's very unlikely this will be merged upstream. To make things worst, no one, including me, wants to maintain such a userspace driver.

And that's why I wrote a new one that does not use an abstraction. It still depends on new API in the parser. This is something we might revisit kernel, as these parameters are very HW specific. On GStreamer side, I'm sharing code with MS DxVA support, and hope we can bring more accelerator around this in the future. It's the model FFMPEG and Chromium have adopted, and what makes it more mature today. About Chromium, there is a branch using the new driver, but I can't find it at the moment.

>
> regards
> Philipp
>