All of lore.kernel.org
 help / color / mirror / Atom feed
* how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?
@ 2020-06-30 11:16 Martin Kepplinger
  2020-06-30 12:59 ` Ezequiel Garcia
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Kepplinger @ 2020-06-30 11:16 UTC (permalink / raw)
  To: ezequiel; +Cc: linux-media

Hi Ezequiel,

On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file
using the VPU.

I'm running the following tree that doesn't change much, but adds the
VPU dts description:
https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
(output of "v4l-ctl --all" at the bottom).


I now run your version of ffmpeg:
https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
(with minor build fixes) but simply doing "ffplay <file>" seems like not
using /dev/video0

Is this supposed to work? If so, do I need to know anything about ffmpeg
to be able to test this? How do you verify that the v4l2 mem2mem device
(vpu) is being used?

thanks a lot,
                                martin



purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
Driver Info:
	Driver name      : hantro-vpu
	Card type        : nxp,imx8mq-vpu-dec
	Bus info         : platform: hantro-vpu
	Driver version   : 5.8.0
	Capabilities     : 0x84204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04204000
		Video Memory-to-Memory Multiplanar
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : hantro-vpu
	Model            : hantro-vpu
	Serial           :
	Bus info         : platform: hantro-vpu
	Media version    : 5.8.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 5.8.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
Entity Info:
	ID               : 0x00000001 (1)
	Name             : nxp,imx8mq-vpu-dec-source
	Function         : V4L2 I/O
	Pad 0x01000002   : 0: Source
	  Link 0x02000008: to remote pad 0x1000004 of entity
'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
Priority: 2
Format Video Capture Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 48
	   Size Image     : 3456
Format Video Output Multiplanar:
	Width/Height      : 48/48
	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
	Field             : None
	Number of planes  : 1
	Flags             :
	Colorspace        : JPEG
	Transfer Function : Default
	YCbCr/HSV Encoding: Default
	Quantization      : Default
	Plane 0           :
	   Bytes per Line : 0
	   Size Image     : 4608

Codec Controls

        mpeg_2_slice_parameters 0x009909fa (unknown): type=103
flags=has-payload
   mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
flags=has-payload
                   h264_profile 0x00990a6b (menu)   : min=0 max=4
default=2 value=2
    h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
flags=has-payload
     h264_picture_parameter_set 0x00990ce9 (unknown): type=111
flags=has-payload
            h264_scaling_matrix 0x00990cea (unknown): type=112
flags=has-payload
          h264_slice_parameters 0x00990ceb (unknown): type=113
flags=has-payload
         h264_decode_parameters 0x00990cec (unknown): type=114
flags=has-payload
               h264_decode_mode 0x00990ced (menu)   : min=1 max=1
default=1 value=1
                h264_start_code 0x00990cee (menu)   : min=1 max=1
default=1 value=1
               vp8_frame_header 0x009910d0 (unknown): type=301
flags=has-payload

^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?
  2020-06-30 11:16 how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu? Martin Kepplinger
@ 2020-06-30 12:59 ` Ezequiel Garcia
  2020-06-30 13:37   ` Martin Kepplinger
  0 siblings, 1 reply; 4+ messages in thread
From: Ezequiel Garcia @ 2020-06-30 12:59 UTC (permalink / raw)
  To: Martin Kepplinger; +Cc: linux-media

Hi Martin,

On Tue, 2020-06-30 at 13:16 +0200, Martin Kepplinger wrote:
> Hi Ezequiel,
> 
> On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file
> using the VPU.
> 
> I'm running the following tree that doesn't change much, but adds the
> VPU dts description:
> https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
> (output of "v4l-ctl --all" at the bottom).
> 

OK, this means the VPU is enabled with the Hantro driver, so far so good.

> 
> I now run your version of ffmpeg:
> https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
> (with minor build fixes) but simply doing "ffplay <file>" seems like not
> using /dev/video0
> 
> Is this supposed to work? If so, do I need to know anything about ffmpeg
> to be able to test this? How do you verify that the v4l2 mem2mem device
> (vpu) is being used?
> 

Not sure what state the branch is on, and also IIRC you should use that together
with mpv for proper zero-copy operation.

I think the libre elec community can help you with that, as they maintain
proper ffmpeg/mpv branches.

Do you need this to work with ffmpeg, or is GStreamer also OK?

Thanks,
Ezequiel

> thanks a lot,
>                                 martin
> 
> 
> 
> purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
> Driver Info:
> 	Driver name      : hantro-vpu
> 	Card type        : nxp,imx8mq-vpu-dec
> 	Bus info         : platform: hantro-vpu
> 	Driver version   : 5.8.0
> 	Capabilities     : 0x84204000
> 		Video Memory-to-Memory Multiplanar
> 		Streaming
> 		Extended Pix Format
> 		Device Capabilities
> 	Device Caps      : 0x04204000
> 		Video Memory-to-Memory Multiplanar
> 		Streaming
> 		Extended Pix Format
> Media Driver Info:
> 	Driver name      : hantro-vpu
> 	Model            : hantro-vpu
> 	Serial           :
> 	Bus info         : platform: hantro-vpu
> 	Media version    : 5.8.0
> 	Hardware revision: 0x00000000 (0)
> 	Driver version   : 5.8.0
> Interface Info:
> 	ID               : 0x0300000c
> 	Type             : V4L Video
> Entity Info:
> 	ID               : 0x00000001 (1)
> 	Name             : nxp,imx8mq-vpu-dec-source
> 	Function         : V4L2 I/O
> 	Pad 0x01000002   : 0: Source
> 	  Link 0x02000008: to remote pad 0x1000004 of entity
> 'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
> Priority: 2
> Format Video Capture Multiplanar:
> 	Width/Height      : 48/48
> 	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
> 	Field             : None
> 	Number of planes  : 1
> 	Flags             :
> 	Colorspace        : JPEG
> 	Transfer Function : Default
> 	YCbCr/HSV Encoding: Default
> 	Quantization      : Default
> 	Plane 0           :
> 	   Bytes per Line : 48
> 	   Size Image     : 3456
> Format Video Output Multiplanar:
> 	Width/Height      : 48/48
> 	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
> 	Field             : None
> 	Number of planes  : 1
> 	Flags             :
> 	Colorspace        : JPEG
> 	Transfer Function : Default
> 	YCbCr/HSV Encoding: Default
> 	Quantization      : Default
> 	Plane 0           :
> 	   Bytes per Line : 0
> 	   Size Image     : 4608
> 
> Codec Controls
> 
>         mpeg_2_slice_parameters 0x009909fa (unknown): type=103
> flags=has-payload
>    mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
> flags=has-payload
>                    h264_profile 0x00990a6b (menu)   : min=0 max=4
> default=2 value=2
>     h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
> flags=has-payload
>      h264_picture_parameter_set 0x00990ce9 (unknown): type=111
> flags=has-payload
>             h264_scaling_matrix 0x00990cea (unknown): type=112
> flags=has-payload
>           h264_slice_parameters 0x00990ceb (unknown): type=113
> flags=has-payload
>          h264_decode_parameters 0x00990cec (unknown): type=114
> flags=has-payload
>                h264_decode_mode 0x00990ced (menu)   : min=1 max=1
> default=1 value=1
>                 h264_start_code 0x00990cee (menu)   : min=1 max=1
> default=1 value=1
>                vp8_frame_header 0x009910d0 (unknown): type=301
> flags=has-payload



^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?
  2020-06-30 12:59 ` Ezequiel Garcia
@ 2020-06-30 13:37   ` Martin Kepplinger
  2020-06-30 14:05     ` Ezequiel Garcia
  0 siblings, 1 reply; 4+ messages in thread
From: Martin Kepplinger @ 2020-06-30 13:37 UTC (permalink / raw)
  To: Ezequiel Garcia; +Cc: linux-media

On 30.06.20 14:59, Ezequiel Garcia wrote:
> Hi Martin,
> 
> On Tue, 2020-06-30 at 13:16 +0200, Martin Kepplinger wrote:
>> Hi Ezequiel,
>>
>> On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file
>> using the VPU.
>>
>> I'm running the following tree that doesn't change much, but adds the
>> VPU dts description:
>> https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
>> (output of "v4l-ctl --all" at the bottom).
>>
> 
> OK, this means the VPU is enabled with the Hantro driver, so far so good.
> 
>>
>> I now run your version of ffmpeg:
>> https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
>> (with minor build fixes) but simply doing "ffplay <file>" seems like not
>> using /dev/video0
>>
>> Is this supposed to work? If so, do I need to know anything about ffmpeg
>> to be able to test this? How do you verify that the v4l2 mem2mem device
>> (vpu) is being used?
>>
> 
> Not sure what state the branch is on, and also IIRC you should use that together
> with mpv for proper zero-copy operation.
> 
> I think the libre elec community can help you with that, as they maintain
> proper ffmpeg/mpv branches.

thanks for the hint.

> 
> Do you need this to work with ffmpeg, or is GStreamer also OK?

gstreamer would be equally fine (or easier as I've used that before) for
me. Last time I checked I think I'd have to build many more libraries
from source in order to test though. That's the only reason why I've
tried ffmpeg.

> 
> Thanks,
> Ezequiel
> 
>> thanks a lot,
>>                                 martin
>>
>>
>>
>> purism@pureos:~/ffmpeg_sources/ffmpeg$ v4l2-ctl --all
>> Driver Info:
>> 	Driver name      : hantro-vpu
>> 	Card type        : nxp,imx8mq-vpu-dec
>> 	Bus info         : platform: hantro-vpu
>> 	Driver version   : 5.8.0
>> 	Capabilities     : 0x84204000
>> 		Video Memory-to-Memory Multiplanar
>> 		Streaming
>> 		Extended Pix Format
>> 		Device Capabilities
>> 	Device Caps      : 0x04204000
>> 		Video Memory-to-Memory Multiplanar
>> 		Streaming
>> 		Extended Pix Format
>> Media Driver Info:
>> 	Driver name      : hantro-vpu
>> 	Model            : hantro-vpu
>> 	Serial           :
>> 	Bus info         : platform: hantro-vpu
>> 	Media version    : 5.8.0
>> 	Hardware revision: 0x00000000 (0)
>> 	Driver version   : 5.8.0
>> Interface Info:
>> 	ID               : 0x0300000c
>> 	Type             : V4L Video
>> Entity Info:
>> 	ID               : 0x00000001 (1)
>> 	Name             : nxp,imx8mq-vpu-dec-source
>> 	Function         : V4L2 I/O
>> 	Pad 0x01000002   : 0: Source
>> 	  Link 0x02000008: to remote pad 0x1000004 of entity
>> 'nxp,imx8mq-vpu-dec-proc': Data, Enabled, Immutable
>> Priority: 2
>> Format Video Capture Multiplanar:
>> 	Width/Height      : 48/48
>> 	Pixel Format      : 'NV12' (Y/CbCr 4:2:0)
>> 	Field             : None
>> 	Number of planes  : 1
>> 	Flags             :
>> 	Colorspace        : JPEG
>> 	Transfer Function : Default
>> 	YCbCr/HSV Encoding: Default
>> 	Quantization      : Default
>> 	Plane 0           :
>> 	   Bytes per Line : 48
>> 	   Size Image     : 3456
>> Format Video Output Multiplanar:
>> 	Width/Height      : 48/48
>> 	Pixel Format      : 'MG2S' (MPEG-2 Parsed Slice Data)
>> 	Field             : None
>> 	Number of planes  : 1
>> 	Flags             :
>> 	Colorspace        : JPEG
>> 	Transfer Function : Default
>> 	YCbCr/HSV Encoding: Default
>> 	Quantization      : Default
>> 	Plane 0           :
>> 	   Bytes per Line : 0
>> 	   Size Image     : 4608
>>
>> Codec Controls
>>
>>         mpeg_2_slice_parameters 0x009909fa (unknown): type=103
>> flags=has-payload
>>    mpeg_2_quantization_matrices 0x009909fb (unknown): type=104
>> flags=has-payload
>>                    h264_profile 0x00990a6b (menu)   : min=0 max=4
>> default=2 value=2
>>     h264_sequence_parameter_set 0x00990ce8 (unknown): type=110
>> flags=has-payload
>>      h264_picture_parameter_set 0x00990ce9 (unknown): type=111
>> flags=has-payload
>>             h264_scaling_matrix 0x00990cea (unknown): type=112
>> flags=has-payload
>>           h264_slice_parameters 0x00990ceb (unknown): type=113
>> flags=has-payload
>>          h264_decode_parameters 0x00990cec (unknown): type=114
>> flags=has-payload
>>                h264_decode_mode 0x00990ced (menu)   : min=1 max=1
>> default=1 value=1
>>                 h264_start_code 0x00990cee (menu)   : min=1 max=1
>> default=1 value=1
>>                vp8_frame_header 0x009910d0 (unknown): type=301
>> flags=has-payload
> 
> 

^ permalink raw reply	[flat|nested] 4+ messages in thread
* Re: how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu?
  2020-06-30 13:37   ` Martin Kepplinger
@ 2020-06-30 14:05     ` Ezequiel Garcia
  0 siblings, 0 replies; 4+ messages in thread
From: Ezequiel Garcia @ 2020-06-30 14:05 UTC (permalink / raw)
  To: Martin Kepplinger; +Cc: linux-media

On Tue, 2020-06-30 at 15:37 +0200, Martin Kepplinger wrote:
> On 30.06.20 14:59, Ezequiel Garcia wrote:
> > Hi Martin,
> > 
> > On Tue, 2020-06-30 at 13:16 +0200, Martin Kepplinger wrote:
> > > Hi Ezequiel,
> > > 
> > > On the Librem 5 (imx8mq) I try to decode a (h264) full hd video file
> > > using the VPU.
> > > 
> > > I'm running the following tree that doesn't change much, but adds the
> > > VPU dts description:
> > > https://source.puri.sm/martin.kepplinger/linux-next/-/commits/5.8-rc3/librem5___vpu
> > > (output of "v4l-ctl --all" at the bottom).
> > > 
> > 
> > OK, this means the VPU is enabled with the Hantro driver, so far so good.
> > 
> > > I now run your version of ffmpeg:
> > > https://gitlab.collabora.com/ezequiel/ffmpeg/-/commits/stateless-mpeg2-vp8-h264-v4
> > > (with minor build fixes) but simply doing "ffplay <file>" seems like not
> > > using /dev/video0
> > > 
> > > Is this supposed to work? If so, do I need to know anything about ffmpeg
> > > to be able to test this? How do you verify that the v4l2 mem2mem device
> > > (vpu) is being used?
> > > 
> > 
> > Not sure what state the branch is on, and also IIRC you should use that together
> > with mpv for proper zero-copy operation.
> > 
> > I think the libre elec community can help you with that, as they maintain
> > proper ffmpeg/mpv branches.
> 
> thanks for the hint.
> 
> > Do you need this to work with ffmpeg, or is GStreamer also OK?
> 
> gstreamer would be equally fine (or easier as I've used that before) for
> me. Last time I checked I think I'd have to build many more libraries
> from source in order to test though. That's the only reason why I've
> tried ffmpeg.
> 

In my experience, using gst-build is not too hard;
combined with meson, it's really fast to cross build.

There are a couple articles:

https://www.collabora.com/news-and-blog/blog/2020/03/19/getting-started-with-gstreamer-gst-build/
https://www.collabora.com/news-and-blog/blog/2020/05/15/cross-compiling-with-gst-build-and-gstreamer/

.. or tl;dr off the top of my head, hope it works:

(host) cd whatever_your_nfs_rootfs_is/gst-build
(host) meson --cross-file /home/user/meson-cross/arm64.txt build
(host) ninja -C build

(The second article explains how to generate a cross-file)

Then on the target:

(target) cd gst-build/
(target) ./gst-uninstalled.py

Few examples that have worked out of the box. The GL ones
might or might not work, depending on your support.

gst-launch-1.0 filesrc location=/$1  ! parsebin ! decodebin3 ! fakevideosink
gst-launch-1.0 -ev filesrc location=$1 ! decodebin3  ! videoconvert ! video/x-raw,format=YUY2 ! glupload ! glfiltercube ! glimagesink
gst-launch-1.0 -ev filesrc location=$1 ! decodebin3  ! videoconvert ! video/x-raw,format=NV12 ! kmssink

You'll have VPU interrupts and should see nice CPU usage. The GStreamer filter element that decodebin3 should pick up is "v4l2slh264dec".

Let us know if it works :-)

Thanks,
Ezequiel


^ permalink raw reply	[flat|nested] 4+ messages in thread
end of thread, other threads:[~2020-06-30 14:06 UTC | newest]

Thread overview: 4+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2020-06-30 11:16 how to use stateless-mpeg2-vp8-h264-v4 on imx8m with vpu? Martin Kepplinger
2020-06-30 12:59 ` Ezequiel Garcia
2020-06-30 13:37   ` Martin Kepplinger
2020-06-30 14:05     ` Ezequiel Garcia

This is an external index of several public inboxes,
see mirroring instructions on how to clone and mirror
all data and code used by this external index.