Re: [RFC 0/5] arm64: imx8mm: Enable Hantro VPUs

From: Adam Ford <aford173@gmail.com>
To: Tim Harvey <tharvey@gateworks.com>
Cc: Ezequiel Garcia <ezequiel@vanguardiasur.com.ar>,
	Nicolas Dufresne <nicolas@ndufresne.ca>,
	 linux-media <linux-media@vger.kernel.org>,
	 Schrempf Frieder <frieder.schrempf@kontron.de>,
	Marek Vasut <marek.vasut@gmail.com>,
	 Jagan Teki <jagan@amarulasolutions.com>,
	Adam Ford-BE <aford@beaconembedded.com>,
	 cstevens@beaconembedded.com,
	Philipp Zabel <p.zabel@pengutronix.de>,
	 Mauro Carvalho Chehab <mchehab@kernel.org>,
	Rob Herring <robh+dt@kernel.org>, Shawn Guo <shawnguo@kernel.org>,
	 Sascha Hauer <s.hauer@pengutronix.de>,
	Pengutronix Kernel Team <kernel@pengutronix.de>,
	 Fabio Estevam <festevam@gmail.com>,
	NXP Linux Team <linux-imx@nxp.com>,
	 Greg Kroah-Hartman <gregkh@linuxfoundation.org>,
	Heiko Stuebner <heiko@sntech.de>,
	 Lucas Stach <l.stach@pengutronix.de>,
	Joakim Zhang <qiangqing.zhang@nxp.com>,
	 Alice Guo <alice.guo@nxp.com>, Peng Fan <peng.fan@nxp.com>,
	 "open list:HANTRO VPU CODEC DRIVER"
	<linux-rockchip@lists.infradead.org>,
	 "open list:OPEN FIRMWARE AND FLATTENED DEVICE TREE BINDINGS"
	<devicetree@vger.kernel.org>,
	 "moderated list:ARM/FREESCALE IMX / MXC ARM ARCHITECTURE"
	<linux-arm-kernel@lists.infradead.org>,
	 open list <linux-kernel@vger.kernel.org>,
	 "open list:STAGING SUBSYSTEM" <linux-staging@lists.linux.dev>
Subject: Re: [RFC 0/5] arm64: imx8mm: Enable Hantro VPUs
Date: Tue, 30 Nov 2021 14:33:01 -0600	[thread overview]
Message-ID: <CAHCN7xJrM9uZUnmx65uTxWEo6HAkjozd3kD3UoEv-pYd5DV4QA@mail.gmail.com> (raw)
In-Reply-To: <CAJ+vNU2we5mGXgYsR6CfimvFXZsc0zktR3fDa-h6RRa02jTT0g@mail.gmail.com>

On Tue, Nov 30, 2021 at 1:28 PM Tim Harvey <tharvey@gateworks.com> wrote:
>
> On Tue, Nov 30, 2021 at 6:00 AM Ezequiel Garcia
> <ezequiel@vanguardiasur.com.ar> wrote:
> >
> > Hi Tim,
> >
> > On Mon, 29 Nov 2021 at 16:36, Tim Harvey <tharvey@gateworks.com> wrote:
> > >
> > > On Mon, Nov 29, 2021 at 10:59 AM Adam Ford <aford173@gmail.com> wrote:
> > ..
> > > >
> > >
> > > Adam,
> > >
> > > What deps did you install in order to get v4l2codecs building? I
> > > installed libgudev-1.0-dev based on Nicolas' suggestion and rebuilt
> > > (not sure if I needed to re-configure somehow) but there is still
> > > nothing in build/subprojects/gst-plugins-bad/sys/v4l2codecs/. A 'meson
> > > configure' tells me that v4l2codecs is set to 'auto' but I'm not sure
> > > how to find out what dependencies are needed or what may be missing.
> > >
> >
> > At least in my case (Centps-derivative), this is what I've done:
> >
> > ...
> > gst-plugins-bad| Run-time dependency gudev-1.0 found: NO (tried
> > pkgconfig and cmake)
> >
> > Installed gudev ... and then:
> >
> > ...
> > gst-plugins-bad| Dependency gudev-1.0 found: YES 232 (cached)
> > ...
> > gst-plugins-bad 1.19.3.1
> >
> >     Plugins               : accurip, adpcmdec, adpcmenc, aiff, asfmux,
> > audiobuffersplit, audiofxbad, audiomixmatrix, audiolatency,
> > audiovisualizers, autoconvert, bayer,
> >                             camerabin, codecalpha, coloreffects,
> > debugutilsbad, dvbsubenc, dvbsuboverlay, dvdspu, faceoverlay,
> > festival, fieldanalysis, freeverb, frei0r,
> >                             gaudieffects, gdp, geometrictransform,
> > id3tag, inter, interlace, ivfparse, ivtc, jp2kdecimator, jpegformat,
> > rfbsrc, midi, mpegpsdemux,
> >                             mpegpsmux, mpegtsdemux, mpegtsmux, mxf,
> > netsim, rtponvif, pcapparse, pnm, proxy, legacyrawparse,
> > removesilence, rist, rtmp2, rtpmanagerbad,
> >                             sdpelem, segmentclip, siren, smooth,
> > speed, subenc, switchbin, timecode, transcode, videofiltersbad,
> > videoframe_audiolevel, videoparsersbad,
> >                             videosignal, vmnc, y4mdec, decklink, dvb,
> > fbdevsink, ipcpipeline, nvcodec, shm, v4l2codecs, hls, sctp
> >
> > GStreamer current master build fails. It's a known issue which will be
> > fixed today:
> >
> > [...]
> > [8/9] Compiling C object
> > subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > FAILED: subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > cc -Isubprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p
> > -Isubprojects/gst-plugins-bad/sys/v4l2codecs
> > -I../subprojects/gst-plugins-bad/sys/v4l2codecs
> > -Isubprojects/gst-plugins-bad -I../subprojects/gst-plugins-bad
> > -Isubprojects/gstreamer/libs -I../subprojects/gstreamer/libs
> > -Isubprojects/gstreamer -I../subprojects/gstreamer
> > -Isubprojects/gst-plugins-bad/gst-libs
> > -I../subprojects/gst-plugins-bad/gst-libs
> > -Isubprojects/gst-plugins-base/gst-libs
> > -I../subprojects/gst-plugins-base/gst-libs -Isubprojects/orc
> > -I../subprojects/orc -Isubprojects/gstreamer/gst
> > -Isubprojects/gst-plugins-base/gst-libs/gst/video
> > -Isubprojects/gst-plugins-base/gst-libs/gst/pbutils
> > -Isubprojects/gst-plugins-base/gst-libs/gst/audio
> > -Isubprojects/gst-plugins-base/gst-libs/gst/tag
> > -I/usr/include/glib-2.0 -I/usr/lib64/glib-2.0/include
> > -I/usr/include/gudev-1.0 -fdiagnostics-color=always
> > -D_FILE_OFFSET_BITS=64 -Wall -Winvalid-pch -O2 -g -fvisibility=hidden
> > -fno-strict-aliasing -DG_DISABLE_DEPRECATED -Wmissing-prototypes
> > -Wdeclaration-after-statement -Wold-style-definition
> > -Wmissing-declarations -Wredundant-decls -Wwrite-strings -Wformat
> > -Wformat-security -Winit-self -Wmissing-include-dirs -Waddress
> > -Wno-multichar -Wvla -Wpointer-arith -fPIC -pthread -DHAVE_CONFIG_H
> > -MD -MQ subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > -MF subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o.d
> > -o subprojects/gst-plugins-bad/sys/v4l2codecs/libgstv4l2codecs.so.p/gstv4l2codecvp9dec.c.o
> > -c ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:92:3:
> > error: unknown type name ‘grefcount’
> >    grefcount ref_count;
> >    ^~~~~~~~~
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > function ‘gst_v4l2_codec_vp9_dec_picture_data_new’:
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:106:3:
> > warning: implicit declaration of function ‘g_ref_count_init’; did you
> > mean ‘g_cond_init’? [-Wimplicit-function-declaration]
> >    g_ref_count_init (&pic_data->ref_count);
> >    ^~~~~~~~~~~~~~~~
> >    g_cond_init
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > function ‘gst_v4l2_codec_vp9_dec_picture_data_ref’:
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:118:3:
> > warning: implicit declaration of function ‘g_ref_count_inc’; did you
> > mean ‘g_strv_contains’? [-Wimplicit-function-declaration]
> >    g_ref_count_inc (&data->ref_count);
> >    ^~~~~~~~~~~~~~~
> >    g_strv_contains
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c: In
> > function ‘gst_v4l2_codec_vp9_dec_picture_data_unref’:
> > ../subprojects/gst-plugins-bad/sys/v4l2codecs/gstv4l2codecvp9dec.c:125:7:
> > warning: implicit declaration of function ‘g_ref_count_dec’
> > [-Wimplicit-function-declaration]
> >    if (g_ref_count_dec (&data->ref_count)) {
> >        ^~~~~~~~~~~~~~~
> > ninja: build stopped: subcommand failed.
> >
> > Hope this helps get you started!
> > Ezequiel
>
> Ezequiel and Nicolas,
>
> Thanks - I did manage to get gstreamer 1.19.3 built successfully with
> v4l2codecs finally by getting the correct dependencies. I've attempted
> to software encode from another system and decode/display on the IMX8M
> Mini but thus far have not been successful.

I will post a V2 last today with the Mini's post-processing removed.
Someone, I apologize that I forget who, mentioned it was fused out of
the Mini, so the testing I've been doing was with that removed and I
removed the H1 encoder since the Mini doesn't support JPEG encoding.

I don't understand the YAML very well, and I'm struggling with the
dt-bindings which is the main reason I havne't submitted a formal
patch yet.

To date, I have only tested with v4l2-compliance and with the fluster
app that was recommended.

>
> I see that v4l2codecs plugin v4l2slh264dec/v4l2slmpeg2dec/v4l2slvp8dec
> and these all can output video/x-raw NV12/YUY2 which kmssink should
> accept so I'm attempting the following :
>
> # vp8 encode from x86
> gst-launch-1.0 -v videotestsrc ! video/x-raw,width=800,height=480 !
> vp8enc ! rtpvp8pay ! udpsink host=172.24.33.15 port=9001
> # vp8 decode on imx8mm@172.24.33.15 which has a 800x480 display
> [gst-main] root@focal-venice:~/gstreamer/build# gst-launch-1.0 -v
> udpsrc port=9001 caps = "application/x-rtp, media=(string)video,
> clock-rate=(int)90000, encoding-name=(string)VP8, payload=(int)96,
> ssrc=(uint)2745262155, timestamp-offset=(uint)2515032683,
> seqnum-offset=(uint)19579, a-framerate=(string)30" ! rtpvp8depay !
> v4l2slvp8dec ! kmssink
> Setting pipeline to PAUSED ...
> Pipeline is live and does not need PREROLL ...
> /GstPipeline:pipeline0/GstKMSSink:kmssink0: display-width = 800
> /GstPipeline:pipeline0/GstKMSSink:kmssink0: display-height = 480
> Pipeline is PREROLLED ...
> Setting pipeline to PLAYING ...
> /GstPipeline:pipeline0/GstUDPSrc:udpsrc0.GstPad:src: caps =
> application/x-rtp, media=(string)video, clock-rate=(int)90000,
> encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)2745262155,
> timestamp-offset=(uint)2515032683, seqnum-offset=(uint)19579,
> a-framerate=(string)30
> New clock: GstSystemClock
> /GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:sink: caps =
> application/x-rtp, media=(string)video, clock-rate=(int)90000,
> encoding-name=(string)VP8, payload=(int)96, ssrc=(uint)2745262155,
> timestamp-offset=(uint)2515032683, seqnum-offset=(uint)19579,
> a-framerate=(string)30
> /GstPipeline:pipeline0/GstRtpVP8Depay:rtpvp8depay0.GstPad:src: caps =
> video/x-vp8, framerate=(fraction)0/1, height=(int)480, width=(int)800,
> profile=(string)0
> ERROR: from element /GstPipeline:pipeline0/GstUDPSrc:udpsrc0: Internal
> data stream error.
> Additional debug info:
> ../subprojects/gstreamer/libs/gst/base/gstbasesrc.c(3127):
> gst_base_src_loop (): /GstPipeline:pipeline0/GstUDPSrc:udpsrc0:
> streaming stopped, reason not-negotiated (-4)
> Execution ended after 0:00:02.076839644
> Setting pipeline to NULL ...
> Freeing pipeline ...
>
> I'm getting the same thing when trying to use h264.
>
> I've never quite been able to grasp how to debug GStreamer's
> negotiation issues. If I end with fakesink it appears to decode so it
> must be the v4l2slvp8dec to kmssink. I tried forcing the pixel format
> using 'v4l2slvp8dec ! "video/x-raw,format=(string)NV12" ! kmssink' but
> I still get the negotiation error.
>
> What interrupts should I be seeing in /proc/interrupts? I don't see
> anything vpu/hantro related there.

I don't have my Mini with me right now, but with my current patch set
that I mentioned above, I am able to see the interrupts for the
video-codec@38300000 increase each time I run a test with fluster.

>
> I also want to make sure I have a basic understanding of the vpu
> drivers and usersapce on the IMX8M Mini. The IMX6Q/DL that I'm more
> familiar with has a vpu that is supported by the GStreamer video4linux
> plugin which shows the following (on GStreamer 1.16.2):
>   v4l2jpegenc: V4L2 JPEG Encoder
>   v4l2jpegdec: V4L2 JPEG Decoder
>   v4l2h264enc: V4L2 H.264 Encoder
>   v4l2mpeg4enc: V4L2 MPEG4 Encoder
>   v4l2mpeg4dec: V4L2 MPEG4 Decoder
>   v4l2mpeg2dec: V4L2 MPEG2 Decoder
>   v4l2h264dec: V4L2 H264 Decoder
> The IMX6Q/DL also has an IPU that has an M2M driver that provides the
> following for scaling/colorspace conversion:
>   v4l2convert: V4L2 Video Converter
>
> I believe what I'm reading is that the IMX8M Mini Hantro codecs are
> 'stateful' where more software is required to drive them and is
> supported by the newer v4l2codecs plugin. I haven't been able to
> understand what kernel version/requirements the v4l2codecs plugin
> users/requires.
>
> I'm also trying to understand how we can get scaling/colorspace
> conversion on the IMX8M Mini. The IMX8M lacks an IPU... is there some
> way to utilize scaling/colorspace conversion from the 2D GPU bound to
> the etnaviv driver?
>
> Best regards,
>
> Tim