From: Nicolas Dufresne <nicolas.dufresne@collabora.com>
To: Paul Kocialkowski <paul.kocialkowski@bootlin.com>
Cc: "Michael Grzeschik" <mgr@pengutronix.de>,
linux-kernel@vger.kernel.org, linux-media@vger.kernel.org,
"Hans Verkuil" <hverkuil@xs4all.nl>,
"Sakari Ailus" <sakari.ailus@iki.fi>,
"Andrzej Pietrasiewicz" <andrzej.p@collabora.com>,
"Michael Tretter" <m.tretter@pengutronix.de>,
"Jernej Škrabec" <jernej.skrabec@gmail.com>,
"Chen-Yu Tsai" <wens@csie.org>,
"Samuel Holland" <samuel@sholland.org>,
"Thomas Petazzoni" <thomas.petazzoni@bootlin.com>
Subject: Re: Stateless Encoding uAPI Discussion and Proposal
Date: Wed, 26 Jul 2023 16:02:49 -0400 [thread overview]
Message-ID: <4afabd718f632965314794946d4b31cec739a3d6.camel@collabora.com> (raw)
In-Reply-To: <ZL-RWOfUYh5VbUo1@aptenodytes>
Le mardi 25 juillet 2023 à 11:09 +0200, Paul Kocialkowski a écrit :
> Hi Nicolas,
>
> On Mon 24 Jul 23, 10:03, Nicolas Dufresne wrote:
> > Le vendredi 21 juillet 2023 à 20:19 +0200, Michael Grzeschik a écrit :
> > > > As a result, we cannot expect that any given encoder is able to produce frames
> > > > for any set of headers. Reporting related constraints and limitations (beyond
> > > > profile/level) seems quite difficult and error-prone.
> > > >
> > > > So it seems that keeping header generation in-kernel only (close to where the
> > > > hardware is actually configured) is the safest approach.
> > >
> > > For the case with the rkvenc, the headers are also not created by the
> > > kernel driver. Instead we use the gst_h264_bit_writer_sps/pps functions
> > > that are part of the codecparsers module.
> >
> > One level of granularity we can add is split headers (like SPS/PPS) and
> > slice/frame headers.
>
> Do you mean asking the driver to return a buffer with only SPS/PPS and then
> return another buffer with the slice/frame header?
>
> Looks like there's already a control for it: V4L2_CID_MPEG_VIDEO_HEADER_MODE
> which takes either
> - V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE: looks like what you're suggesting
> - V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME: usual case
>
> So that could certainly be supported to easily allow userspace to stuff extra
> NALUs in-between.
Good point, indeed.
>
> > It remains that in some cases, like HEVC, when the slice
> > header is byte aligned, it can be nice to be able to handle it at application
> > side in order to avoid limiting SVC support (and other creative features) by our
> > API/abstraction limitations.
>
> Do you see something in the headers that we expect the kernel to generate that
> would need specific changes to support features like SVC?
Getting the kernel to set the layer IDs, unless we have a full SVC configuration
would just be extra indirections. That being said, if we mention HEVC, these IDs
can be modified in-place as they use a fixed number of bytes. If you can split
the headers appart, generating per layer headers in application makes a lot of
sense.
Traditionally, slice headers are made by stateless accelerators, but not the
SPS/PPS and friend.
>
> From what I can see there's a svc_extension_flag that's only set for specific
> NALUs (prefix_nal_unit/lice_layer_extension) so these could be inserted by
> userspace.
>
> Also I'm not very knowledgeable about SVC so it's not very clear to me if it's
> possible to take an encoder that doesn't support SVC and turn the resulting
> stream into something SVC-ready by adding extra NAL units or if the encoder
> should be a lot more involved.
You can use any encoders to create a temporal SVC. Its only about the
referencing pattern, made so you can reduce the framerate (dividing by 2
usually).
For spatial layer, the encoders need scaling capabilities. I'm not totally sure
how multi-view work, but this is most likely just using left eye as reference
(not having an I frame ever for the second eye).
>
> Also do you know if we have stateful codecs supporting SVC?
We don't at the moment, they all produce headers with layer id hardcoded to 0 as
far as I'm aware. The general plan (if it had continued) might have been to
offer a memu based control, and drivers could offer from a list of preset SVC
pattern. Mimicking what browsers needs:
https://www.w3.org/TR/webrtc-svc/
>
> > I think a certain level of "per CODEC" reasoning is
> > also needed. Just like, I would not want to have to ask the kernel to generate
> > user data SEI and other in-band data.
>
> Yeah it looks like there is definitely a need for adding extra NALUs from
> userspace without passing that data to the kernel.
>
> Cheers,
>
> Paul
>
next prev parent reply other threads:[~2023-07-26 20:03 UTC|newest]
Thread overview: 29+ messages / expand[flat|nested] mbox.gz Atom feed top
2023-07-11 17:12 Stateless Encoding uAPI Discussion and Proposal Paul Kocialkowski
2023-07-11 18:18 ` Nicolas Dufresne
2023-07-12 14:07 ` Paul Kocialkowski
2023-07-25 3:33 ` Hsia-Jun Li
2023-07-25 12:15 ` Paul Kocialkowski
2023-07-26 2:49 ` Hsia-Jun Li
2023-07-26 19:53 ` Nicolas Dufresne
2023-07-27 2:45 ` Hsia-Jun Li
2023-07-27 17:10 ` Nicolas Dufresne
2023-07-26 8:18 ` Hans Verkuil
2023-08-09 14:43 ` Paul Kocialkowski
2023-08-09 17:24 ` Andrzej Pietrasiewicz
2023-07-21 18:19 ` Michael Grzeschik
2023-07-24 14:03 ` Nicolas Dufresne
2023-07-25 9:09 ` Paul Kocialkowski
2023-07-26 20:02 ` Nicolas Dufresne [this message]
2023-08-10 13:44 ` Paul Kocialkowski
2023-08-10 14:34 ` Nicolas Dufresne
2023-08-11 20:08 ` Paul Kocialkowski
2023-08-21 15:13 ` Nicolas Dufresne
2023-08-22 8:30 ` Hsia-Jun Li
2023-08-22 20:31 ` Nicolas Dufresne
2023-08-23 3:04 ` Hsia-Jun Li
2023-08-30 15:10 ` Nicolas Dufresne
2023-08-30 16:51 ` Randy Li
2023-08-30 15:18 ` Nicolas Dufresne
2023-08-31 9:32 ` Hsia-Jun Li
2023-08-23 8:05 ` Paul Kocialkowski
2023-11-15 13:19 ` Paul Kocialkowski
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=4afabd718f632965314794946d4b31cec739a3d6.camel@collabora.com \
--to=nicolas.dufresne@collabora.com \
--cc=andrzej.p@collabora.com \
--cc=hverkuil@xs4all.nl \
--cc=jernej.skrabec@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-media@vger.kernel.org \
--cc=m.tretter@pengutronix.de \
--cc=mgr@pengutronix.de \
--cc=paul.kocialkowski@bootlin.com \
--cc=sakari.ailus@iki.fi \
--cc=samuel@sholland.org \
--cc=thomas.petazzoni@bootlin.com \
--cc=wens@csie.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).