Re: Stateless Encoding uAPI Discussion and Proposal

[Paul Kocialkowski <paul.kocialkowski@bootlin.com>]

[-- Attachment #1: Type: text/plain, Size: 2633 bytes --]

Hi Nicolas,

On Mon 24 Jul 23, 10:03, Nicolas Dufresne wrote:
> Le vendredi 21 juillet 2023 à 20:19 +0200, Michael Grzeschik a écrit :
> > > As a result, we cannot expect that any given encoder is able to produce frames
> > > for any set of headers. Reporting related constraints and limitations (beyond
> > > profile/level) seems quite difficult and error-prone.
> > > 
> > > So it seems that keeping header generation in-kernel only (close to where the
> > > hardware is actually configured) is the safest approach.
> > 
> > For the case with the rkvenc, the headers are also not created by the
> > kernel driver. Instead we use the gst_h264_bit_writer_sps/pps functions
> > that are part of the codecparsers module.
> 
> One level of granularity we can add is split headers (like SPS/PPS) and
> slice/frame headers.

Do you mean asking the driver to return a buffer with only SPS/PPS and then
return another buffer with the slice/frame header?

Looks like there's already a control for it: V4L2_CID_MPEG_VIDEO_HEADER_MODE
which takes either
- V4L2_MPEG_VIDEO_HEADER_MODE_SEPARATE: looks like what you're suggesting
- V4L2_MPEG_VIDEO_HEADER_MODE_JOINED_WITH_1ST_FRAME: usual case

So that could certainly be supported to easily allow userspace to stuff extra
NALUs in-between.

> It remains that in some cases, like HEVC, when the slice
> header is byte aligned, it can be nice to be able to handle it at application
> side in order to avoid limiting SVC support (and other creative features) by our
> API/abstraction limitations.

Do you see something in the headers that we expect the kernel to generate that
would need specific changes to support features like SVC?

From what I can see there's a svc_extension_flag that's only set for specific
NALUs (prefix_nal_unit/lice_layer_extension) so these could be inserted by
userspace.

Also I'm not very knowledgeable about SVC so it's not very clear to me if it's
possible to take an encoder that doesn't support SVC and turn the resulting
stream into something SVC-ready by adding extra NAL units or if the encoder
should be a lot more involved.

Also do you know if we have stateful codecs supporting SVC?

> I think a certain level of "per CODEC" reasoning is
> also needed. Just like, I would not want to have to ask the kernel to generate
> user data SEI and other in-band data.

Yeah it looks like there is definitely a need for adding extra NALUs from
userspace without passing that data to the kernel.

Cheers,

Paul

-- 
Paul Kocialkowski, Bootlin
Embedded Linux and kernel engineering
https://bootlin.com

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 488 bytes --]