Re: Stateless Encoding uAPI Discussion and Proposal

[Andrzej Pietrasiewicz <andrzej.p@collabora.com>]

Hi Paul & Hans,

W dniu 9.08.2023 o 16:43, Paul Kocialkowski pisze:
> Hi Hans,
> 
> On Wed 26 Jul 23, 10:18, Hans Verkuil wrote:
>> On 11/07/2023 20:18, Nicolas Dufresne wrote:
>>> Le mardi 11 juillet 2023 à 19:12 +0200, Paul Kocialkowski a écrit :
>>>> Hi everyone!
>>>>
>>>> After various discussions following Andrzej's talk at EOSS, feedback from the
>>>> Media Summit (which I could not attend unfortunately) and various direct
>>>> discussions, I have compiled some thoughts and ideas about stateless encoders
>>>> support with various proposals. This is the result of a few years of interest
>>>> in the topic, after working on a PoC for the Hantro H1 using the hantro driver,
>>>> which turned out to have numerous design issues.
>>>>
>>>> I am now working on a H.264 encoder driver for Allwinner platforms (currently
>>>> focusing on the V3/V3s), which already provides some usable bitstream and will
>>>> be published soon.
>>>>
>>>> This is a very long email where I've tried to split things into distinct topics
>>>> and explain a few concepts to make sure everyone is on the same page.
>>>>
>>>> # Bitstream Headers
>>>>
>>>> Stateless encoders typically do not generate all the bitstream headers and
>>>> sometimes no header at all (e.g. Allwinner encoder does not even produce slice
>>>> headers). There's often some hardware block that makes bit-level writing to the
>>>> destination buffer easier (deals with alignment, etc).
>>>>
>>>> The values of the bitstream headers must be in line with how the compressed
>>>> data bitstream is generated and generally follow the codec specification.
>>>> Some encoders might allow configuring all the fields found in the headers,
>>>> others may only allow configuring a few or have specific constraints regarding
>>>> which values are allowed.
>>>>
>>>> As a result, we cannot expect that any given encoder is able to produce frames
>>>> for any set of headers. Reporting related constraints and limitations (beyond
>>>> profile/level) seems quite difficult and error-prone.
>>>>
>>>> So it seems that keeping header generation in-kernel only (close to where the
>>>> hardware is actually configured) is the safest approach.
>>>
>>> This seems to match with what happened with the Hantro VP8 proof of concept. The
>>> encoder does not produce the frame header, but also, it produces 2 encoded
>>> buffers which cannot be made contiguous at the hardware level. This notion of
>>> plane in coded data wasn't something that blended well with the rest of the API
>>> and we didn't want to copy in the kernel while the userspace would also be
>>> forced to copy to align the headers. Our conclusion was that it was best to
>>> generate the headers and copy both segment before delivering to userspace. I
>>> suspect this type of situation will be quite common.
>>>
>>>>
>>>> # Codec Features
>>>>
>>>> Codecs have many variable features that can be enabled or not and specific
>>>> configuration fields that can take various values. There is usually some
>>>> top-level indication of profile/level that restricts what can be used.
>>>>
>>>> This is a very similar situation to stateful encoding, where codec-specific
>>>> controls are used to report and set profile/level and configure these aspects.
>>>> A particularly nice thing about it is that we can reuse these existing controls
>>>> and add new ones in the future for features that are not yet covered.
>>>>
>>>> This approach feels more flexible than designing new structures with a selected
>>>> set of parameters (that could match the existing controls) for each codec.
>>>
>>> Though, reading more into this emails, we still have a fair amount of controls
>>> to design and add, probably some compound controls too ?
>>
>> I expect that for stateless encoders support for read-only requests will be needed:
>>
>> https://patchwork.linuxtv.org/project/linux-media/list/?series=5647
>>
>> I worked on that in the past together with dynamic control arrays. The dynamic
>> array part was merged, but the read-only request part wasn't (there was never a
>> driver that actually needed it).
>>
>> I don't know if that series still applies, but if there is a need for it then I
>> can rebase it and post an RFCv3.
> 
> So if I understand this correctly (from a quick look), this would be to allow
> stateless encoder drivers to attach a particular control value to a specific
> returned frame?
> 
> I guess this would be a good match to return statistics about the encoded frame.
> However that would probably be expressed in a hardware-specific way so it
> seems preferable to not expose this to userspace and handle it in-kernel
> instead.
> 
> What's really important for userspace to know (in order to do user-side
> rate-control, which we definitely want to support) is the resulting bitstream
> size. This is already available with bytesused.
> 
> So all in all I think we're good with the current status of request support.

Yup. I agree. Initially, while working on VP8 encoding we introduced (read-only)
requests on the capture queue, but they turned out not to be useful in this
context and we removed them.

Regards.

Andrzej

> 
> Cheers,
> 
> Paul
>