Re: [ANN] Media sessions in Lyon in October: codecs

[Ricardo Ribalda Delgado <ricardo.ribalda@gmail.com>]

HI Nicolas

Sorry to hijack the thread. Do you know if anyone at AMD is working on
making a v4l driver for the encoder? Or they want to continue with
their mesa approach?

Converting a mesa-vappi to v4l is something doable by the mortals?
Just changing the API or is a complete rewrite of the code?

Best regards!

On Mon, Oct 7, 2019 at 2:05 AM Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
>
> Le jeudi 26 septembre 2019 à 19:21 +0900, Tomasz Figa a écrit :
> > On Mon, Sep 23, 2019 at 11:13 PM Hans Verkuil <hverkuil@xs4all.nl> wrote:
> > > Hi all,
> > >
> > > Since we have three separate half-day sessions for different topics I decided
> > > to split the announcement for this in three emails as well, so these things
> > > can be discussed in separate threads.
> > >
> > > All sessions are in room Terreaux VIP Lounge - Level 0.
> > > There is a maximum of 15 people.
> > >
> > > The first session deals with the codec API and is on Tuesday morning from
> > > 8:30 (tentative, might change) to 12:00 (we have to vacate the room at that
> > > time).
> > >
> > > Confirmed attendees for this session:
> > >
> > > Boris Brezillon <boris.brezillon@collabora.com>
> > > Alexandre Courbot <acourbot@chromium.org>
> > > Nicolas Dufresne <nicolas@ndufresne.ca>
> > > Tomasz Figa <tfiga@chromium.org>
> > > Ezequiel Garcia <ezequiel@collabora.com>
> > > Daniel Gomez <daniel@qtec.com>
> > > Dafna Hirschfeld <dafna.hirschfeld@collabora.com>
> > > Eugen Hristev <Eugen.Hristev@microchip.com>
> > > Paul Kocialkowski <paul.kocialkowski@bootlin.com>
> > > Helen Koike <helen.koike@collabora.com>
> > > Michael Tretter <m.tretter@pengutronix.de>
> > > Hans Verkuil <hverkuil@xs4all.nl>
> > >
> > > Tentative:
> > >
> > > Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > Jacopo Mondi <jacopo@jmondi.org>
> > >
> > > Jacopo, please confirm if you want to attend this session. I didn't find
> > > an email with explicit confirmation, so it was probably done via irc. But since
> > > this session is getting close to capacity I would prefer to keep attendance to
> > > those are actually working with codecs (or will work with it in the near future).
> > >
> > > If I missed someone, or you are on the list but won't attend after all, then
> > > please let me know.
> > >
> > >
> > >
> > > Agenda:
> > >
> > > - Status of any pending patches related to codec support.
> > >
> > > - Requirements of moving codec drivers out of staging.
> > >
> > > - Finalize the stateful encoder API. There are two pieces that need
> > >   to be defined:
> > >
> > > 1) Setting the frame rate so bitrate control can make sense, since
> > >    they need to know this information. This is also relevant for the
> > >    stateless codec (and this may have to change on a per-frame basis
> > >    for stateless codecs!).
> > >
> > >    This can either be implemented via ENUM_FRAMEINTERVALS for the coded
> > >    pixelformats and S_PARM support, or we just add a new control for this.
> > >    E.g. V4L2_CID_MPEG_VIDEO_FRAME_INTERVAL (or perhaps FRAME_RATE). If we
> > >    go for a control, then we need to consider the unit. We can use a
> > >    fraction as well. See this series that puts in the foundation for that:
> > >    https://patchwork.linuxtv.org/cover/58857/
> > >
> > >    I am inclined to go with a control, since the semantics don't really
> > >    match ENUM_FRAMEINTERVALS/S_PARM. These ioctls still need to be supported
> > >    for legacy drivers. Open question: some drivers (mediatek, hva, coda)
> > >    require S_PARM(OUTPUT), some (venus) allow both S_PARM(CAPTURE) and
> > >    S_PARM(OUTPUT). I am inclined to allow both since this is not a CAPTURE
> > >    vs OUTPUT thing, it is global to both queues.
> > >
> > > 2) Interactions between OUTPUT and CAPTURE formats.
> > >
> > >    The main problem is what to do if the capture sizeimage is too small
> > >    for the OUTPUT resolution when streaming starts.
> > >
> > >    Proposal: width and height of S_FMT(OUTPUT) are used to
> > >    calculate a minimum sizeimage (app may request more). This is
> > >    driver-specific. (Is it? Or is this codec-specific?)
> > >
> > >    V4L2_FMT_FLAG_FIXED_RESOLUTION is always set for codec formats
> > >    for the encoder (i.e. we don't support mid-stream resolution
> > >    changes for now) and V4L2_EVENT_SOURCE_CHANGE is not
> > >    supported. See https://patchwork.linuxtv.org/patch/56478/ for
> > >    the patch adding this flag.
> > >
> > >    Of course, if we start to support mid-stream resolution
> > >    changes (or other changes that require a source change event),
> > >    then this flag should be dropped by the encoder driver and
> > >    documentation on how to handle the source change event should
> > >    be documented in the encoder spec. I prefer to postpone this
> > >    until we have an encoder than can actually do mid-stream
> > >    resolution changes.
> > >
> > >    If sizeimage of the OUTPUT is too small for the CAPTURE
> > >    resolution and V4L2_EVENT_SOURCE_CHANGE is not supported,
> > >    then the second STREAMON (either CAPTURE or OUTPUT) will
> > >    return -ENOMEM since there is not enough memory to do the
> > >    encode.
> > >
> > >    If V4L2_FMT_FLAG_FIXED_RESOLUTION is set (i.e. that should
> > >    be the case for all current encoders), then any bitrate controls
> > >    will be limited in range to what the current state (CAPTURE and
> > >    OUTPUT formats and frame rate) supports.
> > >
> > > - Stateless encoders?
> >
> > This could indeed be a good topic. The hantro driver currently only
> > supports JPEG encoding, but the hardware also supports H.264 and VP8
> > at least. It, however, handles only the core parts of the encoding,
> > i.e. generating the actual encoded frames (slices) without headers. It
> > also doesn't do any bitrate control or scene change detection on its
> > own and expects quality parameters (QP) or keyframe requests to come
> > from the software.
> >
> > I'm not sure if there is any other hardware with similar constraints
> > that could use V4L2, but I believe the Intel video encoder supported
> > by VAAPI has a similar design.
>
> I'll try to gather some information next week about that to prepare a
> little. As of now, we have the Rockchip mpp library and the ChromeOS
> version (which is reusing the previous one code). Then the Intel and
> AMD VAAPI drivers (which is implemented in FFMPEG and GStreamer).
>
> Maybe Paul can provide some known information about the CEDRUS (if
> any), even though this is probably harder to gather. We can also study
> software encoders like OpenH264, x264, libvpx etc. to see if there is a
> trend of parameters between the state manager and the low level
> encoders.
>
> Overall goal are, I believe (draft):
>  - Find out if there is a common set of per frame encoding parameter
>  - Find out if bitrate control can be reused for multiple HW
>  - Decide if we do in-kernel bitrate control or not
>  - Decide if we keep bitstream header crafting external (unlike Hantro
> JPEG encoder. but like VAAPI)
>  - Decide if we provide (and maintain) a libv4l2 plugin like ChromeOS
> folks opted for.
>
> >
> > Best regards,
> > Tomasz
> >
> > > - Anything else? (I have a feeling I missed a codec-related topic, but
> > >   I can't find it in my mailbox)
> > >
> > > Regards,
> > >
> > >         Hans
>


-- 
Ricardo Ribalda