Re: [PATCHv3 1/5] media: docs-rst: Document memory-to-memory video encoder interface

[Tomasz Figa <tfiga@chromium.org>]

On Fri, May 29, 2020 at 11:57 AM Stanimir Varbanov
<stanimir.varbanov@linaro.org> wrote:
>
>
>
> On 5/26/20 1:09 PM, Hans Verkuil wrote:
> > From: Tomasz Figa <tfiga@chromium.org>
> >
> > Due to complexity of the video encoding process, the V4L2 drivers of
> > stateful encoder hardware require specific sequences of V4L2 API calls
> > to be followed. These include capability enumeration, initialization,
> > encoding, encode parameters change, drain and reset.
> >
> > Specifics of the above have been discussed during Media Workshops at
> > LinuxCon Europe 2012 in Barcelona and then later Embedded Linux
> > Conference Europe 2014 in Düsseldorf. The de facto Codec API that
> > originated at those events was later implemented by the drivers we already
> > have merged in mainline, such as s5p-mfc or coda.
> >
> > The only thing missing was the real specification included as a part of
> > Linux Media documentation. Fix it now and document the encoder part of
> > the Codec API.
> >
> > Signed-off-by: Tomasz Figa <tfiga@chromium.org>
> > Signed-off-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> > ---
> >  .../userspace-api/media/v4l/dev-encoder.rst   | 728 ++++++++++++++++++
> >  .../userspace-api/media/v4l/dev-mem2mem.rst   |   1 +
> >  .../userspace-api/media/v4l/pixfmt-v4l2.rst   |   5 +
> >  .../userspace-api/media/v4l/v4l2.rst          |   2 +
> >  .../media/v4l/vidioc-encoder-cmd.rst          |  51 +-
> >  5 files changed, 767 insertions(+), 20 deletions(-)
> >  create mode 100644 Documentation/userspace-api/media/v4l/dev-encoder.rst
> >
>
> <cut>
>
> > +
> > +Reset
> > +=====
> > +
> > +The client may want to request the encoder to reinitialize the encoding, so
> > +that the following stream data becomes independent from the stream data
> > +generated before. Depending on the coded format, that may imply that:
> > +
> > +* encoded frames produced after the restart must not reference any frames
> > +  produced before the stop, e.g. no long term references for H.264/HEVC,
> > +
> > +* any headers that must be included in a standalone stream must be produced
> > +  again, e.g. SPS and PPS for H.264/HEVC.
> > +
> > +This can be achieved by performing the reset sequence.
> > +
> > +1. Perform the `Drain` sequence to ensure all the in-flight encoding finishes
> > +   and respective buffers are dequeued.
> > +
> > +2. Stop streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMOFF`. This
> > +   will return all currently queued ``CAPTURE`` buffers to the client, without
> > +   valid frame data.
> > +
> > +3. Start streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMON` and
> > +   continue with regular encoding sequence. The encoded frames produced into
> > +   ``CAPTURE`` buffers from now on will contain a standalone stream that can be
> > +   decoded without the need for frames encoded before the reset sequence,
> > +   starting at the first ``OUTPUT`` buffer queued after issuing the
> > +   `V4L2_ENC_CMD_STOP` of the `Drain` sequence.
> > +
> > +This sequence may be also used to change encoding parameters for encoders
> > +without the ability to change the parameters on the fly.
>
> How the v4l2 client knows which parameters could be changed on the fly
> and which cannot? Controls should return -EBUSY?

Yes, but I guess you found the answer already. :) I guess we could add
a reference to the "Encoding Parameter Changes" section.

>
> Also could that Reset be used to change the pixel format and probably
> colorspace?

I guess it would indeed have to be used for any changes that would
affect the parameters of the encoded stream, e.g. coded resolution or
coded colorspace.

For any changes which wouldn't affect the encoded stream, e.g. a
source format change with the same crop size or a source colorspace
change while keeping the coded colorspace the same (i.e. hardware
doing the conversion), the OUTPUT queue could be reconfigured on its
own without the need to do anything to the CAPTURE queue.

Best regards,
Tomasz