Re: [RFC] [media] mem2mem: add support for hardware buffered queue

[Philipp Zabel <p.zabel@pengutronix.de>]

Hi Kamil,

Am Mittwoch, den 29.05.2013, 11:54 +0200 schrieb Kamil Debski:
> Hi Philipp, Hans,
> 
> > On mem2mem decoders with a hardware bitstream ringbuffer, to drain the
> > buffer at the end of the stream, remaining frames might need to be
> > decoded without additional input buffers being provided, and after
> > calling streamoff on the v4l2 output queue. This also allows a driver
> > to copy input buffers into their bitstream ringbuffer and immediately
> > mark them as done to be dequeued.
> > 
> > The motivation for this patch is hardware assisted h.264 reordering
> > support in the coda driver. For high profile streams, the coda can hold
> > back out-of-order frames, causing a few mem2mem device runs in the
> > beginning, that don't produce any decompressed buffer at the v4l2
> > capture side. At the same time, the last few frames can be decoded from
> > the bitstream with mem2mem device runs that don't need a new input
> > buffer at the v4l2 output side. A streamoff on the v4l2 output side can
> > be used to put the decoder into the ringbuffer draining end-of-stream
> > mode.
> 
> If I remember correctly the main goal of introducing the m2m framework
> was to support simple mem2mem devices where one input buffer = one output
> buffer. In other cases m2m was not to be used. 

The m2m context / queue handling and job scheduling are very useful even
for drivers that don't always produce one CAPTURE buffer from one OUTPUT
buffer, just as you drescribe below.
The CODA encoder path fits the m2m model perfectly. I'd prefer not to
duplicate most of mem2mem just because the decoder doesn't.

There's two things that this patch allows me to do:
a) running mem2mem device_run with an empty OUTPUT queue, which is
   something I'd really like to make possible.
b) running mem2mem device_run with the OUTPUT queue in STREAM OFF, which
   I needed to get the remaining buffers out. But maybe there is a
   better way to do this while keeping the output queue streaming.

> An example of such mem2mem driver, which does not use m2m framework is
> MFC. It uses videobuf2 directly and it is wholly up to the driver how
> will it control the queues, stream on/off and so on. You can then have
> one OUTPUT buffer generate multiple CAPTURE buffer, multiple OUTPUT
> buffers generate a single CAPTURE buffer. Consume OUTPUT buffer without
> generating CAPTURE buffer (e.g. when starting decoding) and produce
> CAPTURE buffers without consuming OUTPUT buffers (e.g. when finishing
> decoding).
>
> I think that stream off should not be used to signal EOS. For this we
> have EOS event. You mentioned the EOS buffer flag. This is the idea
> originally proposed by Andrzej Hajda, after some lengthy discussions
> with v4l community this idea was changed to use an EOS event.

I'm not set on using STREAMOFF to signal the stream-end condition to the
hardware, but after switching to stream-end mode, no new frames should
be queued, so I thought it fit quite well. It also allows to prepare the
OUTPUT buffers (S_FMT/REQBUFS) for the next STREAMON while the CAPTURE
side is still draining the bitstream, although that's probably not a
very useful feature.
I could instead have userspace signal the driver via an EOS buffer flag
or any other mechanism. Then the OUTPUT queue would be kept streaming,
but hold back all buffers queued via QBUF until the last buffer is
dequeued from the CAPTURE queue.

> I was all for the EOS buffer flag, but after discussion with Mauro I
> understood his arguments. We can get back to this discussion, if we
> are sure that events are not enough. Please also note that we need to
> keep backward compatibility.

Maybe I've missed something, but I thought the EOS signal is only for
the driver to signal to userspace that the currently dequeued frame is
the last one?
I need userspace to signal to the driver that it won't queue any new
OUTPUT buffers, but still wants to dequeue the remaining CAPTURE buffers
until the bitstream buffer is empty.

> Original EOS buffer flag patches by Andrzej and part of the discussion
> can be found here:
> 1) https://linuxtv.org/patch/10624/
> 2) https://linuxtv.org/patch/11373/
> 
> Best wishes,
> Kamil Debski

regards
Philipp