[PATCH v2 00/14] V4L2 Explicit Synchronization support

* [PATCH v2 00/14] V4L2 Explicit Synchronization support
@ 2017-09-01  1:50 Gustavo Padovan
  2017-09-01  1:50 ` [PATCH v2 01/14] [media] vb2: add explicit fence user API Gustavo Padovan
                   ` (13 more replies)
  0 siblings, 14 replies; 20+ messages in thread
From: Gustavo Padovan @ 2017-09-01  1:50 UTC (permalink / raw)
  To: linux-media
  Cc: Hans Verkuil, Mauro Carvalho Chehab, Shuah Khan, Gustavo Padovan

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Hi,

Explicit Synchronization allows us to control the synchronization of
shared buffers from userspace by passing fences to the kernel and/or
receiving them from it. Fences passed to the kernel are named in-fences
and the kernel should wait them to signal before using the buffer, i.e.,
queueing it to the driver. On the other side, the kernel can create
out-fences for the buffers it queues to the drivers, out-fences signal
when the driver is finished with buffer, that is, the buffer is ready.

The in-fences and out-fences are communicated at the VIDIOC_QBUF ioctl
using the V4L2_BUF_FLAG_IN_FENCE and V4L2_BUF_FLAG_OUT_FENCE buffer
flags and the fence_fd field. If an in-fence needs to be passed to the
kernel, fence_fd should be set to the fence file descriptor number and
the V4L2_BUF_FLAG_IN_FENCE should be set as well.

To get an out-fence back from V4L2 the V4L2_BUF_FLAG_OUT_FENCE flag
should be set and the fence_fd field will be returned with out-fence
file descriptor related to the next fence to be queued internally to the
V4L2 driver. That means the out-fence may not be associated with the
buffer in the current VIDIOC_QBUF ioctl call because the ordering in
which videobuf2 core queues the buffer to the drivers can’t be
guaranteed. To become aware of the buffer with which the out-fence will
be attached the V4L2_EVENT_BUF_QUEUED should be used. It will trigger an
event for every buffer queued to the V4L2 driver.

Note that the fence_fd field is both an input and output argument here
with different meaning on each direction. As input argument it carries
an in-fence and as output argument it carries an out-fence.

It only works for ordered queues for now, see open question at the end
of the letter.

Test tool can be found at:
https://git.collabora.com/cgit/user/padovan/v4l2-test.git/

The Patches
-----------

The first patch proposes an userspace API for fences, then on patch 2 we
prepare to the addition of in-fences in patch 3, by introducing the
infrastructure on vb2 to wait on an in-fence signal before queueing the
buffer in the driver.

Patch 4 fix uvc v4l2 event handling and patch 5 configure q->dev for
vivid drivers to enable to subscribe and dequeue events on it.

Patches 6-8 enables support to notify BUF_QUEUED events, i.e., let
userspace know that particular buffer was enqueued in the driver. This
is needed, because we return the out-fence fd as an out argument in
QBUF, but at the time it returns we don't know to which buffer the fence
will be attached thus the BUF_QUEUED event tells which buffer is
associated to the fence received in QBUF by userspace.

Patches 9-10 add support to mark queues as ordered. Finally patches 11
and 12 add more fence infrastructure to support out-fences and patch 13
adds support to out-fences. Patch 14 adds overall Documentation about
Explicit Synchronization.

Main Changes
------------

* There is now a .buffer_queued hook on vb2_ops to support notification to
  V4L2 users of buffer queued events
* When handling in-fences we never call vb2_start_streaming() in the fence
  callback, so on the QBUF that is going to trigger the start of the streaming
  we wait synchronously  for the fence to signal before calling
  vb2_start_streaming()
* out-fences: change in behavior: now the out-fence returned via  QBUF
  represents the out-fence of the next buffer to be queued
  to the driver. The buffer id comes out of the BUF_QUEUED event.

All other changes are recorded on the patches' commit messages.

Open Questions
--------------

* non-ordered devices, like m2m: I've been thinking a lot about those
  and one possibility is to have a way to tell userspace that the queue
  is not ordered and then associate the fence with the current buffer in
  QBUF instead of the next one to be queued. Of course, there won't be
  any ordering between the fences. But it may be enough for userspace to
  take advantage of Explicit Synchronization in such cases. Any
  thoughts?

* OUTPUT devices and in-fence. If I understood OUTPUT devices correctly
  it is desirable to queue the buffers to the driver in the same order
  we received them from userspace. If that is correct, shouldn't we add
  some mechanism to prevent buffer whose fence signaled to jump ahead of
  others?

TODO
----

* Gstreamer patch to support fences for some real testing

Gustavo Padovan (13):
  [media] vb2: add explicit fence user API
  [media] vb2: check earlier if stream can be started
  [media] vb2: add in-fence support to QBUF
  [media] uvc: enable subscriptions to other events
  [media] vivid: assign the specific device to the vb2_queue->dev
  [media] v4l: add V4L2_EVENT_BUF_QUEUED event
  [media] vb2: add .buffer_queued() to notify queueing in the driver
  [media] v4l: add support to BUF_QUEUED event
  [media] vb2: add 'ordered' property to queues
  [media] vivid: mark vivid queues as ordered
  [media] vb2: add infrastructure to support out-fences
  [media] vb2: add out-fence support to QBUF
  [media] v4l: Document explicit synchronization behaviour

Javier Martinez Canillas (1):
  [media] vb2: add videobuf2 dma-buf fence helpers

 Documentation/media/uapi/v4l/buffer.rst         |  19 +++
 Documentation/media/uapi/v4l/vidioc-dqevent.rst |  18 ++
 Documentation/media/uapi/v4l/vidioc-qbuf.rst    |  30 ++++
 Documentation/media/videodev2.h.rst.exceptions  |   1 +
 drivers/media/platform/vivid/vivid-core.c       |  15 +-
 drivers/media/usb/cpia2/cpia2_v4l.c             |   2 +-
 drivers/media/usb/uvc/uvc_v4l2.c                |   2 +-
 drivers/media/v4l2-core/Kconfig                 |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |   4 +-
 drivers/media/v4l2-core/v4l2-ctrls.c            |   6 +-
 drivers/media/v4l2-core/videobuf2-core.c        | 213 ++++++++++++++++++++++--
 drivers/media/v4l2-core/videobuf2-v4l2.c        |  51 +++++-
 include/media/videobuf2-core.h                  |  35 +++-
 include/media/videobuf2-fence.h                 |  49 ++++++
 include/uapi/linux/videodev2.h                  |  14 +-
 15 files changed, 428 insertions(+), 32 deletions(-)
 create mode 100644 include/media/videobuf2-fence.h

-- 
2.13.5

^ permalink raw reply	[flat|nested] 20+ messages in thread