[PATCH v10 00/16] V4L2 Explicit Synchronization

[PATCH v10 00/16] V4L2 Explicit Synchronization

[Ezequiel Garcia]

Hi all,

The most relevant change for this round is that all the work
done in the fence callback is now moved to a workqueue,
that runs with the queue lock held.
 
Although this introduces some latency, it is however needed
to take the vb2_queue mutex and safely call vb2 ops.

Given the fence callback can be called fully asynchronously,
and given it needs to be associated with a vb2_buffer,
we now need to refcount vb2_buffer. This allows to safely "attach"
the vb2_buffer to the fence callback.

To prevent annoying deadlocks, and because the fence callback
is called with the fence spinlock, it's best to avoid calling
dma_fence_put in the fence callback itself. So the fence is now
put in the DQBUF operation (or in cancel paths).

Hopefully, I took care of all the feedback provided by
Hans and Brian on v9. Please let me know if you guys catch
anything else.

Thanks!

Changes from v9:

 * Move fence callback to workqueue, and call vb2_start_streaming if needed.
 * Refcount vb2_buffer.
 * Check return of get_unused_fd.
 * Increase seqno for out-fences that reuse the context.
 * Go back to is_unordered callback.
 * Mark unordered formats in cobalt driver.
 * Improve CAP_FENCES check.
 * Minor documentation changes.

Ezequiel Garcia (1):
  videobuf2: Make struct vb2_buffer refcounted

Gustavo Padovan (15):
  xilinx: regroup caps on querycap
  hackrf: group device capabilities
  omap3isp: group device capabilities
  vb2: move vb2_ops functions to videobuf2-core.[ch]
  vb2: add is_unordered callback for drivers
  v4l: add unordered flag to format description ioctl
  v4l: mark unordered formats
  cobalt: add .is_unordered() for cobalt
  vb2: mark codec drivers as unordered
  vb2: add explicit fence user API
  vb2: add in-fence support to QBUF
  vb2: add out-fence support to QBUF
  v4l: introduce the fences capability
  v4l: Add V4L2_CAP_FENCES to drivers
  v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst            |  48 ++-
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst   |   7 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  53 ++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst   |  12 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst   |   3 +
 drivers/media/common/videobuf2/Kconfig             |   1 +
 drivers/media/common/videobuf2/videobuf2-core.c    | 376 ++++++++++++++++++---
 drivers/media/common/videobuf2/videobuf2-v4l2.c    |  65 +++-
 drivers/media/dvb-core/dvb_vb2.c                   |   2 +-
 drivers/media/pci/cobalt/cobalt-v4l2.c             |   4 +
 drivers/media/platform/coda/coda-common.c          |   1 +
 drivers/media/platform/exynos-gsc/gsc-m2m.c        |   3 +-
 drivers/media/platform/m2m-deinterlace.c           |   3 +-
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       |   3 +-
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   1 +
 .../media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c |   2 +-
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   1 +
 .../media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c |   3 +-
 drivers/media/platform/mx2_emmaprp.c               |   3 +-
 drivers/media/platform/omap3isp/ispvideo.c         |  10 +-
 drivers/media/platform/qcom/venus/vdec.c           |   4 +-
 drivers/media/platform/qcom/venus/venc.c           |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       |   1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       |   1 +
 drivers/media/platform/sh_veu.c                    |   3 +-
 drivers/media/platform/xilinx/xilinx-dma.c         |  10 +-
 drivers/media/usb/hackrf/hackrf.c                  |  11 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |   4 +-
 drivers/media/v4l2-core/v4l2-ioctl.c               |  98 +++++-
 include/media/videobuf2-core.h                     |  76 ++++-
 include/media/videobuf2-v4l2.h                     |  18 -
 include/uapi/linux/videodev2.h                     |  10 +-
 32 files changed, 708 insertions(+), 133 deletions(-)

-- 
2.16.3

[PATCH v10 01/16] videobuf2: Make struct vb2_buffer refcounted

[Ezequiel Garcia]

The in-fence implementation involves having a per-buffer fence callback,
that triggers on the fence signal. The fence callback is called asynchronously
and needs a valid reference to the associated ideobuf2 buffer.

Allow this by making the vb2_buffer refcounted, so it can be passed
to other contexts.

Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 27 ++++++++++++++++++++++---
 include/media/videobuf2-core.h                  |  7 +++++--
 2 files changed, 29 insertions(+), 5 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..f1feb45c1e37 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -190,6 +190,26 @@ module_param(debug, int, 0644);
 static void __vb2_queue_cancel(struct vb2_queue *q);
 static void __enqueue_in_driver(struct vb2_buffer *vb);
 
+static void __vb2_buffer_free(struct kref *kref)
+{
+	struct vb2_buffer *vb =
+		container_of(kref, struct vb2_buffer, refcount);
+	kfree(vb);
+}
+
+static void __vb2_buffer_put(struct vb2_buffer *vb)
+{
+	if (vb)
+		kref_put(&vb->refcount, __vb2_buffer_free);
+}
+
+static struct vb2_buffer *__vb2_buffer_get(struct vb2_buffer *vb)
+{
+	if (vb)
+		kref_get(&vb->refcount);
+	return vb;
+}
+
 /*
  * __vb2_buf_mem_alloc() - allocate video memory for the given buffer
  */
@@ -346,6 +366,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 			break;
 		}
 
+		kref_init(&vb->refcount);
 		vb->state = VB2_BUF_STATE_DEQUEUED;
 		vb->vb2_queue = q;
 		vb->num_planes = num_planes;
@@ -365,7 +386,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 				dprintk(1, "failed allocating memory for buffer %d\n",
 					buffer);
 				q->bufs[vb->index] = NULL;
-				kfree(vb);
+				__vb2_buffer_put(vb);
 				break;
 			}
 			__setup_offsets(vb);
@@ -380,7 +401,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 					buffer, vb);
 				__vb2_buf_mem_free(vb);
 				q->bufs[vb->index] = NULL;
-				kfree(vb);
+				__vb2_buffer_put(vb);
 				break;
 			}
 		}
@@ -520,7 +541,7 @@ static int __vb2_queue_free(struct vb2_queue *q, unsigned int buffers)
 	/* Free videobuf buffers */
 	for (buffer = q->num_buffers - buffers; buffer < q->num_buffers;
 	     ++buffer) {
-		kfree(q->bufs[buffer]);
+		__vb2_buffer_put(q->bufs[buffer]);
 		q->bufs[buffer] = NULL;
 	}
 
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f6818f732f34..baa4632c7e59 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -12,11 +12,12 @@
 #ifndef _MEDIA_VIDEOBUF2_CORE_H
 #define _MEDIA_VIDEOBUF2_CORE_H
 
+#include <linux/bitops.h>
+#include <linux/dma-buf.h>
+#include <linux/kref.h>
 #include <linux/mm_types.h>
 #include <linux/mutex.h>
 #include <linux/poll.h>
-#include <linux/dma-buf.h>
-#include <linux/bitops.h>
 
 #define VB2_MAX_FRAME	(32)
 #define VB2_MAX_PLANES	(8)
@@ -249,6 +250,7 @@ struct vb2_buffer {
 
 	/* private: internal use only
 	 *
+	 * refcount:		refcount for this buffer
 	 * state:		current buffer state; do not change
 	 * queued_entry:	entry on the queued buffers list, which holds
 	 *			all buffers queued from userspace
@@ -256,6 +258,7 @@ struct vb2_buffer {
 	 *			to be dequeued to userspace
 	 * vb2_plane:		per-plane information; do not change
 	 */
+	struct kref		refcount;
 	enum vb2_buffer_state	state;
 
 	struct vb2_plane	planes[VB2_MAX_PLANES];
-- 
2.16.3

Re: [PATCH v10 01/16] videobuf2: Make struct vb2_buffer refcounted

[sathyam panda]

Hello,

On 5/21/18, Ezequiel Garcia <ezequiel@collabora.com> wrote:
> The in-fence implementation involves having a per-buffer fence callback,
> that triggers on the fence signal. The fence callback is called
> asynchronously
> and needs a valid reference to the associated ideobuf2 buffer.
>
> Allow this by making the vb2_buffer refcounted, so it can be passed
> to other contexts.
>

-Is it really required, because when a queued buffer with an in_fence
is deallocated, firstly queue is cancelled.
-And __vb2_dqbuf is called which calls dma_fence_remove_callback.
-So if fence callback has been called -__vb2_dqbuf will wait to
acquire fence lock.
-So during execution of fence callback, buffers and queue are still valid.
-And if __vb2_dqbuf remove callback first ,then dma_fence_signal will
wait for lock
- so there won't be any fence callback to call for that buffer when
dma_fence_signal resumes.

> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 27
> ++++++++++++++++++++++---
>  include/media/videobuf2-core.h                  |  7 +++++--
>  2 files changed, 29 insertions(+), 5 deletions(-)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> b/drivers/media/common/videobuf2/videobuf2-core.c
> index d3f7bb33a54d..f1feb45c1e37 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -190,6 +190,26 @@ module_param(debug, int, 0644);
>  static void __vb2_queue_cancel(struct vb2_queue *q);
>  static void __enqueue_in_driver(struct vb2_buffer *vb);
>
> +static void __vb2_buffer_free(struct kref *kref)
> +{
> +	struct vb2_buffer *vb =
> +		container_of(kref, struct vb2_buffer, refcount);
> +	kfree(vb);
> +}
> +
> +static void __vb2_buffer_put(struct vb2_buffer *vb)
> +{
> +	if (vb)
> +		kref_put(&vb->refcount, __vb2_buffer_free);
> +}
> +
> +static struct vb2_buffer *__vb2_buffer_get(struct vb2_buffer *vb)
> +{
> +	if (vb)
> +		kref_get(&vb->refcount);
> +	return vb;
> +}
> +
>  /*
>   * __vb2_buf_mem_alloc() - allocate video memory for the given buffer
>   */
> @@ -346,6 +366,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum
> vb2_memory memory,
>  			break;
>  		}
>
> +		kref_init(&vb->refcount);
>  		vb->state = VB2_BUF_STATE_DEQUEUED;
>  		vb->vb2_queue = q;
>  		vb->num_planes = num_planes;
> @@ -365,7 +386,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum
> vb2_memory memory,
>  				dprintk(1, "failed allocating memory for buffer %d\n",
>  					buffer);
>  				q->bufs[vb->index] = NULL;
> -				kfree(vb);
> +				__vb2_buffer_put(vb);
>  				break;
>  			}
>  			__setup_offsets(vb);
> @@ -380,7 +401,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum
> vb2_memory memory,
>  					buffer, vb);
>  				__vb2_buf_mem_free(vb);
>  				q->bufs[vb->index] = NULL;
> -				kfree(vb);
> +				__vb2_buffer_put(vb);
>  				break;
>  			}
>  		}
> @@ -520,7 +541,7 @@ static int __vb2_queue_free(struct vb2_queue *q,
> unsigned int buffers)
>  	/* Free videobuf buffers */
>  	for (buffer = q->num_buffers - buffers; buffer < q->num_buffers;
>  	     ++buffer) {
> -		kfree(q->bufs[buffer]);
> +		__vb2_buffer_put(q->bufs[buffer]);
>  		q->bufs[buffer] = NULL;
>  	}
>
> diff --git a/include/media/videobuf2-core.h
> b/include/media/videobuf2-core.h
> index f6818f732f34..baa4632c7e59 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -12,11 +12,12 @@
>  #ifndef _MEDIA_VIDEOBUF2_CORE_H
>  #define _MEDIA_VIDEOBUF2_CORE_H
>
> +#include <linux/bitops.h>
> +#include <linux/dma-buf.h>
> +#include <linux/kref.h>
>  #include <linux/mm_types.h>
>  #include <linux/mutex.h>
>  #include <linux/poll.h>
> -#include <linux/dma-buf.h>
> -#include <linux/bitops.h>
>
>  #define VB2_MAX_FRAME	(32)
>  #define VB2_MAX_PLANES	(8)
> @@ -249,6 +250,7 @@ struct vb2_buffer {
>
>  	/* private: internal use only
>  	 *
> +	 * refcount:		refcount for this buffer
>  	 * state:		current buffer state; do not change
>  	 * queued_entry:	entry on the queued buffers list, which holds
>  	 *			all buffers queued from userspace
> @@ -256,6 +258,7 @@ struct vb2_buffer {
>  	 *			to be dequeued to userspace
>  	 * vb2_plane:		per-plane information; do not change
>  	 */
> +	struct kref		refcount;
>  	enum vb2_buffer_state	state;
>
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>

Regards,
 Sathyam

Re: [PATCH v10 01/16] videobuf2: Make struct vb2_buffer refcounted

[Ezequiel Garcia]

On Fri, 2018-05-25 at 12:11 +0530, sathyam panda wrote:
> Hello,
> 
> On 5/21/18, Ezequiel Garcia <ezequiel@collabora.com> wrote:
> > The in-fence implementation involves having a per-buffer fence callback,
> > that triggers on the fence signal. The fence callback is called
> > asynchronously
> > and needs a valid reference to the associated ideobuf2 buffer.
> > 
> > Allow this by making the vb2_buffer refcounted, so it can be passed
> > to other contexts.
> > 
> 
> -Is it really required, because when a queued buffer with an in_fence
> is deallocated, firstly queue is cancelled.
> -And __vb2_dqbuf is called which calls dma_fence_remove_callback.
> -So if fence callback has been called -__vb2_dqbuf will wait to
> acquire fence lock.
> -So during execution of fence callback, buffers and queue are still valid.
> -And if __vb2_dqbuf remove callback first ,then dma_fence_signal will
> wait for lock
> - so there won't be any fence callback to call for that buffer when
> dma_fence_signal resumes.
> 

Hi Sathyam,

Thanks for your review! The refcount is definitely required,
as the fence callback only schedules a workqueue, which is
completely asynchronous with respect to the rest of the
ioctls.

In particular, the workqueue is not synchronized with
vb2_core_queue_release.

Also, another subtle detail, dma_fence_remove_callback
can fail to remove the callback.

Thanks,
Eze

Re: [PATCH v10 01/16] videobuf2: Make struct vb2_buffer refcounted

[sathyam panda]

Hi Ezequiel,

On 5/29/18, Ezequiel Garcia <ezequiel@collabora.com> wrote:
> On Fri, 2018-05-25 at 12:11 +0530, sathyam panda wrote:
>> Hello,
>>
>> On 5/21/18, Ezequiel Garcia <ezequiel@collabora.com> wrote:
>> > The in-fence implementation involves having a per-buffer fence
>> > callback,
>> > that triggers on the fence signal. The fence callback is called
>> > asynchronously
>> > and needs a valid reference to the associated ideobuf2 buffer.
>> >
>> > Allow this by making the vb2_buffer refcounted, so it can be passed
>> > to other contexts.
>> >
>>
>> -Is it really required, because when a queued buffer with an in_fence
>> is deallocated, firstly queue is cancelled.
>> -And __vb2_dqbuf is called which calls dma_fence_remove_callback.
>> -So if fence callback has been called -__vb2_dqbuf will wait to
>> acquire fence lock.
>> -So during execution of fence callback, buffers and queue are still
>> valid.
>> -And if __vb2_dqbuf remove callback first ,then dma_fence_signal will
>> wait for lock
>> - so there won't be any fence callback to call for that buffer when
>> dma_fence_signal resumes.
>>
>
> Hi Sathyam,
>
> Thanks for your review! The refcount is definitely required,
> as the fence callback only schedules a workqueue, which is
> completely asynchronous with respect to the rest of the
> ioctls.
>
> In particular, the workqueue is not synchronized with
> vb2_core_queue_release.
>
> Also, another subtle detail, dma_fence_remove_callback
> can fail to remove the callback.
>
> Thanks,
> Eze
>

- Sorry, I didn't pay attention to the use of workqueue, thanks for
bringing that into my notice. Now it make sense.
- I have a doubt about the queue since many driver specific structures
have queues which they release with kfree() after calling
vb2_queue_release().
- And since vb2_core_queue_release decrements the buffer refcount only
once if fence is already signalled, so buffer is still valid, but
queue isn't as the memory is deallocated.
- So is it safe to access queue in the __qbuf_work. Please correct me
if I am wrong.

Regards,
Sathyam

[PATCH v10 02/16] xilinx: regroup caps on querycap

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

To better organize the code we concentrate the setting of
V4L2_CAP_STREAMING in one place.

v2: move cap->capabilities assignment down (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 522cdfdd3345..d041f94be832 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,13 +494,15 @@ xvip_dma_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	struct v4l2_fh *vfh = file->private_data;
 	struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
-			  | dma->xdev->v4l2_caps;
+	cap->device_caps = V4L2_CAP_STREAMING;
 
 	if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
+
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
+			  | dma->xdev->v4l2_caps;
 
 	strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
 	strlcpy(cap->card, dma->video.name, sizeof(cap->card));
-- 
2.16.3

[PATCH v10 03/16] hackrf: group device capabilities

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING and V4L2_CAP_READWRITE
everywhere, set device_caps earlier with these values.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/usb/hackrf/hackrf.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/media/usb/hackrf/hackrf.c b/drivers/media/usb/hackrf/hackrf.c
index 7eb53517a82f..6d692fb3e8dd 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,18 +909,15 @@ static int hackrf_querycap(struct file *file, void *fh,
 
 	dev_dbg(&intf->dev, "\n");
 
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
 	if (vdev->vfl_dir == VFL_DIR_RX)
-		cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
-
+		cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
 	else
-		cap->device_caps = V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+		cap->device_caps |= V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR;
 
 	cap->capabilities = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
 			    V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-			    V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
-			    V4L2_CAP_DEVICE_CAPS;
+			    V4L2_CAP_DEVICE_CAPS | cap->device_caps;
 	strlcpy(cap->driver, KBUILD_MODNAME, sizeof(cap->driver));
 	strlcpy(cap->card, dev->rx_vdev.name, sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
-- 
2.16.3

[PATCH v10 04/16] omap3isp: group device capabilities

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
earlier with this value.

v2: move cap->capabilities assignment down (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/omap3isp/ispvideo.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..db9aae222134 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,13 +658,15 @@ isp_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	strlcpy(cap->card, video->video.name, sizeof(cap->card));
 	strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-	cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
-		| V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+	cap->device_caps = V4L2_CAP_STREAMING;
 
 	if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
+
+	cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
+		V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
 }
-- 
2.16.3

[PATCH v10 05/16] vb2: move vb2_ops functions to videobuf2-core.[ch]

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

vb2_ops_wait_prepare() and vb2_ops_wait_finish() were in the
wrong file.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 14 ++++++++++++++
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 14 --------------
 include/media/videobuf2-core.h                  | 18 ++++++++++++++++++
 include/media/videobuf2-v4l2.h                  | 18 ------------------
 4 files changed, 32 insertions(+), 32 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index f1feb45c1e37..61e7b6407586 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -677,6 +677,20 @@ int vb2_verify_memory_type(struct vb2_queue *q,
 }
 EXPORT_SYMBOL(vb2_verify_memory_type);
 
+/* vb2_ops helpers. Only use if vq->lock is non-NULL. */
+
+void vb2_ops_wait_prepare(struct vb2_queue *vq)
+{
+	mutex_unlock(vq->lock);
+}
+EXPORT_SYMBOL_GPL(vb2_ops_wait_prepare);
+
+void vb2_ops_wait_finish(struct vb2_queue *vq)
+{
+	mutex_lock(vq->lock);
+}
+EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
+
 int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
 		unsigned int *count)
 {
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..64503615d00b 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -947,20 +947,6 @@ unsigned long vb2_fop_get_unmapped_area(struct file *file, unsigned long addr,
 EXPORT_SYMBOL_GPL(vb2_fop_get_unmapped_area);
 #endif
 
-/* vb2_ops helpers. Only use if vq->lock is non-NULL. */
-
-void vb2_ops_wait_prepare(struct vb2_queue *vq)
-{
-	mutex_unlock(vq->lock);
-}
-EXPORT_SYMBOL_GPL(vb2_ops_wait_prepare);
-
-void vb2_ops_wait_finish(struct vb2_queue *vq)
-{
-	mutex_lock(vq->lock);
-}
-EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
-
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak <pawel@osciak.com>, Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index baa4632c7e59..137f72702101 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -403,6 +403,24 @@ struct vb2_ops {
 	void (*buf_queue)(struct vb2_buffer *vb);
 };
 
+/**
+ * vb2_ops_wait_prepare - helper function to lock a struct &vb2_queue
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * ..note:: only use if vq->lock is non-NULL.
+ */
+void vb2_ops_wait_prepare(struct vb2_queue *vq);
+
+/**
+ * vb2_ops_wait_finish - helper function to unlock a struct &vb2_queue
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * ..note:: only use if vq->lock is non-NULL.
+ */
+void vb2_ops_wait_finish(struct vb2_queue *vq);
+
 /**
  * struct vb2_buf_ops - driver-specific callbacks.
  *
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..cf83b01dc44e 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -273,22 +273,4 @@ unsigned long vb2_fop_get_unmapped_area(struct file *file, unsigned long addr,
 		unsigned long len, unsigned long pgoff, unsigned long flags);
 #endif
 
-/**
- * vb2_ops_wait_prepare - helper function to lock a struct &vb2_queue
- *
- * @vq: pointer to &struct vb2_queue
- *
- * ..note:: only use if vq->lock is non-NULL.
- */
-void vb2_ops_wait_prepare(struct vb2_queue *vq);
-
-/**
- * vb2_ops_wait_finish - helper function to unlock a struct &vb2_queue
- *
- * @vq: pointer to &struct vb2_queue
- *
- * ..note:: only use if vq->lock is non-NULL.
- */
-void vb2_ops_wait_finish(struct vb2_queue *vq);
-
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.16.3

[PATCH v10 06/16] vb2: add is_unordered callback for drivers

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

v5: rename it to vb2_ops_is_unordered() (Hans Verkuil)

v4: go back to a bitfield property for the unordered property.

v3: - make it bool (Hans)
    - create vb2_ops_set_unordered() helper

v2: - improve comments for is_unordered flag (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c |  6 ++++++
 include/media/videobuf2-core.h                  | 16 ++++++++++++++++
 2 files changed, 22 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 61e7b6407586..a9a0a9d1decb 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -691,6 +691,12 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
 }
 EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
 
+bool vb2_ops_is_unordered(struct vb2_queue *q)
+{
+	return true;
+}
+EXPORT_SYMBOL_GPL(vb2_ops_is_unordered);
+
 int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
 		unsigned int *count)
 {
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 137f72702101..71538ae2c255 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -376,6 +376,10 @@ struct vb2_buffer {
  *			callback by calling vb2_buffer_done() with either
  *			%VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  *			vb2_wait_for_all_buffers() function
+ * @is_unordered:	tell if the queue is unordered, i.e. buffers can be
+ *			dequeued in a different order from how they were queued.
+ *			The default is assumed to be ordered and this function
+ *			only needs to be implemented for unordered queues.
  * @buf_queue:		passes buffer vb to the driver; driver may start
  *			hardware operation on this buffer; driver should give
  *			the buffer back by calling vb2_buffer_done() function;
@@ -399,6 +403,7 @@ struct vb2_ops {
 
 	int (*start_streaming)(struct vb2_queue *q, unsigned int count);
 	void (*stop_streaming)(struct vb2_queue *q);
+	bool (*is_unordered)(struct vb2_queue *q);
 
 	void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -421,6 +426,16 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
  */
 void vb2_ops_wait_finish(struct vb2_queue *vq);
 
+/**
+ * vb2_ops_is_unordered - helper function to check if queue is unordered
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * This helper just returns true to notify that the driver can't deal with
+ * ordered queues.
+ */
+bool vb2_ops_is_unordered(struct vb2_queue *q);
+
 /**
  * struct vb2_buf_ops - driver-specific callbacks.
  *
@@ -590,6 +605,7 @@ struct vb2_queue {
 	u32				cnt_wait_finish;
 	u32				cnt_start_streaming;
 	u32				cnt_stop_streaming;
+	u32				cnt_is_unordered;
 #endif
 };
 
-- 
2.16.3

[PATCH v10 07/16] v4l: add unordered flag to format description ioctl

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

v2: Improve documentation (Hans)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 7 +++++++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 8 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..df8e039b9ac2 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,13 @@ one until ``EINVAL`` is returned.
       - This format is not native to the device but emulated through
 	software (usually libv4l2), where possible try to use a native
 	format instead for better performance.
+    * - ``V4L2_FMT_FLAG_UNORDERED``
+      - 0x0004
+      - This format doesn't guarantee ordered buffer handling. I.e. the order
+	in which buffers are dequeued with
+	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` may be different
+	from the order in which they were queued with
+	:ref:`VIDIOC_QBUF <VIDIOC_QBUF>`.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 600877be5c22..a8842a5ca636 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -717,6 +717,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
 	/* Frame Size and frame rate enumeration */
 /*
-- 
2.16.3

[PATCH v10 08/16] v4l: mark unordered formats

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
mark the appropriate formats.

v2: Set unordered flag before calling the driver callback.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 74 +++++++++++++++++++++++++++---------
 1 file changed, 57 insertions(+), 17 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f48c505550e0..2135ac235a96 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1102,6 +1102,27 @@ static int v4l_enumoutput(const struct v4l2_ioctl_ops *ops,
 	return ops->vidioc_enum_output(file, fh, p);
 }
 
+static void v4l_fill_unordered_fmtdesc(struct v4l2_fmtdesc *fmt)
+{
+	switch (fmt->pixelformat) {
+	case V4L2_PIX_FMT_MPEG:
+	case V4L2_PIX_FMT_H264:
+	case V4L2_PIX_FMT_H264_NO_SC:
+	case V4L2_PIX_FMT_H264_MVC:
+	case V4L2_PIX_FMT_H263:
+	case V4L2_PIX_FMT_MPEG1:
+	case V4L2_PIX_FMT_MPEG2:
+	case V4L2_PIX_FMT_MPEG4:
+	case V4L2_PIX_FMT_XVID:
+	case V4L2_PIX_FMT_VC1_ANNEX_G:
+	case V4L2_PIX_FMT_VC1_ANNEX_L:
+	case V4L2_PIX_FMT_VP8:
+	case V4L2_PIX_FMT_VP9:
+	case V4L2_PIX_FMT_HEVC:
+		fmt->flags |= V4L2_FMT_FLAG_UNORDERED;
+	}
+}
+
 static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 {
 	const unsigned sz = sizeof(fmt->description);
@@ -1310,61 +1331,80 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 
 	if (descr)
 		WARN_ON(strlcpy(fmt->description, descr, sz) >= sz);
-	fmt->flags = flags;
+	fmt->flags |= flags;
 }
 
-static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
-				struct file *file, void *fh, void *arg)
-{
-	struct v4l2_fmtdesc *p = arg;
-	int ret = check_fmt(file, p->type);
 
-	if (ret)
-		return ret;
-	ret = -EINVAL;
+static int __vidioc_enum_fmt(const struct v4l2_ioctl_ops *ops,
+			     struct v4l2_fmtdesc *p,
+			     struct file *file, void *fh)
+{
+	int ret = 0;
 
 	switch (p->type) {
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
 		if (unlikely(!ops->vidioc_enum_fmt_vid_cap))
 			break;
-		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
 		if (unlikely(!ops->vidioc_enum_fmt_vid_cap_mplane))
 			break;
-		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
 		if (unlikely(!ops->vidioc_enum_fmt_vid_overlay))
 			break;
-		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
 		if (unlikely(!ops->vidioc_enum_fmt_vid_out))
 			break;
-		ret = ops->vidioc_enum_fmt_vid_out(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_vid_out(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
 		if (unlikely(!ops->vidioc_enum_fmt_vid_out_mplane))
 			break;
-		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_SDR_CAPTURE:
 		if (unlikely(!ops->vidioc_enum_fmt_sdr_cap))
 			break;
-		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_SDR_OUTPUT:
 		if (unlikely(!ops->vidioc_enum_fmt_sdr_out))
 			break;
-		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, p);
 		break;
 	case V4L2_BUF_TYPE_META_CAPTURE:
 		if (unlikely(!ops->vidioc_enum_fmt_meta_cap))
 			break;
-		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, arg);
+		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, p);
 		break;
 	}
+	return ret;
+}
+
+static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
+				struct file *file, void *fh, void *arg)
+{
+	struct v4l2_fmtdesc *p = arg;
+	int ret = check_fmt(file, p->type);
+
+	if (ret)
+		return ret;
+	ret = -EINVAL;
+
+	ret = __vidioc_enum_fmt(ops, p, file, fh);
+	if (ret)
+		return ret;
+	/*
+	 * Set the unordered flag and call the driver
+	 * again so it has the chance to clear the flag.
+	 */
+	v4l_fill_unordered_fmtdesc(p);
+	ret = __vidioc_enum_fmt(ops, p, file, fh);
 	if (ret == 0)
 		v4l_fill_fmtdesc(p);
 	return ret;
-- 
2.16.3

Re: [PATCH v10 08/16] v4l: mark unordered formats

[Hans Verkuil]

On 21/05/18 18:59, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
> mark the appropriate formats.
> 
> v2: Set unordered flag before calling the driver callback.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 74 +++++++++++++++++++++++++++---------
>  1 file changed, 57 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f48c505550e0..2135ac235a96 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1102,6 +1102,27 @@ static int v4l_enumoutput(const struct v4l2_ioctl_ops *ops,
>  	return ops->vidioc_enum_output(file, fh, p);
>  }
>  
> +static void v4l_fill_unordered_fmtdesc(struct v4l2_fmtdesc *fmt)
> +{
> +	switch (fmt->pixelformat) {
> +	case V4L2_PIX_FMT_MPEG:
> +	case V4L2_PIX_FMT_H264:
> +	case V4L2_PIX_FMT_H264_NO_SC:
> +	case V4L2_PIX_FMT_H264_MVC:
> +	case V4L2_PIX_FMT_H263:
> +	case V4L2_PIX_FMT_MPEG1:
> +	case V4L2_PIX_FMT_MPEG2:
> +	case V4L2_PIX_FMT_MPEG4:
> +	case V4L2_PIX_FMT_XVID:
> +	case V4L2_PIX_FMT_VC1_ANNEX_G:
> +	case V4L2_PIX_FMT_VC1_ANNEX_L:
> +	case V4L2_PIX_FMT_VP8:
> +	case V4L2_PIX_FMT_VP9:
> +	case V4L2_PIX_FMT_HEVC:
> +		fmt->flags |= V4L2_FMT_FLAG_UNORDERED;

Please add a break here. This prevents potential future errors if new cases
are added below this line.

> +	}
> +}
> +
>  static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  {
>  	const unsigned sz = sizeof(fmt->description);
> @@ -1310,61 +1331,80 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  
>  	if (descr)
>  		WARN_ON(strlcpy(fmt->description, descr, sz) >= sz);
> -	fmt->flags = flags;
> +	fmt->flags |= flags;
>  }
>  
> -static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
> -				struct file *file, void *fh, void *arg)
> -{
> -	struct v4l2_fmtdesc *p = arg;
> -	int ret = check_fmt(file, p->type);
>  
> -	if (ret)
> -		return ret;
> -	ret = -EINVAL;
> +static int __vidioc_enum_fmt(const struct v4l2_ioctl_ops *ops,
> +			     struct v4l2_fmtdesc *p,
> +			     struct file *file, void *fh)
> +{
> +	int ret = 0;
>  
>  	switch (p->type) {
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap))
>  			break;
> -		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap_mplane))
>  			break;
> -		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>  		if (unlikely(!ops->vidioc_enum_fmt_vid_overlay))
>  			break;
> -		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>  		if (unlikely(!ops->vidioc_enum_fmt_vid_out))
>  			break;
> -		ret = ops->vidioc_enum_fmt_vid_out(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_vid_out(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>  		if (unlikely(!ops->vidioc_enum_fmt_vid_out_mplane))
>  			break;
> -		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_SDR_CAPTURE:
>  		if (unlikely(!ops->vidioc_enum_fmt_sdr_cap))
>  			break;
> -		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_SDR_OUTPUT:
>  		if (unlikely(!ops->vidioc_enum_fmt_sdr_out))
>  			break;
> -		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, p);
>  		break;
>  	case V4L2_BUF_TYPE_META_CAPTURE:
>  		if (unlikely(!ops->vidioc_enum_fmt_meta_cap))
>  			break;
> -		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, arg);
> +		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, p);
>  		break;
>  	}
> +	return ret;
> +}
> +
> +static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
> +				struct file *file, void *fh, void *arg)
> +{
> +	struct v4l2_fmtdesc *p = arg;
> +	int ret = check_fmt(file, p->type);
> +
> +	if (ret)
> +		return ret;
> +	ret = -EINVAL;

Why set ret when it is overwritten below?

> +
> +	ret = __vidioc_enum_fmt(ops, p, file, fh);
> +	if (ret)
> +		return ret;

Huh? Why call the driver twice? As far as I can see you can just drop
these three lines above.

Regards,

	Hans

> +	/*
> +	 * Set the unordered flag and call the driver
> +	 * again so it has the chance to clear the flag.
> +	 */
> +	v4l_fill_unordered_fmtdesc(p);
> +	ret = __vidioc_enum_fmt(ops, p, file, fh);
>  	if (ret == 0)
>  		v4l_fill_fmtdesc(p);
>  	return ret;
> 

Re: [PATCH v10 08/16] v4l: mark unordered formats

[Ezequiel Garcia]

On Tue, 2018-05-22 at 13:55 +0200, Hans Verkuil wrote:
> On 21/05/18 18:59, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
> > mark the appropriate formats.
> > 
> > v2: Set unordered flag before calling the driver callback.
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-ioctl.c | 74 +++++++++++++++++++++++++++---------
> >  1 file changed, 57 insertions(+), 17 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> > index f48c505550e0..2135ac235a96 100644
> > --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> > +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> > @@ -1102,6 +1102,27 @@ static int v4l_enumoutput(const struct v4l2_ioctl_ops *ops,
> >  	return ops->vidioc_enum_output(file, fh, p);
> >  }
> >  
> > +static void v4l_fill_unordered_fmtdesc(struct v4l2_fmtdesc *fmt)
> > +{
> > +	switch (fmt->pixelformat) {
> > +	case V4L2_PIX_FMT_MPEG:
> > +	case V4L2_PIX_FMT_H264:
> > +	case V4L2_PIX_FMT_H264_NO_SC:
> > +	case V4L2_PIX_FMT_H264_MVC:
> > +	case V4L2_PIX_FMT_H263:
> > +	case V4L2_PIX_FMT_MPEG1:
> > +	case V4L2_PIX_FMT_MPEG2:
> > +	case V4L2_PIX_FMT_MPEG4:
> > +	case V4L2_PIX_FMT_XVID:
> > +	case V4L2_PIX_FMT_VC1_ANNEX_G:
> > +	case V4L2_PIX_FMT_VC1_ANNEX_L:
> > +	case V4L2_PIX_FMT_VP8:
> > +	case V4L2_PIX_FMT_VP9:
> > +	case V4L2_PIX_FMT_HEVC:
> > +		fmt->flags |= V4L2_FMT_FLAG_UNORDERED;
> 
> Please add a break here. This prevents potential future errors if new cases
> are added below this line.
> 

Sure.

> > +	}
> > +}
> > +
> >  static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  {
> >  	const unsigned sz = sizeof(fmt->description);
> > @@ -1310,61 +1331,80 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
> >  
> >  	if (descr)
> >  		WARN_ON(strlcpy(fmt->description, descr, sz) >= sz);
> > -	fmt->flags = flags;
> > +	fmt->flags |= flags;
> >  }
> >  
> > -static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
> > -				struct file *file, void *fh, void *arg)
> > -{
> > -	struct v4l2_fmtdesc *p = arg;
> > -	int ret = check_fmt(file, p->type);
> >  
> > -	if (ret)
> > -		return ret;
> > -	ret = -EINVAL;
> > +static int __vidioc_enum_fmt(const struct v4l2_ioctl_ops *ops,
> > +			     struct v4l2_fmtdesc *p,
> > +			     struct file *file, void *fh)
> > +{
> > +	int ret = 0;
> >  
> >  	switch (p->type) {
> >  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
> >  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
> >  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap_mplane))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
> >  		if (unlikely(!ops->vidioc_enum_fmt_vid_overlay))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
> >  		if (unlikely(!ops->vidioc_enum_fmt_vid_out))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_vid_out(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_vid_out(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
> >  		if (unlikely(!ops->vidioc_enum_fmt_vid_out_mplane))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_SDR_CAPTURE:
> >  		if (unlikely(!ops->vidioc_enum_fmt_sdr_cap))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_SDR_OUTPUT:
> >  		if (unlikely(!ops->vidioc_enum_fmt_sdr_out))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, p);
> >  		break;
> >  	case V4L2_BUF_TYPE_META_CAPTURE:
> >  		if (unlikely(!ops->vidioc_enum_fmt_meta_cap))
> >  			break;
> > -		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, arg);
> > +		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, p);
> >  		break;
> >  	}
> > +	return ret;
> > +}
> > +
> > +static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
> > +				struct file *file, void *fh, void *arg)
> > +{
> > +	struct v4l2_fmtdesc *p = arg;
> > +	int ret = check_fmt(file, p->type);
> > +
> > +	if (ret)
> > +		return ret;
> > +	ret = -EINVAL;
> 
> Why set ret when it is overwritten below?
> 

Oops, seems to be some leftover code.

> > +
> > +	ret = __vidioc_enum_fmt(ops, p, file, fh);
> > +	if (ret)
> > +		return ret;
> 
> Huh? Why call the driver twice? As far as I can see you can just drop
> these three lines above.
> 
> 

Well, because I thought this was the outcome of v9 [1]. Let me quote you:

""
I realized that this is a problem since this function is called *after*
the driver. So the driver has no chance to clear this flag if it knows
that the queue is always ordered.
""

So, we first call the driver to get struct v4l2_fmtdesc pixelformat field
filled by the driver, then we call v4l_fill_unordered_fmtdesc()
to set FLAG_UNORDERED if needed, and finally we call the driver again
so it can clear the FLAG_UNORDERED.

Does that make sense?

[1] https://lkml.org/lkml/2018/5/7/349

Re: [PATCH v10 08/16] v4l: mark unordered formats

[Hans Verkuil]

On 23/05/18 12:30, Ezequiel Garcia wrote:
> On Tue, 2018-05-22 at 13:55 +0200, Hans Verkuil wrote:
>> On 21/05/18 18:59, Ezequiel Garcia wrote:
>>> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>>>
>>> Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
>>> mark the appropriate formats.
>>>
>>> v2: Set unordered flag before calling the driver callback.
>>>
>>> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>>> ---
>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 74 +++++++++++++++++++++++++++---------
>>>  1 file changed, 57 insertions(+), 17 deletions(-)
>>>
>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> index f48c505550e0..2135ac235a96 100644
>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> @@ -1102,6 +1102,27 @@ static int v4l_enumoutput(const struct v4l2_ioctl_ops *ops,
>>>  	return ops->vidioc_enum_output(file, fh, p);
>>>  }
>>>  
>>> +static void v4l_fill_unordered_fmtdesc(struct v4l2_fmtdesc *fmt)
>>> +{
>>> +	switch (fmt->pixelformat) {
>>> +	case V4L2_PIX_FMT_MPEG:
>>> +	case V4L2_PIX_FMT_H264:
>>> +	case V4L2_PIX_FMT_H264_NO_SC:
>>> +	case V4L2_PIX_FMT_H264_MVC:
>>> +	case V4L2_PIX_FMT_H263:
>>> +	case V4L2_PIX_FMT_MPEG1:
>>> +	case V4L2_PIX_FMT_MPEG2:
>>> +	case V4L2_PIX_FMT_MPEG4:
>>> +	case V4L2_PIX_FMT_XVID:
>>> +	case V4L2_PIX_FMT_VC1_ANNEX_G:
>>> +	case V4L2_PIX_FMT_VC1_ANNEX_L:
>>> +	case V4L2_PIX_FMT_VP8:
>>> +	case V4L2_PIX_FMT_VP9:
>>> +	case V4L2_PIX_FMT_HEVC:
>>> +		fmt->flags |= V4L2_FMT_FLAG_UNORDERED;
>>
>> Please add a break here. This prevents potential future errors if new cases
>> are added below this line.
>>
> 
> Sure.
> 
>>> +	}
>>> +}
>>> +
>>>  static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>>>  {
>>>  	const unsigned sz = sizeof(fmt->description);
>>> @@ -1310,61 +1331,80 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>>>  
>>>  	if (descr)
>>>  		WARN_ON(strlcpy(fmt->description, descr, sz) >= sz);
>>> -	fmt->flags = flags;
>>> +	fmt->flags |= flags;
>>>  }
>>>  
>>> -static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
>>> -				struct file *file, void *fh, void *arg)
>>> -{
>>> -	struct v4l2_fmtdesc *p = arg;
>>> -	int ret = check_fmt(file, p->type);
>>>  
>>> -	if (ret)
>>> -		return ret;
>>> -	ret = -EINVAL;
>>> +static int __vidioc_enum_fmt(const struct v4l2_ioctl_ops *ops,
>>> +			     struct v4l2_fmtdesc *p,
>>> +			     struct file *file, void *fh)
>>> +{
>>> +	int ret = 0;
>>>  
>>>  	switch (p->type) {
>>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_vid_cap(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_vid_cap_mplane))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_vid_cap_mplane(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_VIDEO_OVERLAY:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_vid_overlay))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_vid_overlay(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_vid_out))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_vid_out(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_vid_out(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_vid_out_mplane))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_vid_out_mplane(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_SDR_CAPTURE:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_sdr_cap))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_sdr_cap(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_SDR_OUTPUT:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_sdr_out))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_sdr_out(file, fh, p);
>>>  		break;
>>>  	case V4L2_BUF_TYPE_META_CAPTURE:
>>>  		if (unlikely(!ops->vidioc_enum_fmt_meta_cap))
>>>  			break;
>>> -		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, arg);
>>> +		ret = ops->vidioc_enum_fmt_meta_cap(file, fh, p);
>>>  		break;
>>>  	}
>>> +	return ret;
>>> +}
>>> +
>>> +static int v4l_enum_fmt(const struct v4l2_ioctl_ops *ops,
>>> +				struct file *file, void *fh, void *arg)
>>> +{
>>> +	struct v4l2_fmtdesc *p = arg;
>>> +	int ret = check_fmt(file, p->type);
>>> +
>>> +	if (ret)
>>> +		return ret;
>>> +	ret = -EINVAL;
>>
>> Why set ret when it is overwritten below?
>>
> 
> Oops, seems to be some leftover code.
> 
>>> +
>>> +	ret = __vidioc_enum_fmt(ops, p, file, fh);
>>> +	if (ret)
>>> +		return ret;
>>
>> Huh? Why call the driver twice? As far as I can see you can just drop
>> these three lines above.
>>
>>
> 
> Well, because I thought this was the outcome of v9 [1]. Let me quote you:
> 
> ""
> I realized that this is a problem since this function is called *after*
> the driver. So the driver has no chance to clear this flag if it knows
> that the queue is always ordered.
> ""
> 
> So, we first call the driver to get struct v4l2_fmtdesc pixelformat field
> filled by the driver, then we call v4l_fill_unordered_fmtdesc()
> to set FLAG_UNORDERED if needed, and finally we call the driver again
> so it can clear the FLAG_UNORDERED.
> 
> Does that make sense?

Not what I meant. v4l_fill_unordered_fmtdesc() should be called first, then
the driver (which can now fiddle with the flag if it wants to) and finally
v4l_fill_fmtdesc(p) is called.

Drivers that support these compressed formats need to be checked though:
they shouldn't overwrite flags unconditionally.

Regards,

	Hans

> 
> [1] https://lkml.org/lkml/2018/5/7/349
> 

[PATCH v10 09/16] cobalt: add .is_unordered() for cobalt

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

The cobalt driver may reorder the capture buffers so we need to report
it as such.

v3: set formats as unordered
v2: use vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 4 ++++
 1 file changed, 4 insertions(+)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
index e2a4c705d353..ccca1a96df90 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
 	.stop_streaming = cobalt_stop_streaming,
 	.wait_prepare = vb2_ops_wait_prepare,
 	.wait_finish = vb2_ops_wait_finish,
+	.is_unordered = vb2_ops_is_unordered,
 };
 
 /* V4L2 ioctls */
@@ -695,14 +696,17 @@ static int cobalt_enum_fmt_vid_cap(struct file *file, void *priv_fh,
 	case 0:
 		strlcpy(f->description, "YUV 4:2:2", sizeof(f->description));
 		f->pixelformat = V4L2_PIX_FMT_YUYV;
+		f->flags |= V4L2_FMT_FLAG_UNORDERED;
 		break;
 	case 1:
 		strlcpy(f->description, "RGB24", sizeof(f->description));
 		f->pixelformat = V4L2_PIX_FMT_RGB24;
+		f->flags |= V4L2_FMT_FLAG_UNORDERED;
 		break;
 	case 2:
 		strlcpy(f->description, "RGB32", sizeof(f->description));
 		f->pixelformat = V4L2_PIX_FMT_BGR32;
+		f->flags |= V4L2_FMT_FLAG_UNORDERED;
 		break;
 	default:
 		return -EINVAL;
-- 
2.16.3

Re: [PATCH v10 09/16] cobalt: add .is_unordered() for cobalt

[Hans Verkuil]

On 21/05/18 18:59, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> The cobalt driver may reorder the capture buffers so we need to report
> it as such.
> 
> v3: set formats as unordered
> v2: use vb2_ops_set_unordered() helper
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/pci/cobalt/cobalt-v4l2.c | 4 ++++
>  1 file changed, 4 insertions(+)
> 
> diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
> index e2a4c705d353..ccca1a96df90 100644
> --- a/drivers/media/pci/cobalt/cobalt-v4l2.c
> +++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
> @@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
>  	.stop_streaming = cobalt_stop_streaming,
>  	.wait_prepare = vb2_ops_wait_prepare,
>  	.wait_finish = vb2_ops_wait_finish,
> +	.is_unordered = vb2_ops_is_unordered,
>  };
>  
>  /* V4L2 ioctls */
> @@ -695,14 +696,17 @@ static int cobalt_enum_fmt_vid_cap(struct file *file, void *priv_fh,
>  	case 0:
>  		strlcpy(f->description, "YUV 4:2:2", sizeof(f->description));
>  		f->pixelformat = V4L2_PIX_FMT_YUYV;
> +		f->flags |= V4L2_FMT_FLAG_UNORDERED;
>  		break;
>  	case 1:
>  		strlcpy(f->description, "RGB24", sizeof(f->description));
>  		f->pixelformat = V4L2_PIX_FMT_RGB24;
> +		f->flags |= V4L2_FMT_FLAG_UNORDERED;
>  		break;
>  	case 2:
>  		strlcpy(f->description, "RGB32", sizeof(f->description));
>  		f->pixelformat = V4L2_PIX_FMT_BGR32;
> +		f->flags |= V4L2_FMT_FLAG_UNORDERED;

Rather than adding this for every case, just move it out of the switch and
set it just before the 'return 0'.

Regards,

	Hans

>  		break;
>  	default:
>  		return -EINVAL;
> 

[PATCH v10 10/16] vb2: mark codec drivers as unordered

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

In preparation to have full support to explicit fence we are
marking codec as non-ordered preventively. It is easier and safer from an
uAPI point of view to move from unordered to ordered than the opposite.

v2: mark only codec drivers as unordered (Nicolas and Hans)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/platform/coda/coda-common.c          | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
 drivers/media/platform/qcom/venus/vdec.c           | 1 +
 drivers/media/platform/qcom/venus/venc.c           | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       | 1 +
 7 files changed, 7 insertions(+)

diff --git a/drivers/media/platform/coda/coda-common.c b/drivers/media/platform/coda/coda-common.c
index 04e35d70ce2e..cac36a11efa2 100644
--- a/drivers/media/platform/coda/coda-common.c
+++ b/drivers/media/platform/coda/coda-common.c
@@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
 	.stop_streaming		= coda_stop_streaming,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
+	.is_unordered		= vb2_ops_is_unordered,
 };
 
 static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
index 86f0a7134365..a4a02f3790fa 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
@@ -1445,6 +1445,7 @@ static const struct vb2_ops mtk_vdec_vb2_ops = {
 	.buf_finish	= vb2ops_vdec_buf_finish,
 	.start_streaming	= vb2ops_vdec_start_streaming,
 	.stop_streaming	= vb2ops_vdec_stop_streaming,
+	.is_unordered	= vb2_ops_is_unordered,
 };
 
 const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops = {
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
index 1b1a28abbf1f..d37d670346b9 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
@@ -931,6 +931,7 @@ static const struct vb2_ops mtk_venc_vb2_ops = {
 	.wait_finish		= vb2_ops_wait_finish,
 	.start_streaming	= vb2ops_venc_start_streaming,
 	.stop_streaming		= vb2ops_venc_stop_streaming,
+	.is_unordered		= vb2_ops_is_unordered,
 };
 
 static int mtk_venc_encode_header(void *priv)
diff --git a/drivers/media/platform/qcom/venus/vdec.c b/drivers/media/platform/qcom/venus/vdec.c
index 49bbd1861d3a..8d7b4fc95880 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -794,6 +794,7 @@ static const struct vb2_ops vdec_vb2_ops = {
 	.start_streaming = vdec_start_streaming,
 	.stop_streaming = venus_helper_vb2_stop_streaming,
 	.buf_queue = venus_helper_vb2_buf_queue,
+	.is_unordered = vb2_ops_is_unordered,
 };
 
 static void vdec_buf_done(struct venus_inst *inst, unsigned int buf_type,
diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
index 6b2ce479584e..713c79ba9639 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -983,6 +983,7 @@ static const struct vb2_ops venc_vb2_ops = {
 	.start_streaming = venc_start_streaming,
 	.stop_streaming = venus_helper_vb2_stop_streaming,
 	.buf_queue = venus_helper_vb2_buf_queue,
+	.is_unordered = vb2_ops_is_unordered,
 };
 
 static void venc_buf_done(struct venus_inst *inst, unsigned int buf_type,
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
index 5cf4d9921264..4402a5d621b2 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
@@ -1105,6 +1105,7 @@ static struct vb2_ops s5p_mfc_dec_qops = {
 	.start_streaming	= s5p_mfc_start_streaming,
 	.stop_streaming		= s5p_mfc_stop_streaming,
 	.buf_queue		= s5p_mfc_buf_queue,
+	.is_unordered		= vb2_ops_is_unordered,
 };
 
 const struct s5p_mfc_codec_ops *get_dec_codec_ops(void)
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
index 5c0462ca9993..376bd8eab8d8 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
@@ -2613,6 +2613,7 @@ static struct vb2_ops s5p_mfc_enc_qops = {
 	.start_streaming	= s5p_mfc_start_streaming,
 	.stop_streaming		= s5p_mfc_stop_streaming,
 	.buf_queue		= s5p_mfc_buf_queue,
+	.is_unordered		= vb2_ops_is_unordered,
 };
 
 const struct s5p_mfc_codec_ops *get_enc_codec_ops(void)
-- 
2.16.3

[PATCH v10 11/16] vb2: add explicit fence user API

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel or return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending an in-fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v8: return -1 if new flags are set.
v7: minor fixes on the Documentation (Hans Verkuil)

v6: big improvement on doc (Hans Verkuil)

v5: - keep using reserved2 field for cpia2
    - set fence_fd to 0 for now, for compat with userspace(Mauro)

v4: make it a union with reserved2 and fence_fd (Hans Verkuil)

v3: make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst         | 48 ++++++++++++++++++++++---
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 +++-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
 include/uapi/linux/videodev2.h                  |  8 ++++-
 4 files changed, 58 insertions(+), 8 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..971b7453040c 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -300,11 +300,23 @@ struct v4l2_buffer
 	multi-planar API the application sets this to the number of
 	elements in the ``planes`` array. The driver will fill in the
 	actual number of valid elements in that array.
-    * - __u32
-      - ``reserved2``
+    * - __s32
+      - ``fence_fd``
       -
-      - A place holder for future extensions. Drivers and applications
-	must set this to 0.
+      - Used to communicate a fence file descriptor from userspace to kernel
+	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
+	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
+	be used and this field set to the fence file descriptor of the in-fence.
+	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
+
+        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
+	must be set, the kernel will return the out-fence file descriptor in
+	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
+        an error.
+
+	For all other ioctls V4L2 sets this field to -1 if
+	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
+	otherwise this field is set to 0 for backward compatibility.
     * - __u32
       - ``reserved``
       -
@@ -648,6 +660,34 @@ Buffer Flags
       - Start Of Exposure. The buffer timestamp has been taken when the
 	exposure of the frame has begun. This is only valid for the
 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+      - ``V4L2_BUF_FLAG_IN_FENCE``
+      - 0x00200000
+      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
+	buffer won't be queued to the driver until the fence signals. The order
+	in which buffers are queued is guaranteed to be preserved, so any
+	buffers queued after this buffer will also be blocked until this fence
+	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
+	other ioctls the driver just reports the value of the flag.
+
+        If the fence signals the flag is cleared and not reported anymore.
+	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
+    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+      - ``V4L2_BUF_FLAG_OUT_FENCE``
+      - 0x00400000
+      - Request for a fence to be attached to the buffer. The driver will fill
+	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
+	<VIDIOC_QBUF>` returns. This flag must be set before calling
+	``VIDIOC_QBUF``. This flag is only an input, and is not set by the kernel.
+
+        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
+	error.
+
+        Note that it is valid to set both ``V4L2_BUF_FLAG_IN_FENCE`` and
+        `V4L2_BUF_FLAG_OUT_FENCE`` flags. In such case, the ``fence_fd``
+        field is used to both set the in-fence and return the out-fence.
 
 
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 64503615d00b..8312f61adfa6 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,9 +203,13 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
 	b->reserved = 0;
 
+	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
+		b->fence_fd = -1;
+	else
+		b->fence_fd = 0;
+
 	if (q->is_multiplanar) {
 		/*
 		 * Fill in plane-related data if userspace provided an array
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 4312935f1dfc..93c752459aec 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -388,7 +388,7 @@ struct v4l2_buffer32 {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__s32			fence_fd;
 	__u32			reserved;
 };
 
@@ -606,7 +606,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
 	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
 	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
 	    assign_in_user(&up->sequence, &kp->sequence) ||
-	    assign_in_user(&up->reserved2, &kp->reserved2) ||
+	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
 	    assign_in_user(&up->reserved, &kp->reserved) ||
 	    get_user(length, &kp->length) ||
 	    put_user(length, &up->length))
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8842a5ca636..1f18dc68ecab 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -934,7 +934,10 @@ struct v4l2_buffer {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	union {
+		__s32		fence_fd;
+		__u32		reserved2;
+	};
 	__u32			reserved;
 };
 
@@ -971,6 +974,9 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+/* Explicit synchronization */
+#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
+#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.16.3

Re: [PATCH v10 11/16] vb2: add explicit fence user API

[Hans Verkuil]

On 21/05/18 18:59, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Turn the reserved2 field into fence_fd that we will use to send
> an in-fence to the kernel or return an out-fence from the kernel to
> userspace.
> 
> Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> when sending an in-fence to the kernel to be waited on, and
> V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> 
> v8: return -1 if new flags are set.
> v7: minor fixes on the Documentation (Hans Verkuil)
> 
> v6: big improvement on doc (Hans Verkuil)
> 
> v5: - keep using reserved2 field for cpia2
>     - set fence_fd to 0 for now, for compat with userspace(Mauro)
> 
> v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> 
> v3: make the out_fence refer to the current buffer (Hans Verkuil)
> 
> v2: add documentation
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/buffer.rst         | 48 ++++++++++++++++++++++---
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 +++-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
>  include/uapi/linux/videodev2.h                  |  8 ++++-
>  4 files changed, 58 insertions(+), 8 deletions(-)
> 
> diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> index e2c85ddc990b..971b7453040c 100644
> --- a/Documentation/media/uapi/v4l/buffer.rst
> +++ b/Documentation/media/uapi/v4l/buffer.rst
> @@ -300,11 +300,23 @@ struct v4l2_buffer
>  	multi-planar API the application sets this to the number of
>  	elements in the ``planes`` array. The driver will fill in the
>  	actual number of valid elements in that array.
> -    * - __u32
> -      - ``reserved2``
> +    * - __s32
> +      - ``fence_fd``
>        -
> -      - A place holder for future extensions. Drivers and applications
> -	must set this to 0.
> +      - Used to communicate a fence file descriptor from userspace to kernel
> +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> +	be used and this field set to the fence file descriptor of the in-fence.
> +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> +
> +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> +	must be set, the kernel will return the out-fence file descriptor in
> +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> +        an error.
> +
> +	For all other ioctls V4L2 sets this field to -1 if
> +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> +	otherwise this field is set to 0 for backward compatibility.
>      * - __u32
>        - ``reserved``
>        -
> @@ -648,6 +660,34 @@ Buffer Flags
>        - Start Of Exposure. The buffer timestamp has been taken when the
>  	exposure of the frame has begun. This is only valid for the
>  	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_IN_FENCE``
> +      - 0x00200000
> +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> +	buffer won't be queued to the driver until the fence signals. The order
> +	in which buffers are queued is guaranteed to be preserved, so any
> +	buffers queued after this buffer will also be blocked until this fence
> +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> +	other ioctls the driver just reports the value of the flag.
> +
> +        If the fence signals the flag is cleared and not reported anymore.
> +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> +      - 0x00400000
> +      - Request for a fence to be attached to the buffer. The driver will fill
> +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> +	``VIDIOC_QBUF``. This flag is only an input, and is not set by the kernel.
> +
> +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> +	error.
> +
> +        Note that it is valid to set both ``V4L2_BUF_FLAG_IN_FENCE`` and
> +        `V4L2_BUF_FLAG_OUT_FENCE`` flags. In such case, the ``fence_fd``
> +        field is used to both set the in-fence and return the out-fence.
>  
>  
>  
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 64503615d00b..8312f61adfa6 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -203,9 +203,13 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->reserved2 = 0;
>  	b->reserved = 0;
>  
> +	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
> +		b->fence_fd = -1;
> +	else
> +		b->fence_fd = 0;

I don't see why we can't just always set fence_fd to -1.

There is no need for backwards compatibility that I can see.

Regards,

	Hans

> +
>  	if (q->is_multiplanar) {
>  		/*
>  		 * Fill in plane-related data if userspace provided an array
> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> index 4312935f1dfc..93c752459aec 100644
> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> @@ -388,7 +388,7 @@ struct v4l2_buffer32 {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	__s32			fence_fd;
>  	__u32			reserved;
>  };
>  
> @@ -606,7 +606,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
>  	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
>  	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
>  	    assign_in_user(&up->sequence, &kp->sequence) ||
> -	    assign_in_user(&up->reserved2, &kp->reserved2) ||
> +	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
>  	    assign_in_user(&up->reserved, &kp->reserved) ||
>  	    get_user(length, &kp->length) ||
>  	    put_user(length, &up->length))
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index a8842a5ca636..1f18dc68ecab 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -934,7 +934,10 @@ struct v4l2_buffer {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	union {
> +		__s32		fence_fd;
> +		__u32		reserved2;
> +	};
>  	__u32			reserved;
>  };
>  
> @@ -971,6 +974,9 @@ struct v4l2_buffer {
>  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
>  /* mem2mem encoder/decoder */
>  #define V4L2_BUF_FLAG_LAST			0x00100000
> +/* Explicit synchronization */
> +#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
> +#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
>  
>  /**
>   * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
> 

Re: [PATCH v10 11/16] vb2: add explicit fence user API

[Ezequiel Garcia]

On Tue, 2018-05-22 at 14:05 +0200, Hans Verkuil wrote:
> On 21/05/18 18:59, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Turn the reserved2 field into fence_fd that we will use to send
> > an in-fence to the kernel or return an out-fence from the kernel to
> > userspace.
> > 
> > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > when sending an in-fence to the kernel to be waited on, and
> > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > 
> > v8: return -1 if new flags are set.
> > v7: minor fixes on the Documentation (Hans Verkuil)
> > 
> > v6: big improvement on doc (Hans Verkuil)
> > 
> > v5: - keep using reserved2 field for cpia2
> >     - set fence_fd to 0 for now, for compat with userspace(Mauro)
> > 
> > v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> > 
> > v3: make the out_fence refer to the current buffer (Hans Verkuil)
> > 
> > v2: add documentation
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > ---
> >  Documentation/media/uapi/v4l/buffer.rst         | 48 ++++++++++++++++++++++---
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 +++-
> >  drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
> >  include/uapi/linux/videodev2.h                  |  8 ++++-
> >  4 files changed, 58 insertions(+), 8 deletions(-)
> > 
> > diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> > index e2c85ddc990b..971b7453040c 100644
> > --- a/Documentation/media/uapi/v4l/buffer.rst
> > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > @@ -300,11 +300,23 @@ struct v4l2_buffer
> >  	multi-planar API the application sets this to the number of
> >  	elements in the ``planes`` array. The driver will fill in the
> >  	actual number of valid elements in that array.
> > -    * - __u32
> > -      - ``reserved2``
> > +    * - __s32
> > +      - ``fence_fd``
> >        -
> > -      - A place holder for future extensions. Drivers and applications
> > -	must set this to 0.
> > +      - Used to communicate a fence file descriptor from userspace to kernel
> > +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> > +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> > +	be used and this field set to the fence file descriptor of the in-fence.
> > +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> > +
> > +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> > +	must be set, the kernel will return the out-fence file descriptor in
> > +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> > +        an error.
> > +
> > +	For all other ioctls V4L2 sets this field to -1 if
> > +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> > +	otherwise this field is set to 0 for backward compatibility.
> >      * - __u32
> >        - ``reserved``
> >        -
> > @@ -648,6 +660,34 @@ Buffer Flags
> >        - Start Of Exposure. The buffer timestamp has been taken when the
> >  	exposure of the frame has begun. This is only valid for the
> >  	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > +
> > +      - ``V4L2_BUF_FLAG_IN_FENCE``
> > +      - 0x00200000
> > +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> > +	buffer won't be queued to the driver until the fence signals. The order
> > +	in which buffers are queued is guaranteed to be preserved, so any
> > +	buffers queued after this buffer will also be blocked until this fence
> > +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> > +	other ioctls the driver just reports the value of the flag.
> > +
> > +        If the fence signals the flag is cleared and not reported anymore.
> > +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> > +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > +
> > +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> > +      - 0x00400000
> > +      - Request for a fence to be attached to the buffer. The driver will fill
> > +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> > +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> > +	``VIDIOC_QBUF``. This flag is only an input, and is not set by the kernel.
> > +
> > +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> > +	error.
> > +
> > +        Note that it is valid to set both ``V4L2_BUF_FLAG_IN_FENCE`` and
> > +        `V4L2_BUF_FLAG_OUT_FENCE`` flags. In such case, the ``fence_fd``
> > +        field is used to both set the in-fence and return the out-fence.
> >  
> >  
> >  
> > diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > index 64503615d00b..8312f61adfa6 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > @@ -203,9 +203,13 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> >  	b->timestamp = ns_to_timeval(vb->timestamp);
> >  	b->timecode = vbuf->timecode;
> >  	b->sequence = vbuf->sequence;
> > -	b->reserved2 = 0;
> >  	b->reserved = 0;
> >  
> > +	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
> > +		b->fence_fd = -1;
> > +	else
> > +		b->fence_fd = 0;
> 
> I don't see why we can't just always set fence_fd to -1.
> 
> There is no need for backwards compatibility that I can see.
> 
> 

I thought we were trying to not break any users expecting reserved2
to be zeroed.

Unless that's silly?

[PATCH v10 12/16] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queued to the driver
out of the order they were queued from userspace. That means that even if
its fence signals it must wait for all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v13: - cleanup implementation.
     - remove wrong Kconfig changes.
     - print noisy warning on unexpected enqueue conditioin
     - schedule a vb2_start_streaming work from the fence callback

v12: fixed dvb_vb2.c usage of vb2_core_qbuf.

v11: - minor doc/comments fixes (Hans Verkuil)
     - reviewed the in-fence path at __fill_v4l2_buffer()

v10: - rename fence to in_fence in many places
     - handle fences signalling with error better (Hans Verkuil)

v9: - improve comments and docs (Hans Verkuil)
    - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
    - move in-fences code that was in the out-fences patch here (Alex)

v8: - improve comments about fences with errors

v7: - get rid of the fence array stuff for ordering and just use
      get_num_buffers_ready() (Hans)
    - fix issue of queuing the buffer twice (Hans)
    - avoid the dma_fence_wait() in core_qbuf() (Alex)
    - merge preparation commit in

v6: - With fences always keep the order userspace queues the buffers.
    - Protect in_fence manipulation with a lock (Brian Starkey)
    - check if fences have the same context before adding a fence array
    - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
    - Clean up fence if __set_in_fence() fails (Brian Starkey)
    - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
      needed (Brian Starkey)
    - keep backward compat on the reserved2 field (Brian Starkey)
    - protect fence callback removal with lock (Brian Starkey)

v4: - Add a comment about dma_fence_add_callback() not returning a
      error (Hans)
    - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
    - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
    - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
    - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
    - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
      vb2_start_streaming() (Hans)
    - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
    - Queue buffers to the driver as soon as they are ready (Hans)
    - call fill_user_buffer() after queuing the buffer (Hans)
    - add err: label to clean up fence
    - add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
    - remove ternary if at vb2_qbuf() return (Mauro)
    - do not change if conditions behaviour (Mauro)

v2: - fix vb2_queue_or_prepare_buf() ret check
    - remove check for VB2_MEMORY_DMABUF only (Javier)
    - check num of ready buffers to start streaming
    - when queueing, start from the first ready buffer
    - handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/common/videobuf2/Kconfig          |   1 +
 drivers/media/common/videobuf2/videobuf2-core.c | 224 ++++++++++++++++++++----
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  37 +++-
 drivers/media/dvb-core/dvb_vb2.c                |   2 +-
 include/media/videobuf2-core.h                  |  19 +-
 5 files changed, 242 insertions(+), 41 deletions(-)

diff --git a/drivers/media/common/videobuf2/Kconfig b/drivers/media/common/videobuf2/Kconfig
index 17c32ea58395..27ad9e8a268b 100644
--- a/drivers/media/common/videobuf2/Kconfig
+++ b/drivers/media/common/videobuf2/Kconfig
@@ -1,6 +1,7 @@
 # Used by drivers that need Videobuf2 modules
 config VIDEOBUF2_CORE
 	select DMA_SHARED_BUFFER
+	select SYNC_FILE
 	tristate
 
 config VIDEOBUF2_V4L2
diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index a9a0a9d1decb..86b5ebe25263 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -27,6 +27,7 @@
 #include <linux/kthread.h>
 
 #include <media/videobuf2-core.h>
+#include <media/v4l2-ioctl.h>
 #include <media/v4l2-mc.h>
 
 #include <trace/events/vb2.h>
@@ -189,6 +190,7 @@ module_param(debug, int, 0644);
 
 static void __vb2_queue_cancel(struct vb2_queue *q);
 static void __enqueue_in_driver(struct vb2_buffer *vb);
+static void __qbuf_work(struct work_struct *work);
 
 static void __vb2_buffer_free(struct kref *kref)
 {
@@ -373,6 +375,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 		vb->index = q->num_buffers + buffer;
 		vb->type = q->type;
 		vb->memory = memory;
+		INIT_WORK(&vb->qbuf_work, __qbuf_work);
 		for (plane = 0; plane < num_planes; ++plane) {
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
@@ -932,21 +935,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
 }
 EXPORT_SYMBOL_GPL(vb2_plane_cookie);
 
-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned long flags;
 	unsigned int plane;
 
-	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
-		return;
-
-	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
-		    state != VB2_BUF_STATE_ERROR &&
-		    state != VB2_BUF_STATE_QUEUED &&
-		    state != VB2_BUF_STATE_REQUEUEING))
-		state = VB2_BUF_STATE_ERROR;
-
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Although this is not a callback, it still does have to balance
@@ -962,6 +956,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
 
 	spin_lock_irqsave(&q->done_lock, flags);
+	if (vb->state == VB2_BUF_STATE_ACTIVE)
+		atomic_dec(&q->owned_by_drv_count);
+
 	if (state == VB2_BUF_STATE_QUEUED ||
 	    state == VB2_BUF_STATE_REQUEUEING) {
 		vb->state = VB2_BUF_STATE_QUEUED;
@@ -970,7 +967,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		list_add_tail(&vb->done_entry, &q->done_list);
 		vb->state = state;
 	}
-	atomic_dec(&q->owned_by_drv_count);
+
 	spin_unlock_irqrestore(&q->done_lock, flags);
 
 	trace_vb2_buf_done(q, vb);
@@ -987,6 +984,47 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		wake_up(&q->done_wq);
 		break;
 	}
+
+}
+
+void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+{
+	struct vb2_buffer *next;
+	struct vb2_queue *q;
+
+	dprintk(4, "called done on buffer %d state: %d -> %d\n",
+			vb->index, vb->state, state);
+
+	if (vb->state == VB2_BUF_STATE_ERROR)
+		return;
+	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
+		return;
+
+	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
+		    state != VB2_BUF_STATE_ERROR &&
+		    state != VB2_BUF_STATE_QUEUED &&
+		    state != VB2_BUF_STATE_REQUEUEING))
+		state = VB2_BUF_STATE_ERROR;
+
+	vb2_process_buffer_done(vb, state);
+
+	/*
+	 * Check if there is any buffer with an in-fence error in the next
+	 * position of the queue. Buffers whose in-fence signaled with error
+	 * are not queued to the driver and kept on the queue until the buffer
+	 * before them is done.
+	 * So here we process any existing buffers with in-fence errors and
+	 * wake up userspace.
+	 */
+	q = vb->vb2_queue;
+	next = list_next_entry(vb, queued_entry);
+	list_for_each_entry_from(next, &q->queued_list, queued_entry) {
+		if (!next->in_fence || !next->in_fence->error)
+			break;
+		if (next->in_fence && next->in_fence->error &&
+		    vb->state != VB2_BUF_STATE_ERROR)
+			vb2_process_buffer_done(next, VB2_BUF_STATE_ERROR);
+	}
 }
 EXPORT_SYMBOL_GPL(vb2_buffer_done);
 
@@ -1271,6 +1309,8 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 
+	if (WARN_ON(vb->state == VB2_BUF_STATE_ACTIVE))
+		return;
 	vb->state = VB2_BUF_STATE_ACTIVE;
 	atomic_inc(&q->owned_by_drv_count);
 
@@ -1322,6 +1362,25 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
 	return 0;
 }
 
+static int __get_num_ready_buffers(struct vb2_queue *q)
+{
+	struct vb2_buffer *vb;
+	int ready_count = 0;
+
+	/*
+	 * Count num of buffers ready in front of the queued_list.
+	 * We want to stop counting when we find a buffer with an
+	 * unsignaled fence.
+	 */
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
+			break;
+		ready_count++;
+	}
+
+	return ready_count;
+}
+
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 {
 	struct vb2_buffer *vb;
@@ -1338,7 +1397,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 	if (ret)
 		return ret;
 
-	/* Fill buffer information for the userspace */
+	/* Fill buffer information for userspace */
 	call_void_bufop(q, fill_user_buffer, vb, pb);
 
 	dprintk(2, "prepare of buffer %d succeeded\n", vb->index);
@@ -1364,11 +1423,16 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	int ret;
 
 	/*
-	 * If any buffers were queued before streamon,
-	 * we can now pass them to driver for processing.
+	 * Activate all buffers currently queued,
+	 * until we get an unsignaled fence.
 	 */
-	list_for_each_entry(vb, &q->queued_list, queued_entry)
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (vb->state != VB2_BUF_STATE_QUEUED)
+			continue;
+		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
+			break;
 		__enqueue_in_driver(vb);
+	}
 
 	/* Tell the driver to start streaming */
 	q->start_streaming_called = 1;
@@ -1410,7 +1474,41 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+static void __qbuf_work(struct work_struct *work)
+{
+	struct vb2_buffer *vb;
+	struct vb2_queue *q;
+
+	vb = container_of(work, struct vb2_buffer, qbuf_work);
+	q = vb->vb2_queue;
+
+	if (q->lock)
+		mutex_lock(q->lock);
+	/* By the time we run, the buffer state could have changed,
+	 * so we need to check it's in the queued state.
+	 * This is the only valid state.
+	 */
+	if (q->start_streaming_called && vb->state == VB2_BUF_STATE_QUEUED)
+		__enqueue_in_driver(vb);
+
+	if (q->streaming && !q->start_streaming_called &&
+	    __get_num_ready_buffers(q) >= q->min_buffers_needed)
+		vb2_start_streaming(q);
+
+	if (q->lock)
+		mutex_unlock(q->lock);
+	__vb2_buffer_put(vb);
+}
+
+static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
+
+	schedule_work(&vb->qbuf_work);
+}
+
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence)
 {
 	struct vb2_buffer *vb;
 	int ret;
@@ -1421,16 +1519,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	case VB2_BUF_STATE_DEQUEUED:
 		ret = __buf_prepare(vb, pb);
 		if (ret)
-			return ret;
+			goto err_release;
 		break;
 	case VB2_BUF_STATE_PREPARED:
 		break;
 	case VB2_BUF_STATE_PREPARING:
 		dprintk(1, "buffer still being prepared\n");
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err_release;
 	default:
 		dprintk(1, "invalid buffer state %d\n", vb->state);
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err_release;
 	}
 
 	/*
@@ -1448,15 +1548,44 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	trace_vb2_qbuf(q, vb);
 
 	/*
-	 * If already streaming, give the buffer to driver for processing.
-	 * If not, the buffer will be given to driver on next streamon.
+	 * If the fence didn't signal yet, we setup a callback to queue
+	 * the buffer once the fence signals. If the fence already signaled,
+	 * then we lose the reference and queue the buffer.
 	 */
-	if (q->start_streaming_called)
-		__enqueue_in_driver(vb);
+	if (in_fence) {
+		WARN_ON(vb->in_fence);
+		vb->in_fence = in_fence;
+		__vb2_buffer_get(vb);
+		ret = dma_fence_add_callback(in_fence, &vb->fence_cb,
+					     vb2_qbuf_fence_cb);
+		/* is the fence signaled? */
+		if (ret == -ENOENT) {
+			if (in_fence->error)
+				/* error-signaled fence, reject this buffer */
+				goto err_release;
+			dma_fence_put(in_fence);
+			vb->in_fence = NULL;
+			__vb2_buffer_put(vb);
+		} else if (ret) {
+			goto err_release;
+		}
+	}
 
-	/* Fill buffer information for the userspace */
-	if (pb)
-		call_void_bufop(q, fill_user_buffer, vb, pb);
+	/*
+	 * If already streaming and there is no fence to wait on
+	 * give the buffer to driver for processing.
+	 */
+	if (q->start_streaming_called) {
+		struct vb2_buffer *b;
+
+		list_for_each_entry(b, &q->queued_list, queued_entry) {
+			if (b->state != VB2_BUF_STATE_QUEUED)
+				continue;
+			if (b->in_fence && dma_fence_get_status(b->in_fence) == 0)
+				break;
+			__enqueue_in_driver(b);
+		}
+	}
 
 	/*
 	 * If streamon has been called, and we haven't yet called
@@ -1465,14 +1594,32 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	 * then we can finally call start_streaming().
 	 */
 	if (q->streaming && !q->start_streaming_called &&
-	    q->queued_count >= q->min_buffers_needed) {
+	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = vb2_start_streaming(q);
 		if (ret)
-			return ret;
+			goto err_release;
 	}
 
+	/* Fill buffer information for userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
 	return 0;
+
+err_release:
+	/* Fill buffer information for userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
+	if (in_fence) {
+		dma_fence_put(in_fence);
+		vb->in_fence = NULL;
+		__vb2_buffer_put(vb);
+	}
+
+	return ret;
+
 }
 EXPORT_SYMBOL_GPL(vb2_core_qbuf);
 
@@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
 		return;
 
 	vb->state = VB2_BUF_STATE_DEQUEUED;
-
+	if (vb->in_fence) {
+		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
+			__vb2_buffer_put(vb);
+		dma_fence_put(vb->in_fence);
+		vb->in_fence = NULL;
+	}
 	/* unmap DMABUF buffer */
 	if (q->memory == VB2_MEMORY_DMABUF)
 		for (i = 0; i < vb->num_planes; ++i) {
@@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
 	if (pindex)
 		*pindex = vb->index;
 
-	/* Fill buffer information for the userspace */
+	/* Fill buffer information for userspace */
 	if (pb)
 		call_void_bufop(q, fill_user_buffer, vb, pb);
 
@@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
 		for (i = 0; i < q->num_buffers; ++i)
 			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
-				pr_warn("driver bug: stop_streaming operation is leaving buf %p in active state\n",
-					q->bufs[i]);
+				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p in active state\n",
+					q->bufs[i]->index, q->bufs[i]);
 				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
 			}
 		/* Must be zero now */
@@ -1783,7 +1935,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
 	 * Tell driver to start streaming provided sufficient buffers
 	 * are available.
 	 */
-	if (q->queued_count >= q->min_buffers_needed) {
+	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = v4l_vb2q_enable_media_source(q);
 		if (ret)
 			return ret;
@@ -2305,7 +2457,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2484,7 +2636,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2587,7 +2739,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 8312f61adfa6..ac14cc8ab1c5 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -23,6 +23,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/v4l2-dev.h>
 #include <media/v4l2-fh.h>
@@ -178,6 +179,22 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 		return -EINVAL;
 	}
 
+	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
+	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
+		return -EINVAL;
+	}
+
+	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
+		return -EINVAL;
+	}
+
+	if ((b->flags & V4L2_BUF_FLAG_IN_FENCE) && !q->lock) {
+		pr_warn("%s: IN_FENCE flag set, but no queue lock\n", opname);
+		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
+	}
+
 	return __verify_planes_array(q->bufs[b->index], b);
 }
 
@@ -210,6 +227,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	else
 		b->fence_fd = 0;
 
+	if (vb->in_fence)
+		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
+	else
+		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
+
 	if (q->is_multiplanar) {
 		/*
 		 * Fill in plane-related data if userspace provided an array
@@ -566,6 +588,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	struct dma_fence *in_fence = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -574,7 +597,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	}
 
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	if (ret)
+		return ret;
+
+	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
+		in_fence = sync_file_get_fence(b->fence_fd);
+		if (!in_fence) {
+			dprintk(1, "failed to get in-fence from fd %d\n",
+				b->fence_fd);
+			return -EINVAL;
+		}
+	}
+
+	return vb2_core_qbuf(q, b->index, b, in_fence);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index b811adf88afa..7da53f10db1a 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
 {
 	int ret;
 
-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
 	if (ret) {
 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
 			b->index, ret);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 71538ae2c255..ddd555f59dbf 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -14,6 +14,7 @@
 
 #include <linux/bitops.h>
 #include <linux/dma-buf.h>
+#include <linux/dma-fence.h>
 #include <linux/kref.h>
 #include <linux/mm_types.h>
 #include <linux/mutex.h>
@@ -257,6 +258,10 @@ struct vb2_buffer {
 	 * done_entry:		entry on the list that stores all buffers ready
 	 *			to be dequeued to userspace
 	 * vb2_plane:		per-plane information; do not change
+	 * in_fence:		fence received from vb2 client to wait on before
+	 *			using the buffer (queueing to the driver)
+	 * fence_cb:		fence callback information
+	 * qbuf_work:		work for deferred qbuf operation
 	 */
 	struct kref		refcount;
 	enum vb2_buffer_state	state;
@@ -264,6 +269,11 @@ struct vb2_buffer {
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 	struct list_head	queued_entry;
 	struct list_head	done_entry;
+
+	struct dma_fence	*in_fence;
+	struct dma_fence_cb	fence_cb;
+	struct work_struct	qbuf_work;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -488,8 +498,9 @@ struct vb2_buf_ops {
  * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
  *		driver can set this to a mutex to let the v4l2 core serialize
  *		the queuing ioctls. If the driver wants to handle locking
- *		itself, then this should be set to NULL. This lock is not used
- *		by the videobuf2 core API.
+ *		itself, then this should be set to NULL. This lock is required
+ *		by the videobuf2 core only on the fence-deferred start streaming
+ *		path.
  * @owner:	The filehandle that 'owns' the buffers, i.e. the filehandle
  *		that called reqbufs, create_buffers or started fileio.
  *		This field is not used by the videobuf2 core API, but it allows
@@ -789,6 +800,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @index:	id number of the buffer
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
+ * @in_fence:	in-fence to wait on before queueing the buffer
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
@@ -803,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * Return: returns zero on success; an error code otherwise.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.16.3

Re: [PATCH v10 12/16] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 21/05/18 18:59, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Receive in-fence from userspace and add support for waiting on them
> before queueing the buffer to the driver. Buffers can't be queued to the
> driver before its fences signal. And a buffer can't be queued to the driver
> out of the order they were queued from userspace. That means that even if
> its fence signals it must wait for all other buffers, ahead of it in the queue,
> to signal first.
> 
> If the fence for some buffer fails we do not queue it to the driver,
> instead we mark it as error and wait until the previous buffer is done
> to notify userspace of the error. We wait here to deliver the buffers back
> to userspace in order.
> 
> v13: - cleanup implementation.
>      - remove wrong Kconfig changes.
>      - print noisy warning on unexpected enqueue conditioin
>      - schedule a vb2_start_streaming work from the fence callback
> 
> v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> 
> v11: - minor doc/comments fixes (Hans Verkuil)
>      - reviewed the in-fence path at __fill_v4l2_buffer()
> 
> v10: - rename fence to in_fence in many places
>      - handle fences signalling with error better (Hans Verkuil)
> 
> v9: - improve comments and docs (Hans Verkuil)
>     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>     - move in-fences code that was in the out-fences patch here (Alex)
> 
> v8: - improve comments about fences with errors
> 
> v7: - get rid of the fence array stuff for ordering and just use
>       get_num_buffers_ready() (Hans)
>     - fix issue of queuing the buffer twice (Hans)
>     - avoid the dma_fence_wait() in core_qbuf() (Alex)
>     - merge preparation commit in
> 
> v6: - With fences always keep the order userspace queues the buffers.
>     - Protect in_fence manipulation with a lock (Brian Starkey)
>     - check if fences have the same context before adding a fence array
>     - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>     - Clean up fence if __set_in_fence() fails (Brian Starkey)
>     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> 
> v5: - use fence_array to keep buffers ordered in vb2 core when
>       needed (Brian Starkey)
>     - keep backward compat on the reserved2 field (Brian Starkey)
>     - protect fence callback removal with lock (Brian Starkey)
> 
> v4: - Add a comment about dma_fence_add_callback() not returning a
>       error (Hans)
>     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>     - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>       vb2_start_streaming() (Hans)
>     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>     - Queue buffers to the driver as soon as they are ready (Hans)
>     - call fill_user_buffer() after queuing the buffer (Hans)
>     - add err: label to clean up fence
>     - add dma_fence_wait() before calling vb2_start_streaming()
> 
> v3: - document fence parameter
>     - remove ternary if at vb2_qbuf() return (Mauro)
>     - do not change if conditions behaviour (Mauro)
> 
> v2: - fix vb2_queue_or_prepare_buf() ret check
>     - remove check for VB2_MEMORY_DMABUF only (Javier)
>     - check num of ready buffers to start streaming
>     - when queueing, start from the first ready buffer
>     - handle queue cancel
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/Kconfig          |   1 +
>  drivers/media/common/videobuf2/videobuf2-core.c | 224 ++++++++++++++++++++----
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  37 +++-
>  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
>  include/media/videobuf2-core.h                  |  19 +-
>  5 files changed, 242 insertions(+), 41 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/Kconfig b/drivers/media/common/videobuf2/Kconfig
> index 17c32ea58395..27ad9e8a268b 100644
> --- a/drivers/media/common/videobuf2/Kconfig
> +++ b/drivers/media/common/videobuf2/Kconfig
> @@ -1,6 +1,7 @@
>  # Used by drivers that need Videobuf2 modules
>  config VIDEOBUF2_CORE
>  	select DMA_SHARED_BUFFER
> +	select SYNC_FILE
>  	tristate
>  
>  config VIDEOBUF2_V4L2
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index a9a0a9d1decb..86b5ebe25263 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -27,6 +27,7 @@
>  #include <linux/kthread.h>
>  
>  #include <media/videobuf2-core.h>
> +#include <media/v4l2-ioctl.h>
>  #include <media/v4l2-mc.h>
>  
>  #include <trace/events/vb2.h>
> @@ -189,6 +190,7 @@ module_param(debug, int, 0644);
>  
>  static void __vb2_queue_cancel(struct vb2_queue *q);
>  static void __enqueue_in_driver(struct vb2_buffer *vb);
> +static void __qbuf_work(struct work_struct *work);
>  
>  static void __vb2_buffer_free(struct kref *kref)
>  {
> @@ -373,6 +375,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  		vb->index = q->num_buffers + buffer;
>  		vb->type = q->type;
>  		vb->memory = memory;
> +		INIT_WORK(&vb->qbuf_work, __qbuf_work);
>  		for (plane = 0; plane < num_planes; ++plane) {
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
> @@ -932,21 +935,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>  }
>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>  
> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  	unsigned long flags;
>  	unsigned int plane;
>  
> -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> -		return;
> -
> -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> -		    state != VB2_BUF_STATE_ERROR &&
> -		    state != VB2_BUF_STATE_QUEUED &&
> -		    state != VB2_BUF_STATE_REQUEUEING))
> -		state = VB2_BUF_STATE_ERROR;
> -
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Although this is not a callback, it still does have to balance
> @@ -962,6 +956,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>  
>  	spin_lock_irqsave(&q->done_lock, flags);
> +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> +		atomic_dec(&q->owned_by_drv_count);
> +
>  	if (state == VB2_BUF_STATE_QUEUED ||
>  	    state == VB2_BUF_STATE_REQUEUEING) {
>  		vb->state = VB2_BUF_STATE_QUEUED;
> @@ -970,7 +967,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		list_add_tail(&vb->done_entry, &q->done_list);
>  		vb->state = state;
>  	}
> -	atomic_dec(&q->owned_by_drv_count);
> +
>  	spin_unlock_irqrestore(&q->done_lock, flags);
>  
>  	trace_vb2_buf_done(q, vb);
> @@ -987,6 +984,47 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		wake_up(&q->done_wq);
>  		break;
>  	}
> +
> +}
> +
> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +{
> +	struct vb2_buffer *next;
> +	struct vb2_queue *q;
> +
> +	dprintk(4, "called done on buffer %d state: %d -> %d\n",
> +			vb->index, vb->state, state);
> +
> +	if (vb->state == VB2_BUF_STATE_ERROR)
> +		return;
> +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> +		return;
> +
> +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> +		    state != VB2_BUF_STATE_ERROR &&
> +		    state != VB2_BUF_STATE_QUEUED &&
> +		    state != VB2_BUF_STATE_REQUEUEING))
> +		state = VB2_BUF_STATE_ERROR;
> +
> +	vb2_process_buffer_done(vb, state);
> +
> +	/*
> +	 * Check if there is any buffer with an in-fence error in the next
> +	 * position of the queue. Buffers whose in-fence signaled with error
> +	 * are not queued to the driver and kept on the queue until the buffer
> +	 * before them is done.
> +	 * So here we process any existing buffers with in-fence errors and
> +	 * wake up userspace.
> +	 */
> +	q = vb->vb2_queue;
> +	next = list_next_entry(vb, queued_entry);
> +	list_for_each_entry_from(next, &q->queued_list, queued_entry) {
> +		if (!next->in_fence || !next->in_fence->error)
> +			break;
> +		if (next->in_fence && next->in_fence->error &&
> +		    vb->state != VB2_BUF_STATE_ERROR)
> +			vb2_process_buffer_done(next, VB2_BUF_STATE_ERROR);
> +	}
>  }
>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>  
> @@ -1271,6 +1309,8 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  
> +	if (WARN_ON(vb->state == VB2_BUF_STATE_ACTIVE))
> +		return;
>  	vb->state = VB2_BUF_STATE_ACTIVE;
>  	atomic_inc(&q->owned_by_drv_count);
>  
> @@ -1322,6 +1362,25 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>  	return 0;
>  }
>  
> +static int __get_num_ready_buffers(struct vb2_queue *q)
> +{
> +	struct vb2_buffer *vb;
> +	int ready_count = 0;
> +
> +	/*
> +	 * Count num of buffers ready in front of the queued_list.
> +	 * We want to stop counting when we find a buffer with an
> +	 * unsignaled fence.
> +	 */
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> +			break;
> +		ready_count++;
> +	}
> +
> +	return ready_count;
> +}
> +
>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  {
>  	struct vb2_buffer *vb;
> @@ -1338,7 +1397,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  	if (ret)
>  		return ret;
>  
> -	/* Fill buffer information for the userspace */
> +	/* Fill buffer information for userspace */
>  	call_void_bufop(q, fill_user_buffer, vb, pb);
>  
>  	dprintk(2, "prepare of buffer %d succeeded\n", vb->index);
> @@ -1364,11 +1423,16 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	int ret;
>  
>  	/*
> -	 * If any buffers were queued before streamon,
> -	 * we can now pass them to driver for processing.
> +	 * Activate all buffers currently queued,
> +	 * until we get an unsignaled fence.
>  	 */
> -	list_for_each_entry(vb, &q->queued_list, queued_entry)
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->state != VB2_BUF_STATE_QUEUED)

I prefer: if (vb->state == VB2_BUF_STATE_ERROR)

It also needs a comment.

> +			continue;
> +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> +			break;
>  		__enqueue_in_driver(vb);
> +	}
>  
>  	/* Tell the driver to start streaming */
>  	q->start_streaming_called = 1;
> @@ -1410,7 +1474,41 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	return ret;
>  }
>  
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> +static void __qbuf_work(struct work_struct *work)
> +{
> +	struct vb2_buffer *vb;
> +	struct vb2_queue *q;
> +
> +	vb = container_of(work, struct vb2_buffer, qbuf_work);
> +	q = vb->vb2_queue;
> +
> +	if (q->lock)
> +		mutex_lock(q->lock);
> +	/* By the time we run, the buffer state could have changed,
> +	 * so we need to check it's in the queued state.
> +	 * This is the only valid state.
> +	 */
> +	if (q->start_streaming_called && vb->state == VB2_BUF_STATE_QUEUED)
> +		__enqueue_in_driver(vb);
> +
> +	if (q->streaming && !q->start_streaming_called &&
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed)
> +		vb2_start_streaming(q);

What happens if vb2_start_streaming fails?

> +
> +	if (q->lock)
> +		mutex_unlock(q->lock);
> +	__vb2_buffer_put(vb);
> +}
> +
> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> +{
> +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> +
> +	schedule_work(&vb->qbuf_work);
> +}
> +
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence)
>  {
>  	struct vb2_buffer *vb;
>  	int ret;
> @@ -1421,16 +1519,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	case VB2_BUF_STATE_DEQUEUED:
>  		ret = __buf_prepare(vb, pb);
>  		if (ret)
> -			return ret;
> +			goto err_release;
>  		break;
>  	case VB2_BUF_STATE_PREPARED:
>  		break;
>  	case VB2_BUF_STATE_PREPARING:
>  		dprintk(1, "buffer still being prepared\n");
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err_release;
>  	default:
>  		dprintk(1, "invalid buffer state %d\n", vb->state);
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err_release;
>  	}
>  
>  	/*
> @@ -1448,15 +1548,44 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	trace_vb2_qbuf(q, vb);
>  
>  	/*
> -	 * If already streaming, give the buffer to driver for processing.
> -	 * If not, the buffer will be given to driver on next streamon.
> +	 * If the fence didn't signal yet, we setup a callback to queue
> +	 * the buffer once the fence signals. If the fence already signaled,
> +	 * then we lose the reference and queue the buffer.
>  	 */
> -	if (q->start_streaming_called)
> -		__enqueue_in_driver(vb);
> +	if (in_fence) {
> +		WARN_ON(vb->in_fence);
> +		vb->in_fence = in_fence;
> +		__vb2_buffer_get(vb);
> +		ret = dma_fence_add_callback(in_fence, &vb->fence_cb,
> +					     vb2_qbuf_fence_cb);
> +		/* is the fence signaled? */
> +		if (ret == -ENOENT) {
> +			if (in_fence->error)
> +				/* error-signaled fence, reject this buffer */
> +				goto err_release;
> +			dma_fence_put(in_fence);
> +			vb->in_fence = NULL;
> +			__vb2_buffer_put(vb);
> +		} else if (ret) {
> +			goto err_release;
> +		}
> +	}
>  
> -	/* Fill buffer information for the userspace */
> -	if (pb)
> -		call_void_bufop(q, fill_user_buffer, vb, pb);
> +	/*
> +	 * If already streaming and there is no fence to wait on
> +	 * give the buffer to driver for processing.
> +	 */
> +	if (q->start_streaming_called) {
> +		struct vb2_buffer *b;
> +
> +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> +			if (b->state != VB2_BUF_STATE_QUEUED)

b->state == VB2_BUF_STATE_ERROR

> +				continue;
> +			if (b->in_fence && dma_fence_get_status(b->in_fence) == 0)
> +				break;
> +			__enqueue_in_driver(b);
> +		}
> +	}
>  
>  	/*
>  	 * If streamon has been called, and we haven't yet called
> @@ -1465,14 +1594,32 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	 * then we can finally call start_streaming().
>  	 */
>  	if (q->streaming && !q->start_streaming_called &&
> -	    q->queued_count >= q->min_buffers_needed) {
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = vb2_start_streaming(q);
>  		if (ret)
> -			return ret;
> +			goto err_release;
>  	}
>  
> +	/* Fill buffer information for userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
>  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>  	return 0;
> +
> +err_release:
> +	/* Fill buffer information for userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
> +	if (in_fence) {
> +		dma_fence_put(in_fence);
> +		vb->in_fence = NULL;
> +		__vb2_buffer_put(vb);
> +	}
> +
> +	return ret;
> +
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>  
> @@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
>  		return;
>  
>  	vb->state = VB2_BUF_STATE_DEQUEUED;
> -
> +	if (vb->in_fence) {
> +		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
> +			__vb2_buffer_put(vb);
> +		dma_fence_put(vb->in_fence);
> +		vb->in_fence = NULL;
> +	}
>  	/* unmap DMABUF buffer */
>  	if (q->memory == VB2_MEMORY_DMABUF)
>  		for (i = 0; i < vb->num_planes; ++i) {
> @@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
>  	if (pindex)
>  		*pindex = vb->index;
>  
> -	/* Fill buffer information for the userspace */
> +	/* Fill buffer information for userspace */
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
>  
> @@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
>  		for (i = 0; i < q->num_buffers; ++i)
>  			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
> -				pr_warn("driver bug: stop_streaming operation is leaving buf %p in active state\n",
> -					q->bufs[i]);
> +				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p in active state\n",
> +					q->bufs[i]->index, q->bufs[i]);
>  				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
>  			}

Shouldn't any pending fences be canceled here?

I feel uncomfortable with the refcounting of buffers, I'd rather that when we
cancel the queue all fences for buffers are removed/canceled/whatever.

Is there any reason for refcounting if we cancel all pending fences here?

Note that besides canceling fences you also need to cancel/flush __qbuf_work.

>  		/* Must be zero now */
> @@ -1783,7 +1935,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>  	 * Tell driver to start streaming provided sufficient buffers
>  	 * are available.
>  	 */
> -	if (q->queued_count >= q->min_buffers_needed) {
> +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = v4l_vb2q_enable_media_source(q);
>  		if (ret)
>  			return ret;
> @@ -2305,7 +2457,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2484,7 +2636,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2587,7 +2739,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 8312f61adfa6..ac14cc8ab1c5 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -23,6 +23,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/v4l2-dev.h>
>  #include <media/v4l2-fh.h>
> @@ -178,6 +179,22 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>  		return -EINVAL;
>  	}
>  
> +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
> +		return -EINVAL;

I'm not sure we need this. If V4L2_BUF_FLAG_IN_FENCE is not set, then fence_fd
can just be ignored (and it is set to -1 anyway when the ioctl returns).

> +	}
> +
> +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
> +		return -EINVAL;
> +	}
> +
> +	if ((b->flags & V4L2_BUF_FLAG_IN_FENCE) && !q->lock) {
> +		pr_warn("%s: IN_FENCE flag set, but no queue lock\n", opname);
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +	}
> +
>  	return __verify_planes_array(q->bufs[b->index], b);
>  }
>  
> @@ -210,6 +227,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	else
>  		b->fence_fd = 0;
>  
> +	if (vb->in_fence)
> +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> +	else
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +
>  	if (q->is_multiplanar) {
>  		/*
>  		 * Fill in plane-related data if userspace provided an array
> @@ -566,6 +588,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	struct dma_fence *in_fence = NULL;
>  	int ret;
>  
>  	if (vb2_fileio_is_active(q)) {
> @@ -574,7 +597,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  	}
>  
>  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> +	if (ret)
> +		return ret;
> +
> +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> +		in_fence = sync_file_get_fence(b->fence_fd);
> +		if (!in_fence) {
> +			dprintk(1, "failed to get in-fence from fd %d\n",
> +				b->fence_fd);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return vb2_core_qbuf(q, b->index, b, in_fence);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
> index b811adf88afa..7da53f10db1a 100644
> --- a/drivers/media/dvb-core/dvb_vb2.c
> +++ b/drivers/media/dvb-core/dvb_vb2.c
> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
>  {
>  	int ret;
>  
> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>  	if (ret) {
>  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>  			b->index, ret);
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 71538ae2c255..ddd555f59dbf 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -14,6 +14,7 @@
>  
>  #include <linux/bitops.h>
>  #include <linux/dma-buf.h>
> +#include <linux/dma-fence.h>
>  #include <linux/kref.h>
>  #include <linux/mm_types.h>
>  #include <linux/mutex.h>
> @@ -257,6 +258,10 @@ struct vb2_buffer {
>  	 * done_entry:		entry on the list that stores all buffers ready
>  	 *			to be dequeued to userspace
>  	 * vb2_plane:		per-plane information; do not change
> +	 * in_fence:		fence received from vb2 client to wait on before
> +	 *			using the buffer (queueing to the driver)
> +	 * fence_cb:		fence callback information
> +	 * qbuf_work:		work for deferred qbuf operation
>  	 */
>  	struct kref		refcount;
>  	enum vb2_buffer_state	state;
> @@ -264,6 +269,11 @@ struct vb2_buffer {
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>  	struct list_head	queued_entry;
>  	struct list_head	done_entry;
> +
> +	struct dma_fence	*in_fence;
> +	struct dma_fence_cb	fence_cb;
> +	struct work_struct	qbuf_work;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -488,8 +498,9 @@ struct vb2_buf_ops {
>   * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
>   *		driver can set this to a mutex to let the v4l2 core serialize
>   *		the queuing ioctls. If the driver wants to handle locking
> - *		itself, then this should be set to NULL. This lock is not used
> - *		by the videobuf2 core API.
> + *		itself, then this should be set to NULL. This lock is required
> + *		by the videobuf2 core only on the fence-deferred start streaming
> + *		path.
>   * @owner:	The filehandle that 'owns' the buffers, i.e. the filehandle
>   *		that called reqbufs, create_buffers or started fileio.
>   *		This field is not used by the videobuf2 core API, but it allows
> @@ -789,6 +800,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * @index:	id number of the buffer
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> + * @in_fence:	in-fence to wait on before queueing the buffer
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> @@ -803,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   *
>   * Return: returns zero on success; an error code otherwise.
>   */
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Regards,

	Hans

Re: [PATCH v10 12/16] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

On Tue, 2018-05-22 at 14:37 +0200, Hans Verkuil wrote:
> On 21/05/18 18:59, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queued to the driver
> > out of the order they were queued from userspace. That means that even if
> > its fence signals it must wait for all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v13: - cleanup implementation.
> >      - remove wrong Kconfig changes.
> >      - print noisy warning on unexpected enqueue conditioin
> >      - schedule a vb2_start_streaming work from the fence callback
> > 
> > v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> > 
> > v11: - minor doc/comments fixes (Hans Verkuil)
> >      - reviewed the in-fence path at __fill_v4l2_buffer()
> > 
> > v10: - rename fence to in_fence in many places
> >      - handle fences signalling with error better (Hans Verkuil)
> > 
> > v9: - improve comments and docs (Hans Verkuil)
> >     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
> >     - move in-fences code that was in the out-fences patch here (Alex)
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7: - get rid of the fence array stuff for ordering and just use
> >       get_num_buffers_ready() (Hans)
> >     - fix issue of queuing the buffer twice (Hans)
> >     - avoid the dma_fence_wait() in core_qbuf() (Alex)
> >     - merge preparation commit in
> > 
> > v6: - With fences always keep the order userspace queues the buffers.
> >     - Protect in_fence manipulation with a lock (Brian Starkey)
> >     - check if fences have the same context before adding a fence array
> >     - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> >     - Clean up fence if __set_in_fence() fails (Brian Starkey)
> >     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> >       needed (Brian Starkey)
> >     - keep backward compat on the reserved2 field (Brian Starkey)
> >     - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4: - Add a comment about dma_fence_add_callback() not returning a
> >       error (Hans)
> >     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> >     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> >     - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> >     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> >     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> >       vb2_start_streaming() (Hans)
> >     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> >     - Queue buffers to the driver as soon as they are ready (Hans)
> >     - call fill_user_buffer() after queuing the buffer (Hans)
> >     - add err: label to clean up fence
> >     - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> >     - remove ternary if at vb2_qbuf() return (Mauro)
> >     - do not change if conditions behaviour (Mauro)
> > 
> > v2: - fix vb2_queue_or_prepare_buf() ret check
> >     - remove check for VB2_MEMORY_DMABUF only (Javier)
> >     - check num of ready buffers to start streaming
> >     - when queueing, start from the first ready buffer
> >     - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  drivers/media/common/videobuf2/Kconfig          |   1 +
> >  drivers/media/common/videobuf2/videobuf2-core.c | 224 ++++++++++++++++++++----
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  37 +++-
> >  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
> >  include/media/videobuf2-core.h                  |  19 +-
> >  5 files changed, 242 insertions(+), 41 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf2/Kconfig b/drivers/media/common/videobuf2/Kconfig
> > index 17c32ea58395..27ad9e8a268b 100644
> > --- a/drivers/media/common/videobuf2/Kconfig
> > +++ b/drivers/media/common/videobuf2/Kconfig
> > @@ -1,6 +1,7 @@
> >  # Used by drivers that need Videobuf2 modules
> >  config VIDEOBUF2_CORE
> >  	select DMA_SHARED_BUFFER
> > +	select SYNC_FILE
> >  	tristate
> >  
> >  config VIDEOBUF2_V4L2
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > index a9a0a9d1decb..86b5ebe25263 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -27,6 +27,7 @@
> >  #include <linux/kthread.h>
> >  
> >  #include <media/videobuf2-core.h>
> > +#include <media/v4l2-ioctl.h>
> >  #include <media/v4l2-mc.h>
> >  
> >  #include <trace/events/vb2.h>
> > @@ -189,6 +190,7 @@ module_param(debug, int, 0644);
> >  
> >  static void __vb2_queue_cancel(struct vb2_queue *q);
> >  static void __enqueue_in_driver(struct vb2_buffer *vb);
> > +static void __qbuf_work(struct work_struct *work);
> >  
> >  static void __vb2_buffer_free(struct kref *kref)
> >  {
> > @@ -373,6 +375,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
> >  		vb->index = q->num_buffers + buffer;
> >  		vb->type = q->type;
> >  		vb->memory = memory;
> > +		INIT_WORK(&vb->qbuf_work, __qbuf_work);
> >  		for (plane = 0; plane < num_planes; ++plane) {
> >  			vb->planes[plane].length = plane_sizes[plane];
> >  			vb->planes[plane].min_length = plane_sizes[plane];
> > @@ -932,21 +935,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
> >  
> > -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> > +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  	unsigned long flags;
> >  	unsigned int plane;
> >  
> > -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > -		return;
> > -
> > -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > -		    state != VB2_BUF_STATE_ERROR &&
> > -		    state != VB2_BUF_STATE_QUEUED &&
> > -		    state != VB2_BUF_STATE_REQUEUEING))
> > -		state = VB2_BUF_STATE_ERROR;
> > -
> >  #ifdef CONFIG_VIDEO_ADV_DEBUG
> >  	/*
> >  	 * Although this is not a callback, it still does have to balance
> > @@ -962,6 +956,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
> >  
> >  	spin_lock_irqsave(&q->done_lock, flags);
> > +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> > +		atomic_dec(&q->owned_by_drv_count);
> > +
> >  	if (state == VB2_BUF_STATE_QUEUED ||
> >  	    state == VB2_BUF_STATE_REQUEUEING) {
> >  		vb->state = VB2_BUF_STATE_QUEUED;
> > @@ -970,7 +967,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		list_add_tail(&vb->done_entry, &q->done_list);
> >  		vb->state = state;
> >  	}
> > -	atomic_dec(&q->owned_by_drv_count);
> > +
> >  	spin_unlock_irqrestore(&q->done_lock, flags);
> >  
> >  	trace_vb2_buf_done(q, vb);
> > @@ -987,6 +984,47 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		wake_up(&q->done_wq);
> >  		break;
> >  	}
> > +
> > +}
> > +
> > +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> > +{
> > +	struct vb2_buffer *next;
> > +	struct vb2_queue *q;
> > +
> > +	dprintk(4, "called done on buffer %d state: %d -> %d\n",
> > +			vb->index, vb->state, state);
> > +
> > +	if (vb->state == VB2_BUF_STATE_ERROR)
> > +		return;
> > +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > +		return;
> > +
> > +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > +		    state != VB2_BUF_STATE_ERROR &&
> > +		    state != VB2_BUF_STATE_QUEUED &&
> > +		    state != VB2_BUF_STATE_REQUEUEING))
> > +		state = VB2_BUF_STATE_ERROR;
> > +
> > +	vb2_process_buffer_done(vb, state);
> > +
> > +	/*
> > +	 * Check if there is any buffer with an in-fence error in the next
> > +	 * position of the queue. Buffers whose in-fence signaled with error
> > +	 * are not queued to the driver and kept on the queue until the buffer
> > +	 * before them is done.
> > +	 * So here we process any existing buffers with in-fence errors and
> > +	 * wake up userspace.
> > +	 */
> > +	q = vb->vb2_queue;
> > +	next = list_next_entry(vb, queued_entry);
> > +	list_for_each_entry_from(next, &q->queued_list, queued_entry) {
> > +		if (!next->in_fence || !next->in_fence->error)
> > +			break;
> > +		if (next->in_fence && next->in_fence->error &&
> > +		    vb->state != VB2_BUF_STATE_ERROR)
> > +			vb2_process_buffer_done(next, VB2_BUF_STATE_ERROR);
> > +	}
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_buffer_done);
> >  
> > @@ -1271,6 +1309,8 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  
> > +	if (WARN_ON(vb->state == VB2_BUF_STATE_ACTIVE))
> > +		return;
> >  	vb->state = VB2_BUF_STATE_ACTIVE;
> >  	atomic_inc(&q->owned_by_drv_count);
> >  
> > @@ -1322,6 +1362,25 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
> >  	return 0;
> >  }
> >  
> > +static int __get_num_ready_buffers(struct vb2_queue *q)
> > +{
> > +	struct vb2_buffer *vb;
> > +	int ready_count = 0;
> > +
> > +	/*
> > +	 * Count num of buffers ready in front of the queued_list.
> > +	 * We want to stop counting when we find a buffer with an
> > +	 * unsignaled fence.
> > +	 */
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> > +			break;
> > +		ready_count++;
> > +	}
> > +
> > +	return ready_count;
> > +}
> > +
> >  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> >  {
> >  	struct vb2_buffer *vb;
> > @@ -1338,7 +1397,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	if (ret)
> >  		return ret;
> >  
> > -	/* Fill buffer information for the userspace */
> > +	/* Fill buffer information for userspace */
> >  	call_void_bufop(q, fill_user_buffer, vb, pb);
> >  
> >  	dprintk(2, "prepare of buffer %d succeeded\n", vb->index);
> > @@ -1364,11 +1423,16 @@ static int vb2_start_streaming(struct vb2_queue *q)
> >  	int ret;
> >  
> >  	/*
> > -	 * If any buffers were queued before streamon,
> > -	 * we can now pass them to driver for processing.
> > +	 * Activate all buffers currently queued,
> > +	 * until we get an unsignaled fence.
> >  	 */
> > -	list_for_each_entry(vb, &q->queued_list, queued_entry)
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		if (vb->state != VB2_BUF_STATE_QUEUED)
> 
> I prefer: if (vb->state == VB2_BUF_STATE_ERROR)
> 
> It also needs a comment.
> 


> > +			continue;
> > +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> > +			break;
> >  		__enqueue_in_driver(vb);
> > +	}
> >  
> >  	/* Tell the driver to start streaming */
> >  	q->start_streaming_called = 1;
> > @@ -1410,7 +1474,41 @@ static int vb2_start_streaming(struct vb2_queue *q)
> >  	return ret;
> >  }
> >  
> > -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> > +static void __qbuf_work(struct work_struct *work)
> > +{
> > +	struct vb2_buffer *vb;
> > +	struct vb2_queue *q;
> > +
> > +	vb = container_of(work, struct vb2_buffer, qbuf_work);
> > +	q = vb->vb2_queue;
> > +
> > +	if (q->lock)
> > +		mutex_lock(q->lock);
> > +	/* By the time we run, the buffer state could have changed,
> > +	 * so we need to check it's in the queued state.
> > +	 * This is the only valid state.
> > +	 */
> > +	if (q->start_streaming_called && vb->state == VB2_BUF_STATE_QUEUED)
> > +		__enqueue_in_driver(vb);
> > +
> > +	if (q->streaming && !q->start_streaming_called &&
> > +	    __get_num_ready_buffers(q) >= q->min_buffers_needed)
> > +		vb2_start_streaming(q);
> 
> What happens if vb2_start_streaming fails?
> 
> > +
> > +	if (q->lock)
> > +		mutex_unlock(q->lock);
> > +	__vb2_buffer_put(vb);
> > +}
> > +
> > +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> > +{
> > +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> > +
> > +	schedule_work(&vb->qbuf_work);
> > +}
> > +
> > +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> > +		  struct dma_fence *in_fence)
> >  {
> >  	struct vb2_buffer *vb;
> >  	int ret;
> > @@ -1421,16 +1519,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	case VB2_BUF_STATE_DEQUEUED:
> >  		ret = __buf_prepare(vb, pb);
> >  		if (ret)
> > -			return ret;
> > +			goto err_release;
> >  		break;
> >  	case VB2_BUF_STATE_PREPARED:
> >  		break;
> >  	case VB2_BUF_STATE_PREPARING:
> >  		dprintk(1, "buffer still being prepared\n");
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err_release;
> >  	default:
> >  		dprintk(1, "invalid buffer state %d\n", vb->state);
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err_release;
> >  	}
> >  
> >  	/*
> > @@ -1448,15 +1548,44 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	trace_vb2_qbuf(q, vb);
> >  
> >  	/*
> > -	 * If already streaming, give the buffer to driver for processing.
> > -	 * If not, the buffer will be given to driver on next streamon.
> > +	 * If the fence didn't signal yet, we setup a callback to queue
> > +	 * the buffer once the fence signals. If the fence already signaled,
> > +	 * then we lose the reference and queue the buffer.
> >  	 */
> > -	if (q->start_streaming_called)
> > -		__enqueue_in_driver(vb);
> > +	if (in_fence) {
> > +		WARN_ON(vb->in_fence);
> > +		vb->in_fence = in_fence;
> > +		__vb2_buffer_get(vb);
> > +		ret = dma_fence_add_callback(in_fence, &vb->fence_cb,
> > +					     vb2_qbuf_fence_cb);
> > +		/* is the fence signaled? */
> > +		if (ret == -ENOENT) {
> > +			if (in_fence->error)
> > +				/* error-signaled fence, reject this buffer */
> > +				goto err_release;
> > +			dma_fence_put(in_fence);
> > +			vb->in_fence = NULL;
> > +			__vb2_buffer_put(vb);
> > +		} else if (ret) {
> > +			goto err_release;
> > +		}
> > +	}
> >  
> > -	/* Fill buffer information for the userspace */
> > -	if (pb)
> > -		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +	/*
> > +	 * If already streaming and there is no fence to wait on
> > +	 * give the buffer to driver for processing.
> > +	 */
> > +	if (q->start_streaming_called) {
> > +		struct vb2_buffer *b;
> > +
> > +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> > +			if (b->state != VB2_BUF_STATE_QUEUED)
> 
> b->state == VB2_BUF_STATE_ERROR
> 
> > +				continue;
> > +			if (b->in_fence && dma_fence_get_status(b->in_fence) == 0)
> > +				break;
> > +			__enqueue_in_driver(b);
> > +		}
> > +	}
> >  
> >  	/*
> >  	 * If streamon has been called, and we haven't yet called
> > @@ -1465,14 +1594,32 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	 * then we can finally call start_streaming().
> >  	 */
> >  	if (q->streaming && !q->start_streaming_called &&
> > -	    q->queued_count >= q->min_buffers_needed) {
> > +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
> >  		ret = vb2_start_streaming(q);
> >  		if (ret)
> > -			return ret;
> > +			goto err_release;
> >  	}
> >  
> > +	/* Fill buffer information for userspace */
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> >  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
> >  	return 0;
> > +
> > +err_release:
> > +	/* Fill buffer information for userspace */
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> > +	if (in_fence) {
> > +		dma_fence_put(in_fence);
> > +		vb->in_fence = NULL;
> > +		__vb2_buffer_put(vb);
> > +	}
> > +
> > +	return ret;
> > +
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
> >  
> > @@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
> >  		return;
> >  
> >  	vb->state = VB2_BUF_STATE_DEQUEUED;
> > -
> > +	if (vb->in_fence) {
> > +		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
> > +			__vb2_buffer_put(vb);
> > +		dma_fence_put(vb->in_fence);
> > +		vb->in_fence = NULL;
> > +	}
> >  	/* unmap DMABUF buffer */
> >  	if (q->memory == VB2_MEMORY_DMABUF)
> >  		for (i = 0; i < vb->num_planes; ++i) {
> > @@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
> >  	if (pindex)
> >  		*pindex = vb->index;
> >  
> > -	/* Fill buffer information for the userspace */
> > +	/* Fill buffer information for userspace */
> >  	if (pb)
> >  		call_void_bufop(q, fill_user_buffer, vb, pb);
> >  
> > @@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> >  	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
> >  		for (i = 0; i < q->num_buffers; ++i)
> >  			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
> > -				pr_warn("driver bug: stop_streaming operation is leaving buf %p in active state\n",
> > -					q->bufs[i]);
> > +				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p in active
> > state\n",
> > +					q->bufs[i]->index, q->bufs[i]);
> >  				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
> >  			}
> 
> Shouldn't any pending fences be canceled here?
> 

No, we don't have to flush -- that's the reason of the refcount :)
The qbuf_work won't do anything if all the buffers are returned
by the driver (with error or done state), and if !streaming.

Also, note that's why qbuf_work checks for the queued state, and not
for the error state.

> I feel uncomfortable with the refcounting of buffers, I'd rather that when we
> cancel the queue all fences for buffers are removed/canceled/whatever.
> 
> Is there any reason for refcounting if we cancel all pending fences here?
> 
> Note that besides canceling fences you also need to cancel/flush __qbuf_work.
> 
> 

Like I said above, I'm trying to avoid cancel/flushing the workqueue.
Currently, I believe it works fine without any flushing, provided we refcount
the buffers.

The problem with cancelling the workqueue, is that you need to unlock the queue
lock, to avoid a deadlock. It seemed to me that having a refcount is more natural.

Thoughts?

Re: [PATCH v10 12/16] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 22/05/18 18:22, Ezequiel Garcia wrote:
>>> @@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
>>>  		return;
>>>  
>>>  	vb->state = VB2_BUF_STATE_DEQUEUED;
>>> -
>>> +	if (vb->in_fence) {
>>> +		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
>>> +			__vb2_buffer_put(vb);
>>> +		dma_fence_put(vb->in_fence);
>>> +		vb->in_fence = NULL;
>>> +	}
>>>  	/* unmap DMABUF buffer */
>>>  	if (q->memory == VB2_MEMORY_DMABUF)
>>>  		for (i = 0; i < vb->num_planes; ++i) {
>>> @@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
>>>  	if (pindex)
>>>  		*pindex = vb->index;
>>>  
>>> -	/* Fill buffer information for the userspace */
>>> +	/* Fill buffer information for userspace */
>>>  	if (pb)
>>>  		call_void_bufop(q, fill_user_buffer, vb, pb);
>>>  
>>> @@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>>>  	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
>>>  		for (i = 0; i < q->num_buffers; ++i)
>>>  			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
>>> -				pr_warn("driver bug: stop_streaming operation is leaving buf %p in active state\n",
>>> -					q->bufs[i]);
>>> +				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p in active
>>> state\n",
>>> +					q->bufs[i]->index, q->bufs[i]);
>>>  				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
>>>  			}
>>
>> Shouldn't any pending fences be canceled here?
>>
> 
> No, we don't have to flush -- that's the reason of the refcount :)
> The qbuf_work won't do anything if all the buffers are returned
> by the driver (with error or done state), and if !streaming.
> 
> Also, note that's why qbuf_work checks for the queued state, and not
> for the error state.
> 
>> I feel uncomfortable with the refcounting of buffers, I'd rather that when we
>> cancel the queue all fences for buffers are removed/canceled/whatever.
>>
>> Is there any reason for refcounting if we cancel all pending fences here?
>>
>> Note that besides canceling fences you also need to cancel/flush __qbuf_work.
>>
>>
> 
> Like I said above, I'm trying to avoid cancel/flushing the workqueue.
> Currently, I believe it works fine without any flushing, provided we refcount
> the buffers.
> 
> The problem with cancelling the workqueue, is that you need to unlock the queue
> lock, to avoid a deadlock. It seemed to me that having a refcount is more natural.
> 
> Thoughts?
> 

I'll take another look tomorrow morning. Do you have a public git tree containing
this series that I can browse?

Regards,

	Hans

Re: [PATCH v10 12/16] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

On Tue, 2018-05-22 at 18:48 +0200, Hans Verkuil wrote:
> On 22/05/18 18:22, Ezequiel Garcia wrote:
> > > > @@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
> > > >  		return;
> > > >  
> > > >  	vb->state = VB2_BUF_STATE_DEQUEUED;
> > > > -
> > > > +	if (vb->in_fence) {
> > > > +		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
> > > > +			__vb2_buffer_put(vb);
> > > > +		dma_fence_put(vb->in_fence);
> > > > +		vb->in_fence = NULL;
> > > > +	}
> > > >  	/* unmap DMABUF buffer */
> > > >  	if (q->memory == VB2_MEMORY_DMABUF)
> > > >  		for (i = 0; i < vb->num_planes; ++i) {
> > > > @@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
> > > >  	if (pindex)
> > > >  		*pindex = vb->index;
> > > >  
> > > > -	/* Fill buffer information for the userspace */
> > > > +	/* Fill buffer information for userspace */
> > > >  	if (pb)
> > > >  		call_void_bufop(q, fill_user_buffer, vb, pb);
> > > >  
> > > > @@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> > > >  	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
> > > >  		for (i = 0; i < q->num_buffers; ++i)
> > > >  			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
> > > > -				pr_warn("driver bug: stop_streaming operation is leaving buf %p in active
> > > > state\n",
> > > > -					q->bufs[i]);
> > > > +				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p in active
> > > > state\n",
> > > > +					q->bufs[i]->index, q->bufs[i]);
> > > >  				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
> > > >  			}
> > > 
> > > Shouldn't any pending fences be canceled here?
> > > 
> > 
> > No, we don't have to flush -- that's the reason of the refcount :)
> > The qbuf_work won't do anything if all the buffers are returned
> > by the driver (with error or done state), and if !streaming.
> > 
> > Also, note that's why qbuf_work checks for the queued state, and not
> > for the error state.
> > 
> > > I feel uncomfortable with the refcounting of buffers, I'd rather that when we
> > > cancel the queue all fences for buffers are removed/canceled/whatever.
> > > 
> > > Is there any reason for refcounting if we cancel all pending fences here?
> > > 
> > > Note that besides canceling fences you also need to cancel/flush __qbuf_work.
> > > 
> > > 
> > 
> > Like I said above, I'm trying to avoid cancel/flushing the workqueue.
> > Currently, I believe it works fine without any flushing, provided we refcount
> > the buffers.
> > 
> > The problem with cancelling the workqueue, is that you need to unlock the queue
> > lock, to avoid a deadlock. It seemed to me that having a refcount is more natural.
> > 
> > Thoughts?
> > 
> 
> I'll take another look tomorrow morning. Do you have a public git tree containing
> this series that I can browse?
> 
> 

Sure, there you go http://git.infradead.org/users/ezequielg/linux/shortlog/refs/heads/fences_v10_v4.17-rc1

Re: [PATCH v10 12/16] vb2: add in-fence support to QBUF

[sathyam panda]

Hello,

On 5/21/18, Ezequiel Garcia <ezequiel@collabora.com> wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
> Receive in-fence from userspace and add support for waiting on them
> before queueing the buffer to the driver. Buffers can't be queued to the
> driver before its fences signal. And a buffer can't be queued to the driver
> out of the order they were queued from userspace. That means that even if
> its fence signals it must wait for all other buffers, ahead of it in the
> queue,
> to signal first.
>
> If the fence for some buffer fails we do not queue it to the driver,
> instead we mark it as error and wait until the previous buffer is done
> to notify userspace of the error. We wait here to deliver the buffers back
> to userspace in order.
>
> v13: - cleanup implementation.
>      - remove wrong Kconfig changes.
>      - print noisy warning on unexpected enqueue conditioin
>      - schedule a vb2_start_streaming work from the fence callback
>
> v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
>
> v11: - minor doc/comments fixes (Hans Verkuil)
>      - reviewed the in-fence path at __fill_v4l2_buffer()
>
> v10: - rename fence to in_fence in many places
>      - handle fences signalling with error better (Hans Verkuil)
>
> v9: - improve comments and docs (Hans Verkuil)
>     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>     - move in-fences code that was in the out-fences patch here (Alex)
>
> v8: - improve comments about fences with errors
>
> v7: - get rid of the fence array stuff for ordering and just use
>       get_num_buffers_ready() (Hans)
>     - fix issue of queuing the buffer twice (Hans)
>     - avoid the dma_fence_wait() in core_qbuf() (Alex)
>     - merge preparation commit in
>
> v6: - With fences always keep the order userspace queues the buffers.
>     - Protect in_fence manipulation with a lock (Brian Starkey)
>     - check if fences have the same context before adding a fence array
>     - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>     - Clean up fence if __set_in_fence() fails (Brian Starkey)
>     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
>
> v5: - use fence_array to keep buffers ordered in vb2 core when
>       needed (Brian Starkey)
>     - keep backward compat on the reserved2 field (Brian Starkey)
>     - protect fence callback removal with lock (Brian Starkey)
>
> v4: - Add a comment about dma_fence_add_callback() not returning a
>       error (Hans)
>     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>     - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>       vb2_start_streaming() (Hans)
>     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>     - Queue buffers to the driver as soon as they are ready (Hans)
>     - call fill_user_buffer() after queuing the buffer (Hans)
>     - add err: label to clean up fence
>     - add dma_fence_wait() before calling vb2_start_streaming()
>
> v3: - document fence parameter
>     - remove ternary if at vb2_qbuf() return (Mauro)
>     - do not change if conditions behaviour (Mauro)
>
> v2: - fix vb2_queue_or_prepare_buf() ret check
>     - remove check for VB2_MEMORY_DMABUF only (Javier)
>     - check num of ready buffers to start streaming
>     - when queueing, start from the first ready buffer
>     - handle queue cancel
>
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/Kconfig          |   1 +
>  drivers/media/common/videobuf2/videobuf2-core.c | 224
> ++++++++++++++++++++----
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  37 +++-
>  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
>  include/media/videobuf2-core.h                  |  19 +-
>  5 files changed, 242 insertions(+), 41 deletions(-)
>
> diff --git a/drivers/media/common/videobuf2/Kconfig
> b/drivers/media/common/videobuf2/Kconfig
> index 17c32ea58395..27ad9e8a268b 100644
> --- a/drivers/media/common/videobuf2/Kconfig
> +++ b/drivers/media/common/videobuf2/Kconfig
> @@ -1,6 +1,7 @@
>  # Used by drivers that need Videobuf2 modules
>  config VIDEOBUF2_CORE
>  	select DMA_SHARED_BUFFER
> +	select SYNC_FILE
>  	tristate
>
>  config VIDEOBUF2_V4L2
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> b/drivers/media/common/videobuf2/videobuf2-core.c
> index a9a0a9d1decb..86b5ebe25263 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -27,6 +27,7 @@
>  #include <linux/kthread.h>
>
>  #include <media/videobuf2-core.h>
> +#include <media/v4l2-ioctl.h>
>  #include <media/v4l2-mc.h>
>
>  #include <trace/events/vb2.h>
> @@ -189,6 +190,7 @@ module_param(debug, int, 0644);
>
>  static void __vb2_queue_cancel(struct vb2_queue *q);
>  static void __enqueue_in_driver(struct vb2_buffer *vb);
> +static void __qbuf_work(struct work_struct *work);
>
>  static void __vb2_buffer_free(struct kref *kref)
>  {
> @@ -373,6 +375,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum
> vb2_memory memory,
>  		vb->index = q->num_buffers + buffer;
>  		vb->type = q->type;
>  		vb->memory = memory;
> +		INIT_WORK(&vb->qbuf_work, __qbuf_work);
>  		for (plane = 0; plane < num_planes; ++plane) {
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
> @@ -932,21 +935,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned
> int plane_no)
>  }
>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>
> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum
> vb2_buffer_state state)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  	unsigned long flags;
>  	unsigned int plane;
>
> -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> -		return;
> -
> -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> -		    state != VB2_BUF_STATE_ERROR &&
> -		    state != VB2_BUF_STATE_QUEUED &&
> -		    state != VB2_BUF_STATE_REQUEUEING))
> -		state = VB2_BUF_STATE_ERROR;
> -
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Although this is not a callback, it still does have to balance
> @@ -962,6 +956,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum
> vb2_buffer_state state)
>  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>
>  	spin_lock_irqsave(&q->done_lock, flags);
> +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> +		atomic_dec(&q->owned_by_drv_count);
> +
>  	if (state == VB2_BUF_STATE_QUEUED ||
>  	    state == VB2_BUF_STATE_REQUEUEING) {
>  		vb->state = VB2_BUF_STATE_QUEUED;
> @@ -970,7 +967,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum
> vb2_buffer_state state)
>  		list_add_tail(&vb->done_entry, &q->done_list);
>  		vb->state = state;
>  	}
> -	atomic_dec(&q->owned_by_drv_count);
> +
>  	spin_unlock_irqrestore(&q->done_lock, flags);
>
>  	trace_vb2_buf_done(q, vb);
> @@ -987,6 +984,47 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum
> vb2_buffer_state state)
>  		wake_up(&q->done_wq);
>  		break;
>  	}
> +
> +}
> +
> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +{
> +	struct vb2_buffer *next;
> +	struct vb2_queue *q;
> +
> +	dprintk(4, "called done on buffer %d state: %d -> %d\n",
> +			vb->index, vb->state, state);
> +
> +	if (vb->state == VB2_BUF_STATE_ERROR)
> +		return;
> +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> +		return;
> +
> +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> +		    state != VB2_BUF_STATE_ERROR &&
> +		    state != VB2_BUF_STATE_QUEUED &&
> +		    state != VB2_BUF_STATE_REQUEUEING))
> +		state = VB2_BUF_STATE_ERROR;
> +
> +	vb2_process_buffer_done(vb, state);
> +
> +	/*
> +	 * Check if there is any buffer with an in-fence error in the next
> +	 * position of the queue. Buffers whose in-fence signaled with error
> +	 * are not queued to the driver and kept on the queue until the buffer
> +	 * before them is done.
> +	 * So here we process any existing buffers with in-fence errors and
> +	 * wake up userspace.
> +	 */
> +	q = vb->vb2_queue;
> +	next = list_next_entry(vb, queued_entry);
> +	list_for_each_entry_from(next, &q->queued_list, queued_entry) {
> +		if (!next->in_fence || !next->in_fence->error)
> +			break;
> +		if (next->in_fence && next->in_fence->error &&
> +		    vb->state != VB2_BUF_STATE_ERROR)
> +			vb2_process_buffer_done(next, VB2_BUF_STATE_ERROR);
> +	}
>  }
>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>
> @@ -1271,6 +1309,8 @@ static void __enqueue_in_driver(struct vb2_buffer
> *vb)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>
> +	if (WARN_ON(vb->state == VB2_BUF_STATE_ACTIVE))
> +		return;
>  	vb->state = VB2_BUF_STATE_ACTIVE;
>  	atomic_inc(&q->owned_by_drv_count);
>
> @@ -1322,6 +1362,25 @@ static int __buf_prepare(struct vb2_buffer *vb, const
> void *pb)
>  	return 0;
>  }
>
> +static int __get_num_ready_buffers(struct vb2_queue *q)
> +{
> +	struct vb2_buffer *vb;
> +	int ready_count = 0;
> +
> +	/*
> +	 * Count num of buffers ready in front of the queued_list.
> +	 * We want to stop counting when we find a buffer with an
> +	 * unsignaled fence.
> +	 */
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> +			break;
> +		ready_count++;
> +	}
> +
> +	return ready_count;
> +}
> +
>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void
> *pb)
>  {
>  	struct vb2_buffer *vb;
> @@ -1338,7 +1397,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned
> int index, void *pb)
>  	if (ret)
>  		return ret;
>
> -	/* Fill buffer information for the userspace */
> +	/* Fill buffer information for userspace */
>  	call_void_bufop(q, fill_user_buffer, vb, pb);
>
>  	dprintk(2, "prepare of buffer %d succeeded\n", vb->index);
> @@ -1364,11 +1423,16 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	int ret;
>
>  	/*
> -	 * If any buffers were queued before streamon,
> -	 * we can now pass them to driver for processing.
> +	 * Activate all buffers currently queued,
> +	 * until we get an unsignaled fence.
>  	 */
> -	list_for_each_entry(vb, &q->queued_list, queued_entry)
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->state != VB2_BUF_STATE_QUEUED)
> +			continue;
> +		if (vb->in_fence && dma_fence_get_status(vb->in_fence) == 0)
> +			break;
>  		__enqueue_in_driver(vb);
> +	}
>
>  	/* Tell the driver to start streaming */
>  	q->start_streaming_called = 1;
> @@ -1410,7 +1474,41 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	return ret;
>  }
>
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> +static void __qbuf_work(struct work_struct *work)
> +{
> +	struct vb2_buffer *vb;
> +	struct vb2_queue *q;
> +
> +	vb = container_of(work, struct vb2_buffer, qbuf_work);
> +	q = vb->vb2_queue;
> +
> +	if (q->lock)
> +		mutex_lock(q->lock);
> +	/* By the time we run, the buffer state could have changed,
> +	 * so we need to check it's in the queued state.
> +	 * This is the only valid state.
> +	 */
> +	if (q->start_streaming_called && vb->state == VB2_BUF_STATE_QUEUED)
> +		__enqueue_in_driver(vb);

What if fence is signalled out of order, e.g. fence 3 is signalled
after fence 1.
In that case it'll go ahead and enqueue without checking if all others
buffers before this have been signalled or not.

> +
> +	if (q->streaming && !q->start_streaming_called &&
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed)
> +		vb2_start_streaming(q);
> +
> +	if (q->lock)
> +		mutex_unlock(q->lock);
> +	__vb2_buffer_put(vb);
> +}
> +
> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb
> *cb)
> +{
> +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> +
> +	schedule_work(&vb->qbuf_work);
> +}
> +
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence)
>  {
>  	struct vb2_buffer *vb;
>  	int ret;
> @@ -1421,16 +1519,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int
> index, void *pb)
>  	case VB2_BUF_STATE_DEQUEUED:
>  		ret = __buf_prepare(vb, pb);
>  		if (ret)
> -			return ret;
> +			goto err_release;
>  		break;
>  	case VB2_BUF_STATE_PREPARED:
>  		break;
>  	case VB2_BUF_STATE_PREPARING:
>  		dprintk(1, "buffer still being prepared\n");
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err_release;
>  	default:
>  		dprintk(1, "invalid buffer state %d\n", vb->state);
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err_release;
>  	}
>
>  	/*
> @@ -1448,15 +1548,44 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int
> index, void *pb)
>  	trace_vb2_qbuf(q, vb);
>
>  	/*
> -	 * If already streaming, give the buffer to driver for processing.
> -	 * If not, the buffer will be given to driver on next streamon.
> +	 * If the fence didn't signal yet, we setup a callback to queue
> +	 * the buffer once the fence signals. If the fence already signaled,
> +	 * then we lose the reference and queue the buffer.
>  	 */
> -	if (q->start_streaming_called)
> -		__enqueue_in_driver(vb);
> +	if (in_fence) {
> +		WARN_ON(vb->in_fence);
> +		vb->in_fence = in_fence;
> +		__vb2_buffer_get(vb);
> +		ret = dma_fence_add_callback(in_fence, &vb->fence_cb,
> +					     vb2_qbuf_fence_cb);
> +		/* is the fence signaled? */
> +		if (ret == -ENOENT) {
> +			if (in_fence->error)
> +				/* error-signaled fence, reject this buffer */
> +				goto err_release;
> +			dma_fence_put(in_fence);
> +			vb->in_fence = NULL;
> +			__vb2_buffer_put(vb);
> +		} else if (ret) {
> +			goto err_release;
> +		}
> +	}
>
> -	/* Fill buffer information for the userspace */
> -	if (pb)
> -		call_void_bufop(q, fill_user_buffer, vb, pb);
> +	/*
> +	 * If already streaming and there is no fence to wait on
> +	 * give the buffer to driver for processing.
> +	 */
> +	if (q->start_streaming_called) {
> +		struct vb2_buffer *b;
> +
> +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> +			if (b->state != VB2_BUF_STATE_QUEUED)
> +				continue;
> +			if (b->in_fence && dma_fence_get_status(b->in_fence) == 0)
> +				break;
> +			__enqueue_in_driver(b);
> +		}
> +	}

Instead of checking the whole queue again,

- we can check for first buffer before the current buffer in the queue
and if state is VB2_BUF_STATE_ACTIVE or VB2_BUF_STATE_DONE, then
enqueue all buffers starting from current buffer whose state!=error
and fence signalled till we find first buffer with fence not signaled
yet.

- And if previous buffer to the current buffer is in state =
VB2_BUF_STATE_ERROR, continue checking in backwards till you find a
buffer with state!= Error or reach head of queue.

- while traveling backward in queue list, If reached head of queue,
then start enqueue from current buffer exactly like in 1st point.

- Else if buffer found with state!=error &&
state==VB2_BUF_STATE_QUEUED, then donot enqueue current buffer.

Should change time complexity from O(n^2) to O(n)

>
>  	/*
>  	 * If streamon has been called, and we haven't yet called
> @@ -1465,14 +1594,32 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int
> index, void *pb)
>  	 * then we can finally call start_streaming().
>  	 */
>  	if (q->streaming && !q->start_streaming_called &&
> -	    q->queued_count >= q->min_buffers_needed) {
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = vb2_start_streaming(q);
>  		if (ret)
> -			return ret;
> +			goto err_release;
>  	}
>
> +	/* Fill buffer information for userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
>  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>  	return 0;
> +
> +err_release:
> +	/* Fill buffer information for userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
> +	if (in_fence) {
> +		dma_fence_put(in_fence);
> +		vb->in_fence = NULL;
> +		__vb2_buffer_put(vb);
> +	}
> +
> +	return ret;
> +
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>
> @@ -1615,7 +1762,12 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
>  		return;
>
>  	vb->state = VB2_BUF_STATE_DEQUEUED;
> -
> +	if (vb->in_fence) {
> +		if (dma_fence_remove_callback(vb->in_fence, &vb->fence_cb))
> +			__vb2_buffer_put(vb);
> +		dma_fence_put(vb->in_fence);
> +		vb->in_fence = NULL;
> +	}
>  	/* unmap DMABUF buffer */
>  	if (q->memory == VB2_MEMORY_DMABUF)
>  		for (i = 0; i < vb->num_planes; ++i) {
> @@ -1653,7 +1805,7 @@ int vb2_core_dqbuf(struct vb2_queue *q, unsigned int
> *pindex, void *pb,
>  	if (pindex)
>  		*pindex = vb->index;
>
> -	/* Fill buffer information for the userspace */
> +	/* Fill buffer information for userspace */
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
>
> @@ -1700,8 +1852,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	if (WARN_ON(atomic_read(&q->owned_by_drv_count))) {
>  		for (i = 0; i < q->num_buffers; ++i)
>  			if (q->bufs[i]->state == VB2_BUF_STATE_ACTIVE) {
> -				pr_warn("driver bug: stop_streaming operation is leaving buf %p in
> active state\n",
> -					q->bufs[i]);
> +				pr_warn("driver bug: stop_streaming operation is leaving buf[%d] 0x%p
> in active state\n",
> +					q->bufs[i]->index, q->bufs[i]);
>  				vb2_buffer_done(q->bufs[i], VB2_BUF_STATE_ERROR);
>  			}
>  		/* Must be zero now */
> @@ -1783,7 +1935,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned
> int type)
>  	 * Tell driver to start streaming provided sufficient buffers
>  	 * are available.
>  	 */
> -	if (q->queued_count >= q->min_buffers_needed) {
> +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = v4l_vb2q_enable_media_source(q);
>  		if (ret)
>  			return ret;
> @@ -2305,7 +2457,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int
> read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2484,7 +2636,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue
> *q, char __user *data, size_
>
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2587,7 +2739,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 8312f61adfa6..ac14cc8ab1c5 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -23,6 +23,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>
>  #include <media/v4l2-dev.h>
>  #include <media/v4l2-fh.h>
> @@ -178,6 +179,22 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue
> *q, struct v4l2_buffer *b,
>  		return -EINVAL;
>  	}
>
> +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
> +		return -EINVAL;
> +	}
> +
> +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
> +		return -EINVAL;
> +	}
> +
> +	if ((b->flags & V4L2_BUF_FLAG_IN_FENCE) && !q->lock) {
> +		pr_warn("%s: IN_FENCE flag set, but no queue lock\n", opname);
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +	}
> +
>  	return __verify_planes_array(q->bufs[b->index], b);
>  }
>
> @@ -210,6 +227,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb,
> void *pb)
>  	else
>  		b->fence_fd = 0;
>
> +	if (vb->in_fence)
> +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> +	else
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +
>  	if (q->is_multiplanar) {
>  		/*
>  		 * Fill in plane-related data if userspace provided an array
> @@ -566,6 +588,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	struct dma_fence *in_fence = NULL;
>  	int ret;
>
>  	if (vb2_fileio_is_active(q)) {
> @@ -574,7 +597,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer
> *b)
>  	}
>
>  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> +	if (ret)
> +		return ret;
> +
> +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> +		in_fence = sync_file_get_fence(b->fence_fd);
> +		if (!in_fence) {
> +			dprintk(1, "failed to get in-fence from fd %d\n",
> +				b->fence_fd);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return vb2_core_qbuf(q, b->index, b, in_fence);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>
> diff --git a/drivers/media/dvb-core/dvb_vb2.c
> b/drivers/media/dvb-core/dvb_vb2.c
> index b811adf88afa..7da53f10db1a 100644
> --- a/drivers/media/dvb-core/dvb_vb2.c
> +++ b/drivers/media/dvb-core/dvb_vb2.c
> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct
> dmx_buffer *b)
>  {
>  	int ret;
>
> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>  	if (ret) {
>  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>  			b->index, ret);
> diff --git a/include/media/videobuf2-core.h
> b/include/media/videobuf2-core.h
> index 71538ae2c255..ddd555f59dbf 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -14,6 +14,7 @@
>
>  #include <linux/bitops.h>
>  #include <linux/dma-buf.h>
> +#include <linux/dma-fence.h>
>  #include <linux/kref.h>
>  #include <linux/mm_types.h>
>  #include <linux/mutex.h>
> @@ -257,6 +258,10 @@ struct vb2_buffer {
>  	 * done_entry:		entry on the list that stores all buffers ready
>  	 *			to be dequeued to userspace
>  	 * vb2_plane:		per-plane information; do not change
> +	 * in_fence:		fence received from vb2 client to wait on before
> +	 *			using the buffer (queueing to the driver)
> +	 * fence_cb:		fence callback information
> +	 * qbuf_work:		work for deferred qbuf operation
>  	 */
>  	struct kref		refcount;
>  	enum vb2_buffer_state	state;
> @@ -264,6 +269,11 @@ struct vb2_buffer {
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>  	struct list_head	queued_entry;
>  	struct list_head	done_entry;
> +
> +	struct dma_fence	*in_fence;
> +	struct dma_fence_cb	fence_cb;
> +	struct work_struct	qbuf_work;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -488,8 +498,9 @@ struct vb2_buf_ops {
>   * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
>   *		driver can set this to a mutex to let the v4l2 core serialize
>   *		the queuing ioctls. If the driver wants to handle locking
> - *		itself, then this should be set to NULL. This lock is not used
> - *		by the videobuf2 core API.
> + *		itself, then this should be set to NULL. This lock is required
> + *		by the videobuf2 core only on the fence-deferred start streaming
> + *		path.
>   * @owner:	The filehandle that 'owns' the buffers, i.e. the filehandle
>   *		that called reqbufs, create_buffers or started fileio.
>   *		This field is not used by the videobuf2 core API, but it allows
> @@ -789,6 +800,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned
> int index, void *pb);
>   * @index:	id number of the buffer
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> + * @in_fence:	in-fence to wait on before queueing the buffer
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is
> called
>   * internally by VB2 by an API-specific handler, like
> ``videobuf2-v4l2.h``.
> @@ -803,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned
> int index, void *pb);
>   *
>   * Return: returns zero on success; an error code otherwise.
>   */
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence);
>
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
>

Regards,
 Sathyam

[PATCH v10 13/16] vb2: add out-fence support to QBUF

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace in the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v12: - Pass the fence_fd in vb2_qbuf for clarity.
     - Increase fence seqno if fence context if shared
     - Check get_unused_fd return

v11: - Return fence_fd to userpace only in the QBUF ioctl.
     - Rework implementation to avoid storing the sync_file
       as state, which is not really needed.

v10: - use -EIO for fence error (Hans Verkuil)
     - add comment around fence context creation (Hans Verkuil)

v9: - remove in-fences changes from this patch (Alex Courbot)
    - improve fence context creation (Hans Verkuil)
    - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)

v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
    - fix crash when checking not using fences in vb2_buffer_done()

v7: - merge patch that add the infrastructure to out-fences into
      this one (Alex Courbot)
    - Do not install the fd if there is no fence. (Alex Courbot)
    - do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
      ordering in vb2 for queueing in the driver, so the event is not
      necessary anymore and the out_fence_fd is sent back to userspace
      on QBUF call return arg
    - do not allow requeueing with out-fences, instead mark the buffer
      with an error and wake up to userspace.
    - send the out_fence_fd back to userspace on the fence_fd field

v5: - delay fd_install to DQ_EVENT (Hans)
    - if queue is fully ordered send OUT_FENCE event right away
      (Brian)
    - rename 'q->ordered' to 'q->ordered_in_driver'
    - merge change to implement OUT_FENCE event here

v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
    - set the OUT_FENCE flag if there is a fence pending (Hans)
    - call fd_install() after vb2_core_qbuf() (Hans)
    - clean up fence if vb2_core_qbuf() fails (Hans)
    - add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 113 +++++++++++++++++++++++-
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  10 ++-
 drivers/media/dvb-core/dvb_vb2.c                |   2 +-
 include/media/videobuf2-core.h                  |  20 ++++-
 4 files changed, 136 insertions(+), 9 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 86b5ebe25263..edc2fdaf56de 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/videobuf2-core.h>
 #include <media/v4l2-ioctl.h>
@@ -380,6 +381,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
 		}
+		vb->out_fence_fd = -1;
 		q->bufs[vb->index] = vb;
 
 		/* Allocate video buffer memory for the MMAP type */
@@ -976,10 +978,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
 	case VB2_BUF_STATE_QUEUED:
 		return;
 	case VB2_BUF_STATE_REQUEUEING:
+		/* Requeuing with explicit synchronization, spit warning */
+		WARN_ON_ONCE(vb->out_fence);
+
 		if (q->start_streaming_called)
 			__enqueue_in_driver(vb);
 		return;
 	default:
+		if (vb->out_fence) {
+			if (state == VB2_BUF_STATE_ERROR)
+				dma_fence_set_error(vb->out_fence, -EIO);
+			dma_fence_signal(vb->out_fence);
+			dma_fence_put(vb->out_fence);
+			vb->out_fence = NULL;
+			vb->out_fence_fd = -1;
+		}
+
 		/* Inform any processes that may be waiting for buffers */
 		wake_up(&q->done_wq);
 		break;
@@ -1406,6 +1420,76 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+	return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+	return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+	return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+	.get_driver_name = vb2_fence_get_driver_name,
+	.get_timeline_name = vb2_fence_get_timeline_name,
+	.enable_signaling = vb2_fence_enable_signaling,
+	.wait = dma_fence_default_wait,
+};
+
+static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
+{
+	struct sync_file *sync_file;
+	unsigned int seqno;
+	int ret, fd;
+
+	fd = get_unused_fd_flags(O_CLOEXEC);
+	if (fd < 0)
+		return fd;
+
+	/*
+	 * The same context can be used only if the queue is ordered,
+	 * so if the queue is ordered create one when the queueing start,
+	 * otherwise create one for every buffer
+	 */
+	if (call_qop(q, is_unordered, q)) {
+		q->out_fence_context = dma_fence_context_alloc(1);
+		seqno = 1;
+	} else {
+		seqno = q->out_fence_seqno++;
+	}
+
+	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
+	if (!vb->out_fence) {
+		ret = -ENOMEM;
+		goto err_put_fd;
+	}
+	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
+		       q->out_fence_context, seqno);
+
+	sync_file = sync_file_create(vb->out_fence);
+	if (!sync_file) {
+		ret = -ENOMEM;
+		goto err_free_fence;
+	}
+	fd_install(fd, sync_file->file);
+	vb->out_fence_fd = fd;
+	return 0;
+
+err_free_fence:
+	dma_fence_put(vb->out_fence);
+	vb->out_fence = NULL;
+err_put_fd:
+	put_unused_fd(vb->out_fence_fd);
+	vb->out_fence_fd = -1;
+	return ret;
+}
+
 /*
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1508,7 +1592,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
 }
 
 int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
-		  struct dma_fence *in_fence)
+		  struct dma_fence *in_fence, int *out_fence_fd)
 {
 	struct vb2_buffer *vb;
 	int ret;
@@ -1540,6 +1624,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 	list_add_tail(&vb->queued_entry, &q->queued_list);
 	q->queued_count++;
 	q->waiting_for_buffers = false;
+	q->queueing_started = 1;
 	vb->state = VB2_BUF_STATE_QUEUED;
 
 	if (pb)
@@ -1600,6 +1685,20 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 			goto err_release;
 	}
 
+	if (out_fence_fd) {
+		*out_fence_fd = -1;
+		ret = vb2_setup_out_fence(q, vb);
+		if (ret) {
+			dprintk(1, "failed to set up out-fence\n");
+			goto err_release;
+		}
+
+		/* If we have successfully created an out-fence,
+		 * return its fd.
+		 */
+		*out_fence_fd = vb->out_fence_fd;
+	}
+
 	/* Fill buffer information for userspace */
 	if (pb)
 		call_void_bufop(q, fill_user_buffer, vb, pb);
@@ -1864,6 +1963,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	q->start_streaming_called = 0;
 	q->queued_count = 0;
 	q->error = 0;
+	q->queueing_started = 0;
 
 	/*
 	 * Remove all buffers from videobuf's list...
@@ -2206,7 +2306,12 @@ int vb2_core_queue_init(struct vb2_queue *q)
 	spin_lock_init(&q->done_lock);
 	mutex_init(&q->mmap_lock);
 	init_waitqueue_head(&q->done_wq);
+	spin_lock_init(&q->out_fence_lock);
 
+	if (!call_qop(q, is_unordered, q)) {
+		q->out_fence_context = dma_fence_context_alloc(1);
+		q->out_fence_seqno = 1;
+	}
 	q->memory = VB2_MEMORY_UNKNOWN;
 
 	if (q->buf_struct_size == 0)
@@ -2457,7 +2562,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2636,7 +2741,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2739,7 +2844,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index ac14cc8ab1c5..b4908d0432c6 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -588,8 +588,9 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	bool get_out_fence = b->flags & V4L2_BUF_FLAG_OUT_FENCE;
 	struct dma_fence *in_fence = NULL;
-	int ret;
+	int ret, fence_fd;
 
 	if (vb2_fileio_is_active(q)) {
 		dprintk(1, "file io in progress\n");
@@ -609,7 +610,12 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		}
 	}
 
-	return vb2_core_qbuf(q, b->index, b, in_fence);
+	ret = vb2_core_qbuf(q, b->index, b, in_fence,
+			    get_out_fence ? &fence_fd : NULL);
+	if (ret)
+		return ret;
+	b->fence_fd = fence_fd;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index 7da53f10db1a..5c1523a8accc 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
 {
 	int ret;
 
-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, NULL);
 	if (ret) {
 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
 			b->index, ret);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index ddd555f59dbf..6abe133d1c9a 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -262,6 +262,9 @@ struct vb2_buffer {
 	 *			using the buffer (queueing to the driver)
 	 * fence_cb:		fence callback information
 	 * qbuf_work:		work for deferred qbuf operation
+	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
+	 * out_fence:		the out-fence associated with the buffer once
+	 *			it is queued to the driver.
 	 */
 	struct kref		refcount;
 	enum vb2_buffer_state	state;
@@ -274,6 +277,9 @@ struct vb2_buffer {
 	struct dma_fence_cb	fence_cb;
 	struct work_struct	qbuf_work;
 
+	int			out_fence_fd;
+	struct dma_fence	*out_fence;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -550,6 +556,10 @@ struct vb2_buf_ops {
  * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
  *		last decoded buffer was already dequeued. Set for capture queues
  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
+ * @queueing_started: if queueing has started. Currently used to determine
+ *		if an out_fence_context is needed.
+ * @out_fence_context: the fence context for the out fences
+ * @out_fence_seqno: the fence seqno to use, if the context is shared
  * @fileio:	file io emulator internal data, used only if emulator is active
  * @threadio:	thread io internal data, used only if thread is active
  */
@@ -602,6 +612,11 @@ struct vb2_queue {
 	unsigned int			is_output:1;
 	unsigned int			copy_timestamp:1;
 	unsigned int			last_buffer_dequeued:1;
+	unsigned int			queueing_started:1;
+
+	u64				out_fence_context;
+	unsigned int			out_fence_seqno;
+	spinlock_t			out_fence_lock;
 
 	struct vb2_fileio_data		*fileio;
 	struct vb2_threadio_data	*threadio;
@@ -800,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @index:	id number of the buffer
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
- * @in_fence:	in-fence to wait on before queueing the buffer
+ * @in_fence:	optioanl in-fence to wait on before queueing the buffer
+ * @out_fence_fd: optional pointer to store a newly created out-fence fd
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
@@ -816,7 +832,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * Return: returns zero on success; an error code otherwise.
  */
 int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
-		  struct dma_fence *in_fence);
+		  struct dma_fence *in_fence, int *out_fence_fd);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.16.3

Re: [PATCH v10 13/16] vb2: add out-fence support to QBUF

[Hans Verkuil]

On 21/05/18 18:59, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> an out_fence and send its fd to userspace in the fence_fd field as a
> return arg for the QBUF call.
> 
> The fence is signaled on buffer_done(), when the job on the buffer is
> finished.
> 
> v12: - Pass the fence_fd in vb2_qbuf for clarity.
>      - Increase fence seqno if fence context if shared
>      - Check get_unused_fd return

v12 while this this PATCH v10? That's a bit confusing.

> 
> v11: - Return fence_fd to userpace only in the QBUF ioctl.
>      - Rework implementation to avoid storing the sync_file
>        as state, which is not really needed.
> 
> v10: - use -EIO for fence error (Hans Verkuil)
>      - add comment around fence context creation (Hans Verkuil)
> 
> v9: - remove in-fences changes from this patch (Alex Courbot)
>     - improve fence context creation (Hans Verkuil)
>     - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
> 
> v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
>     - fix crash when checking not using fences in vb2_buffer_done()
> 
> v7: - merge patch that add the infrastructure to out-fences into
>       this one (Alex Courbot)
>     - Do not install the fd if there is no fence. (Alex Courbot)
>     - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
> 
> v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
>       ordering in vb2 for queueing in the driver, so the event is not
>       necessary anymore and the out_fence_fd is sent back to userspace
>       on QBUF call return arg
>     - do not allow requeueing with out-fences, instead mark the buffer
>       with an error and wake up to userspace.
>     - send the out_fence_fd back to userspace on the fence_fd field
> 
> v5: - delay fd_install to DQ_EVENT (Hans)
>     - if queue is fully ordered send OUT_FENCE event right away
>       (Brian)
>     - rename 'q->ordered' to 'q->ordered_in_driver'
>     - merge change to implement OUT_FENCE event here
> 
> v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)
> 
> v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
>     - set the OUT_FENCE flag if there is a fence pending (Hans)
>     - call fd_install() after vb2_core_qbuf() (Hans)
>     - clean up fence if vb2_core_qbuf() fails (Hans)
>     - add list to store sync_file and fence for the next queued buffer
> 
> v2: check if the queue is ordered.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 113 +++++++++++++++++++++++-
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  10 ++-
>  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
>  include/media/videobuf2-core.h                  |  20 ++++-
>  4 files changed, 136 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 86b5ebe25263..edc2fdaf56de 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/videobuf2-core.h>
>  #include <media/v4l2-ioctl.h>
> @@ -380,6 +381,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
>  		}
> +		vb->out_fence_fd = -1;
>  		q->bufs[vb->index] = vb;
>  
>  		/* Allocate video buffer memory for the MMAP type */
> @@ -976,10 +978,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
>  	case VB2_BUF_STATE_QUEUED:
>  		return;
>  	case VB2_BUF_STATE_REQUEUEING:
> +		/* Requeuing with explicit synchronization, spit warning */
> +		WARN_ON_ONCE(vb->out_fence);
> +
>  		if (q->start_streaming_called)
>  			__enqueue_in_driver(vb);
>  		return;
>  	default:
> +		if (vb->out_fence) {
> +			if (state == VB2_BUF_STATE_ERROR)
> +				dma_fence_set_error(vb->out_fence, -EIO);
> +			dma_fence_signal(vb->out_fence);
> +			dma_fence_put(vb->out_fence);
> +			vb->out_fence = NULL;
> +			vb->out_fence_fd = -1;
> +		}
> +
>  		/* Inform any processes that may be waiting for buffers */
>  		wake_up(&q->done_wq);
>  		break;
> @@ -1406,6 +1420,76 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>  
> +static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence";
> +}
> +
> +static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence_timeline";
> +}
> +
> +static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
> +{
> +	return true;
> +}

Does inline make sense since this functions are used as ops below?
I think that can be dropped.

Regards,

	Hans

> +
> +static const struct dma_fence_ops vb2_fence_ops = {
> +	.get_driver_name = vb2_fence_get_driver_name,
> +	.get_timeline_name = vb2_fence_get_timeline_name,
> +	.enable_signaling = vb2_fence_enable_signaling,
> +	.wait = dma_fence_default_wait,
> +};
> +
> +static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
> +{
> +	struct sync_file *sync_file;
> +	unsigned int seqno;
> +	int ret, fd;
> +
> +	fd = get_unused_fd_flags(O_CLOEXEC);
> +	if (fd < 0)
> +		return fd;
> +
> +	/*
> +	 * The same context can be used only if the queue is ordered,
> +	 * so if the queue is ordered create one when the queueing start,
> +	 * otherwise create one for every buffer
> +	 */
> +	if (call_qop(q, is_unordered, q)) {
> +		q->out_fence_context = dma_fence_context_alloc(1);
> +		seqno = 1;
> +	} else {
> +		seqno = q->out_fence_seqno++;
> +	}
> +
> +	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
> +	if (!vb->out_fence) {
> +		ret = -ENOMEM;
> +		goto err_put_fd;
> +	}
> +	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
> +		       q->out_fence_context, seqno);
> +
> +	sync_file = sync_file_create(vb->out_fence);
> +	if (!sync_file) {
> +		ret = -ENOMEM;
> +		goto err_free_fence;
> +	}
> +	fd_install(fd, sync_file->file);
> +	vb->out_fence_fd = fd;
> +	return 0;
> +
> +err_free_fence:
> +	dma_fence_put(vb->out_fence);
> +	vb->out_fence = NULL;
> +err_put_fd:
> +	put_unused_fd(vb->out_fence_fd);
> +	vb->out_fence_fd = -1;
> +	return ret;
> +}
> +
>  /*
>   * vb2_start_streaming() - Attempt to start streaming.
>   * @q:		videobuf2 queue
> @@ -1508,7 +1592,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>  }
>  
>  int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> -		  struct dma_fence *in_fence)
> +		  struct dma_fence *in_fence, int *out_fence_fd)
>  {
>  	struct vb2_buffer *vb;
>  	int ret;
> @@ -1540,6 +1624,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  	list_add_tail(&vb->queued_entry, &q->queued_list);
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
> +	q->queueing_started = 1;
>  	vb->state = VB2_BUF_STATE_QUEUED;
>  
>  	if (pb)
> @@ -1600,6 +1685,20 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  			goto err_release;
>  	}
>  
> +	if (out_fence_fd) {
> +		*out_fence_fd = -1;
> +		ret = vb2_setup_out_fence(q, vb);
> +		if (ret) {
> +			dprintk(1, "failed to set up out-fence\n");
> +			goto err_release;
> +		}
> +
> +		/* If we have successfully created an out-fence,
> +		 * return its fd.
> +		 */
> +		*out_fence_fd = vb->out_fence_fd;
> +	}
> +
>  	/* Fill buffer information for userspace */
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
> @@ -1864,6 +1963,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	q->start_streaming_called = 0;
>  	q->queued_count = 0;
>  	q->error = 0;
> +	q->queueing_started = 0;
>  
>  	/*
>  	 * Remove all buffers from videobuf's list...
> @@ -2206,7 +2306,12 @@ int vb2_core_queue_init(struct vb2_queue *q)
>  	spin_lock_init(&q->done_lock);
>  	mutex_init(&q->mmap_lock);
>  	init_waitqueue_head(&q->done_wq);
> +	spin_lock_init(&q->out_fence_lock);
>  
> +	if (!call_qop(q, is_unordered, q)) {
> +		q->out_fence_context = dma_fence_context_alloc(1);
> +		q->out_fence_seqno = 1;
> +	}
>  	q->memory = VB2_MEMORY_UNKNOWN;
>  
>  	if (q->buf_struct_size == 0)
> @@ -2457,7 +2562,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2636,7 +2741,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2739,7 +2844,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index ac14cc8ab1c5..b4908d0432c6 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -588,8 +588,9 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	bool get_out_fence = b->flags & V4L2_BUF_FLAG_OUT_FENCE;
>  	struct dma_fence *in_fence = NULL;
> -	int ret;
> +	int ret, fence_fd;
>  
>  	if (vb2_fileio_is_active(q)) {
>  		dprintk(1, "file io in progress\n");
> @@ -609,7 +610,12 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  		}
>  	}
>  
> -	return vb2_core_qbuf(q, b->index, b, in_fence);
> +	ret = vb2_core_qbuf(q, b->index, b, in_fence,
> +			    get_out_fence ? &fence_fd : NULL);
> +	if (ret)
> +		return ret;
> +	b->fence_fd = fence_fd;
> +	return 0;
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
> index 7da53f10db1a..5c1523a8accc 100644
> --- a/drivers/media/dvb-core/dvb_vb2.c
> +++ b/drivers/media/dvb-core/dvb_vb2.c
> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
>  {
>  	int ret;
>  
> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, NULL);
>  	if (ret) {
>  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>  			b->index, ret);
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index ddd555f59dbf..6abe133d1c9a 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -262,6 +262,9 @@ struct vb2_buffer {
>  	 *			using the buffer (queueing to the driver)
>  	 * fence_cb:		fence callback information
>  	 * qbuf_work:		work for deferred qbuf operation
> +	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
> +	 * out_fence:		the out-fence associated with the buffer once
> +	 *			it is queued to the driver.
>  	 */
>  	struct kref		refcount;
>  	enum vb2_buffer_state	state;
> @@ -274,6 +277,9 @@ struct vb2_buffer {
>  	struct dma_fence_cb	fence_cb;
>  	struct work_struct	qbuf_work;
>  
> +	int			out_fence_fd;
> +	struct dma_fence	*out_fence;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -550,6 +556,10 @@ struct vb2_buf_ops {
>   * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
>   *		last decoded buffer was already dequeued. Set for capture queues
>   *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
> + * @queueing_started: if queueing has started. Currently used to determine
> + *		if an out_fence_context is needed.
> + * @out_fence_context: the fence context for the out fences
> + * @out_fence_seqno: the fence seqno to use, if the context is shared
>   * @fileio:	file io emulator internal data, used only if emulator is active
>   * @threadio:	thread io internal data, used only if thread is active
>   */
> @@ -602,6 +612,11 @@ struct vb2_queue {
>  	unsigned int			is_output:1;
>  	unsigned int			copy_timestamp:1;
>  	unsigned int			last_buffer_dequeued:1;
> +	unsigned int			queueing_started:1;
> +
> +	u64				out_fence_context;
> +	unsigned int			out_fence_seqno;
> +	spinlock_t			out_fence_lock;
>  
>  	struct vb2_fileio_data		*fileio;
>  	struct vb2_threadio_data	*threadio;
> @@ -800,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * @index:	id number of the buffer
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> - * @in_fence:	in-fence to wait on before queueing the buffer
> + * @in_fence:	optioanl in-fence to wait on before queueing the buffer
> + * @out_fence_fd: optional pointer to store a newly created out-fence fd
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> @@ -816,7 +832,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * Return: returns zero on success; an error code otherwise.
>   */
>  int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> -		  struct dma_fence *in_fence);
> +		  struct dma_fence *in_fence, int *out_fence_fd);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Re: [PATCH v10 13/16] vb2: add out-fence support to QBUF

[Brian Starkey]

Hi Ezequiel,

Not sure if this patch series is still relevant, but I spotted a
couple more things below.

On Mon, May 21, 2018 at 01:59:43PM -0300, Ezequiel Garcia wrote:
>From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
>If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
>an out_fence and send its fd to userspace in the fence_fd field as a
>return arg for the QBUF call.
>
>The fence is signaled on buffer_done(), when the job on the buffer is
>finished.
>
>v12: - Pass the fence_fd in vb2_qbuf for clarity.
>     - Increase fence seqno if fence context if shared
>     - Check get_unused_fd return
>
>v11: - Return fence_fd to userpace only in the QBUF ioctl.
>     - Rework implementation to avoid storing the sync_file
>       as state, which is not really needed.
>
>v10: - use -EIO for fence error (Hans Verkuil)
>     - add comment around fence context creation (Hans Verkuil)
>
>v9: - remove in-fences changes from this patch (Alex Courbot)
>    - improve fence context creation (Hans Verkuil)
>    - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
>
>v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
>    - fix crash when checking not using fences in vb2_buffer_done()
>
>v7: - merge patch that add the infrastructure to out-fences into
>      this one (Alex Courbot)
>    - Do not install the fd if there is no fence. (Alex Courbot)
>    - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
>
>v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
>      ordering in vb2 for queueing in the driver, so the event is not
>      necessary anymore and the out_fence_fd is sent back to userspace
>      on QBUF call return arg
>    - do not allow requeueing with out-fences, instead mark the buffer
>      with an error and wake up to userspace.
>    - send the out_fence_fd back to userspace on the fence_fd field
>
>v5: - delay fd_install to DQ_EVENT (Hans)
>    - if queue is fully ordered send OUT_FENCE event right away
>      (Brian)
>    - rename 'q->ordered' to 'q->ordered_in_driver'
>    - merge change to implement OUT_FENCE event here
>
>v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)
>
>v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
>    - set the OUT_FENCE flag if there is a fence pending (Hans)
>    - call fd_install() after vb2_core_qbuf() (Hans)
>    - clean up fence if vb2_core_qbuf() fails (Hans)
>    - add list to store sync_file and fence for the next queued buffer
>
>v2: check if the queue is ordered.
>
>Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>---
> drivers/media/common/videobuf2/videobuf2-core.c | 113 +++++++++++++++++++++++-
> drivers/media/common/videobuf2/videobuf2-v4l2.c |  10 ++-
> drivers/media/dvb-core/dvb_vb2.c                |   2 +-
> include/media/videobuf2-core.h                  |  20 ++++-
> 4 files changed, 136 insertions(+), 9 deletions(-)
>
>diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>index 86b5ebe25263..edc2fdaf56de 100644
>--- a/drivers/media/common/videobuf2/videobuf2-core.c
>+++ b/drivers/media/common/videobuf2/videobuf2-core.c
>@@ -25,6 +25,7 @@
> #include <linux/sched.h>
> #include <linux/freezer.h>
> #include <linux/kthread.h>
>+#include <linux/sync_file.h>
>
> #include <media/videobuf2-core.h>
> #include <media/v4l2-ioctl.h>
>@@ -380,6 +381,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
> 			vb->planes[plane].length = plane_sizes[plane];
> 			vb->planes[plane].min_length = plane_sizes[plane];
> 		}
>+		vb->out_fence_fd = -1;
> 		q->bufs[vb->index] = vb;
>
> 		/* Allocate video buffer memory for the MMAP type */
>@@ -976,10 +978,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
> 	case VB2_BUF_STATE_QUEUED:
> 		return;
> 	case VB2_BUF_STATE_REQUEUEING:
>+		/* Requeuing with explicit synchronization, spit warning */
>+		WARN_ON_ONCE(vb->out_fence);
>+
> 		if (q->start_streaming_called)
> 			__enqueue_in_driver(vb);
> 		return;
> 	default:
>+		if (vb->out_fence) {
>+			if (state == VB2_BUF_STATE_ERROR)
>+				dma_fence_set_error(vb->out_fence, -EIO);
>+			dma_fence_signal(vb->out_fence);
>+			dma_fence_put(vb->out_fence);
>+			vb->out_fence = NULL;
>+			vb->out_fence_fd = -1;
>+		}
>+
> 		/* Inform any processes that may be waiting for buffers */
> 		wake_up(&q->done_wq);
> 		break;
>@@ -1406,6 +1420,76 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> }
> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>
>+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
>+{
>+	return "vb2_fence";
>+}
>+
>+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
>+{
>+	return "vb2_fence_timeline";
>+}
>+
>+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
>+{
>+	return true;
>+}
>+
>+static const struct dma_fence_ops vb2_fence_ops = {
>+	.get_driver_name = vb2_fence_get_driver_name,
>+	.get_timeline_name = vb2_fence_get_timeline_name,
>+	.enable_signaling = vb2_fence_enable_signaling,
>+	.wait = dma_fence_default_wait,
>+};
>+
>+static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
>+{
>+	struct sync_file *sync_file;
>+	unsigned int seqno;
>+	int ret, fd;
>+
>+	fd = get_unused_fd_flags(O_CLOEXEC);
>+	if (fd < 0)
>+		return fd;
>+
>+	/*
>+	 * The same context can be used only if the queue is ordered,
>+	 * so if the queue is ordered create one when the queueing start,
>+	 * otherwise create one for every buffer
>+	 */
>+	if (call_qop(q, is_unordered, q)) {
>+		q->out_fence_context = dma_fence_context_alloc(1);
>+		seqno = 1;
>+	} else {
>+		seqno = q->out_fence_seqno++;
>+	}
>+
>+	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
>+	if (!vb->out_fence) {
>+		ret = -ENOMEM;
>+		goto err_put_fd;
>+	}
>+	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
>+		       q->out_fence_context, seqno);
>+
>+	sync_file = sync_file_create(vb->out_fence);
>+	if (!sync_file) {
>+		ret = -ENOMEM;
>+		goto err_free_fence;
>+	}
>+	fd_install(fd, sync_file->file);
>+	vb->out_fence_fd = fd;
>+	return 0;
>+
>+err_free_fence:
>+	dma_fence_put(vb->out_fence);
>+	vb->out_fence = NULL;
>+err_put_fd:
>+	put_unused_fd(vb->out_fence_fd);
>+	vb->out_fence_fd = -1;
>+	return ret;
>+}
>+
> /*
>  * vb2_start_streaming() - Attempt to start streaming.
>  * @q:		videobuf2 queue
>@@ -1508,7 +1592,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> }
>
> int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>-		  struct dma_fence *in_fence)
>+		  struct dma_fence *in_fence, int *out_fence_fd)
> {
> 	struct vb2_buffer *vb;
> 	int ret;
>@@ -1540,6 +1624,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> 	list_add_tail(&vb->queued_entry, &q->queued_list);
> 	q->queued_count++;
> 	q->waiting_for_buffers = false;
>+	q->queueing_started = 1;
> 	vb->state = VB2_BUF_STATE_QUEUED;
>
> 	if (pb)
>@@ -1600,6 +1685,20 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> 			goto err_release;
> 	}
>
>+	if (out_fence_fd) {
>+		*out_fence_fd = -1;
>+		ret = vb2_setup_out_fence(q, vb);
>+		if (ret) {
>+			dprintk(1, "failed to set up out-fence\n");
>+			goto err_release;
>+		}
>+
>+		/* If we have successfully created an out-fence,
>+		 * return its fd.
>+		 */
>+		*out_fence_fd = vb->out_fence_fd;

I'm actually not sure why we need vb->out_fence_fd at all - perhaps
the fence_fd pointer can just be passed all the way down to
vb2_setup_out_fence.

Certainly after this point (and after returning to userspace) the
vb->out_fence_fd seems to be "dangling" without a reference, and I
didn't see anywhere else that it's needed.

>+	}
>+
> 	/* Fill buffer information for userspace */
> 	if (pb)
> 		call_void_bufop(q, fill_user_buffer, vb, pb);
>@@ -1864,6 +1963,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> 	q->start_streaming_called = 0;
> 	q->queued_count = 0;
> 	q->error = 0;
>+	q->queueing_started = 0;
>
> 	/*
> 	 * Remove all buffers from videobuf's list...
>@@ -2206,7 +2306,12 @@ int vb2_core_queue_init(struct vb2_queue *q)
> 	spin_lock_init(&q->done_lock);
> 	mutex_init(&q->mmap_lock);
> 	init_waitqueue_head(&q->done_wq);
>+	spin_lock_init(&q->out_fence_lock);
>
>+	if (!call_qop(q, is_unordered, q)) {
>+		q->out_fence_context = dma_fence_context_alloc(1);
>+		q->out_fence_seqno = 1;
>+	}
> 	q->memory = VB2_MEMORY_UNKNOWN;
>
> 	if (q->buf_struct_size == 0)
>@@ -2457,7 +2562,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
> 		 * Queue all buffers.
> 		 */
> 		for (i = 0; i < q->num_buffers; i++) {
>-			ret = vb2_core_qbuf(q, i, NULL, NULL);
>+			ret = vb2_core_qbuf(q, i, NULL, NULL, NULL);
> 			if (ret)
> 				goto err_reqbufs;
> 			fileio->bufs[i].queued = 1;
>@@ -2636,7 +2741,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>
> 		if (copy_timestamp)
> 			b->timestamp = ktime_get_ns();
>-		ret = vb2_core_qbuf(q, index, NULL, NULL);
>+		ret = vb2_core_qbuf(q, index, NULL, NULL, NULL);
> 		dprintk(5, "vb2_dbuf result: %d\n", ret);
> 		if (ret)
> 			return ret;
>@@ -2739,7 +2844,7 @@ static int vb2_thread(void *data)
> 		if (copy_timestamp)
> 			vb->timestamp = ktime_get_ns();
> 		if (!threadio->stop)
>-			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, NULL);
> 		call_void_qop(q, wait_prepare, q);
> 		if (ret || threadio->stop)
> 			break;
>diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>index ac14cc8ab1c5..b4908d0432c6 100644
>--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>@@ -588,8 +588,9 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>
> int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> {
>+	bool get_out_fence = b->flags & V4L2_BUF_FLAG_OUT_FENCE;
> 	struct dma_fence *in_fence = NULL;
>-	int ret;
>+	int ret, fence_fd;
>
> 	if (vb2_fileio_is_active(q)) {
> 		dprintk(1, "file io in progress\n");
>@@ -609,7 +610,12 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> 		}
> 	}
>
>-	return vb2_core_qbuf(q, b->index, b, in_fence);
>+	ret = vb2_core_qbuf(q, b->index, b, in_fence,
>+			    get_out_fence ? &fence_fd : NULL);
>+	if (ret)
>+		return ret;

If vb2_start_streaming() fails at the end of vb2_core_qbuf, then it
looks like the fence is installed in fence_fd, but the return here
means that the fd never gets given to userspace, and so the fd and the
fence gets leaked.

Thanks,
-Brian

>+	b->fence_fd = fence_fd;
>+	return 0;
> }
> EXPORT_SYMBOL_GPL(vb2_qbuf);
>
>diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
>index 7da53f10db1a..5c1523a8accc 100644
>--- a/drivers/media/dvb-core/dvb_vb2.c
>+++ b/drivers/media/dvb-core/dvb_vb2.c
>@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
> {
> 	int ret;
>
>-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, NULL);
> 	if (ret) {
> 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
> 			b->index, ret);
>diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>index ddd555f59dbf..6abe133d1c9a 100644
>--- a/include/media/videobuf2-core.h
>+++ b/include/media/videobuf2-core.h
>@@ -262,6 +262,9 @@ struct vb2_buffer {
> 	 *			using the buffer (queueing to the driver)
> 	 * fence_cb:		fence callback information
> 	 * qbuf_work:		work for deferred qbuf operation
>+	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
>+	 * out_fence:		the out-fence associated with the buffer once
>+	 *			it is queued to the driver.
> 	 */
> 	struct kref		refcount;
> 	enum vb2_buffer_state	state;
>@@ -274,6 +277,9 @@ struct vb2_buffer {
> 	struct dma_fence_cb	fence_cb;
> 	struct work_struct	qbuf_work;
>
>+	int			out_fence_fd;
>+	struct dma_fence	*out_fence;
>+
> #ifdef CONFIG_VIDEO_ADV_DEBUG
> 	/*
> 	 * Counters for how often these buffer-related ops are
>@@ -550,6 +556,10 @@ struct vb2_buf_ops {
>  * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
>  *		last decoded buffer was already dequeued. Set for capture queues
>  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
>+ * @queueing_started: if queueing has started. Currently used to determine
>+ *		if an out_fence_context is needed.
>+ * @out_fence_context: the fence context for the out fences
>+ * @out_fence_seqno: the fence seqno to use, if the context is shared
>  * @fileio:	file io emulator internal data, used only if emulator is active
>  * @threadio:	thread io internal data, used only if thread is active
>  */
>@@ -602,6 +612,11 @@ struct vb2_queue {
> 	unsigned int			is_output:1;
> 	unsigned int			copy_timestamp:1;
> 	unsigned int			last_buffer_dequeued:1;
>+	unsigned int			queueing_started:1;
>+
>+	u64				out_fence_context;
>+	unsigned int			out_fence_seqno;
>+	spinlock_t			out_fence_lock;
>
> 	struct vb2_fileio_data		*fileio;
> 	struct vb2_threadio_data	*threadio;
>@@ -800,7 +815,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  * @index:	id number of the buffer
>  * @pb:		buffer structure passed from userspace to
>  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
>- * @in_fence:	in-fence to wait on before queueing the buffer
>+ * @in_fence:	optioanl in-fence to wait on before queueing the buffer
>+ * @out_fence_fd: optional pointer to store a newly created out-fence fd
>  *
>  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
>@@ -816,7 +832,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  * Return: returns zero on success; an error code otherwise.
>  */
> int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>-		  struct dma_fence *in_fence);
>+		  struct dma_fence *in_fence, int *out_fence_fd);
>
> /**
>  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
>-- 
>2.16.3
>

[PATCH v10 14/16] v4l: introduce the fences capability

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers capable of using fences (vb2 drivers) should report the
V4L2_CAP_FENCES to userspace, so add this flag to the uapi.

v2: minor doc/english fix (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst b/Documentation/media/uapi/v4l/vidioc-querycap.rst
index 66fb1b3d6e6e..df3ad57f07a3 100644
--- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
@@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
     * - ``V4L2_CAP_TOUCH``
       - 0x10000000
       - This is a touch device.
+    * - ``V4L2_CAP_FENCES``
+      - 0x20000000
+      - The device supports explicit synchronization.
     * - ``V4L2_CAP_DEVICE_CAPS``
       - 0x80000000
       - The driver fills the ``device_caps`` field. This capability can
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1f18dc68ecab..cab35fca7c7f 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -460,6 +460,7 @@ struct v4l2_capability {
 #define V4L2_CAP_STREAMING              0x04000000  /* streaming I/O ioctls */
 
 #define V4L2_CAP_TOUCH                  0x10000000  /* Is a touch device */
+#define V4L2_CAP_FENCES                 0x20000000  /* Supports explicit synchronization */
 
 #define V4L2_CAP_DEVICE_CAPS            0x80000000  /* sets device capabilities field */
 
-- 
2.16.3

[PATCH v10 15/16] v4l: Add V4L2_CAP_FENCES to drivers

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers that use videobuf2 are capable of using fences and
should report that to userspace.

v10: - Add CAPS_FENCES to drivers that don't use fh->m2m_ctx.
     - Keep the ifdef V4L2_MEM2MEM_DEV.
     - Set CAPS_FENCES after vidioc_querycap.
v9: Add in the core.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/platform/exynos-gsc/gsc-m2m.c        |  3 ++-
 drivers/media/platform/m2m-deinterlace.c           |  3 ++-
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       |  3 ++-
 .../media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c |  2 +-
 .../media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c |  3 ++-
 drivers/media/platform/mx2_emmaprp.c               |  3 ++-
 drivers/media/platform/qcom/venus/vdec.c           |  3 ++-
 drivers/media/platform/qcom/venus/venc.c           |  3 ++-
 drivers/media/platform/sh_veu.c                    |  3 ++-
 drivers/media/v4l2-core/v4l2-ioctl.c               | 24 +++++++++++++++++++++-
 10 files changed, 40 insertions(+), 10 deletions(-)

diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c b/drivers/media/platform/exynos-gsc/gsc-m2m.c
index e9ff27949a91..fbf072c8aedd 100644
--- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
+++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
@@ -298,7 +298,8 @@ static int gsc_m2m_querycap(struct file *file, void *fh,
 	strlcpy(cap->card, GSC_MODULE_NAME " gscaler", sizeof(cap->card));
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 		 dev_name(&gsc->pdev->dev));
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/m2m-deinterlace.c b/drivers/media/platform/m2m-deinterlace.c
index 1e4195144f39..b7421de1268f 100644
--- a/drivers/media/platform/m2m-deinterlace.c
+++ b/drivers/media/platform/m2m-deinterlace.c
@@ -461,7 +461,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	 * and are scheduled for removal.
 	 */
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT |
-			   V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+			   V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
index 583d47724ee8..7aba2ba128ba 100644
--- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
+++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
@@ -1242,7 +1242,8 @@ int mtk_mdp_register_m2m_device(struct mtk_mdp_dev *mdp)
 		ret = -ENOMEM;
 		goto err_video_alloc;
 	}
-	mdp->vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	mdp->vdev->device_caps = V4L2_CAP_FENCES | V4L2_CAP_VIDEO_M2M_MPLANE |
+				 V4L2_CAP_STREAMING;
 	mdp->vdev->fops = &mtk_mdp_m2m_fops;
 	mdp->vdev->ioctl_ops = &mtk_mdp_m2m_ioctl_ops;
 	mdp->vdev->release = video_device_release;
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c
index 4334b7394861..75318a4129ea 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec_drv.c
@@ -321,7 +321,7 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
 	vfd_dec->v4l2_dev	= &dev->v4l2_dev;
 	vfd_dec->vfl_dir	= VFL_DIR_M2M;
 	vfd_dec->device_caps	= V4L2_CAP_VIDEO_M2M_MPLANE |
-			V4L2_CAP_STREAMING;
+			V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 
 	snprintf(vfd_dec->name, sizeof(vfd_dec->name), "%s",
 		MTK_VCODEC_DEC_NAME);
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c
index 83f859e8509c..1c9d4e7262bb 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc_drv.c
@@ -339,7 +339,8 @@ static int mtk_vcodec_probe(struct platform_device *pdev)
 	vfd_enc->v4l2_dev       = &dev->v4l2_dev;
 	vfd_enc->vfl_dir        = VFL_DIR_M2M;
 	vfd_enc->device_caps	= V4L2_CAP_VIDEO_M2M_MPLANE |
-					V4L2_CAP_STREAMING;
+					V4L2_CAP_STREAMING |
+					V4L2_CAP_FENCES;
 
 	snprintf(vfd_enc->name, sizeof(vfd_enc->name), "%s",
 		 MTK_VCODEC_ENC_NAME);
diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c
index 5a8eff60e95f..230ad00fc62d 100644
--- a/drivers/media/platform/mx2_emmaprp.c
+++ b/drivers/media/platform/mx2_emmaprp.c
@@ -401,7 +401,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 {
 	strncpy(cap->driver, MEM2MEM_NAME, sizeof(cap->driver) - 1);
 	strncpy(cap->card, MEM2MEM_NAME, sizeof(cap->card) - 1);
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/qcom/venus/vdec.c b/drivers/media/platform/qcom/venus/vdec.c
index 8d7b4fc95880..9da748d05bd9 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -1099,7 +1099,8 @@ static int vdec_probe(struct platform_device *pdev)
 	vdev->ioctl_ops = &vdec_ioctl_ops;
 	vdev->vfl_dir = VFL_DIR_M2M;
 	vdev->v4l2_dev = &core->v4l2_dev;
-	vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING |
+			    V4L2_CAP_FENCES;
 
 	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
 	if (ret)
diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
index 713c79ba9639..41b25ef2d4dc 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -1243,7 +1243,8 @@ static int venc_probe(struct platform_device *pdev)
 	vdev->ioctl_ops = &venc_ioctl_ops;
 	vdev->vfl_dir = VFL_DIR_M2M;
 	vdev->v4l2_dev = &core->v4l2_dev;
-	vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	vdev->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING |
+			    V4L2_CAP_FENCES;
 
 	ret = video_register_device(vdev, VFL_TYPE_GRABBER, -1);
 	if (ret)
diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c
index 1a0cde017fdf..4ad7ec5a197a 100644
--- a/drivers/media/platform/sh_veu.c
+++ b/drivers/media/platform/sh_veu.c
@@ -351,7 +351,8 @@ static int sh_veu_querycap(struct file *file, void *priv,
 	strlcpy(cap->driver, "sh-veu", sizeof(cap->driver));
 	strlcpy(cap->card, "sh-mobile VEU", sizeof(cap->card));
 	strlcpy(cap->bus_info, "platform:sh-veu", sizeof(cap->bus_info));
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2135ac235a96..4c5e95a0fe6f 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -29,6 +29,7 @@
 #include <media/v4l2-device.h>
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mc.h>
+#include <media/v4l2-mem2mem.h>
 
 #include <trace/events/v4l2.h>
 
@@ -1002,6 +1003,8 @@ static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 {
 	struct v4l2_capability *cap = (struct v4l2_capability *)arg;
 	struct video_device *vfd = video_devdata(file);
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
 	int ret;
 
 	cap->version = LINUX_VERSION_CODE;
@@ -1010,7 +1013,26 @@ static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 
 	ret = ops->vidioc_querycap(file, fh, cap);
 
-	cap->capabilities |= V4L2_CAP_EXT_PIX_FORMAT;
+	/* If a streaming device has a queue or a m2m context,
+	 * then the device supports explicit sync.
+	 */
+	if (cap->device_caps & V4L2_CAP_STREAMING) {
+		/*
+		 * TODO: Drop these additional queue lock
+		 * checks once we make vb2_queue lock
+		 * mandatory.
+		 */
+		if (vfd->queue && vfd->queue->lock)
+			cap->device_caps |= V4L2_CAP_FENCES;
+#if IS_ENABLED(CONFIG_V4L2_MEM2MEM_DEV)
+		if (vfh && vfh->m2m_ctx &&
+		    vfh->m2m_ctx->cap_q_ctx.q.lock &&
+		    vfh->m2m_ctx->out_q_ctx.q.lock)
+			cap->device_caps |= V4L2_CAP_FENCES;
+#endif
+	}
+	cap->capabilities |= cap->device_caps | V4L2_CAP_EXT_PIX_FORMAT;
+
 	/*
 	 * Drivers MUST fill in device_caps, so check for this and
 	 * warn if it was forgotten.
-- 
2.16.3

[PATCH v10 16/16] v4l: Document explicit synchronization behavior

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it

v9: assorted improvements.

v8: amend querybuf documentation.

v7: minor issues and English improvements (Hans Verkuil)

v6: Close some gaps in the docs (Hans)

v5: - Remove V4L2_CAP_ORDERED
    - Add doc about V4L2_FMT_FLAG_UNORDERED

v4: - Document ordering behavior for in-fences
    - Document V4L2_CAP_ORDERED capability
    - Remove doc about OUT_FENCE event
    - Document immediate return of out-fence in QBUF

v3: - make the out_fence refer to the current buffer (Hans)
    - Note what happens when the IN_FENCE is not set (Hans)

v2: - mention that fences are files (Hans)
    - rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 53 +++++++++++++++++++++++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
 2 files changed, 61 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..f09e21247a6c 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,57 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+------------------------
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer. On the other
+side, the kernel can create out-fences for the buffers it queues to the
+drivers. Out-fences signal when the driver is finished with buffer, i.e., the
+buffer is ready. The fences are represented as a file and passed as a file
+descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with an error.
+
+It is guaranteed that all buffers queued with an in-fence will be queued to
+the drivers in the same order. Fences may signal out of order, so this guarantee
+is necessary to not change ordering. While waiting for a fence to signal,
+all buffers queued afterwards will also be blocked until that fence signals.
+
+If the in-fence signals with an error the buffer will be marked with
+``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
+Even with the error the order of dequeueing the buffers is preserved.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument in the `fence_fd` field.
+
+Note the same `fence_fd` field is used for both sending the in-fence as
+at input argument and to receive the out-fence as a return argument. A buffer can
+have both an in-fence and an out-fence.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. They will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that when using this format, the order in which buffers are dequeued can
+be different from the order in which they were queued.
+
+Ordering is important to fences because it can optimize the pipeline with
+other drivers like a DRM/KMS display driver. For example, if a capture from the
+camera is happening in an ordered manner one can send the capture buffer
+out-fence to the DRM/KMS driver and rest assured that the buffers will be shown on
+the screen in the correct order. If the queue is not ordered, then such
+arrangements with other drivers may not be possible.
 
 Return Value
 ============
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..cda73b43c334 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
+one. The ``reserved`` field must be set to 0. When
 using the :ref:`multi-planar API <planar-apis>`, the ``m.planes``
 field must contain a userspace pointer to an array of struct
 :c:type:`v4l2_plane` and the ``length`` field has to be set
@@ -64,8 +64,14 @@ elements will be used instead and the ``length`` field of struct
 array elements. The driver may or may not set the remaining fields and
 flags, they are meaningless in this context.
 
-The struct :c:type:`v4l2_buffer` structure is specified in
-:ref:`buffer`.
+When using in-fences, the ``V4L2_BUF_FLAG_IN_FENCE`` will be set if the
+in-fence didn't signal at the time of the
+:ref:`VIDIOC_QUERYBUF`. Similarly, the ``V4L2_BUF_FLAG_OUT_FENCE`` will be
+set if there's a pending out-fence for the buffer. Note that the ``fence_fd``
+field is not set, because the file descriptor only makes sense
+for the process that received the out-fence in the :ref:`VIDIOC_QBUF` response.
+
+The struct :c:type:`v4l2_buffer` structure is specified in :ref:`buffer`.
 
 
 Return Value
-- 
2.16.3