[PATCH v9 00/15] V4L2 Explicit Synchronization

[PATCH v9 00/15] V4L2 Explicit Synchronization

[Ezequiel Garcia]

Hi all,

Gustavo has asked to me to take care of the final
issues with this series.

I'm working on adding some fences tests to v4l2-compliance,
which I'll be posting shortly.

So, here's a new version of the "video4linux meet fences"
series. This new round hopefully addresses all the feedback
received in v8.

There are some additional changes:

 * Removed the vb2_ops_is_unordered callback,
   and instead use a simpler unordered bitfield.
   After inspecting the code, I really saw no reason
   for having a callback. Please correct me if I missed
   anything.

 * Reworked the out-fence setup in vb2_core_qbuf,
   which results in getting rid of the sync_file
   state. After inspecting the code, it became apparent
   that the sync_file wasn't meant to be part of the
   buffer state. The resulting code is simpler.

 * Avoid returning fence file descriptors anywhere
   but in the QBUF result. Also, fixed the documentation
   to be clear about this.

Worth mentioning, I've decided to drop the is-signaled
flag suggested by Hans. It'll be easier to review and
merge if we keep this simple. We can always add stuff
later.

Gustavo Padovan (15):
  xilinx: regroup caps on querycap
  hackrf: group device capabilities
  omap3isp: group device capabilities
  vb2: move vb2_ops functions to videobuf2-core.[ch]
  vb2: add unordered vb2_queue property for drivers
  v4l: add unordered flag to format description ioctl
  v4l: mark unordered formats
  cobalt: set queue as unordered
  vb2: mark codec drivers as unordered
  vb2: add explicit fence user API
  vb2: add in-fence support to QBUF
  vb2: add out-fence support to QBUF
  v4l: introduce the fences capability
  v4l: Add V4L2_CAP_FENCES to drivers
  v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst            |  45 +++-
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst   |   7 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  54 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst   |  12 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst   |   3 +
 drivers/media/common/videobuf2/videobuf2-core.c    | 298 +++++++++++++++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c    |  66 +++--
 drivers/media/dvb-core/dvb_vb2.c                   |   2 +-
 drivers/media/pci/cobalt/cobalt-v4l2.c             |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   2 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   1 +
 drivers/media/platform/omap3isp/ispvideo.c         |  10 +-
 drivers/media/platform/qcom/venus/venc.c           |   2 +
 drivers/media/platform/s5p-mfc/s5p_mfc.c           |   2 +
 drivers/media/platform/xilinx/xilinx-dma.c         |  10 +-
 drivers/media/usb/hackrf/hackrf.c                  |  11 +-
 drivers/media/v4l2-core/Kconfig                    |  33 +++
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |   4 +-
 drivers/media/v4l2-core/v4l2-ioctl.c               |  63 +++--
 include/media/v4l2-fh.h                            |   2 -
 include/media/videobuf2-core.h                     |  49 +++-
 include/media/videobuf2-v4l2.h                     |  18 --
 include/uapi/linux/videodev2.h                     |  10 +-
 23 files changed, 590 insertions(+), 115 deletions(-)

-- 
2.16.3

[PATCH v9 01/15] xilinx: regroup caps on querycap

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

To better organize the code we concentrate the setting of
V4L2_CAP_STREAMING in one place.

v2: move cap->capabilities assignment down (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 522cdfdd3345..d041f94be832 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,13 +494,15 @@ xvip_dma_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	struct v4l2_fh *vfh = file->private_data;
 	struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
-			  | dma->xdev->v4l2_caps;
+	cap->device_caps = V4L2_CAP_STREAMING;
 
 	if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
+
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
+			  | dma->xdev->v4l2_caps;
 
 	strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
 	strlcpy(cap->card, dma->video.name, sizeof(cap->card));
-- 
2.16.3

[PATCH v9 02/15] hackrf: group device capabilities

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING and V4L2_CAP_READWRITE
everywhere, set device_caps earlier with these values.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/usb/hackrf/hackrf.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/media/usb/hackrf/hackrf.c b/drivers/media/usb/hackrf/hackrf.c
index 7eb53517a82f..6d692fb3e8dd 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,18 +909,15 @@ static int hackrf_querycap(struct file *file, void *fh,
 
 	dev_dbg(&intf->dev, "\n");
 
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
 	if (vdev->vfl_dir == VFL_DIR_RX)
-		cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
-
+		cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
 	else
-		cap->device_caps = V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+		cap->device_caps |= V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR;
 
 	cap->capabilities = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
 			    V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-			    V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
-			    V4L2_CAP_DEVICE_CAPS;
+			    V4L2_CAP_DEVICE_CAPS | cap->device_caps;
 	strlcpy(cap->driver, KBUILD_MODNAME, sizeof(cap->driver));
 	strlcpy(cap->card, dev->rx_vdev.name, sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
-- 
2.16.3

[PATCH v9 03/15] omap3isp: group device capabilities

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
earlier with this value.

v2: move cap->capabilities assignment down (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/omap3isp/ispvideo.c | 10 ++++++----
 1 file changed, 6 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..db9aae222134 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,13 +658,15 @@ isp_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	strlcpy(cap->card, video->video.name, sizeof(cap->card));
 	strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-	cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
-		| V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+	cap->device_caps = V4L2_CAP_STREAMING;
 
 	if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
+
+	cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
+		V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
 }
-- 
2.16.3

[PATCH v9 04/15] vb2: move vb2_ops functions to videobuf2-core.[ch]

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

vb2_ops_wait_prepare() and vb2_ops_wait_finish() were in the
wrong file.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 14 ++++++++++++++
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 14 --------------
 include/media/videobuf2-core.h                  | 18 ++++++++++++++++++
 include/media/videobuf2-v4l2.h                  | 18 ------------------
 4 files changed, 32 insertions(+), 32 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..6b8e083893ad 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -656,6 +656,20 @@ int vb2_verify_memory_type(struct vb2_queue *q,
 }
 EXPORT_SYMBOL(vb2_verify_memory_type);
 
+/* vb2_ops helpers. Only use if vq->lock is non-NULL. */
+
+void vb2_ops_wait_prepare(struct vb2_queue *vq)
+{
+	mutex_unlock(vq->lock);
+}
+EXPORT_SYMBOL_GPL(vb2_ops_wait_prepare);
+
+void vb2_ops_wait_finish(struct vb2_queue *vq)
+{
+	mutex_lock(vq->lock);
+}
+EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
+
 int vb2_core_reqbufs(struct vb2_queue *q, enum vb2_memory memory,
 		unsigned int *count)
 {
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..64503615d00b 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -947,20 +947,6 @@ unsigned long vb2_fop_get_unmapped_area(struct file *file, unsigned long addr,
 EXPORT_SYMBOL_GPL(vb2_fop_get_unmapped_area);
 #endif
 
-/* vb2_ops helpers. Only use if vq->lock is non-NULL. */
-
-void vb2_ops_wait_prepare(struct vb2_queue *vq)
-{
-	mutex_unlock(vq->lock);
-}
-EXPORT_SYMBOL_GPL(vb2_ops_wait_prepare);
-
-void vb2_ops_wait_finish(struct vb2_queue *vq)
-{
-	mutex_lock(vq->lock);
-}
-EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
-
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak <pawel@osciak.com>, Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f6818f732f34..f9633de0386c 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -400,6 +400,24 @@ struct vb2_ops {
 	void (*buf_queue)(struct vb2_buffer *vb);
 };
 
+/**
+ * vb2_ops_wait_prepare - helper function to lock a struct &vb2_queue
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * ..note:: only use if vq->lock is non-NULL.
+ */
+void vb2_ops_wait_prepare(struct vb2_queue *vq);
+
+/**
+ * vb2_ops_wait_finish - helper function to unlock a struct &vb2_queue
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * ..note:: only use if vq->lock is non-NULL.
+ */
+void vb2_ops_wait_finish(struct vb2_queue *vq);
+
 /**
  * struct vb2_buf_ops - driver-specific callbacks.
  *
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..cf83b01dc44e 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -273,22 +273,4 @@ unsigned long vb2_fop_get_unmapped_area(struct file *file, unsigned long addr,
 		unsigned long len, unsigned long pgoff, unsigned long flags);
 #endif
 
-/**
- * vb2_ops_wait_prepare - helper function to lock a struct &vb2_queue
- *
- * @vq: pointer to &struct vb2_queue
- *
- * ..note:: only use if vq->lock is non-NULL.
- */
-void vb2_ops_wait_prepare(struct vb2_queue *vq);
-
-/**
- * vb2_ops_wait_finish - helper function to unlock a struct &vb2_queue
- *
- * @vq: pointer to &struct vb2_queue
- *
- * ..note:: only use if vq->lock is non-NULL.
- */
-void vb2_ops_wait_finish(struct vb2_queue *vq);
-
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.16.3

[PATCH v9 05/15] vb2: add unordered vb2_queue property for drivers

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

v5: go back to a bitfield property for the unordered property.

v4: rename it to vb2_ops_is_unordered() (Hans Verkuil)

v3: - make it bool (Hans)
    - create vb2_ops_set_unordered() helper

v2: improve comments for is_unordered flag (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 include/media/videobuf2-core.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f9633de0386c..364e4cb41b10 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -467,6 +467,8 @@ struct vb2_buf_ops {
  * @quirk_poll_must_check_waiting_for_buffers: Return %EPOLLERR at poll when QBUF
  *              has not been called. This is a vb1 idiom that has been adopted
  *              also by vb2.
+ * @unordered:	tell if the queue is unordered, i.e. buffers can be
+ *		dequeued in a different order from how they were queued.
  * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
  *		driver can set this to a mutex to let the v4l2 core serialize
  *		the queuing ioctls. If the driver wants to handle locking
@@ -533,6 +535,7 @@ struct vb2_queue {
 	unsigned			fileio_read_once:1;
 	unsigned			fileio_write_immediately:1;
 	unsigned			allow_zero_bytesused:1;
+	unsigned			unordered:1;
 	unsigned		   quirk_poll_must_check_waiting_for_buffers:1;
 
 	struct mutex			*lock;
-- 
2.16.3

Re: [PATCH v9 05/15] vb2: add unordered vb2_queue property for drivers

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Explicit synchronization benefits a lot from ordered queues, they fit
> better in a pipeline with DRM for example so create a opt-in way for
> drivers notify videobuf2 that the queue is unordered.
> 
> v5: go back to a bitfield property for the unordered property.

This needs to be a callback. Whether or not the queue is unordered depends
on the current format (see the next two patches). So if a queue can e.g.
capture both raw/MJPEG and compressed (MPEG-like) data, then it has to call
a callback.

Regards,

	Hans

> 
> v4: rename it to vb2_ops_is_unordered() (Hans Verkuil)
> 
> v3: - make it bool (Hans)
>     - create vb2_ops_set_unordered() helper
> 
> v2: improve comments for is_unordered flag (Hans Verkuil)
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  include/media/videobuf2-core.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index f9633de0386c..364e4cb41b10 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -467,6 +467,8 @@ struct vb2_buf_ops {
>   * @quirk_poll_must_check_waiting_for_buffers: Return %EPOLLERR at poll when QBUF
>   *              has not been called. This is a vb1 idiom that has been adopted
>   *              also by vb2.
> + * @unordered:	tell if the queue is unordered, i.e. buffers can be
> + *		dequeued in a different order from how they were queued.
>   * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
>   *		driver can set this to a mutex to let the v4l2 core serialize
>   *		the queuing ioctls. If the driver wants to handle locking
> @@ -533,6 +535,7 @@ struct vb2_queue {
>  	unsigned			fileio_read_once:1;
>  	unsigned			fileio_write_immediately:1;
>  	unsigned			allow_zero_bytesused:1;
> +	unsigned			unordered:1;
>  	unsigned		   quirk_poll_must_check_waiting_for_buffers:1;
>  
>  	struct mutex			*lock;
> 

[PATCH v9 06/15] v4l: add unordered flag to format description ioctl

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

v2: Improve documentation (Hans)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 7 +++++++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 8 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..df8e039b9ac2 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,13 @@ one until ``EINVAL`` is returned.
       - This format is not native to the device but emulated through
 	software (usually libv4l2), where possible try to use a native
 	format instead for better performance.
+    * - ``V4L2_FMT_FLAG_UNORDERED``
+      - 0x0004
+      - This format doesn't guarantee ordered buffer handling. I.e. the order
+	in which buffers are dequeued with
+	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` may be different
+	from the order in which they were queued with
+	:ref:`VIDIOC_QBUF <VIDIOC_QBUF>`.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 600877be5c22..a8842a5ca636 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -717,6 +717,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
 	/* Frame Size and frame rate enumeration */
 /*
-- 
2.16.3

[PATCH v9 07/15] v4l: mark unordered formats

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
mark the appropriate formats.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 55 ++++++++++++++++++++----------------
 1 file changed, 30 insertions(+), 25 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f48c505550e0..f75ad954a6f2 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1260,20 +1260,6 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_MJPEG:	descr = "Motion-JPEG"; break;
 		case V4L2_PIX_FMT_JPEG:		descr = "JFIF JPEG"; break;
 		case V4L2_PIX_FMT_DV:		descr = "1394"; break;
-		case V4L2_PIX_FMT_MPEG:		descr = "MPEG-1/2/4"; break;
-		case V4L2_PIX_FMT_H264:		descr = "H.264"; break;
-		case V4L2_PIX_FMT_H264_NO_SC:	descr = "H.264 (No Start Codes)"; break;
-		case V4L2_PIX_FMT_H264_MVC:	descr = "H.264 MVC"; break;
-		case V4L2_PIX_FMT_H263:		descr = "H.263"; break;
-		case V4L2_PIX_FMT_MPEG1:	descr = "MPEG-1 ES"; break;
-		case V4L2_PIX_FMT_MPEG2:	descr = "MPEG-2 ES"; break;
-		case V4L2_PIX_FMT_MPEG4:	descr = "MPEG-4 part 2 ES"; break;
-		case V4L2_PIX_FMT_XVID:		descr = "Xvid"; break;
-		case V4L2_PIX_FMT_VC1_ANNEX_G:	descr = "VC-1 (SMPTE 412M Annex G)"; break;
-		case V4L2_PIX_FMT_VC1_ANNEX_L:	descr = "VC-1 (SMPTE 412M Annex L)"; break;
-		case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
-		case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
-		case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
 		case V4L2_PIX_FMT_CPIA1:	descr = "GSPCA CPiA YUV"; break;
 		case V4L2_PIX_FMT_WNVA:		descr = "WNVA"; break;
 		case V4L2_PIX_FMT_SN9C10X:	descr = "GSPCA SN9C10X"; break;
@@ -1294,17 +1280,36 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
 		case V4L2_PIX_FMT_S5C_UYVY_JPG:	descr = "S5C73MX interleaved UYVY/JPEG"; break;
 		case V4L2_PIX_FMT_MT21C:	descr = "Mediatek Compressed Format"; break;
 		default:
-			WARN(1, "Unknown pixelformat 0x%08x\n", fmt->pixelformat);
-			if (fmt->description[0])
-				return;
-			flags = 0;
-			snprintf(fmt->description, sz, "%c%c%c%c%s",
-					(char)(fmt->pixelformat & 0x7f),
-					(char)((fmt->pixelformat >> 8) & 0x7f),
-					(char)((fmt->pixelformat >> 16) & 0x7f),
-					(char)((fmt->pixelformat >> 24) & 0x7f),
-					(fmt->pixelformat & (1 << 31)) ? "-BE" : "");
-			break;
+			/* Unordered formats */
+			flags = V4L2_FMT_FLAG_UNORDERED;
+			switch (fmt->pixelformat) {
+			case V4L2_PIX_FMT_MPEG:		descr = "MPEG-1/2/4"; break;
+			case V4L2_PIX_FMT_H264:		descr = "H.264"; break;
+			case V4L2_PIX_FMT_H264_NO_SC:	descr = "H.264 (No Start Codes)"; break;
+			case V4L2_PIX_FMT_H264_MVC:	descr = "H.264 MVC"; break;
+			case V4L2_PIX_FMT_H263:		descr = "H.263"; break;
+			case V4L2_PIX_FMT_MPEG1:	descr = "MPEG-1 ES"; break;
+			case V4L2_PIX_FMT_MPEG2:	descr = "MPEG-2 ES"; break;
+			case V4L2_PIX_FMT_MPEG4:	descr = "MPEG-4 part 2 ES"; break;
+			case V4L2_PIX_FMT_XVID:		descr = "Xvid"; break;
+			case V4L2_PIX_FMT_VC1_ANNEX_G:	descr = "VC-1 (SMPTE 412M Annex G)"; break;
+			case V4L2_PIX_FMT_VC1_ANNEX_L:	descr = "VC-1 (SMPTE 412M Annex L)"; break;
+			case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
+			case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
+			case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
+			default:
+				WARN(1, "Unknown pixelformat 0x%08x\n", fmt->pixelformat);
+				if (fmt->description[0])
+					return;
+				flags = 0;
+				snprintf(fmt->description, sz, "%c%c%c%c%s",
+						(char)(fmt->pixelformat & 0x7f),
+						(char)((fmt->pixelformat >> 8) & 0x7f),
+						(char)((fmt->pixelformat >> 16) & 0x7f),
+						(char)((fmt->pixelformat >> 24) & 0x7f),
+						(fmt->pixelformat & (1 << 31)) ? "-BE" : "");
+				break;
+			}
 		}
 	}
 
-- 
2.16.3

Re: [PATCH v9 07/15] v4l: mark unordered formats

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Now that we've introduced the V4L2_FMT_FLAG_UNORDERED flag,
> mark the appropriate formats.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 55 ++++++++++++++++++++----------------
>  1 file changed, 30 insertions(+), 25 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f48c505550e0..f75ad954a6f2 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1260,20 +1260,6 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  		case V4L2_PIX_FMT_MJPEG:	descr = "Motion-JPEG"; break;
>  		case V4L2_PIX_FMT_JPEG:		descr = "JFIF JPEG"; break;
>  		case V4L2_PIX_FMT_DV:		descr = "1394"; break;
> -		case V4L2_PIX_FMT_MPEG:		descr = "MPEG-1/2/4"; break;
> -		case V4L2_PIX_FMT_H264:		descr = "H.264"; break;
> -		case V4L2_PIX_FMT_H264_NO_SC:	descr = "H.264 (No Start Codes)"; break;
> -		case V4L2_PIX_FMT_H264_MVC:	descr = "H.264 MVC"; break;
> -		case V4L2_PIX_FMT_H263:		descr = "H.263"; break;
> -		case V4L2_PIX_FMT_MPEG1:	descr = "MPEG-1 ES"; break;
> -		case V4L2_PIX_FMT_MPEG2:	descr = "MPEG-2 ES"; break;
> -		case V4L2_PIX_FMT_MPEG4:	descr = "MPEG-4 part 2 ES"; break;
> -		case V4L2_PIX_FMT_XVID:		descr = "Xvid"; break;
> -		case V4L2_PIX_FMT_VC1_ANNEX_G:	descr = "VC-1 (SMPTE 412M Annex G)"; break;
> -		case V4L2_PIX_FMT_VC1_ANNEX_L:	descr = "VC-1 (SMPTE 412M Annex L)"; break;
> -		case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
> -		case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
> -		case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
>  		case V4L2_PIX_FMT_CPIA1:	descr = "GSPCA CPiA YUV"; break;
>  		case V4L2_PIX_FMT_WNVA:		descr = "WNVA"; break;
>  		case V4L2_PIX_FMT_SN9C10X:	descr = "GSPCA SN9C10X"; break;
> @@ -1294,17 +1280,36 @@ static void v4l_fill_fmtdesc(struct v4l2_fmtdesc *fmt)
>  		case V4L2_PIX_FMT_S5C_UYVY_JPG:	descr = "S5C73MX interleaved UYVY/JPEG"; break;
>  		case V4L2_PIX_FMT_MT21C:	descr = "Mediatek Compressed Format"; break;
>  		default:
> -			WARN(1, "Unknown pixelformat 0x%08x\n", fmt->pixelformat);
> -			if (fmt->description[0])
> -				return;
> -			flags = 0;
> -			snprintf(fmt->description, sz, "%c%c%c%c%s",
> -					(char)(fmt->pixelformat & 0x7f),
> -					(char)((fmt->pixelformat >> 8) & 0x7f),
> -					(char)((fmt->pixelformat >> 16) & 0x7f),
> -					(char)((fmt->pixelformat >> 24) & 0x7f),
> -					(fmt->pixelformat & (1 << 31)) ? "-BE" : "");
> -			break;
> +			/* Unordered formats */
> +			flags = V4L2_FMT_FLAG_UNORDERED;

I realized that this is a problem since this function is called *after*
the driver. So the driver has no chance to clear this flag if it knows
that the queue is always ordered.

I think this needs to be split: first set the UNORDERED flag for the selected
formats, then call the driver, then fill in the rest.

Note that this function sets fmt->flags, this should become |=. Otherwise
the UNORDERED flag would be overwritten.

It's a bit messy, but I don't see a better approach. Except by setting the
UNORDERED flag in the drivers, but I prefer this more defensive approach
(i.e. presumed unordered, unless stated otherwise).

Regards,

	Hans

> +			switch (fmt->pixelformat) {
> +			case V4L2_PIX_FMT_MPEG:		descr = "MPEG-1/2/4"; break;
> +			case V4L2_PIX_FMT_H264:		descr = "H.264"; break;
> +			case V4L2_PIX_FMT_H264_NO_SC:	descr = "H.264 (No Start Codes)"; break;
> +			case V4L2_PIX_FMT_H264_MVC:	descr = "H.264 MVC"; break;
> +			case V4L2_PIX_FMT_H263:		descr = "H.263"; break;
> +			case V4L2_PIX_FMT_MPEG1:	descr = "MPEG-1 ES"; break;
> +			case V4L2_PIX_FMT_MPEG2:	descr = "MPEG-2 ES"; break;
> +			case V4L2_PIX_FMT_MPEG4:	descr = "MPEG-4 part 2 ES"; break;
> +			case V4L2_PIX_FMT_XVID:		descr = "Xvid"; break;
> +			case V4L2_PIX_FMT_VC1_ANNEX_G:	descr = "VC-1 (SMPTE 412M Annex G)"; break;
> +			case V4L2_PIX_FMT_VC1_ANNEX_L:	descr = "VC-1 (SMPTE 412M Annex L)"; break;
> +			case V4L2_PIX_FMT_VP8:		descr = "VP8"; break;
> +			case V4L2_PIX_FMT_VP9:		descr = "VP9"; break;
> +			case V4L2_PIX_FMT_HEVC:		descr = "HEVC"; break; /* aka H.265 */
> +			default:
> +				WARN(1, "Unknown pixelformat 0x%08x\n", fmt->pixelformat);
> +				if (fmt->description[0])
> +					return;
> +				flags = 0;
> +				snprintf(fmt->description, sz, "%c%c%c%c%s",
> +						(char)(fmt->pixelformat & 0x7f),
> +						(char)((fmt->pixelformat >> 8) & 0x7f),
> +						(char)((fmt->pixelformat >> 16) & 0x7f),
> +						(char)((fmt->pixelformat >> 24) & 0x7f),
> +						(fmt->pixelformat & (1 << 31)) ? "-BE" : "");
> +				break;
> +			}
>  		}
>  	}
>  
> 

[PATCH v9 08/15] cobalt: set queue as unordered

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

The cobalt driver may reorder the capture buffers so we need to report
it as such.

v3: set unordered as a property
v2: use vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
index e2a4c705d353..8f06cc7f1c81 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -1236,6 +1236,7 @@ static int cobalt_node_register(struct cobalt *cobalt, int node)
 	q->min_buffers_needed = 2;
 	q->lock = &s->lock;
 	q->dev = &cobalt->pci_dev->dev;
+	q->unordered = 1;
 	vdev->queue = q;
 
 	video_set_drvdata(vdev, s);
-- 
2.16.3

Re: [PATCH v9 08/15] cobalt: set queue as unordered

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> The cobalt driver may reorder the capture buffers so we need to report
> it as such.
> 
> v3: set unordered as a property
> v2: use vb2_ops_set_unordered() helper
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
> index e2a4c705d353..8f06cc7f1c81 100644
> --- a/drivers/media/pci/cobalt/cobalt-v4l2.c
> +++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
> @@ -1236,6 +1236,7 @@ static int cobalt_node_register(struct cobalt *cobalt, int node)
>  	q->min_buffers_needed = 2;
>  	q->lock = &s->lock;
>  	q->dev = &cobalt->pci_dev->dev;
> +	q->unordered = 1;
>  	vdev->queue = q;
>  
>  	video_set_drvdata(vdev, s);
> 

As mentioned in my review of v8 of this patch, you also need to mark all formats
in this driver as unordered.

Regards,

	Hans

[PATCH v9 09/15] vb2: mark codec drivers as unordered

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

In preparation to have full support to explicit fence we are
marking codec as non-ordered preventively. It is easier and safer from an
uAPI point of view to move from unordered to ordered than the opposite.

v3: set property instead of callback
v2: mark only codec drivers as unordered (Nicolas and Hans)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 2 ++
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
 drivers/media/platform/qcom/venus/venc.c           | 2 ++
 drivers/media/platform/s5p-mfc/s5p_mfc.c           | 2 ++
 4 files changed, 7 insertions(+)

diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
index 86f0a7134365..c03cefde0c28 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
@@ -1498,6 +1498,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq,
 	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
 	src_vq->lock		= &ctx->dev->dev_mutex;
 	src_vq->dev             = &ctx->dev->plat_dev->dev;
+	src_vq->unordered       = 1;
 
 	ret = vb2_queue_init(src_vq);
 	if (ret) {
@@ -1513,6 +1514,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq,
 	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
 	dst_vq->lock		= &ctx->dev->dev_mutex;
 	dst_vq->dev             = &ctx->dev->plat_dev->dev;
+	src_vq->unordered       = 1;
 
 	ret = vb2_queue_init(dst_vq);
 	if (ret) {
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
index 1b1a28abbf1f..81751c9aa19d 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
@@ -1340,6 +1340,7 @@ int mtk_vcodec_enc_queue_init(void *priv, struct vb2_queue *src_vq,
 	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
 	dst_vq->lock		= &ctx->dev->dev_mutex;
 	dst_vq->dev		= &ctx->dev->plat_dev->dev;
+	dst_vq->unordered	= 1;
 
 	return vb2_queue_init(dst_vq);
 }
diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
index 6b2ce479584e..17ec2d218aa5 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -1053,6 +1053,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue *src_vq,
 	src_vq->buf_struct_size = sizeof(struct venus_buffer);
 	src_vq->allow_zero_bytesused = 1;
 	src_vq->min_buffers_needed = 1;
+	src_vq->unordered = 1;
 	src_vq->dev = inst->core->dev;
 	if (inst->core->res->hfi_version == HFI_VERSION_1XX)
 		src_vq->bidirectional = 1;
@@ -1069,6 +1070,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue *src_vq,
 	dst_vq->buf_struct_size = sizeof(struct venus_buffer);
 	dst_vq->allow_zero_bytesused = 1;
 	dst_vq->min_buffers_needed = 1;
+	src_vq->unordered = 1;
 	dst_vq->dev = inst->core->dev;
 	ret = vb2_queue_init(dst_vq);
 	if (ret) {
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c
index a80251ed3143..6a4251f988ab 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c
@@ -863,6 +863,7 @@ static int s5p_mfc_open(struct file *file)
 	q->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES;
 	q->mem_ops = &vb2_dma_contig_memops;
 	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+	q->unordered = 1;
 	ret = vb2_queue_init(q);
 	if (ret) {
 		mfc_err("Failed to initialize videobuf2 queue(capture)\n");
@@ -898,6 +899,7 @@ static int s5p_mfc_open(struct file *file)
 	q->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES;
 	q->mem_ops = &vb2_dma_contig_memops;
 	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
+	q->unordered = 1;
 	ret = vb2_queue_init(q);
 	if (ret) {
 		mfc_err("Failed to initialize videobuf2 queue(output)\n");
-- 
2.16.3

Re: [PATCH v9 09/15] vb2: mark codec drivers as unordered

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> In preparation to have full support to explicit fence we are
> marking codec as non-ordered preventively. It is easier and safer from an
> uAPI point of view to move from unordered to ordered than the opposite.
> 
> v3: set property instead of callback
> v2: mark only codec drivers as unordered (Nicolas and Hans)
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 2 ++
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
>  drivers/media/platform/qcom/venus/venc.c           | 2 ++
>  drivers/media/platform/s5p-mfc/s5p_mfc.c           | 2 ++

Why were coda, exynos-gsc, exynos4-is and venus/vdec.c removed?
Those were patched in v8.

Regards,

	Hans

>  4 files changed, 7 insertions(+)
> 
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> index 86f0a7134365..c03cefde0c28 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> @@ -1498,6 +1498,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq,
>  	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
>  	src_vq->lock		= &ctx->dev->dev_mutex;
>  	src_vq->dev             = &ctx->dev->plat_dev->dev;
> +	src_vq->unordered       = 1;
>  
>  	ret = vb2_queue_init(src_vq);
>  	if (ret) {
> @@ -1513,6 +1514,7 @@ int mtk_vcodec_dec_queue_init(void *priv, struct vb2_queue *src_vq,
>  	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
>  	dst_vq->lock		= &ctx->dev->dev_mutex;
>  	dst_vq->dev             = &ctx->dev->plat_dev->dev;
> +	src_vq->unordered       = 1;
>  
>  	ret = vb2_queue_init(dst_vq);
>  	if (ret) {
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> index 1b1a28abbf1f..81751c9aa19d 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> @@ -1340,6 +1340,7 @@ int mtk_vcodec_enc_queue_init(void *priv, struct vb2_queue *src_vq,
>  	dst_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
>  	dst_vq->lock		= &ctx->dev->dev_mutex;
>  	dst_vq->dev		= &ctx->dev->plat_dev->dev;
> +	dst_vq->unordered	= 1;
>  
>  	return vb2_queue_init(dst_vq);
>  }
> diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
> index 6b2ce479584e..17ec2d218aa5 100644
> --- a/drivers/media/platform/qcom/venus/venc.c
> +++ b/drivers/media/platform/qcom/venus/venc.c
> @@ -1053,6 +1053,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue *src_vq,
>  	src_vq->buf_struct_size = sizeof(struct venus_buffer);
>  	src_vq->allow_zero_bytesused = 1;
>  	src_vq->min_buffers_needed = 1;
> +	src_vq->unordered = 1;
>  	src_vq->dev = inst->core->dev;
>  	if (inst->core->res->hfi_version == HFI_VERSION_1XX)
>  		src_vq->bidirectional = 1;
> @@ -1069,6 +1070,7 @@ static int m2m_queue_init(void *priv, struct vb2_queue *src_vq,
>  	dst_vq->buf_struct_size = sizeof(struct venus_buffer);
>  	dst_vq->allow_zero_bytesused = 1;
>  	dst_vq->min_buffers_needed = 1;
> +	src_vq->unordered = 1;
>  	dst_vq->dev = inst->core->dev;
>  	ret = vb2_queue_init(dst_vq);
>  	if (ret) {
> diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc.c b/drivers/media/platform/s5p-mfc/s5p_mfc.c
> index a80251ed3143..6a4251f988ab 100644
> --- a/drivers/media/platform/s5p-mfc/s5p_mfc.c
> +++ b/drivers/media/platform/s5p-mfc/s5p_mfc.c
> @@ -863,6 +863,7 @@ static int s5p_mfc_open(struct file *file)
>  	q->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES;
>  	q->mem_ops = &vb2_dma_contig_memops;
>  	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> +	q->unordered = 1;
>  	ret = vb2_queue_init(q);
>  	if (ret) {
>  		mfc_err("Failed to initialize videobuf2 queue(capture)\n");
> @@ -898,6 +899,7 @@ static int s5p_mfc_open(struct file *file)
>  	q->dma_attrs = DMA_ATTR_ALLOC_SINGLE_PAGES;
>  	q->mem_ops = &vb2_dma_contig_memops;
>  	q->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
> +	q->unordered = 1;
>  	ret = vb2_queue_init(q);
>  	if (ret) {
>  		mfc_err("Failed to initialize videobuf2 queue(output)\n");
> 

[PATCH v9 10/15] vb2: add explicit fence user API

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel or return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending an in-fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v7: minor fixes on the Documentation (Hans Verkuil)

v6: big improvement on doc (Hans Verkuil)

v5: - keep using reserved2 field for cpia2
    - set fence_fd to 0 for now, for compat with userspace(Mauro)

v4: make it a union with reserved2 and fence_fd (Hans Verkuil)

v3: make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
 include/uapi/linux/videodev2.h                  |  8 ++++-
 4 files changed, 52 insertions(+), 7 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..be9719cf5745 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,22 @@ struct v4l2_buffer
 	elements in the ``planes`` array. The driver will fill in the
 	actual number of valid elements in that array.
     * - __u32
-      - ``reserved2``
+      - ``fence_fd``
       -
-      - A place holder for future extensions. Drivers and applications
-	must set this to 0.
+      - Used to communicate a fence file descriptors from userspace to kernel
+	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
+	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
+	be used and this field set to the fence file descriptor of the in-fence.
+	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
+
+        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
+	must be set, the kernel will return the out-fence file descriptor in
+	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
+        an error.
+
+	For all other ioctls V4L2 sets this field to -1 if
+	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
+	otherwise this field is set to 0 for backward compatibility.
     * - __u32
       - ``reserved``
       -
@@ -648,6 +660,33 @@ Buffer Flags
       - Start Of Exposure. The buffer timestamp has been taken when the
 	exposure of the frame has begun. This is only valid for the
 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+      - ``V4L2_BUF_FLAG_IN_FENCE``
+      - 0x00200000
+      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
+	buffer won't be queued to the driver until the fence signals. The order
+	in which buffers are queued is guaranteed to be preserved, so any
+	buffers queued after this buffer will also be blocked until this fence
+	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
+	other ioctls the driver just reports the value of the flag.
+
+        If the fence signals the flag is cleared and not reported anymore.
+	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
+
+
+    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+      - ``V4L2_BUF_FLAG_OUT_FENCE``
+      - 0x00400000
+      - Request for a fence to be attached to the buffer. The driver will fill
+	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
+	<VIDIOC_QBUF>` returns. This flag must be set before calling
+	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
+	the flag.
+
+        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
+	error.
 
 
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 64503615d00b..b1c0fa2b0b88 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
+	b->fence_fd = 0;
 	b->reserved = 0;
 
 	if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 4312935f1dfc..93c752459aec 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -388,7 +388,7 @@ struct v4l2_buffer32 {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__s32			fence_fd;
 	__u32			reserved;
 };
 
@@ -606,7 +606,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
 	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
 	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
 	    assign_in_user(&up->sequence, &kp->sequence) ||
-	    assign_in_user(&up->reserved2, &kp->reserved2) ||
+	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
 	    assign_in_user(&up->reserved, &kp->reserved) ||
 	    get_user(length, &kp->length) ||
 	    put_user(length, &up->length))
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index a8842a5ca636..1f18dc68ecab 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -934,7 +934,10 @@ struct v4l2_buffer {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	union {
+		__s32		fence_fd;
+		__u32		reserved2;
+	};
 	__u32			reserved;
 };
 
@@ -971,6 +974,9 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+/* Explicit synchronization */
+#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
+#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.16.3

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Turn the reserved2 field into fence_fd that we will use to send
> an in-fence to the kernel or return an out-fence from the kernel to
> userspace.
> 
> Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> when sending an in-fence to the kernel to be waited on, and
> V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> 
> v7: minor fixes on the Documentation (Hans Verkuil)
> 
> v6: big improvement on doc (Hans Verkuil)
> 
> v5: - keep using reserved2 field for cpia2
>     - set fence_fd to 0 for now, for compat with userspace(Mauro)
> 
> v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> 
> v3: make the out_fence refer to the current buffer (Hans Verkuil)
> 
> v2: add documentation
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
>  include/uapi/linux/videodev2.h                  |  8 ++++-
>  4 files changed, 52 insertions(+), 7 deletions(-)
> 
> diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> index e2c85ddc990b..be9719cf5745 100644
> --- a/Documentation/media/uapi/v4l/buffer.rst
> +++ b/Documentation/media/uapi/v4l/buffer.rst
> @@ -301,10 +301,22 @@ struct v4l2_buffer
>  	elements in the ``planes`` array. The driver will fill in the
>  	actual number of valid elements in that array.
>      * - __u32

Type is now __s32.

> -      - ``reserved2``
> +      - ``fence_fd``
>        -
> -      - A place holder for future extensions. Drivers and applications
> -	must set this to 0.
> +      - Used to communicate a fence file descriptors from userspace to kernel

descriptors -> descriptor

> +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> +	be used and this field set to the fence file descriptor of the in-fence.
> +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> +
> +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> +	must be set, the kernel will return the out-fence file descriptor in
> +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> +        an error.
> +
> +	For all other ioctls V4L2 sets this field to -1 if
> +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> +	otherwise this field is set to 0 for backward compatibility.
>      * - __u32
>        - ``reserved``
>        -
> @@ -648,6 +660,33 @@ Buffer Flags
>        - Start Of Exposure. The buffer timestamp has been taken when the
>  	exposure of the frame has begun. This is only valid for the
>  	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_IN_FENCE``
> +      - 0x00200000
> +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> +	buffer won't be queued to the driver until the fence signals. The order
> +	in which buffers are queued is guaranteed to be preserved, so any
> +	buffers queued after this buffer will also be blocked until this fence
> +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> +	other ioctls the driver just reports the value of the flag.
> +
> +        If the fence signals the flag is cleared and not reported anymore.
> +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> +
> +
> +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> +      - 0x00400000
> +      - Request for a fence to be attached to the buffer. The driver will fill
> +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> +	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
> +	the flag.

What happens once the out fence signals? Is this flag cleared? I think that would
make sense.

> +
> +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> +	error.
>  
>  

I would like to see a mention here that it is valid to set both FENCE flags.

>  
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 64503615d00b..b1c0fa2b0b88 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->reserved2 = 0;
> +	b->fence_fd = 0;
>  	b->reserved = 0;
>  
>  	if (q->is_multiplanar) {
> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> index 4312935f1dfc..93c752459aec 100644
> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> @@ -388,7 +388,7 @@ struct v4l2_buffer32 {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	__s32			fence_fd;
>  	__u32			reserved;
>  };
>  
> @@ -606,7 +606,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
>  	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
>  	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
>  	    assign_in_user(&up->sequence, &kp->sequence) ||
> -	    assign_in_user(&up->reserved2, &kp->reserved2) ||
> +	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
>  	    assign_in_user(&up->reserved, &kp->reserved) ||
>  	    get_user(length, &kp->length) ||
>  	    put_user(length, &up->length))
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index a8842a5ca636..1f18dc68ecab 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -934,7 +934,10 @@ struct v4l2_buffer {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	union {
> +		__s32		fence_fd;
> +		__u32		reserved2;
> +	};
>  	__u32			reserved;
>  };
>  
> @@ -971,6 +974,9 @@ struct v4l2_buffer {
>  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
>  /* mem2mem encoder/decoder */
>  #define V4L2_BUF_FLAG_LAST			0x00100000
> +/* Explicit synchronization */
> +#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
> +#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
>  
>  /**
>   * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
> 

Regards,

	Hans

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Brian Starkey]

Hi Ezequiel,

On Fri, May 04, 2018 at 05:06:07PM -0300, Ezequiel Garcia wrote:
>From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
>Turn the reserved2 field into fence_fd that we will use to send
>an in-fence to the kernel or return an out-fence from the kernel to
>userspace.
>
>Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
>when sending an in-fence to the kernel to be waited on, and
>V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
>
>v7: minor fixes on the Documentation (Hans Verkuil)
>
>v6: big improvement on doc (Hans Verkuil)
>
>v5: - keep using reserved2 field for cpia2
>    - set fence_fd to 0 for now, for compat with userspace(Mauro)
>
>v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
>
>v3: make the out_fence refer to the current buffer (Hans Verkuil)
>
>v2: add documentation
>
>Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>---
> Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
> drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
> drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
> include/uapi/linux/videodev2.h                  |  8 ++++-
> 4 files changed, 52 insertions(+), 7 deletions(-)
>
>diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
>index e2c85ddc990b..be9719cf5745 100644
>--- a/Documentation/media/uapi/v4l/buffer.rst
>+++ b/Documentation/media/uapi/v4l/buffer.rst
>@@ -301,10 +301,22 @@ struct v4l2_buffer
> 	elements in the ``planes`` array. The driver will fill in the
> 	actual number of valid elements in that array.
>     * - __u32
>-      - ``reserved2``
>+      - ``fence_fd``
>       -
>-      - A place holder for future extensions. Drivers and applications
>-	must set this to 0.
>+      - Used to communicate a fence file descriptors from userspace to kernel
>+	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
>+	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
>+	be used and this field set to the fence file descriptor of the in-fence.
>+	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
>+
>+        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
>+	must be set, the kernel will return the out-fence file descriptor in
>+	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
>+        an error.
>+
>+	For all other ioctls V4L2 sets this field to -1 if
>+	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
>+	otherwise this field is set to 0 for backward compatibility.
>     * - __u32
>       - ``reserved``
>       -
>@@ -648,6 +660,33 @@ Buffer Flags
>       - Start Of Exposure. The buffer timestamp has been taken when the
> 	exposure of the frame has begun. This is only valid for the
> 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
>+    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
>+
>+      - ``V4L2_BUF_FLAG_IN_FENCE``
>+      - 0x00200000
>+      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
>+	buffer won't be queued to the driver until the fence signals. The order
>+	in which buffers are queued is guaranteed to be preserved, so any
>+	buffers queued after this buffer will also be blocked until this fence
>+	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
>+	other ioctls the driver just reports the value of the flag.
>+
>+        If the fence signals the flag is cleared and not reported anymore.
>+	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
>+
>+
>+    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
>+
>+      - ``V4L2_BUF_FLAG_OUT_FENCE``
>+      - 0x00400000
>+      - Request for a fence to be attached to the buffer. The driver will fill
>+	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
>+	<VIDIOC_QBUF>` returns. This flag must be set before calling
>+	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
>+	the flag.
>+
>+        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
>+	error.
>

I commented similarly on some of the old patch-sets, and it's a minor
thing, but I still think the ordering of this series is off. It's
strange/wrong to me document all this behaviour, and expose the flags
to userspace, when the functionality isn't implemented yet.

If I apply this patch to the kernel, then the kernel doesn't do what
the (newly added) kernel-doc says it will.

I'd use this sequence:

   * Remove all the stuff in patch 11 which checks/sets FLAG_IN_FENCE
     (i.e. the code added in videobuf2-v4l2.c, except for adding the
     extra argument to vb2_core_qbuf(), which will always be NULL).

   * Remove all the stuff in patch 12 which checks/sets FLAG_OUT_FENCE
     (i.e. the code added in videobuf2-v4l2.c, except for adding the
     extra argument to vb2_core_qbuf(), which will always be false).

   * Move this to be a "new" patch 13, and add back the code to
     videobuf2-v4l2.c, which hooks up the UAPI flags to the
     implementation.

There's also some ordering dependency with the V4L2_CAP_FENCES. I'd
put the V4L2_CAP_FENCES patch first, and it will always be unset.
Then, once everything is in-place, you can finally add the patches
which set it. That also allows you to check and document that the
FLAG_*_FENCE flags are only valid when that cap is set.

That way, the patches depend on each other in the right order (I can't
pick the patch exposing the userspace flags without first having the
patches implementing the behaviour), and userspace can never see
something inconsistent.

Cheers,
-Brian

>
>
>diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>index 64503615d00b..b1c0fa2b0b88 100644
>--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> 	b->timestamp = ns_to_timeval(vb->timestamp);
> 	b->timecode = vbuf->timecode;
> 	b->sequence = vbuf->sequence;
>-	b->reserved2 = 0;
>+	b->fence_fd = 0;
> 	b->reserved = 0;
>
> 	if (q->is_multiplanar) {
>diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>index 4312935f1dfc..93c752459aec 100644
>--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>@@ -388,7 +388,7 @@ struct v4l2_buffer32 {
> 		__s32		fd;
> 	} m;
> 	__u32			length;
>-	__u32			reserved2;
>+	__s32			fence_fd;
> 	__u32			reserved;
> };
>
>@@ -606,7 +606,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
> 	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
> 	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
> 	    assign_in_user(&up->sequence, &kp->sequence) ||
>-	    assign_in_user(&up->reserved2, &kp->reserved2) ||
>+	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
> 	    assign_in_user(&up->reserved, &kp->reserved) ||
> 	    get_user(length, &kp->length) ||
> 	    put_user(length, &up->length))
>diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>index a8842a5ca636..1f18dc68ecab 100644
>--- a/include/uapi/linux/videodev2.h
>+++ b/include/uapi/linux/videodev2.h
>@@ -934,7 +934,10 @@ struct v4l2_buffer {
> 		__s32		fd;
> 	} m;
> 	__u32			length;
>-	__u32			reserved2;
>+	union {
>+		__s32		fence_fd;
>+		__u32		reserved2;
>+	};
> 	__u32			reserved;
> };
>
>@@ -971,6 +974,9 @@ struct v4l2_buffer {
> #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
> /* mem2mem encoder/decoder */
> #define V4L2_BUF_FLAG_LAST			0x00100000
>+/* Explicit synchronization */
>+#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
>+#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
>
> /**
>  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
>-- 
>2.16.3
>

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Ezequiel Garcia]

On Wed, 2018-05-09 at 11:33 +0100, Brian Starkey wrote:
> Hi Ezequiel,
> 
> On Fri, May 04, 2018 at 05:06:07PM -0300, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Turn the reserved2 field into fence_fd that we will use to send
> > an in-fence to the kernel or return an out-fence from the kernel to
> > userspace.
> > 
> > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > when sending an in-fence to the kernel to be waited on, and
> > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > 
> > v7: minor fixes on the Documentation (Hans Verkuil)
> > 
> > v6: big improvement on doc (Hans Verkuil)
> > 
> > v5: - keep using reserved2 field for cpia2
> >    - set fence_fd to 0 for now, for compat with userspace(Mauro)
> > 
> > v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> > 
> > v3: make the out_fence refer to the current buffer (Hans Verkuil)
> > 
> > v2: add documentation
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > ---
> > Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
> > drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
> > drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
> > include/uapi/linux/videodev2.h                  |  8 ++++-
> > 4 files changed, 52 insertions(+), 7 deletions(-)
> > 
> > diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> > index e2c85ddc990b..be9719cf5745 100644
> > --- a/Documentation/media/uapi/v4l/buffer.rst
> > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > @@ -301,10 +301,22 @@ struct v4l2_buffer
> > 	elements in the ``planes`` array. The driver will fill in the
> > 	actual number of valid elements in that array.
> >     * - __u32
> > -      - ``reserved2``
> > +      - ``fence_fd``
> >       -
> > -      - A place holder for future extensions. Drivers and applications
> > -	must set this to 0.
> > +      - Used to communicate a fence file descriptors from userspace to kernel
> > +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> > +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> > +	be used and this field set to the fence file descriptor of the in-fence.
> > +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> > +
> > +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> > +	must be set, the kernel will return the out-fence file descriptor in
> > +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> > +        an error.
> > +
> > +	For all other ioctls V4L2 sets this field to -1 if
> > +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> > +	otherwise this field is set to 0 for backward compatibility.
> >     * - __u32
> >       - ``reserved``
> >       -
> > @@ -648,6 +660,33 @@ Buffer Flags
> >       - Start Of Exposure. The buffer timestamp has been taken when the
> > 	exposure of the frame has begun. This is only valid for the
> > 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > +
> > +      - ``V4L2_BUF_FLAG_IN_FENCE``
> > +      - 0x00200000
> > +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> > +	buffer won't be queued to the driver until the fence signals. The order
> > +	in which buffers are queued is guaranteed to be preserved, so any
> > +	buffers queued after this buffer will also be blocked until this fence
> > +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> > +	other ioctls the driver just reports the value of the flag.
> > +
> > +        If the fence signals the flag is cleared and not reported anymore.
> > +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> > +
> > +
> > +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > +
> > +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> > +      - 0x00400000
> > +      - Request for a fence to be attached to the buffer. The driver will fill
> > +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> > +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> > +	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
> > +	the flag.
> > +
> > +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> > +	error.
> > 
> 
> I commented similarly on some of the old patch-sets, and it's a minor
> thing, but I still think the ordering of this series is off. It's
> strange/wrong to me document all this behaviour, and expose the flags
> to userspace, when the functionality isn't implemented yet.
> 
> If I apply this patch to the kernel, then the kernel doesn't do what
> the (newly added) kernel-doc says it will.
> 

This has never been a problem, and it has always been the canonical
way of doing things.

First the required macros, stubs, documentation and interfaces are added,
and then they are implemented.

I see no reason to go berserk here, unless you see an actual problem?
Or something actually broken?

The only thing I can think of is that we should return fence_fd -1
if the flags are set. We could do it on this patch, and be consistent
with userspace.

Regards,
Eze

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Brian Starkey]

On Wed, May 09, 2018 at 12:52:26PM -0300, Ezequiel Garcia wrote:
>On Wed, 2018-05-09 at 11:33 +0100, Brian Starkey wrote:
>> Hi Ezequiel,
>>
>> On Fri, May 04, 2018 at 05:06:07PM -0300, Ezequiel Garcia wrote:
>> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
>> >
>> > Turn the reserved2 field into fence_fd that we will use to send
>> > an in-fence to the kernel or return an out-fence from the kernel to
>> > userspace.
>> >
>> > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
>> > when sending an in-fence to the kernel to be waited on, and
>> > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
>> >
>> > v7: minor fixes on the Documentation (Hans Verkuil)
>> >
>> > v6: big improvement on doc (Hans Verkuil)
>> >
>> > v5: - keep using reserved2 field for cpia2
>> >    - set fence_fd to 0 for now, for compat with userspace(Mauro)
>> >
>> > v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
>> >
>> > v3: make the out_fence refer to the current buffer (Hans Verkuil)
>> >
>> > v2: add documentation
>> >
>> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>> > ---
>> > Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
>> > drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
>> > drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
>> > include/uapi/linux/videodev2.h                  |  8 ++++-
>> > 4 files changed, 52 insertions(+), 7 deletions(-)
>> >
>> > diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
>> > index e2c85ddc990b..be9719cf5745 100644
>> > --- a/Documentation/media/uapi/v4l/buffer.rst
>> > +++ b/Documentation/media/uapi/v4l/buffer.rst
>> > @@ -301,10 +301,22 @@ struct v4l2_buffer
>> > 	elements in the ``planes`` array. The driver will fill in the
>> > 	actual number of valid elements in that array.
>> >     * - __u32
>> > -      - ``reserved2``
>> > +      - ``fence_fd``
>> >       -
>> > -      - A place holder for future extensions. Drivers and applications
>> > -	must set this to 0.
>> > +      - Used to communicate a fence file descriptors from userspace to kernel
>> > +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
>> > +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
>> > +	be used and this field set to the fence file descriptor of the in-fence.
>> > +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
>> > +
>> > +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
>> > +	must be set, the kernel will return the out-fence file descriptor in
>> > +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
>> > +        an error.
>> > +
>> > +	For all other ioctls V4L2 sets this field to -1 if
>> > +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
>> > +	otherwise this field is set to 0 for backward compatibility.
>> >     * - __u32
>> >       - ``reserved``
>> >       -
>> > @@ -648,6 +660,33 @@ Buffer Flags
>> >       - Start Of Exposure. The buffer timestamp has been taken when the
>> > 	exposure of the frame has begun. This is only valid for the
>> > 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
>> > +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
>> > +
>> > +      - ``V4L2_BUF_FLAG_IN_FENCE``
>> > +      - 0x00200000
>> > +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
>> > +	buffer won't be queued to the driver until the fence signals. The order
>> > +	in which buffers are queued is guaranteed to be preserved, so any
>> > +	buffers queued after this buffer will also be blocked until this fence
>> > +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
>> > +	other ioctls the driver just reports the value of the flag.
>> > +
>> > +        If the fence signals the flag is cleared and not reported anymore.
>> > +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
>> > +
>> > +
>> > +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
>> > +
>> > +      - ``V4L2_BUF_FLAG_OUT_FENCE``
>> > +      - 0x00400000
>> > +      - Request for a fence to be attached to the buffer. The driver will fill
>> > +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
>> > +	<VIDIOC_QBUF>` returns. This flag must be set before calling
>> > +	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
>> > +	the flag.
>> > +
>> > +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
>> > +	error.
>> >
>>
>> I commented similarly on some of the old patch-sets, and it's a minor
>> thing, but I still think the ordering of this series is off. It's
>> strange/wrong to me document all this behaviour, and expose the flags
>> to userspace, when the functionality isn't implemented yet.
>>
>> If I apply this patch to the kernel, then the kernel doesn't do what
>> the (newly added) kernel-doc says it will.
>>
>
>This has never been a problem, and it has always been the canonical
>way of doing things.
>
>First the required macros, stubs, documentation and interfaces are added,
>and then they are implemented.

If you say so, I don't know what sets the standard but that seems
kinda backwards.

I'd expect the "flick the switch, expose to userspace" to always be
the last thing, but I'm happy to be shown examples to the contrary.

>
>I see no reason to go berserk here, unless you see an actual problem?
>Or something actually broken?
>

The only "broken" thing, is as I said - I can apply this patch to a
kernel (any kernel, because there's no dependencies in the code), and
it won't do what the kernel-doc says it will.

Maybe I'm crazy, but shouldn't comments at least be correct at the
point where they are added, even if they become incorrect later
through neglect?

Cheers,
-Brian

>The only thing I can think of is that we should return fence_fd -1
>if the flags are set. We could do it on this patch, and be consistent
>with userspace.
>
>Regards,
>Eze

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Ezequiel Garcia]

On Wed, 2018-05-09 at 17:33 +0100, Brian Starkey wrote:
> On Wed, May 09, 2018 at 12:52:26PM -0300, Ezequiel Garcia wrote:
> > On Wed, 2018-05-09 at 11:33 +0100, Brian Starkey wrote:
> > > Hi Ezequiel,
> > > 
> > > On Fri, May 04, 2018 at 05:06:07PM -0300, Ezequiel Garcia wrote:
> > > > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > > > 
> > > > Turn the reserved2 field into fence_fd that we will use to send
> > > > an in-fence to the kernel or return an out-fence from the kernel to
> > > > userspace.
> > > > 
> > > > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > > > when sending an in-fence to the kernel to be waited on, and
> > > > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > > > 
> > > > v7: minor fixes on the Documentation (Hans Verkuil)
> > > > 
> > > > v6: big improvement on doc (Hans Verkuil)
> > > > 
> > > > v5: - keep using reserved2 field for cpia2
> > > >    - set fence_fd to 0 for now, for compat with userspace(Mauro)
> > > > 
> > > > v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> > > > 
> > > > v3: make the out_fence refer to the current buffer (Hans Verkuil)
> > > > 
> > > > v2: add documentation
> > > > 
> > > > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > > > ---
> > > > Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
> > > > drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
> > > > drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
> > > > include/uapi/linux/videodev2.h                  |  8 ++++-
> > > > 4 files changed, 52 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> > > > index e2c85ddc990b..be9719cf5745 100644
> > > > --- a/Documentation/media/uapi/v4l/buffer.rst
> > > > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > > > @@ -301,10 +301,22 @@ struct v4l2_buffer
> > > > 	elements in the ``planes`` array. The driver will fill in the
> > > > 	actual number of valid elements in that array.
> > > >     * - __u32
> > > > -      - ``reserved2``
> > > > +      - ``fence_fd``
> > > >       -
> > > > -      - A place holder for future extensions. Drivers and applications
> > > > -	must set this to 0.
> > > > +      - Used to communicate a fence file descriptors from userspace to kernel
> > > > +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> > > > +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> > > > +	be used and this field set to the fence file descriptor of the in-fence.
> > > > +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> > > > +
> > > > +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> > > > +	must be set, the kernel will return the out-fence file descriptor in
> > > > +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> > > > +        an error.
> > > > +
> > > > +	For all other ioctls V4L2 sets this field to -1 if
> > > > +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> > > > +	otherwise this field is set to 0 for backward compatibility.
> > > >     * - __u32
> > > >       - ``reserved``
> > > >       -
> > > > @@ -648,6 +660,33 @@ Buffer Flags
> > > >       - Start Of Exposure. The buffer timestamp has been taken when the
> > > > 	exposure of the frame has begun. This is only valid for the
> > > > 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > > > +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > > > +
> > > > +      - ``V4L2_BUF_FLAG_IN_FENCE``
> > > > +      - 0x00200000
> > > > +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> > > > +	buffer won't be queued to the driver until the fence signals. The order
> > > > +	in which buffers are queued is guaranteed to be preserved, so any
> > > > +	buffers queued after this buffer will also be blocked until this fence
> > > > +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> > > > +	other ioctls the driver just reports the value of the flag.
> > > > +
> > > > +        If the fence signals the flag is cleared and not reported anymore.
> > > > +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> > > > +
> > > > +
> > > > +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > > > +
> > > > +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> > > > +      - 0x00400000
> > > > +      - Request for a fence to be attached to the buffer. The driver will fill
> > > > +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> > > > +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> > > > +	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
> > > > +	the flag.
> > > > +
> > > > +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> > > > +	error.
> > > > 
> > > 
> > > I commented similarly on some of the old patch-sets, and it's a minor
> > > thing, but I still think the ordering of this series is off. It's
> > > strange/wrong to me document all this behaviour, and expose the flags
> > > to userspace, when the functionality isn't implemented yet.
> > > 
> > > If I apply this patch to the kernel, then the kernel doesn't do what
> > > the (newly added) kernel-doc says it will.
> > > 
> > 
> > This has never been a problem, and it has always been the canonical
> > way of doing things.
> > 
> > First the required macros, stubs, documentation and interfaces are added,
> > and then they are implemented.
> 
> If you say so, I don't know what sets the standard but that seems
> kinda backwards.
> 
> I'd expect the "flick the switch, expose to userspace" to always be
> the last thing, but I'm happy to be shown examples to the contrary.
> 
> > 
> > I see no reason to go berserk here, unless you see an actual problem?
> > Or something actually broken?
> > 
> 
> The only "broken" thing, is as I said - I can apply this patch to a
> kernel (any kernel, because there's no dependencies in the code), and
> it won't do what the kernel-doc says it will.
> 

I don't think we've ever honored that, but I can be wrong too.

> Maybe I'm crazy, but shouldn't comments at least be correct at the
> point where they are added, even if they become incorrect later
> through neglect?
> 

This is the best example I can give, a similar policy in the
devicetree bindings [1]:

""
  3) The Documentation/ portion of the patch should come in the series before
     the code implementing the binding.
""

[1]  Documentation/devicetree/bindings/submitting-patches.txt

Re: [PATCH v9 10/15] vb2: add explicit fence user API

[Ezequiel Garcia]

On Wed, 2018-05-09 at 17:33 +0100, Brian Starkey wrote:
> On Wed, May 09, 2018 at 12:52:26PM -0300, Ezequiel Garcia wrote:
> > On Wed, 2018-05-09 at 11:33 +0100, Brian Starkey wrote:
> > > Hi Ezequiel,
> > > 
> > > On Fri, May 04, 2018 at 05:06:07PM -0300, Ezequiel Garcia wrote:
> > > > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > > > 
> > > > Turn the reserved2 field into fence_fd that we will use to send
> > > > an in-fence to the kernel or return an out-fence from the kernel to
> > > > userspace.
> > > > 
> > > > Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> > > > when sending an in-fence to the kernel to be waited on, and
> > > > V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> > > > 
> > > > v7: minor fixes on the Documentation (Hans Verkuil)
> > > > 
> > > > v6: big improvement on doc (Hans Verkuil)
> > > > 
> > > > v5: - keep using reserved2 field for cpia2
> > > >    - set fence_fd to 0 for now, for compat with userspace(Mauro)
> > > > 
> > > > v4: make it a union with reserved2 and fence_fd (Hans Verkuil)
> > > > 
> > > > v3: make the out_fence refer to the current buffer (Hans Verkuil)
> > > > 
> > > > v2: add documentation
> > > > 
> > > > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > > > ---
> > > > Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
> > > > drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
> > > > drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
> > > > include/uapi/linux/videodev2.h                  |  8 ++++-
> > > > 4 files changed, 52 insertions(+), 7 deletions(-)
> > > > 
> > > > diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> > > > index e2c85ddc990b..be9719cf5745 100644
> > > > --- a/Documentation/media/uapi/v4l/buffer.rst
> > > > +++ b/Documentation/media/uapi/v4l/buffer.rst
> > > > @@ -301,10 +301,22 @@ struct v4l2_buffer
> > > > 	elements in the ``planes`` array. The driver will fill in the
> > > > 	actual number of valid elements in that array.
> > > >     * - __u32
> > > > -      - ``reserved2``
> > > > +      - ``fence_fd``
> > > >       -
> > > > -      - A place holder for future extensions. Drivers and applications
> > > > -	must set this to 0.
> > > > +      - Used to communicate a fence file descriptors from userspace to kernel
> > > > +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> > > > +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> > > > +	be used and this field set to the fence file descriptor of the in-fence.
> > > > +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
> > > > +
> > > > +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> > > > +	must be set, the kernel will return the out-fence file descriptor in
> > > > +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
> > > > +        an error.
> > > > +
> > > > +	For all other ioctls V4L2 sets this field to -1 if
> > > > +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> > > > +	otherwise this field is set to 0 for backward compatibility.
> > > >     * - __u32
> > > >       - ``reserved``
> > > >       -
> > > > @@ -648,6 +660,33 @@ Buffer Flags
> > > >       - Start Of Exposure. The buffer timestamp has been taken when the
> > > > 	exposure of the frame has begun. This is only valid for the
> > > > 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> > > > +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> > > > +
> > > > +      - ``V4L2_BUF_FLAG_IN_FENCE``
> > > > +      - 0x00200000
> > > > +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> > > > +	buffer won't be queued to the driver until the fence signals. The order
> > > > +	in which buffers are queued is guaranteed to be preserved, so any
> > > > +	buffers queued after this buffer will also be blocked until this fence
> > > > +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> > > > +	other ioctls the driver just reports the value of the flag.
> > > > +
> > > > +        If the fence signals the flag is cleared and not reported anymore.
> > > > +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> > > > +
> > > > +
> > > > +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> > > > +
> > > > +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> > > > +      - 0x00400000
> > > > +      - Request for a fence to be attached to the buffer. The driver will fill
> > > > +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> > > > +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> > > > +	``VIDIOC_QBUF``. For other ioctls the driver just reports the value of
> > > > +	the flag.
> > > > +
> > > > +        If the creation of the out-fence fails ``VIDIOC_QBUF`` returns an
> > > > +	error.
> > > > 
> > > 
> > > I commented similarly on some of the old patch-sets, and it's a minor
> > > thing, but I still think the ordering of this series is off. It's
> > > strange/wrong to me document all this behaviour, and expose the flags
> > > to userspace, when the functionality isn't implemented yet.
> > > 
> > > If I apply this patch to the kernel, then the kernel doesn't do what
> > > the (newly added) kernel-doc says it will.
> > > 
> > 
> > This has never been a problem, and it has always been the canonical
> > way of doing things.
> > 
> > First the required macros, stubs, documentation and interfaces are added,
> > and then they are implemented.
> 
> If you say so, I don't know what sets the standard but that seems
> kinda backwards.
> 
> I'd expect the "flick the switch, expose to userspace" to always be
> the last thing, but I'm happy to be shown examples to the contrary.
> 
> > 
> > I see no reason to go berserk here, unless you see an actual problem?
> > Or something actually broken?
> > 
> 
> The only "broken" thing, is as I said - I can apply this patch to a
> kernel (any kernel, because there's no dependencies in the code), and
> it won't do what the kernel-doc says it will.
> 

I don't think we've ever honored that, but I can be wrong too.

> Maybe I'm crazy, but shouldn't comments at least be correct at the
> point where they are added, even if they become incorrect later
> through neglect?
> 

This is the best example I can give, a similar policy in the
devicetree bindings [1]:

""
  3) The Documentation/ portion of the patch should come in the series before
     the code implementing the binding.
""

[1]  Documentation/devicetree/bindings/submitting-patches.txt

[PATCH v9 11/15] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queued to the driver
out of the order they were queued from userspace. That means that even if
its fence signals it must wait for all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v12: fixed dvb_vb2.c usage of vb2_core_qbuf.

v11: - minor doc/comments fixes (Hans Verkuil)
     - reviewed the in-fence path at __fill_v4l2_buffer()

v10: - rename fence to in_fence in many places
     - handle fences signalling with error better (Hans Verkuil)

v9: - improve comments and docs (Hans Verkuil)
    - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
    - move in-fences code that was in the out-fences patch here (Alex)

v8: - improve comments about fences with errors

v7: - get rid of the fence array stuff for ordering and just use
      get_num_buffers_ready() (Hans)
    - fix issue of queuing the buffer twice (Hans)
    - avoid the dma_fence_wait() in core_qbuf() (Alex)
    - merge preparation commit in

v6: - With fences always keep the order userspace queues the buffers.
    - Protect in_fence manipulation with a lock (Brian Starkey)
    - check if fences have the same context before adding a fence array
    - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
    - Clean up fence if __set_in_fence() fails (Brian Starkey)
    - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5: - use fence_array to keep buffers ordered in vb2 core when
      needed (Brian Starkey)
    - keep backward compat on the reserved2 field (Brian Starkey)
    - protect fence callback removal with lock (Brian Starkey)

v4: - Add a comment about dma_fence_add_callback() not returning a
      error (Hans)
    - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
    - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
    - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
    - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
    - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
      vb2_start_streaming() (Hans)
    - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
    - Queue buffers to the driver as soon as they are ready (Hans)
    - call fill_user_buffer() after queuing the buffer (Hans)
    - add err: label to clean up fence
    - add dma_fence_wait() before calling vb2_start_streaming()

v3: - document fence parameter
    - remove ternary if at vb2_qbuf() return (Mauro)
    - do not change if conditions behaviour (Mauro)

v2: - fix vb2_queue_or_prepare_buf() ret check
    - remove check for VB2_MEMORY_DMABUF only (Javier)
    - check num of ready buffers to start streaming
    - when queueing, start from the first ready buffer
    - handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
 drivers/media/dvb-core/dvb_vb2.c                |   2 +-
 drivers/media/v4l2-core/Kconfig                 |  33 ++++
 include/media/videobuf2-core.h                  |  14 +-
 5 files changed, 249 insertions(+), 31 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 6b8e083893ad..996b99497a98 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 		vb->index = q->num_buffers + buffer;
 		vb->type = q->type;
 		vb->memory = memory;
+		spin_lock_init(&vb->fence_cb_lock);
 		for (plane = 0; plane < num_planes; ++plane) {
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
@@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
 }
 EXPORT_SYMBOL_GPL(vb2_plane_cookie);
 
-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned long flags;
 	unsigned int plane;
 
-	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
-		return;
-
-	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
-		    state != VB2_BUF_STATE_ERROR &&
-		    state != VB2_BUF_STATE_QUEUED &&
-		    state != VB2_BUF_STATE_REQUEUEING))
-		state = VB2_BUF_STATE_ERROR;
 
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
@@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
 
 	spin_lock_irqsave(&q->done_lock, flags);
+	if (vb->state == VB2_BUF_STATE_ACTIVE)
+		atomic_dec(&q->owned_by_drv_count);
+
 	if (state == VB2_BUF_STATE_QUEUED ||
 	    state == VB2_BUF_STATE_REQUEUEING) {
 		vb->state = VB2_BUF_STATE_QUEUED;
@@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		list_add_tail(&vb->done_entry, &q->done_list);
 		vb->state = state;
 	}
-	atomic_dec(&q->owned_by_drv_count);
+
 	spin_unlock_irqrestore(&q->done_lock, flags);
 
 	trace_vb2_buf_done(q, vb);
@@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		wake_up(&q->done_wq);
 		break;
 	}
+
+}
+
+void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+{
+	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
+		return;
+
+	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
+		    state != VB2_BUF_STATE_ERROR &&
+		    state != VB2_BUF_STATE_QUEUED &&
+		    state != VB2_BUF_STATE_REQUEUEING))
+		state = VB2_BUF_STATE_ERROR;
+
+	vb2_process_buffer_done(vb, state);
+
+	/*
+	 * Check if there is any buffer with error in the next position of the queue,
+	 * buffers whose in-fence signaled with error are not queued to the driver
+	 * and kept on the queue until the buffer before them is done, so to not
+	 * deliver buffers back to userspace in the wrong order. Here we process
+	 * any existing buffers with errors and wake up userspace.
+	 */
+	for (;;) {
+		vb = list_next_entry(vb, queued_entry);
+		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
+			break;
+
+		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
+        }
 }
 EXPORT_SYMBOL_GPL(vb2_buffer_done);
 
@@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 
+	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
+		return;
+
 	vb->state = VB2_BUF_STATE_ACTIVE;
 	atomic_inc(&q->owned_by_drv_count);
 
@@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
 	return 0;
 }
 
+static int __get_num_ready_buffers(struct vb2_queue *q)
+{
+	struct vb2_buffer *vb;
+	int ready_count = 0;
+	unsigned long flags;
+
+	/* count num of buffers ready in front of the queued_list */
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
+		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
+			break;
+		ready_count++;
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+	}
+
+	return ready_count;
+}
+
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 {
 	struct vb2_buffer *vb;
@@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
+	struct vb2_queue *q = vb->vb2_queue;
+	unsigned long flags;
+
+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
+	/*
+	 * If the fence signals with an error we mark the buffer as such
+	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
+	 * not queueing it to the driver. However we can't notify the error
+	 * to userspace right now because, at the time this callback run, QBUF
+	 * has returned already.
+	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
+	 * as well.
+	 */
+	if (vb->in_fence->error)
+		vb->state = VB2_BUF_STATE_ERROR;
+
+	dma_fence_put(vb->in_fence);
+	vb->in_fence = NULL;
+
+	if (vb->state == VB2_BUF_STATE_ERROR) {
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+		return;
+	}
+
+	if (q->start_streaming_called)
+		__enqueue_in_driver(vb);
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+}
+
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence)
 {
 	struct vb2_buffer *vb;
+	unsigned long flags;
 	int ret;
 
 	vb = q->bufs[index];
@@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	case VB2_BUF_STATE_DEQUEUED:
 		ret = __buf_prepare(vb, pb);
 		if (ret)
-			return ret;
+			goto err;
 		break;
 	case VB2_BUF_STATE_PREPARED:
 		break;
 	case VB2_BUF_STATE_PREPARING:
 		dprintk(1, "buffer still being prepared\n");
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err;
 	default:
 		dprintk(1, "invalid buffer state %d\n", vb->state);
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err;
 	}
 
 	/*
@@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	q->queued_count++;
 	q->waiting_for_buffers = false;
 	vb->state = VB2_BUF_STATE_QUEUED;
+	vb->in_fence = in_fence;
 
 	if (pb)
 		call_void_bufop(q, copy_timestamp, vb, pb);
@@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	trace_vb2_qbuf(q, vb);
 
 	/*
-	 * If already streaming, give the buffer to driver for processing.
-	 * If not, the buffer will be given to driver on next streamon.
+	 * For explicit synchronization: If the fence didn't signal
+	 * yet we setup a callback to queue the buffer once the fence
+	 * signals and then return successfully. But if the fence
+	 * already signaled we lose the reference we held and queue the
+	 * buffer to the driver.
 	 */
-	if (q->start_streaming_called)
-		__enqueue_in_driver(vb);
+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
+	if (vb->in_fence) {
+		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
+					     vb2_qbuf_fence_cb);
+		/* is the fence signaled? */
+		if (ret == -ENOENT) {
+			dma_fence_put(vb->in_fence);
+			vb->in_fence = NULL;
+		} else if (ret) {
+			goto unlock;
+		}
+	}
 
-	/* Fill buffer information for the userspace */
-	if (pb)
-		call_void_bufop(q, fill_user_buffer, vb, pb);
+	/*
+	 * If already streaming and there is no fence to wait on
+	 * give the buffer to driver for processing.
+	 */
+	if (q->start_streaming_called) {
+		struct vb2_buffer *b;
+
+		list_for_each_entry(b, &q->queued_list, queued_entry) {
+			if (b->state != VB2_BUF_STATE_QUEUED)
+				continue;
+			if (b->in_fence)
+				break;
+			__enqueue_in_driver(b);
+		}
+	}
 
 	/*
 	 * If streamon has been called, and we haven't yet called
@@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	 * then we can finally call start_streaming().
 	 */
 	if (q->streaming && !q->start_streaming_called &&
-	    q->queued_count >= q->min_buffers_needed) {
+	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = vb2_start_streaming(q);
 		if (ret)
-			return ret;
+			goto unlock;
 	}
 
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+
+	/* Fill buffer information for the userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
 	return 0;
+
+unlock:
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+
+err:
+	/* Fill buffer information for the userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
+	if (vb->in_fence) {
+		dma_fence_put(vb->in_fence);
+		vb->in_fence = NULL;
+	}
+
+	return ret;
+
 }
 EXPORT_SYMBOL_GPL(vb2_core_qbuf);
 
@@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
 static void __vb2_queue_cancel(struct vb2_queue *q)
 {
 	unsigned int i;
+	struct vb2_buffer *vb;
+	unsigned long flags;
 
 	/*
 	 * Tell driver to stop all transactions and release all queued
@@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	q->queued_count = 0;
 	q->error = 0;
 
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
+		if (vb->in_fence) {
+			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
+			dma_fence_put(vb->in_fence);
+			vb->in_fence = NULL;
+		}
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+	}
+
 	/*
 	 * Remove all buffers from videobuf's list...
 	 */
@@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
 	 * Tell driver to start streaming provided sufficient buffers
 	 * are available.
 	 */
-	if (q->queued_count >= q->min_buffers_needed) {
+	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = v4l_vb2q_enable_media_source(q);
 		if (ret)
 			return ret;
@@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index b1c0fa2b0b88..74d7062e5285 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -23,6 +23,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/v4l2-dev.h>
 #include <media/v4l2-fh.h>
@@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 		return -EINVAL;
 	}
 
+	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
+	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
+		return -EINVAL;
+	}
+
+	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
+		return -EINVAL;
+	}
+
 	return __verify_planes_array(q->bufs[b->index], b);
 }
 
@@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->fence_fd = 0;
 	b->reserved = 0;
 
+	b->fence_fd = 0;
+	if (vb->in_fence)
+		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
+	else
+		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
+
 	if (q->is_multiplanar) {
 		/*
 		 * Fill in plane-related data if userspace provided an array
@@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	struct dma_fence *in_fence = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	}
 
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	if (ret)
+		return ret;
+
+	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
+		in_fence = sync_file_get_fence(b->fence_fd);
+		if (!in_fence) {
+			dprintk(1, "failed to get in-fence from fd %d\n",
+				b->fence_fd);
+			return -EINVAL;
+		}
+	}
+
+	return vb2_core_qbuf(q, b->index, b, in_fence);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index b811adf88afa..7da53f10db1a 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
 {
 	int ret;
 
-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
 	if (ret) {
 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
 			b->index, ret);
diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
index 8e37e7c5e0f7..79dfb5dfd1fc 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
 config VIDEOBUF_DVB
 	tristate
 	select VIDEOBUF_GEN
+
+# Used by drivers that need Videobuf2 modules
+config VIDEOBUF2_CORE
+	select DMA_SHARED_BUFFER
+	select SYNC_FILE
+	tristate
+
+config VIDEOBUF2_MEMOPS
+	tristate
+	select FRAME_VECTOR
+
+config VIDEOBUF2_DMA_CONTIG
+	tristate
+	depends on HAS_DMA
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+	select DMA_SHARED_BUFFER
+
+config VIDEOBUF2_VMALLOC
+	tristate
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+	select DMA_SHARED_BUFFER
+
+config VIDEOBUF2_DMA_SG
+	tristate
+	depends on HAS_DMA
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+
+config VIDEOBUF2_DVB
+	tristate
+	select VIDEOBUF2_CORE
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 364e4cb41b10..28ce8f66882e 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -17,6 +17,7 @@
 #include <linux/poll.h>
 #include <linux/dma-buf.h>
 #include <linux/bitops.h>
+#include <linux/dma-fence.h>
 
 #define VB2_MAX_FRAME	(32)
 #define VB2_MAX_PLANES	(8)
@@ -255,12 +256,21 @@ struct vb2_buffer {
 	 * done_entry:		entry on the list that stores all buffers ready
 	 *			to be dequeued to userspace
 	 * vb2_plane:		per-plane information; do not change
+	 * in_fence:		fence received from vb2 client to wait on before
+	 *			using the buffer (queueing to the driver)
+	 * fence_cb:		fence callback information
+	 * fence_cb_lock:	protect callback signal/remove
 	 */
 	enum vb2_buffer_state	state;
 
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 	struct list_head	queued_entry;
 	struct list_head	done_entry;
+
+	struct dma_fence	*in_fence;
+	struct dma_fence_cb	fence_cb;
+	spinlock_t              fence_cb_lock;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -773,6 +783,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @index:	id number of the buffer
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
+ * @in_fence:	in-fence to wait on before queueing the buffer
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
@@ -787,7 +798,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * Return: returns zero on success; an error code otherwise.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.16.3

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Receive in-fence from userspace and add support for waiting on them
> before queueing the buffer to the driver. Buffers can't be queued to the
> driver before its fences signal. And a buffer can't be queued to the driver
> out of the order they were queued from userspace. That means that even if
> its fence signals it must wait for all other buffers, ahead of it in the queue,
> to signal first.
> 
> If the fence for some buffer fails we do not queue it to the driver,
> instead we mark it as error and wait until the previous buffer is done
> to notify userspace of the error. We wait here to deliver the buffers back
> to userspace in order.
> 
> v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> 
> v11: - minor doc/comments fixes (Hans Verkuil)
>      - reviewed the in-fence path at __fill_v4l2_buffer()
> 
> v10: - rename fence to in_fence in many places
>      - handle fences signalling with error better (Hans Verkuil)
> 
> v9: - improve comments and docs (Hans Verkuil)
>     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>     - move in-fences code that was in the out-fences patch here (Alex)
> 
> v8: - improve comments about fences with errors
> 
> v7: - get rid of the fence array stuff for ordering and just use
>       get_num_buffers_ready() (Hans)
>     - fix issue of queuing the buffer twice (Hans)
>     - avoid the dma_fence_wait() in core_qbuf() (Alex)
>     - merge preparation commit in
> 
> v6: - With fences always keep the order userspace queues the buffers.
>     - Protect in_fence manipulation with a lock (Brian Starkey)
>     - check if fences have the same context before adding a fence array
>     - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>     - Clean up fence if __set_in_fence() fails (Brian Starkey)
>     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> 
> v5: - use fence_array to keep buffers ordered in vb2 core when
>       needed (Brian Starkey)
>     - keep backward compat on the reserved2 field (Brian Starkey)
>     - protect fence callback removal with lock (Brian Starkey)
> 
> v4: - Add a comment about dma_fence_add_callback() not returning a
>       error (Hans)
>     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>     - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>       vb2_start_streaming() (Hans)
>     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>     - Queue buffers to the driver as soon as they are ready (Hans)
>     - call fill_user_buffer() after queuing the buffer (Hans)
>     - add err: label to clean up fence
>     - add dma_fence_wait() before calling vb2_start_streaming()
> 
> v3: - document fence parameter
>     - remove ternary if at vb2_qbuf() return (Mauro)
>     - do not change if conditions behaviour (Mauro)
> 
> v2: - fix vb2_queue_or_prepare_buf() ret check
>     - remove check for VB2_MEMORY_DMABUF only (Javier)
>     - check num of ready buffers to start streaming
>     - when queueing, start from the first ready buffer
>     - handle queue cancel
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
>  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
>  drivers/media/v4l2-core/Kconfig                 |  33 ++++
>  include/media/videobuf2-core.h                  |  14 +-
>  5 files changed, 249 insertions(+), 31 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 6b8e083893ad..996b99497a98 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  		vb->index = q->num_buffers + buffer;
>  		vb->type = q->type;
>  		vb->memory = memory;
> +		spin_lock_init(&vb->fence_cb_lock);
>  		for (plane = 0; plane < num_planes; ++plane) {
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
> @@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>  }
>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>  
> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  	unsigned long flags;
>  	unsigned int plane;
>  
> -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> -		return;
> -
> -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> -		    state != VB2_BUF_STATE_ERROR &&
> -		    state != VB2_BUF_STATE_QUEUED &&
> -		    state != VB2_BUF_STATE_REQUEUEING))
> -		state = VB2_BUF_STATE_ERROR;
>  
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
> @@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>  
>  	spin_lock_irqsave(&q->done_lock, flags);
> +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> +		atomic_dec(&q->owned_by_drv_count);
> +
>  	if (state == VB2_BUF_STATE_QUEUED ||
>  	    state == VB2_BUF_STATE_REQUEUEING) {
>  		vb->state = VB2_BUF_STATE_QUEUED;
> @@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		list_add_tail(&vb->done_entry, &q->done_list);
>  		vb->state = state;
>  	}
> -	atomic_dec(&q->owned_by_drv_count);
> +
>  	spin_unlock_irqrestore(&q->done_lock, flags);
>  
>  	trace_vb2_buf_done(q, vb);
> @@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		wake_up(&q->done_wq);
>  		break;
>  	}
> +
> +}
> +
> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +{
> +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> +		return;
> +
> +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> +		    state != VB2_BUF_STATE_ERROR &&
> +		    state != VB2_BUF_STATE_QUEUED &&
> +		    state != VB2_BUF_STATE_REQUEUEING))
> +		state = VB2_BUF_STATE_ERROR;
> +
> +	vb2_process_buffer_done(vb, state);
> +
> +	/*
> +	 * Check if there is any buffer with error in the next position of the queue,
> +	 * buffers whose in-fence signaled with error are not queued to the driver
> +	 * and kept on the queue until the buffer before them is done, so to not
> +	 * deliver buffers back to userspace in the wrong order. Here we process
> +	 * any existing buffers with errors and wake up userspace.
> +	 */
> +	for (;;) {

I think this should test if the list is empty before calling list_next_entry().

> +		vb = list_next_entry(vb, queued_entry);
> +		if (!vb || vb->state != VB2_BUF_STATE_ERROR)

The !vb check makes no sense. list_next_entry won't return NULL if there is no
next entry, you need to call list_empty() first.

> +			break;
> +
> +		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
> +        }
>  }
>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>  
> @@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  
> +	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> +		return;
> +
>  	vb->state = VB2_BUF_STATE_ACTIVE;
>  	atomic_inc(&q->owned_by_drv_count);
>  
> @@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>  	return 0;
>  }
>  
> +static int __get_num_ready_buffers(struct vb2_queue *q)
> +{
> +	struct vb2_buffer *vb;
> +	int ready_count = 0;
> +	unsigned long flags;
> +
> +	/* count num of buffers ready in front of the queued_list */
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> +			break;
> +		ready_count++;
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +	}
> +
> +	return ready_count;
> +}
> +
>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  {
>  	struct vb2_buffer *vb;
> @@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	return ret;
>  }
>  
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> +{
> +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> +	struct vb2_queue *q = vb->vb2_queue;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +	/*
> +	 * If the fence signals with an error we mark the buffer as such
> +	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
> +	 * not queueing it to the driver. However we can't notify the error
> +	 * to userspace right now because, at the time this callback run, QBUF
> +	 * has returned already.
> +	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
> +	 * as well.
> +	 */
> +	if (vb->in_fence->error)
> +		vb->state = VB2_BUF_STATE_ERROR;
> +
> +	dma_fence_put(vb->in_fence);
> +	vb->in_fence = NULL;
> +
> +	if (vb->state == VB2_BUF_STATE_ERROR) {
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +		return;
> +	}
> +
> +	if (q->start_streaming_called)

I'd change this to:

	if (q->start_streaming_called &&
	    vb->state != VB2_BUF_STATE_ERROR)

Then you can drop the previous 'if'.

> +		__enqueue_in_driver(vb);
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +}
> +
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence)
>  {
>  	struct vb2_buffer *vb;
> +	unsigned long flags;
>  	int ret;
>  
>  	vb = q->bufs[index];
> @@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	case VB2_BUF_STATE_DEQUEUED:
>  		ret = __buf_prepare(vb, pb);
>  		if (ret)
> -			return ret;
> +			goto err;
>  		break;
>  	case VB2_BUF_STATE_PREPARED:
>  		break;
>  	case VB2_BUF_STATE_PREPARING:
>  		dprintk(1, "buffer still being prepared\n");
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err;
>  	default:
>  		dprintk(1, "invalid buffer state %d\n", vb->state);
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err;
>  	}
>  
>  	/*
> @@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
>  	vb->state = VB2_BUF_STATE_QUEUED;
> +	vb->in_fence = in_fence;
>  
>  	if (pb)
>  		call_void_bufop(q, copy_timestamp, vb, pb);
> @@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	trace_vb2_qbuf(q, vb);
>  
>  	/*
> -	 * If already streaming, give the buffer to driver for processing.
> -	 * If not, the buffer will be given to driver on next streamon.
> +	 * For explicit synchronization: If the fence didn't signal
> +	 * yet we setup a callback to queue the buffer once the fence
> +	 * signals and then return successfully. But if the fence
> +	 * already signaled we lose the reference we held and queue the
> +	 * buffer to the driver.

What happens if the fence signaled an error? Is that error returned to userspace?
(i.e. VIDIOC_QBUF will fail in that case)

>  	 */
> -	if (q->start_streaming_called)
> -		__enqueue_in_driver(vb);
> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +	if (vb->in_fence) {
> +		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
> +					     vb2_qbuf_fence_cb);
> +		/* is the fence signaled? */
> +		if (ret == -ENOENT) {
> +			dma_fence_put(vb->in_fence);
> +			vb->in_fence = NULL;
> +		} else if (ret) {
> +			goto unlock;
> +		}
> +	}
>  
> -	/* Fill buffer information for the userspace */
> -	if (pb)
> -		call_void_bufop(q, fill_user_buffer, vb, pb);
> +	/*
> +	 * If already streaming and there is no fence to wait on
> +	 * give the buffer to driver for processing.
> +	 */
> +	if (q->start_streaming_called) {
> +		struct vb2_buffer *b;
> +
> +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> +			if (b->state != VB2_BUF_STATE_QUEUED)
> +				continue;
> +			if (b->in_fence)
> +				break;
> +			__enqueue_in_driver(b);
> +		}
> +	}
>  
>  	/*
>  	 * If streamon has been called, and we haven't yet called
> @@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	 * then we can finally call start_streaming().
>  	 */
>  	if (q->streaming && !q->start_streaming_called &&
> -	    q->queued_count >= q->min_buffers_needed) {
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = vb2_start_streaming(q);
>  		if (ret)
> -			return ret;
> +			goto unlock;
>  	}
>  
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +
> +	/* Fill buffer information for the userspace */

s/the//

> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
>  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>  	return 0;
> +
> +unlock:
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +
> +err:
> +	/* Fill buffer information for the userspace */

s/the//

> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
> +	if (vb->in_fence) {
> +		dma_fence_put(vb->in_fence);
> +		vb->in_fence = NULL;
> +	}
> +
> +	return ret;
> +
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>  
> @@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
>  static void __vb2_queue_cancel(struct vb2_queue *q)
>  {
>  	unsigned int i;
> +	struct vb2_buffer *vb;
> +	unsigned long flags;
>  
>  	/*
>  	 * Tell driver to stop all transactions and release all queued
> @@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	q->queued_count = 0;
>  	q->error = 0;
>  
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +		if (vb->in_fence) {
> +			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
> +			dma_fence_put(vb->in_fence);
> +			vb->in_fence = NULL;
> +		}
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +	}
> +
>  	/*
>  	 * Remove all buffers from videobuf's list...
>  	 */
> @@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>  	 * Tell driver to start streaming provided sufficient buffers
>  	 * are available.
>  	 */
> -	if (q->queued_count >= q->min_buffers_needed) {
> +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = v4l_vb2q_enable_media_source(q);
>  		if (ret)
>  			return ret;
> @@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index b1c0fa2b0b88..74d7062e5285 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -23,6 +23,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/v4l2-dev.h>
>  #include <media/v4l2-fh.h>
> @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>  		return -EINVAL;
>  	}
>  
> +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
> +		return -EINVAL;
> +	}
> +
> +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
> +		return -EINVAL;
> +	}
> +
>  	return __verify_planes_array(q->bufs[b->index], b);
>  }
>  
> @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->fence_fd = 0;
>  	b->reserved = 0;
>  
> +	b->fence_fd = 0;
> +	if (vb->in_fence)
> +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> +	else
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +
>  	if (q->is_multiplanar) {
>  		/*
>  		 * Fill in plane-related data if userspace provided an array
> @@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	struct dma_fence *in_fence = NULL;
>  	int ret;
>  
>  	if (vb2_fileio_is_active(q)) {
> @@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  	}
>  
>  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> +	if (ret)
> +		return ret;
> +
> +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> +		in_fence = sync_file_get_fence(b->fence_fd);
> +		if (!in_fence) {
> +			dprintk(1, "failed to get in-fence from fd %d\n",
> +				b->fence_fd);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return vb2_core_qbuf(q, b->index, b, in_fence);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
> index b811adf88afa..7da53f10db1a 100644
> --- a/drivers/media/dvb-core/dvb_vb2.c
> +++ b/drivers/media/dvb-core/dvb_vb2.c
> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
>  {
>  	int ret;
>  
> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>  	if (ret) {
>  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>  			b->index, ret);
> diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
> index 8e37e7c5e0f7..79dfb5dfd1fc 100644
> --- a/drivers/media/v4l2-core/Kconfig
> +++ b/drivers/media/v4l2-core/Kconfig
> @@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
>  config VIDEOBUF_DVB
>  	tristate
>  	select VIDEOBUF_GEN
> +
> +# Used by drivers that need Videobuf2 modules
> +config VIDEOBUF2_CORE
> +	select DMA_SHARED_BUFFER
> +	select SYNC_FILE
> +	tristate
> +
> +config VIDEOBUF2_MEMOPS
> +	tristate
> +	select FRAME_VECTOR
> +
> +config VIDEOBUF2_DMA_CONTIG
> +	tristate
> +	depends on HAS_DMA
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +	select DMA_SHARED_BUFFER
> +
> +config VIDEOBUF2_VMALLOC
> +	tristate
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +	select DMA_SHARED_BUFFER
> +
> +config VIDEOBUF2_DMA_SG
> +	tristate
> +	depends on HAS_DMA
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +
> +config VIDEOBUF2_DVB
> +	tristate
> +	select VIDEOBUF2_CORE

This chunk doesn't really belong here. Should be a separate patch with an explanation
why.

> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 364e4cb41b10..28ce8f66882e 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -17,6 +17,7 @@
>  #include <linux/poll.h>
>  #include <linux/dma-buf.h>
>  #include <linux/bitops.h>
> +#include <linux/dma-fence.h>
>  
>  #define VB2_MAX_FRAME	(32)
>  #define VB2_MAX_PLANES	(8)
> @@ -255,12 +256,21 @@ struct vb2_buffer {
>  	 * done_entry:		entry on the list that stores all buffers ready
>  	 *			to be dequeued to userspace
>  	 * vb2_plane:		per-plane information; do not change
> +	 * in_fence:		fence received from vb2 client to wait on before
> +	 *			using the buffer (queueing to the driver)
> +	 * fence_cb:		fence callback information
> +	 * fence_cb_lock:	protect callback signal/remove
>  	 */
>  	enum vb2_buffer_state	state;
>  
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>  	struct list_head	queued_entry;
>  	struct list_head	done_entry;
> +
> +	struct dma_fence	*in_fence;
> +	struct dma_fence_cb	fence_cb;
> +	spinlock_t              fence_cb_lock;
> +

So for the _MPLANE formats this is one fence for all planes. Which makes sense, but how
does drm handle that? Also one fence for all planes?

I think there should be a comment about this somewhere.

>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -773,6 +783,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * @index:	id number of the buffer
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> + * @in_fence:	in-fence to wait on before queueing the buffer
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> @@ -787,7 +798,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   *
>   * Return: returns zero on success; an error code otherwise.
>   */
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Regards,

	Hans

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
> On 04/05/18 22:06, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued to the
> > driver before its fences signal. And a buffer can't be queued to the driver
> > out of the order they were queued from userspace. That means that even if
> > its fence signals it must wait for all other buffers, ahead of it in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the driver,
> > instead we mark it as error and wait until the previous buffer is done
> > to notify userspace of the error. We wait here to deliver the buffers back
> > to userspace in order.
> > 
> > v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> > 
> > v11: - minor doc/comments fixes (Hans Verkuil)
> >      - reviewed the in-fence path at __fill_v4l2_buffer()
> > 
> > v10: - rename fence to in_fence in many places
> >      - handle fences signalling with error better (Hans Verkuil)
> > 
> > v9: - improve comments and docs (Hans Verkuil)
> >     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
> >     - move in-fences code that was in the out-fences patch here (Alex)
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7: - get rid of the fence array stuff for ordering and just use
> >       get_num_buffers_ready() (Hans)
> >     - fix issue of queuing the buffer twice (Hans)
> >     - avoid the dma_fence_wait() in core_qbuf() (Alex)
> >     - merge preparation commit in
> > 
> > v6: - With fences always keep the order userspace queues the buffers.
> >     - Protect in_fence manipulation with a lock (Brian Starkey)
> >     - check if fences have the same context before adding a fence array
> >     - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> >     - Clean up fence if __set_in_fence() fails (Brian Starkey)
> >     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> >       needed (Brian Starkey)
> >     - keep backward compat on the reserved2 field (Brian Starkey)
> >     - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4: - Add a comment about dma_fence_add_callback() not returning a
> >       error (Hans)
> >     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> >     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> >     - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> >     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> >     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> >       vb2_start_streaming() (Hans)
> >     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> >     - Queue buffers to the driver as soon as they are ready (Hans)
> >     - call fill_user_buffer() after queuing the buffer (Hans)
> >     - add err: label to clean up fence
> >     - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> >     - remove ternary if at vb2_qbuf() return (Mauro)
> >     - do not change if conditions behaviour (Mauro)
> > 
> > v2: - fix vb2_queue_or_prepare_buf() ret check
> >     - remove check for VB2_MEMORY_DMABUF only (Javier)
> >     - check num of ready buffers to start streaming
> >     - when queueing, start from the first ready buffer
> >     - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
> >  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
> >  drivers/media/v4l2-core/Kconfig                 |  33 ++++
> >  include/media/videobuf2-core.h                  |  14 +-
> >  5 files changed, 249 insertions(+), 31 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> > index 6b8e083893ad..996b99497a98 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
> >  		vb->index = q->num_buffers + buffer;
> >  		vb->type = q->type;
> >  		vb->memory = memory;
> > +		spin_lock_init(&vb->fence_cb_lock);
> >  		for (plane = 0; plane < num_planes; ++plane) {
> >  			vb->planes[plane].length = plane_sizes[plane];
> >  			vb->planes[plane].min_length = plane_sizes[plane];
> > @@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
> >  
> > -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> > +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  	unsigned long flags;
> >  	unsigned int plane;
> >  
> > -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > -		return;
> > -
> > -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > -		    state != VB2_BUF_STATE_ERROR &&
> > -		    state != VB2_BUF_STATE_QUEUED &&
> > -		    state != VB2_BUF_STATE_REQUEUEING))
> > -		state = VB2_BUF_STATE_ERROR;
> >  
> >  #ifdef CONFIG_VIDEO_ADV_DEBUG
> >  	/*
> > @@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
> >  
> >  	spin_lock_irqsave(&q->done_lock, flags);
> > +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> > +		atomic_dec(&q->owned_by_drv_count);
> > +
> >  	if (state == VB2_BUF_STATE_QUEUED ||
> >  	    state == VB2_BUF_STATE_REQUEUEING) {
> >  		vb->state = VB2_BUF_STATE_QUEUED;
> > @@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		list_add_tail(&vb->done_entry, &q->done_list);
> >  		vb->state = state;
> >  	}
> > -	atomic_dec(&q->owned_by_drv_count);
> > +
> >  	spin_unlock_irqrestore(&q->done_lock, flags);
> >  
> >  	trace_vb2_buf_done(q, vb);
> > @@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> >  		wake_up(&q->done_wq);
> >  		break;
> >  	}
> > +
> > +}
> > +
> > +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> > +{
> > +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > +		return;
> > +
> > +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > +		    state != VB2_BUF_STATE_ERROR &&
> > +		    state != VB2_BUF_STATE_QUEUED &&
> > +		    state != VB2_BUF_STATE_REQUEUEING))
> > +		state = VB2_BUF_STATE_ERROR;
> > +
> > +	vb2_process_buffer_done(vb, state);
> > +
> > +	/*
> > +	 * Check if there is any buffer with error in the next position of the queue,
> > +	 * buffers whose in-fence signaled with error are not queued to the driver
> > +	 * and kept on the queue until the buffer before them is done, so to not
> > +	 * deliver buffers back to userspace in the wrong order. Here we process
> > +	 * any existing buffers with errors and wake up userspace.
> > +	 */
> > +	for (;;) {
> 
> I think this should test if the list is empty before calling list_next_entry().
> 
> > +		vb = list_next_entry(vb, queued_entry);
> > +		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
> 
> The !vb check makes no sense. list_next_entry won't return NULL if there is no
> next entry, you need to call list_empty() first.
> 

Got it.

> > +			break;
> > +
> > +		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
> > +        }
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_buffer_done);
> >  
> > @@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  
> > +	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> > +		return;
> > +
> >  	vb->state = VB2_BUF_STATE_ACTIVE;
> >  	atomic_inc(&q->owned_by_drv_count);
> >  
> > @@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
> >  	return 0;
> >  }
> >  
> > +static int __get_num_ready_buffers(struct vb2_queue *q)
> > +{
> > +	struct vb2_buffer *vb;
> > +	int ready_count = 0;
> > +	unsigned long flags;
> > +
> > +	/* count num of buffers ready in front of the queued_list */
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> > +			break;
> > +		ready_count++;
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +	}
> > +
> > +	return ready_count;
> > +}
> > +
> >  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> >  {
> >  	struct vb2_buffer *vb;
> > @@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
> >  	return ret;
> >  }
> >  
> > -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> > +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> > +{
> > +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> > +	struct vb2_queue *q = vb->vb2_queue;
> > +	unsigned long flags;
> > +
> > +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +	/*
> > +	 * If the fence signals with an error we mark the buffer as such
> > +	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
> > +	 * not queueing it to the driver. However we can't notify the error
> > +	 * to userspace right now because, at the time this callback run, QBUF
> > +	 * has returned already.
> > +	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
> > +	 * as well.
> > +	 */
> > +	if (vb->in_fence->error)
> > +		vb->state = VB2_BUF_STATE_ERROR;
> > +
> > +	dma_fence_put(vb->in_fence);
> > +	vb->in_fence = NULL;
> > +
> > +	if (vb->state == VB2_BUF_STATE_ERROR) {
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +		return;
> > +	}
> > +
> > +	if (q->start_streaming_called)
> 
> I'd change this to:
> 
> 	if (q->start_streaming_called &&
> 	    vb->state != VB2_BUF_STATE_ERROR)
> 
> Then you can drop the previous 'if'.
> 

OK.

> > +		__enqueue_in_driver(vb);
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +}
> > +
> > +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> > +		  struct dma_fence *in_fence)
> >  {
> >  	struct vb2_buffer *vb;
> > +	unsigned long flags;
> >  	int ret;
> >  
> >  	vb = q->bufs[index];
> > @@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	case VB2_BUF_STATE_DEQUEUED:
> >  		ret = __buf_prepare(vb, pb);
> >  		if (ret)
> > -			return ret;
> > +			goto err;
> >  		break;
> >  	case VB2_BUF_STATE_PREPARED:
> >  		break;
> >  	case VB2_BUF_STATE_PREPARING:
> >  		dprintk(1, "buffer still being prepared\n");
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err;
> >  	default:
> >  		dprintk(1, "invalid buffer state %d\n", vb->state);
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err;
> >  	}
> >  
> >  	/*
> > @@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	q->queued_count++;
> >  	q->waiting_for_buffers = false;
> >  	vb->state = VB2_BUF_STATE_QUEUED;
> > +	vb->in_fence = in_fence;
> >  
> >  	if (pb)
> >  		call_void_bufop(q, copy_timestamp, vb, pb);
> > @@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	trace_vb2_qbuf(q, vb);
> >  
> >  	/*
> > -	 * If already streaming, give the buffer to driver for processing.
> > -	 * If not, the buffer will be given to driver on next streamon.
> > +	 * For explicit synchronization: If the fence didn't signal
> > +	 * yet we setup a callback to queue the buffer once the fence
> > +	 * signals and then return successfully. But if the fence
> > +	 * already signaled we lose the reference we held and queue the
> > +	 * buffer to the driver.
> 
> What happens if the fence signaled an error? Is that error returned to userspace?
> (i.e. VIDIOC_QBUF will fail in that case)
> 

Hm, good question. If the fence signals with an error, we won't catch it apparently.
We should fix dma_fence_add_callback to know about signaled vs. error signaled.

> >  	 */
> > -	if (q->start_streaming_called)
> > -		__enqueue_in_driver(vb);
> > +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +	if (vb->in_fence) {
> > +		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
> > +					     vb2_qbuf_fence_cb);
> > +		/* is the fence signaled? */
> > +		if (ret == -ENOENT) {
> > +			dma_fence_put(vb->in_fence);
> > +			vb->in_fence = NULL;
> > +		} else if (ret) {
> > +			goto unlock;
> > +		}
> > +	}
> >  
> > -	/* Fill buffer information for the userspace */
> > -	if (pb)
> > -		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +	/*
> > +	 * If already streaming and there is no fence to wait on
> > +	 * give the buffer to driver for processing.
> > +	 */
> > +	if (q->start_streaming_called) {
> > +		struct vb2_buffer *b;
> > +
> > +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> > +			if (b->state != VB2_BUF_STATE_QUEUED)
> > +				continue;
> > +			if (b->in_fence)
> > +				break;
> > +			__enqueue_in_driver(b);
> > +		}
> > +	}
> >  
> >  	/*
> >  	 * If streamon has been called, and we haven't yet called
> > @@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> >  	 * then we can finally call start_streaming().
> >  	 */
> >  	if (q->streaming && !q->start_streaming_called &&
> > -	    q->queued_count >= q->min_buffers_needed) {
> > +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
> >  		ret = vb2_start_streaming(q);
> >  		if (ret)
> > -			return ret;
> > +			goto unlock;
> >  	}
> >  
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +
> > +	/* Fill buffer information for the userspace */
> 
> s/the//
> 
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> >  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
> >  	return 0;
> > +
> > +unlock:
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +
> > +err:
> > +	/* Fill buffer information for the userspace */
> 
> s/the//
> 
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> > +	if (vb->in_fence) {
> > +		dma_fence_put(vb->in_fence);
> > +		vb->in_fence = NULL;
> > +	}
> > +
> > +	return ret;
> > +
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
> >  
> > @@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
> >  static void __vb2_queue_cancel(struct vb2_queue *q)
> >  {
> >  	unsigned int i;
> > +	struct vb2_buffer *vb;
> > +	unsigned long flags;
> >  
> >  	/*
> >  	 * Tell driver to stop all transactions and release all queued
> > @@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> >  	q->queued_count = 0;
> >  	q->error = 0;
> >  
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +		if (vb->in_fence) {
> > +			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
> > +			dma_fence_put(vb->in_fence);
> > +			vb->in_fence = NULL;
> > +		}
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +	}
> > +
> >  	/*
> >  	 * Remove all buffers from videobuf's list...
> >  	 */
> > @@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
> >  	 * Tell driver to start streaming provided sufficient buffers
> >  	 * are available.
> >  	 */
> > -	if (q->queued_count >= q->min_buffers_needed) {
> > +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
> >  		ret = v4l_vb2q_enable_media_source(q);
> >  		if (ret)
> >  			return ret;
> > @@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
> >  		 * Queue all buffers.
> >  		 */
> >  		for (i = 0; i < q->num_buffers; i++) {
> > -			ret = vb2_core_qbuf(q, i, NULL);
> > +			ret = vb2_core_qbuf(q, i, NULL, NULL);
> >  			if (ret)
> >  				goto err_reqbufs;
> >  			fileio->bufs[i].queued = 1;
> > @@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
> >  
> >  		if (copy_timestamp)
> >  			b->timestamp = ktime_get_ns();
> > -		ret = vb2_core_qbuf(q, index, NULL);
> > +		ret = vb2_core_qbuf(q, index, NULL, NULL);
> >  		dprintk(5, "vb2_dbuf result: %d\n", ret);
> >  		if (ret)
> >  			return ret;
> > @@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
> >  		if (copy_timestamp)
> >  			vb->timestamp = ktime_get_ns();
> >  		if (!threadio->stop)
> > -			ret = vb2_core_qbuf(q, vb->index, NULL);
> > +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
> >  		call_void_qop(q, wait_prepare, q);
> >  		if (ret || threadio->stop)
> >  			break;
> > diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > index b1c0fa2b0b88..74d7062e5285 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/sched.h>
> >  #include <linux/freezer.h>
> >  #include <linux/kthread.h>
> > +#include <linux/sync_file.h>
> >  
> >  #include <media/v4l2-dev.h>
> >  #include <media/v4l2-fh.h>
> > @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
> >  		return -EINVAL;
> >  	}
> >  
> > +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> > +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
> > +		return -EINVAL;
> > +	}
> > +
> > +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
> > +		return -EINVAL;
> > +	}
> > +
> >  	return __verify_planes_array(q->bufs[b->index], b);
> >  }
> >  
> > @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> >  	b->timestamp = ns_to_timeval(vb->timestamp);
> >  	b->timecode = vbuf->timecode;
> >  	b->sequence = vbuf->sequence;
> > -	b->fence_fd = 0;
> >  	b->reserved = 0;
> >  
> > +	b->fence_fd = 0;
> > +	if (vb->in_fence)
> > +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> > +	else
> > +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> > +
> >  	if (q->is_multiplanar) {
> >  		/*
> >  		 * Fill in plane-related data if userspace provided an array
> > @@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
> >  
> >  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> >  {
> > +	struct dma_fence *in_fence = NULL;
> >  	int ret;
> >  
> >  	if (vb2_fileio_is_active(q)) {
> > @@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> >  	}
> >  
> >  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> > -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> > +		in_fence = sync_file_get_fence(b->fence_fd);
> > +		if (!in_fence) {
> > +			dprintk(1, "failed to get in-fence from fd %d\n",
> > +				b->fence_fd);
> > +			return -EINVAL;
> > +		}
> > +	}
> > +
> > +	return vb2_core_qbuf(q, b->index, b, in_fence);
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_qbuf);
> >  
> > diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
> > index b811adf88afa..7da53f10db1a 100644
> > --- a/drivers/media/dvb-core/dvb_vb2.c
> > +++ b/drivers/media/dvb-core/dvb_vb2.c
> > @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
> >  {
> >  	int ret;
> >  
> > -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
> > +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
> >  	if (ret) {
> >  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
> >  			b->index, ret);
> > diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
> > index 8e37e7c5e0f7..79dfb5dfd1fc 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
> >  config VIDEOBUF_DVB
> >  	tristate
> >  	select VIDEOBUF_GEN
> > +
> > +# Used by drivers that need Videobuf2 modules
> > +config VIDEOBUF2_CORE
> > +	select DMA_SHARED_BUFFER
> > +	select SYNC_FILE
> > +	tristate
> > +
> > +config VIDEOBUF2_MEMOPS
> > +	tristate
> > +	select FRAME_VECTOR
> > +
> > +config VIDEOBUF2_DMA_CONTIG
> > +	tristate
> > +	depends on HAS_DMA
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +	select DMA_SHARED_BUFFER
> > +
> > +config VIDEOBUF2_VMALLOC
> > +	tristate
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +	select DMA_SHARED_BUFFER
> > +
> > +config VIDEOBUF2_DMA_SG
> > +	tristate
> > +	depends on HAS_DMA
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +
> > +config VIDEOBUF2_DVB
> > +	tristate
> > +	select VIDEOBUF2_CORE
> 
> This chunk doesn't really belong here. Should be a separate patch with an explanation
> why.
> 

OK.

> > diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> > index 364e4cb41b10..28ce8f66882e 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -17,6 +17,7 @@
> >  #include <linux/poll.h>
> >  #include <linux/dma-buf.h>
> >  #include <linux/bitops.h>
> > +#include <linux/dma-fence.h>
> >  
> >  #define VB2_MAX_FRAME	(32)
> >  #define VB2_MAX_PLANES	(8)
> > @@ -255,12 +256,21 @@ struct vb2_buffer {
> >  	 * done_entry:		entry on the list that stores all buffers ready
> >  	 *			to be dequeued to userspace
> >  	 * vb2_plane:		per-plane information; do not change
> > +	 * in_fence:		fence received from vb2 client to wait on before
> > +	 *			using the buffer (queueing to the driver)
> > +	 * fence_cb:		fence callback information
> > +	 * fence_cb_lock:	protect callback signal/remove
> >  	 */
> >  	enum vb2_buffer_state	state;
> >  
> >  	struct vb2_plane	planes[VB2_MAX_PLANES];
> >  	struct list_head	queued_entry;
> >  	struct list_head	done_entry;
> > +
> > +	struct dma_fence	*in_fence;
> > +	struct dma_fence_cb	fence_cb;
> > +	spinlock_t              fence_cb_lock;
> > +
> 
> So for the _MPLANE formats this is one fence for all planes. Which makes sense, but how
> does drm handle that? Also one fence for all planes?
> 

In drm we have one fence per plane, where as on this proposal
we support one fence per vb2_buffer (one for all planes).

I guess this means we only support single-planar formats por now.
Multi-planar support can be added as a follow-up patch.

> I think there should be a comment about this somewhere.
> 

Agreed.

> >  #ifdef CONFIG_VIDEO_ADV_DEBUG
> >  	/*
> >  	 * Counters for how often these buffer-related ops are
> > @@ -773,6 +783,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
> >   * @index:	id number of the buffer
> >   * @pb:		buffer structure passed from userspace to
> >   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> > + * @in_fence:	in-fence to wait on before queueing the buffer
> >   *
> >   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
> >   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> > @@ -787,7 +798,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
> >   *
> >   * Return: returns zero on success; an error code otherwise.
> >   */
> > -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> > +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> > +		  struct dma_fence *in_fence);
> >  
> >  /**
> >   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> > 
> 
> Regards,
> 
> 	Hans

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 05/08/2018 09:16 PM, Ezequiel Garcia wrote:
> On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
>> On 04/05/18 22:06, Ezequiel Garcia wrote:
>>> @@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>  	trace_vb2_qbuf(q, vb);
>>>  
>>>  	/*
>>> -	 * If already streaming, give the buffer to driver for processing.
>>> -	 * If not, the buffer will be given to driver on next streamon.
>>> +	 * For explicit synchronization: If the fence didn't signal
>>> +	 * yet we setup a callback to queue the buffer once the fence
>>> +	 * signals and then return successfully. But if the fence
>>> +	 * already signaled we lose the reference we held and queue the
>>> +	 * buffer to the driver.
>>
>> What happens if the fence signaled an error? Is that error returned to userspace?
>> (i.e. VIDIOC_QBUF will fail in that case)
>>
> 
> Hm, good question. If the fence signals with an error, we won't catch it apparently.
> We should fix dma_fence_add_callback to know about signaled vs. error signaled.

OK, so in the meantime we need a comment explaining this in the code. Perhaps as
a FIXME or TODO.

Regards,

	Hans

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Gustavo Padovan]

Hi Hans,

On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
> On 04/05/18 22:06, Ezequiel Garcia wrote:
> > From: Gustavo Padovan <gustavo.padovan@collabora.com>
> > 
> > Receive in-fence from userspace and add support for waiting on them
> > before queueing the buffer to the driver. Buffers can't be queued
> > to the
> > driver before its fences signal. And a buffer can't be queued to
> > the driver
> > out of the order they were queued from userspace. That means that
> > even if
> > its fence signals it must wait for all other buffers, ahead of it
> > in the queue,
> > to signal first.
> > 
> > If the fence for some buffer fails we do not queue it to the
> > driver,
> > instead we mark it as error and wait until the previous buffer is
> > done
> > to notify userspace of the error. We wait here to deliver the
> > buffers back
> > to userspace in order.
> > 
> > v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
> > 
> > v11: - minor doc/comments fixes (Hans Verkuil)
> >      - reviewed the in-fence path at __fill_v4l2_buffer()
> > 
> > v10: - rename fence to in_fence in many places
> >      - handle fences signalling with error better (Hans Verkuil)
> > 
> > v9: - improve comments and docs (Hans Verkuil)
> >     - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans
> > Verkuil)
> >     - move in-fences code that was in the out-fences patch here
> > (Alex)
> > 
> > v8: - improve comments about fences with errors
> > 
> > v7: - get rid of the fence array stuff for ordering and just use
> >       get_num_buffers_ready() (Hans)
> >     - fix issue of queuing the buffer twice (Hans)
> >     - avoid the dma_fence_wait() in core_qbuf() (Alex)
> >     - merge preparation commit in
> > 
> > v6: - With fences always keep the order userspace queues the
> > buffers.
> >     - Protect in_fence manipulation with a lock (Brian Starkey)
> >     - check if fences have the same context before adding a fence
> > array
> >     - Fix last_fence ref unbalance in __set_in_fence() (Brian
> > Starkey)
> >     - Clean up fence if __set_in_fence() fails (Brian Starkey)
> >     - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> > 
> > v5: - use fence_array to keep buffers ordered in vb2 core when
> >       needed (Brian Starkey)
> >     - keep backward compat on the reserved2 field (Brian Starkey)
> >     - protect fence callback removal with lock (Brian Starkey)
> > 
> > v4: - Add a comment about dma_fence_add_callback() not returning a
> >       error (Hans)
> >     - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> >     - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> >     - Move dma_fence_is_signaled() check to __enqueue_in_driver()
> > (Hans)
> >     - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> >     - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> >       vb2_start_streaming() (Hans)
> >     - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> >     - Queue buffers to the driver as soon as they are ready (Hans)
> >     - call fill_user_buffer() after queuing the buffer (Hans)
> >     - add err: label to clean up fence
> >     - add dma_fence_wait() before calling vb2_start_streaming()
> > 
> > v3: - document fence parameter
> >     - remove ternary if at vb2_qbuf() return (Mauro)
> >     - do not change if conditions behaviour (Mauro)
> > 
> > v2: - fix vb2_queue_or_prepare_buf() ret check
> >     - remove check for VB2_MEMORY_DMABUF only (Javier)
> >     - check num of ready buffers to start streaming
> >     - when queueing, start from the first ready buffer
> >     - handle queue cancel
> > 
> > Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  drivers/media/common/videobuf2/videobuf2-core.c | 197
> > ++++++++++++++++++++----
> >  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
> >  drivers/media/dvb-core/dvb_vb2.c                |   2 +-
> >  drivers/media/v4l2-core/Kconfig                 |  33 ++++
> >  include/media/videobuf2-core.h                  |  14 +-
> >  5 files changed, 249 insertions(+), 31 deletions(-)
> > 
> > diff --git a/drivers/media/common/videobuf2/videobuf2-core.c
> > b/drivers/media/common/videobuf2/videobuf2-core.c
> > index 6b8e083893ad..996b99497a98 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-core.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> > @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue
> > *q, enum vb2_memory memory,
> >  		vb->index = q->num_buffers + buffer;
> >  		vb->type = q->type;
> >  		vb->memory = memory;
> > +		spin_lock_init(&vb->fence_cb_lock);
> >  		for (plane = 0; plane < num_planes; ++plane) {
> >  			vb->planes[plane].length =
> > plane_sizes[plane];
> >  			vb->planes[plane].min_length =
> > plane_sizes[plane];
> > @@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb,
> > unsigned int plane_no)
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
> >  
> > -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
> > state)
> > +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum
> > vb2_buffer_state state)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  	unsigned long flags;
> >  	unsigned int plane;
> >  
> > -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > -		return;
> > -
> > -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > -		    state != VB2_BUF_STATE_ERROR &&
> > -		    state != VB2_BUF_STATE_QUEUED &&
> > -		    state != VB2_BUF_STATE_REQUEUEING))
> > -		state = VB2_BUF_STATE_ERROR;
> >  
> >  #ifdef CONFIG_VIDEO_ADV_DEBUG
> >  	/*
> > @@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb,
> > enum vb2_buffer_state state)
> >  		call_void_memop(vb, finish, vb-
> > >planes[plane].mem_priv);
> >  
> >  	spin_lock_irqsave(&q->done_lock, flags);
> > +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> > +		atomic_dec(&q->owned_by_drv_count);
> > +
> >  	if (state == VB2_BUF_STATE_QUEUED ||
> >  	    state == VB2_BUF_STATE_REQUEUEING) {
> >  		vb->state = VB2_BUF_STATE_QUEUED;
> > @@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb,
> > enum vb2_buffer_state state)
> >  		list_add_tail(&vb->done_entry, &q->done_list);
> >  		vb->state = state;
> >  	}
> > -	atomic_dec(&q->owned_by_drv_count);
> > +
> >  	spin_unlock_irqrestore(&q->done_lock, flags);
> >  
> >  	trace_vb2_buf_done(q, vb);
> > @@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb,
> > enum vb2_buffer_state state)
> >  		wake_up(&q->done_wq);
> >  		break;
> >  	}
> > +
> > +}
> > +
> > +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
> > state)
> > +{
> > +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> > +		return;
> > +
> > +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> > +		    state != VB2_BUF_STATE_ERROR &&
> > +		    state != VB2_BUF_STATE_QUEUED &&
> > +		    state != VB2_BUF_STATE_REQUEUEING))
> > +		state = VB2_BUF_STATE_ERROR;
> > +
> > +	vb2_process_buffer_done(vb, state);
> > +
> > +	/*
> > +	 * Check if there is any buffer with error in the next
> > position of the queue,
> > +	 * buffers whose in-fence signaled with error are not
> > queued to the driver
> > +	 * and kept on the queue until the buffer before them is
> > done, so to not
> > +	 * deliver buffers back to userspace in the wrong order.
> > Here we process
> > +	 * any existing buffers with errors and wake up userspace.
> > +	 */
> > +	for (;;) {
> 
> I think this should test if the list is empty before calling
> list_next_entry().
> 
> > +		vb = list_next_entry(vb, queued_entry);
> > +		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
> 
> The !vb check makes no sense. list_next_entry won't return NULL if
> there is no
> next entry, you need to call list_empty() first.
> 
> > +			break;
> > +
> > +		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
> > +        }
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_buffer_done);
> >  
> > @@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct
> > vb2_buffer *vb)
> >  {
> >  	struct vb2_queue *q = vb->vb2_queue;
> >  
> > +	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> > +		return;
> > +
> >  	vb->state = VB2_BUF_STATE_ACTIVE;
> >  	atomic_inc(&q->owned_by_drv_count);
> >  
> > @@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer
> > *vb, const void *pb)
> >  	return 0;
> >  }
> >  
> > +static int __get_num_ready_buffers(struct vb2_queue *q)
> > +{
> > +	struct vb2_buffer *vb;
> > +	int ready_count = 0;
> > +	unsigned long flags;
> > +
> > +	/* count num of buffers ready in front of the queued_list
> > */
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +		if (vb->in_fence && !dma_fence_is_signaled(vb-
> > >in_fence))
> > +			break;
> > +		ready_count++;
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +	}
> > +
> > +	return ready_count;
> > +}
> > +
> >  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index,
> > void *pb)
> >  {
> >  	struct vb2_buffer *vb;
> > @@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct
> > vb2_queue *q)
> >  	return ret;
> >  }
> >  
> > -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void
> > *pb)
> > +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct
> > dma_fence_cb *cb)
> > +{
> > +	struct vb2_buffer *vb = container_of(cb, struct
> > vb2_buffer, fence_cb);
> > +	struct vb2_queue *q = vb->vb2_queue;
> > +	unsigned long flags;
> > +
> > +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +	/*
> > +	 * If the fence signals with an error we mark the buffer
> > as such
> > +	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR
> > and
> > +	 * not queueing it to the driver. However we can't notify
> > the error
> > +	 * to userspace right now because, at the time this
> > callback run, QBUF
> > +	 * has returned already.
> > +	 * So we delay that to DQBUF time. See comments in
> > vb2_buffer_done()
> > +	 * as well.
> > +	 */
> > +	if (vb->in_fence->error)
> > +		vb->state = VB2_BUF_STATE_ERROR;
> > +
> > +	dma_fence_put(vb->in_fence);
> > +	vb->in_fence = NULL;
> > +
> > +	if (vb->state == VB2_BUF_STATE_ERROR) {
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +		return;
> > +	}
> > +
> > +	if (q->start_streaming_called)
> 
> I'd change this to:
> 
> 	if (q->start_streaming_called &&
> 	    vb->state != VB2_BUF_STATE_ERROR)
> 
> Then you can drop the previous 'if'.
> 
> > +		__enqueue_in_driver(vb);
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +}
> > +
> > +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void
> > *pb,
> > +		  struct dma_fence *in_fence)
> >  {
> >  	struct vb2_buffer *vb;
> > +	unsigned long flags;
> >  	int ret;
> >  
> >  	vb = q->bufs[index];
> > @@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q,
> > unsigned int index, void *pb)
> >  	case VB2_BUF_STATE_DEQUEUED:
> >  		ret = __buf_prepare(vb, pb);
> >  		if (ret)
> > -			return ret;
> > +			goto err;
> >  		break;
> >  	case VB2_BUF_STATE_PREPARED:
> >  		break;
> >  	case VB2_BUF_STATE_PREPARING:
> >  		dprintk(1, "buffer still being prepared\n");
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err;
> >  	default:
> >  		dprintk(1, "invalid buffer state %d\n", vb-
> > >state);
> > -		return -EINVAL;
> > +		ret = -EINVAL;
> > +		goto err;
> >  	}
> >  
> >  	/*
> > @@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q,
> > unsigned int index, void *pb)
> >  	q->queued_count++;
> >  	q->waiting_for_buffers = false;
> >  	vb->state = VB2_BUF_STATE_QUEUED;
> > +	vb->in_fence = in_fence;
> >  
> >  	if (pb)
> >  		call_void_bufop(q, copy_timestamp, vb, pb);
> > @@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q,
> > unsigned int index, void *pb)
> >  	trace_vb2_qbuf(q, vb);
> >  
> >  	/*
> > -	 * If already streaming, give the buffer to driver for
> > processing.
> > -	 * If not, the buffer will be given to driver on next
> > streamon.
> > +	 * For explicit synchronization: If the fence didn't
> > signal
> > +	 * yet we setup a callback to queue the buffer once the
> > fence
> > +	 * signals and then return successfully. But if the fence
> > +	 * already signaled we lose the reference we held and
> > queue the
> > +	 * buffer to the driver.
> 
> What happens if the fence signaled an error? Is that error returned
> to userspace?
> (i.e. VIDIOC_QBUF will fail in that case)
> 
> >  	 */
> > -	if (q->start_streaming_called)
> > -		__enqueue_in_driver(vb);
> > +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +	if (vb->in_fence) {
> > +		ret = dma_fence_add_callback(vb->in_fence, &vb-
> > >fence_cb,
> > +					     vb2_qbuf_fence_cb);
> > +		/* is the fence signaled? */
> > +		if (ret == -ENOENT) {
> > +			dma_fence_put(vb->in_fence);
> > +			vb->in_fence = NULL;
> > +		} else if (ret) {
> > +			goto unlock;
> > +		}
> > +	}
> >  
> > -	/* Fill buffer information for the userspace */
> > -	if (pb)
> > -		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +	/*
> > +	 * If already streaming and there is no fence to wait on
> > +	 * give the buffer to driver for processing.
> > +	 */
> > +	if (q->start_streaming_called) {
> > +		struct vb2_buffer *b;
> > +
> > +		list_for_each_entry(b, &q->queued_list,
> > queued_entry) {
> > +			if (b->state != VB2_BUF_STATE_QUEUED)
> > +				continue;
> > +			if (b->in_fence)
> > +				break;
> > +			__enqueue_in_driver(b);
> > +		}
> > +	}
> >  
> >  	/*
> >  	 * If streamon has been called, and we haven't yet called
> > @@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q,
> > unsigned int index, void *pb)
> >  	 * then we can finally call start_streaming().
> >  	 */
> >  	if (q->streaming && !q->start_streaming_called &&
> > -	    q->queued_count >= q->min_buffers_needed) {
> > +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
> >  		ret = vb2_start_streaming(q);
> >  		if (ret)
> > -			return ret;
> > +			goto unlock;
> >  	}
> >  
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +
> > +	/* Fill buffer information for the userspace */
> 
> s/the//
> 
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> >  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
> >  	return 0;
> > +
> > +unlock:
> > +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +
> > +err:
> > +	/* Fill buffer information for the userspace */
> 
> s/the//
> 
> > +	if (pb)
> > +		call_void_bufop(q, fill_user_buffer, vb, pb);
> > +
> > +	if (vb->in_fence) {
> > +		dma_fence_put(vb->in_fence);
> > +		vb->in_fence = NULL;
> > +	}
> > +
> > +	return ret;
> > +
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
> >  
> > @@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
> >  static void __vb2_queue_cancel(struct vb2_queue *q)
> >  {
> >  	unsigned int i;
> > +	struct vb2_buffer *vb;
> > +	unsigned long flags;
> >  
> >  	/*
> >  	 * Tell driver to stop all transactions and release all
> > queued
> > @@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct
> > vb2_queue *q)
> >  	q->queued_count = 0;
> >  	q->error = 0;
> >  
> > +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> > +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> > +		if (vb->in_fence) {
> > +			dma_fence_remove_callback(vb->in_fence,
> > &vb->fence_cb);
> > +			dma_fence_put(vb->in_fence);
> > +			vb->in_fence = NULL;
> > +		}
> > +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> > +	}
> > +
> >  	/*
> >  	 * Remove all buffers from videobuf's list...
> >  	 */
> > @@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q,
> > unsigned int type)
> >  	 * Tell driver to start streaming provided sufficient
> > buffers
> >  	 * are available.
> >  	 */
> > -	if (q->queued_count >= q->min_buffers_needed) {
> > +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
> >  		ret = v4l_vb2q_enable_media_source(q);
> >  		if (ret)
> >  			return ret;
> > @@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue
> > *q, int read)
> >  		 * Queue all buffers.
> >  		 */
> >  		for (i = 0; i < q->num_buffers; i++) {
> > -			ret = vb2_core_qbuf(q, i, NULL);
> > +			ret = vb2_core_qbuf(q, i, NULL, NULL);
> >  			if (ret)
> >  				goto err_reqbufs;
> >  			fileio->bufs[i].queued = 1;
> > @@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct
> > vb2_queue *q, char __user *data, size_
> >  
> >  		if (copy_timestamp)
> >  			b->timestamp = ktime_get_ns();
> > -		ret = vb2_core_qbuf(q, index, NULL);
> > +		ret = vb2_core_qbuf(q, index, NULL, NULL);
> >  		dprintk(5, "vb2_dbuf result: %d\n", ret);
> >  		if (ret)
> >  			return ret;
> > @@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
> >  		if (copy_timestamp)
> >  			vb->timestamp = ktime_get_ns();
> >  		if (!threadio->stop)
> > -			ret = vb2_core_qbuf(q, vb->index, NULL);
> > +			ret = vb2_core_qbuf(q, vb->index, NULL,
> > NULL);
> >  		call_void_qop(q, wait_prepare, q);
> >  		if (ret || threadio->stop)
> >  			break;
> > diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > index b1c0fa2b0b88..74d7062e5285 100644
> > --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> > @@ -23,6 +23,7 @@
> >  #include <linux/sched.h>
> >  #include <linux/freezer.h>
> >  #include <linux/kthread.h>
> > +#include <linux/sync_file.h>
> >  
> >  #include <media/v4l2-dev.h>
> >  #include <media/v4l2-fh.h>
> > @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct
> > vb2_queue *q, struct v4l2_buffer *b,
> >  		return -EINVAL;
> >  	}
> >  
> > +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> > +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> > +		dprintk(1, "%s: fence_fd set without IN_FENCE
> > flag\n", opname);
> > +		return -EINVAL;
> > +	}
> > +
> > +	if (b->fence_fd < 0 && (b->flags &
> > V4L2_BUF_FLAG_IN_FENCE)) {
> > +		dprintk(1, "%s: IN_FENCE flag set but no
> > fence_fd\n", opname);
> > +		return -EINVAL;
> > +	}
> > +
> >  	return __verify_planes_array(q->bufs[b->index], b);
> >  }
> >  
> > @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct
> > vb2_buffer *vb, void *pb)
> >  	b->timestamp = ns_to_timeval(vb->timestamp);
> >  	b->timecode = vbuf->timecode;
> >  	b->sequence = vbuf->sequence;
> > -	b->fence_fd = 0;
> >  	b->reserved = 0;
> >  
> > +	b->fence_fd = 0;
> > +	if (vb->in_fence)
> > +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> > +	else
> > +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> > +
> >  	if (q->is_multiplanar) {
> >  		/*
> >  		 * Fill in plane-related data if userspace
> > provided an array
> > @@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
> >  
> >  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> >  {
> > +	struct dma_fence *in_fence = NULL;
> >  	int ret;
> >  
> >  	if (vb2_fileio_is_active(q)) {
> > @@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct
> > v4l2_buffer *b)
> >  	}
> >  
> >  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> > -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> > +	if (ret)
> > +		return ret;
> > +
> > +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> > +		in_fence = sync_file_get_fence(b->fence_fd);
> > +		if (!in_fence) {
> > +			dprintk(1, "failed to get in-fence from fd
> > %d\n",
> > +				b->fence_fd);
> > +			return -EINVAL;
> > +		}
> > +	}
> > +
> > +	return vb2_core_qbuf(q, b->index, b, in_fence);
> >  }
> >  EXPORT_SYMBOL_GPL(vb2_qbuf);
> >  
> > diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-
> > core/dvb_vb2.c
> > index b811adf88afa..7da53f10db1a 100644
> > --- a/drivers/media/dvb-core/dvb_vb2.c
> > +++ b/drivers/media/dvb-core/dvb_vb2.c
> > @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx,
> > struct dmx_buffer *b)
> >  {
> >  	int ret;
> >  
> > -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
> > +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
> >  	if (ret) {
> >  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
> >  			b->index, ret);
> > diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-
> > core/Kconfig
> > index 8e37e7c5e0f7..79dfb5dfd1fc 100644
> > --- a/drivers/media/v4l2-core/Kconfig
> > +++ b/drivers/media/v4l2-core/Kconfig
> > @@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
> >  config VIDEOBUF_DVB
> >  	tristate
> >  	select VIDEOBUF_GEN
> > +
> > +# Used by drivers that need Videobuf2 modules
> > +config VIDEOBUF2_CORE
> > +	select DMA_SHARED_BUFFER
> > +	select SYNC_FILE
> > +	tristate
> > +
> > +config VIDEOBUF2_MEMOPS
> > +	tristate
> > +	select FRAME_VECTOR
> > +
> > +config VIDEOBUF2_DMA_CONTIG
> > +	tristate
> > +	depends on HAS_DMA
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +	select DMA_SHARED_BUFFER
> > +
> > +config VIDEOBUF2_VMALLOC
> > +	tristate
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +	select DMA_SHARED_BUFFER
> > +
> > +config VIDEOBUF2_DMA_SG
> > +	tristate
> > +	depends on HAS_DMA
> > +	select VIDEOBUF2_CORE
> > +	select VIDEOBUF2_MEMOPS
> > +
> > +config VIDEOBUF2_DVB
> > +	tristate
> > +	select VIDEOBUF2_CORE
> 
> This chunk doesn't really belong here. Should be a separate patch
> with an explanation
> why.
> 
> > diff --git a/include/media/videobuf2-core.h
> > b/include/media/videobuf2-core.h
> > index 364e4cb41b10..28ce8f66882e 100644
> > --- a/include/media/videobuf2-core.h
> > +++ b/include/media/videobuf2-core.h
> > @@ -17,6 +17,7 @@
> >  #include <linux/poll.h>
> >  #include <linux/dma-buf.h>
> >  #include <linux/bitops.h>
> > +#include <linux/dma-fence.h>
> >  
> >  #define VB2_MAX_FRAME	(32)
> >  #define VB2_MAX_PLANES	(8)
> > @@ -255,12 +256,21 @@ struct vb2_buffer {
> >  	 * done_entry:		entry on the list that
> > stores all buffers ready
> >  	 *			to be dequeued to userspace
> >  	 * vb2_plane:		per-plane information; do not
> > change
> > +	 * in_fence:		fence received from vb2 client
> > to wait on before
> > +	 *			using the buffer (queueing to
> > the driver)
> > +	 * fence_cb:		fence callback information
> > +	 * fence_cb_lock:	protect callback signal/remove
> >  	 */
> >  	enum vb2_buffer_state	state;
> >  
> >  	struct vb2_plane	planes[VB2_MAX_PLANES];
> >  	struct list_head	queued_entry;
> >  	struct list_head	done_entry;
> > +
> > +	struct dma_fence	*in_fence;
> > +	struct dma_fence_cb	fence_cb;
> > +	spinlock_t              fence_cb_lock;
> > +
> 
> So for the _MPLANE formats this is one fence for all planes. Which
> makes sense, but how
> does drm handle that? Also one fence for all planes?

Yes, this is one fence for all planes. 

The DRM concept for planes is a totally different concept and is
basically a representation of an user definable square on the screen,
and to that plane there in one framebuffer attached - display hw has no
such a multiplanar for the same image AFAICT. So you probably need some
blit to convert the v4l2 multiplanar to a DRM framebuffer.

> 
> I think there should be a comment about this somewhere.

Yes, we've been over this exact discussion a few times :)
Having entirely different things with the same name is quite confusing.

Regards,

Gustavo

-- 
Gustavo Padovan
Principal Software Engineer
Collabora Ltd.

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Brian Starkey]

Hi,

On Tue, May 08, 2018 at 08:18:06PM -0300, Gustavo Padovan wrote:
>
>Hi Hans,
>
>On Mon, 2018-05-07 at 14:07 +0200, Hans Verkuil wrote:
>> On 04/05/18 22:06, Ezequiel Garcia wrote:
>> > From: Gustavo Padovan <gustavo.padovan@collabora.com>

[snip]

>> > diff --git a/include/media/videobuf2-core.h
>> > b/include/media/videobuf2-core.h
>> > index 364e4cb41b10..28ce8f66882e 100644
>> > --- a/include/media/videobuf2-core.h
>> > +++ b/include/media/videobuf2-core.h
>> > @@ -17,6 +17,7 @@
>> >  #include <linux/poll.h>
>> >  #include <linux/dma-buf.h>
>> >  #include <linux/bitops.h>
>> > +#include <linux/dma-fence.h>
>> >
>> >  #define VB2_MAX_FRAME	(32)
>> >  #define VB2_MAX_PLANES	(8)
>> > @@ -255,12 +256,21 @@ struct vb2_buffer {
>> >  	 * done_entry:		entry on the list that
>> > stores all buffers ready
>> >  	 *			to be dequeued to userspace
>> >  	 * vb2_plane:		per-plane information; do not
>> > change
>> > +	 * in_fence:		fence received from vb2 client
>> > to wait on before
>> > +	 *			using the buffer (queueing to
>> > the driver)
>> > +	 * fence_cb:		fence callback information
>> > +	 * fence_cb_lock:	protect callback signal/remove
>> >  	 */
>> >  	enum vb2_buffer_state	state;
>> >
>> >  	struct vb2_plane	planes[VB2_MAX_PLANES];
>> >  	struct list_head	queued_entry;
>> >  	struct list_head	done_entry;
>> > +
>> > +	struct dma_fence	*in_fence;
>> > +	struct dma_fence_cb	fence_cb;
>> > +	spinlock_t              fence_cb_lock;
>> > +
>>
>> So for the _MPLANE formats this is one fence for all planes. Which
>> makes sense, but how
>> does drm handle that? Also one fence for all planes?
>
>Yes, this is one fence for all planes.
>
>The DRM concept for planes is a totally different concept and is
>basically a representation of an user definable square on the screen,
>and to that plane there in one framebuffer attached - display hw has no
>such a multiplanar for the same image AFAICT. So you probably need some
>blit to convert the v4l2 multiplanar to a DRM framebuffer.
>

Lots of display hardware can do multi-planar formats, and there's
space in struct drm_framebuffer for up to 4 buffer handles (e.g. 3
handles are passed for Luma, Cr, and Cb when the framebuffer format is
DRM_FORMAT_YUV420) - like V4L2 MPLANE.

The V4L2 code here matches with the DRM "explicit sync"
(IN_FENCE_FD/OUT_FENCE_PTR) stuff, which is probably what we want.
The main difference is that in DRM, explicit fences aren't associated
with framebuffers, they're associated with the things using the
framebuffers - but practically it doesn't make a difference.

There can be per-buffer "implicit sync" via dma-buf reservation
objects, but I don't think this series should attempt to deal with
that.

Cheers,
-Brian

>>
>> I think there should be a comment about this somewhere.
>
>Yes, we've been over this exact discussion a few times :)
>Having entirely different things with the same name is quite confusing.
>
>Regards,
>
>Gustavo
>
>-- 
>Gustavo Padovan
>Principal Software Engineer
>Collabora Ltd.

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Brian Starkey]

Hi Ezequiel,

On Fri, May 04, 2018 at 05:06:08PM -0300, Ezequiel Garcia wrote:
>From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
>Receive in-fence from userspace and add support for waiting on them
>before queueing the buffer to the driver. Buffers can't be queued to the
>driver before its fences signal. And a buffer can't be queued to the driver
>out of the order they were queued from userspace. That means that even if
>its fence signals it must wait for all other buffers, ahead of it in the queue,
>to signal first.
>
>If the fence for some buffer fails we do not queue it to the driver,
>instead we mark it as error and wait until the previous buffer is done
>to notify userspace of the error. We wait here to deliver the buffers back
>to userspace in order.
>
>v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
>
>v11: - minor doc/comments fixes (Hans Verkuil)
>     - reviewed the in-fence path at __fill_v4l2_buffer()
>
>v10: - rename fence to in_fence in many places
>     - handle fences signalling with error better (Hans Verkuil)
>
>v9: - improve comments and docs (Hans Verkuil)
>    - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>    - move in-fences code that was in the out-fences patch here (Alex)
>
>v8: - improve comments about fences with errors
>
>v7: - get rid of the fence array stuff for ordering and just use
>      get_num_buffers_ready() (Hans)
>    - fix issue of queuing the buffer twice (Hans)
>    - avoid the dma_fence_wait() in core_qbuf() (Alex)
>    - merge preparation commit in
>
>v6: - With fences always keep the order userspace queues the buffers.
>    - Protect in_fence manipulation with a lock (Brian Starkey)
>    - check if fences have the same context before adding a fence array
>    - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>    - Clean up fence if __set_in_fence() fails (Brian Starkey)
>    - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
>
>v5: - use fence_array to keep buffers ordered in vb2 core when
>      needed (Brian Starkey)
>    - keep backward compat on the reserved2 field (Brian Starkey)
>    - protect fence callback removal with lock (Brian Starkey)
>
>v4: - Add a comment about dma_fence_add_callback() not returning a
>      error (Hans)
>    - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>    - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>    - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>    - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>    - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>      vb2_start_streaming() (Hans)
>    - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>    - Queue buffers to the driver as soon as they are ready (Hans)
>    - call fill_user_buffer() after queuing the buffer (Hans)
>    - add err: label to clean up fence
>    - add dma_fence_wait() before calling vb2_start_streaming()
>
>v3: - document fence parameter
>    - remove ternary if at vb2_qbuf() return (Mauro)
>    - do not change if conditions behaviour (Mauro)
>
>v2: - fix vb2_queue_or_prepare_buf() ret check
>    - remove check for VB2_MEMORY_DMABUF only (Javier)
>    - check num of ready buffers to start streaming
>    - when queueing, start from the first ready buffer
>    - handle queue cancel
>
>Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>---
> drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
> drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
> drivers/media/dvb-core/dvb_vb2.c                |   2 +-
> drivers/media/v4l2-core/Kconfig                 |  33 ++++
> include/media/videobuf2-core.h                  |  14 +-
> 5 files changed, 249 insertions(+), 31 deletions(-)
>
>diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>index 6b8e083893ad..996b99497a98 100644
>--- a/drivers/media/common/videobuf2/videobuf2-core.c
>+++ b/drivers/media/common/videobuf2/videobuf2-core.c
>@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
> 		vb->index = q->num_buffers + buffer;
> 		vb->type = q->type;
> 		vb->memory = memory;
>+		spin_lock_init(&vb->fence_cb_lock);
> 		for (plane = 0; plane < num_planes; ++plane) {
> 			vb->planes[plane].length = plane_sizes[plane];
> 			vb->planes[plane].min_length = plane_sizes[plane];
>@@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
> }
> EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>
>-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> {
> 	struct vb2_queue *q = vb->vb2_queue;
> 	unsigned long flags;
> 	unsigned int plane;
>
>-	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>-		return;
>-
>-	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>-		    state != VB2_BUF_STATE_ERROR &&
>-		    state != VB2_BUF_STATE_QUEUED &&
>-		    state != VB2_BUF_STATE_REQUEUEING))
>-		state = VB2_BUF_STATE_ERROR;
>
> #ifdef CONFIG_VIDEO_ADV_DEBUG
> 	/*
>@@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> 		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>
> 	spin_lock_irqsave(&q->done_lock, flags);
>+	if (vb->state == VB2_BUF_STATE_ACTIVE)
>+		atomic_dec(&q->owned_by_drv_count);
>+
> 	if (state == VB2_BUF_STATE_QUEUED ||
> 	    state == VB2_BUF_STATE_REQUEUEING) {
> 		vb->state = VB2_BUF_STATE_QUEUED;
>@@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> 		list_add_tail(&vb->done_entry, &q->done_list);
> 		vb->state = state;
> 	}
>-	atomic_dec(&q->owned_by_drv_count);
>+
> 	spin_unlock_irqrestore(&q->done_lock, flags);
>
> 	trace_vb2_buf_done(q, vb);
>@@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> 		wake_up(&q->done_wq);
> 		break;
> 	}
>+
>+}
>+
>+void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>+{
>+	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>+		return;
>+
>+	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>+		    state != VB2_BUF_STATE_ERROR &&
>+		    state != VB2_BUF_STATE_QUEUED &&
>+		    state != VB2_BUF_STATE_REQUEUEING))
>+		state = VB2_BUF_STATE_ERROR;
>+
>+	vb2_process_buffer_done(vb, state);
>+
>+	/*
>+	 * Check if there is any buffer with error in the next position of the queue,
>+	 * buffers whose in-fence signaled with error are not queued to the driver
>+	 * and kept on the queue until the buffer before them is done, so to not
>+	 * deliver buffers back to userspace in the wrong order. Here we process
>+	 * any existing buffers with errors and wake up userspace.
>+	 */
>+	for (;;) {
>+		vb = list_next_entry(vb, queued_entry);
>+		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
>+			break;
>+
>+		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
>+        }
> }
> EXPORT_SYMBOL_GPL(vb2_buffer_done);
>
>@@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
> {
> 	struct vb2_queue *q = vb->vb2_queue;
>
>+	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>+		return;

Seems like this is not needed now. Either __enqueue_in_driver() is
called when there's no in_fence; or it's called from the fence_cb in
which case we're guaranteed to already be signaled.

Might be worth a loud error if these conditions are ever true - but
the silent return is a leftover from old patches I think.

>+
> 	vb->state = VB2_BUF_STATE_ACTIVE;
> 	atomic_inc(&q->owned_by_drv_count);
>
>@@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
> 	return 0;
> }
>
>+static int __get_num_ready_buffers(struct vb2_queue *q)
>+{
>+	struct vb2_buffer *vb;
>+	int ready_count = 0;
>+	unsigned long flags;
>+
>+	/* count num of buffers ready in front of the queued_list */
>+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
>+		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>+			break;
>+		ready_count++;
>+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+	}
>+
>+	return ready_count;
>+}
>+
> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> {
> 	struct vb2_buffer *vb;
>@@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
> 	return ret;
> }
>
>-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>+static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>+{
>+	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
>+	struct vb2_queue *q = vb->vb2_queue;
>+	unsigned long flags;
>+
>+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
>+	/*
>+	 * If the fence signals with an error we mark the buffer as such
>+	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
>+	 * not queueing it to the driver. However we can't notify the error
>+	 * to userspace right now because, at the time this callback run, QBUF
>+	 * has returned already.
>+	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
>+	 * as well.
>+	 */
>+	if (vb->in_fence->error)
>+		vb->state = VB2_BUF_STATE_ERROR;
>+
>+	dma_fence_put(vb->in_fence);
>+	vb->in_fence = NULL;
>+
>+	if (vb->state == VB2_BUF_STATE_ERROR) {
>+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+		return;
>+	}
>+
>+	if (q->start_streaming_called)
>+		__enqueue_in_driver(vb);
>+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+}
>+
>+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>+		  struct dma_fence *in_fence)
> {
> 	struct vb2_buffer *vb;
>+	unsigned long flags;
> 	int ret;
>
> 	vb = q->bufs[index];
>@@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> 	case VB2_BUF_STATE_DEQUEUED:
> 		ret = __buf_prepare(vb, pb);
> 		if (ret)
>-			return ret;
>+			goto err;
> 		break;
> 	case VB2_BUF_STATE_PREPARED:
> 		break;
> 	case VB2_BUF_STATE_PREPARING:
> 		dprintk(1, "buffer still being prepared\n");
>-		return -EINVAL;
>+		ret = -EINVAL;
>+		goto err;
> 	default:
> 		dprintk(1, "invalid buffer state %d\n", vb->state);
>-		return -EINVAL;
>+		ret = -EINVAL;
>+		goto err;
> 	}
>
> 	/*
>@@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> 	q->queued_count++;
> 	q->waiting_for_buffers = false;
> 	vb->state = VB2_BUF_STATE_QUEUED;
>+	vb->in_fence = in_fence;
>
> 	if (pb)
> 		call_void_bufop(q, copy_timestamp, vb, pb);
>@@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> 	trace_vb2_qbuf(q, vb);
>
> 	/*
>-	 * If already streaming, give the buffer to driver for processing.
>-	 * If not, the buffer will be given to driver on next streamon.
>+	 * For explicit synchronization: If the fence didn't signal
>+	 * yet we setup a callback to queue the buffer once the fence
>+	 * signals and then return successfully. But if the fence
>+	 * already signaled we lose the reference we held and queue the
>+	 * buffer to the driver.
> 	 */
>-	if (q->start_streaming_called)
>-		__enqueue_in_driver(vb);
>+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
>+	if (vb->in_fence) {
>+		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
>+					     vb2_qbuf_fence_cb);
>+		/* is the fence signaled? */
>+		if (ret == -ENOENT) {
>+			dma_fence_put(vb->in_fence);
>+			vb->in_fence = NULL;
>+		} else if (ret) {
>+			goto unlock;
>+		}
>+	}
>
>-	/* Fill buffer information for the userspace */
>-	if (pb)
>-		call_void_bufop(q, fill_user_buffer, vb, pb);
>+	/*
>+	 * If already streaming and there is no fence to wait on
>+	 * give the buffer to driver for processing.
>+	 */
>+	if (q->start_streaming_called) {
>+		struct vb2_buffer *b;
>+
>+		list_for_each_entry(b, &q->queued_list, queued_entry) {
>+			if (b->state != VB2_BUF_STATE_QUEUED)
>+				continue;
>+			if (b->in_fence)
>+				break;
>+			__enqueue_in_driver(b);
>+		}
>+	}
>
> 	/*
> 	 * If streamon has been called, and we haven't yet called
>@@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> 	 * then we can finally call start_streaming().
> 	 */
> 	if (q->streaming && !q->start_streaming_called &&
>-	    q->queued_count >= q->min_buffers_needed) {
>+	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
> 		ret = vb2_start_streaming(q);

If (q->streaming && !q->start_streaming_called), and all buffers have
in_fences which aren't signalled yet, then who calls
vb2_start_streaming()?

It seems like the fence path will always jump over here, and I don't
see a call to vb2_start_streaming() in the fence_cb path. (Sorry if I
missed something).

> 		if (ret)
>-			return ret;
>+			goto unlock;
> 	}
>
>+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+
>+	/* Fill buffer information for the userspace */
>+	if (pb)
>+		call_void_bufop(q, fill_user_buffer, vb, pb);
>+
> 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
> 	return 0;
>+
>+unlock:
>+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+
>+err:
>+	/* Fill buffer information for the userspace */
>+	if (pb)
>+		call_void_bufop(q, fill_user_buffer, vb, pb);
>+
>+	if (vb->in_fence) {
>+		dma_fence_put(vb->in_fence);
>+		vb->in_fence = NULL;
>+	}
>+
>+	return ret;
>+
> }
> EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>
>@@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
> static void __vb2_queue_cancel(struct vb2_queue *q)
> {
> 	unsigned int i;
>+	struct vb2_buffer *vb;
>+	unsigned long flags;
>
> 	/*
> 	 * Tell driver to stop all transactions and release all queued
>@@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> 	q->queued_count = 0;
> 	q->error = 0;
>
>+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
>+		if (vb->in_fence) {
>+			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
>+			dma_fence_put(vb->in_fence);
>+			vb->in_fence = NULL;
>+		}
>+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>+	}
>+
> 	/*
> 	 * Remove all buffers from videobuf's list...
> 	 */
>@@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
> 	 * Tell driver to start streaming provided sufficient buffers
> 	 * are available.
> 	 */
>-	if (q->queued_count >= q->min_buffers_needed) {
>+	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
> 		ret = v4l_vb2q_enable_media_source(q);
> 		if (ret)
> 			return ret;
>@@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
> 		 * Queue all buffers.
> 		 */
> 		for (i = 0; i < q->num_buffers; i++) {
>-			ret = vb2_core_qbuf(q, i, NULL);
>+			ret = vb2_core_qbuf(q, i, NULL, NULL);
> 			if (ret)
> 				goto err_reqbufs;
> 			fileio->bufs[i].queued = 1;
>@@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>
> 		if (copy_timestamp)
> 			b->timestamp = ktime_get_ns();
>-		ret = vb2_core_qbuf(q, index, NULL);
>+		ret = vb2_core_qbuf(q, index, NULL, NULL);
> 		dprintk(5, "vb2_dbuf result: %d\n", ret);
> 		if (ret)
> 			return ret;
>@@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
> 		if (copy_timestamp)
> 			vb->timestamp = ktime_get_ns();
> 		if (!threadio->stop)
>-			ret = vb2_core_qbuf(q, vb->index, NULL);
>+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
> 		call_void_qop(q, wait_prepare, q);
> 		if (ret || threadio->stop)
> 			break;
>diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>index b1c0fa2b0b88..74d7062e5285 100644
>--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>@@ -23,6 +23,7 @@
> #include <linux/sched.h>
> #include <linux/freezer.h>
> #include <linux/kthread.h>
>+#include <linux/sync_file.h>
>
> #include <media/v4l2-dev.h>
> #include <media/v4l2-fh.h>
>@@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
> 		return -EINVAL;
> 	}
>
>+	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
>+	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>+		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
>+		return -EINVAL;
>+	}
>+
>+	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>+		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
>+		return -EINVAL;
>+	}
>+
> 	return __verify_planes_array(q->bufs[b->index], b);
> }
>
>@@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> 	b->timestamp = ns_to_timeval(vb->timestamp);
> 	b->timecode = vbuf->timecode;
> 	b->sequence = vbuf->sequence;
>-	b->fence_fd = 0;
> 	b->reserved = 0;
>
>+	b->fence_fd = 0;

I didn't understand why we're returning 0 instead of -1. Actually the
doc in patch 10 seems to say it will be -1 or 0 depending on whether
we set one of the fence flags? I'm not sure:

    For all other ioctls V4L2 sets this field to -1 if
    ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
    otherwise this field is set to 0 for backward compatibility.

Cheers,
-Brian

>+	if (vb->in_fence)
>+		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>+	else
>+		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
>+
> 	if (q->is_multiplanar) {
> 		/*
> 		 * Fill in plane-related data if userspace provided an array
>@@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>
> int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> {
>+	struct dma_fence *in_fence = NULL;
> 	int ret;
>
> 	if (vb2_fileio_is_active(q)) {
>@@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> 	}
>
> 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
>-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
>+	if (ret)
>+		return ret;
>+
>+	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
>+		in_fence = sync_file_get_fence(b->fence_fd);
>+		if (!in_fence) {
>+			dprintk(1, "failed to get in-fence from fd %d\n",
>+				b->fence_fd);
>+			return -EINVAL;
>+		}
>+	}
>+
>+	return vb2_core_qbuf(q, b->index, b, in_fence);
> }
> EXPORT_SYMBOL_GPL(vb2_qbuf);
>
>diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
>index b811adf88afa..7da53f10db1a 100644
>--- a/drivers/media/dvb-core/dvb_vb2.c
>+++ b/drivers/media/dvb-core/dvb_vb2.c
>@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
> {
> 	int ret;
>
>-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
>+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
> 	if (ret) {
> 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
> 			b->index, ret);
>diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
>index 8e37e7c5e0f7..79dfb5dfd1fc 100644
>--- a/drivers/media/v4l2-core/Kconfig
>+++ b/drivers/media/v4l2-core/Kconfig
>@@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
> config VIDEOBUF_DVB
> 	tristate
> 	select VIDEOBUF_GEN
>+
>+# Used by drivers that need Videobuf2 modules
>+config VIDEOBUF2_CORE
>+	select DMA_SHARED_BUFFER
>+	select SYNC_FILE
>+	tristate
>+
>+config VIDEOBUF2_MEMOPS
>+	tristate
>+	select FRAME_VECTOR
>+
>+config VIDEOBUF2_DMA_CONTIG
>+	tristate
>+	depends on HAS_DMA
>+	select VIDEOBUF2_CORE
>+	select VIDEOBUF2_MEMOPS
>+	select DMA_SHARED_BUFFER
>+
>+config VIDEOBUF2_VMALLOC
>+	tristate
>+	select VIDEOBUF2_CORE
>+	select VIDEOBUF2_MEMOPS
>+	select DMA_SHARED_BUFFER
>+
>+config VIDEOBUF2_DMA_SG
>+	tristate
>+	depends on HAS_DMA
>+	select VIDEOBUF2_CORE
>+	select VIDEOBUF2_MEMOPS
>+
>+config VIDEOBUF2_DVB
>+	tristate
>+	select VIDEOBUF2_CORE
>diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>index 364e4cb41b10..28ce8f66882e 100644
>--- a/include/media/videobuf2-core.h
>+++ b/include/media/videobuf2-core.h
>@@ -17,6 +17,7 @@
> #include <linux/poll.h>
> #include <linux/dma-buf.h>
> #include <linux/bitops.h>
>+#include <linux/dma-fence.h>
>
> #define VB2_MAX_FRAME	(32)
> #define VB2_MAX_PLANES	(8)
>@@ -255,12 +256,21 @@ struct vb2_buffer {
> 	 * done_entry:		entry on the list that stores all buffers ready
> 	 *			to be dequeued to userspace
> 	 * vb2_plane:		per-plane information; do not change
>+	 * in_fence:		fence received from vb2 client to wait on before
>+	 *			using the buffer (queueing to the driver)
>+	 * fence_cb:		fence callback information
>+	 * fence_cb_lock:	protect callback signal/remove
> 	 */
> 	enum vb2_buffer_state	state;
>
> 	struct vb2_plane	planes[VB2_MAX_PLANES];
> 	struct list_head	queued_entry;
> 	struct list_head	done_entry;
>+
>+	struct dma_fence	*in_fence;
>+	struct dma_fence_cb	fence_cb;
>+	spinlock_t              fence_cb_lock;
>+
> #ifdef CONFIG_VIDEO_ADV_DEBUG
> 	/*
> 	 * Counters for how often these buffer-related ops are
>@@ -773,6 +783,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  * @index:	id number of the buffer
>  * @pb:		buffer structure passed from userspace to
>  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
>+ * @in_fence:	in-fence to wait on before queueing the buffer
>  *
>  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
>@@ -787,7 +798,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  *
>  * Return: returns zero on success; an error code otherwise.
>  */
>-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
>+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>+		  struct dma_fence *in_fence);
>
> /**
>  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
>-- 
>2.16.3
>

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 05/09/18 12:36, Brian Starkey wrote:
> Hi Ezequiel,
> 
> On Fri, May 04, 2018 at 05:06:08PM -0300, Ezequiel Garcia wrote:
>> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>>
>> Receive in-fence from userspace and add support for waiting on them
>> before queueing the buffer to the driver. Buffers can't be queued to the
>> driver before its fences signal. And a buffer can't be queued to the driver
>> out of the order they were queued from userspace. That means that even if
>> its fence signals it must wait for all other buffers, ahead of it in the queue,
>> to signal first.
>>
>> If the fence for some buffer fails we do not queue it to the driver,
>> instead we mark it as error and wait until the previous buffer is done
>> to notify userspace of the error. We wait here to deliver the buffers back
>> to userspace in order.
>>
>> v12: fixed dvb_vb2.c usage of vb2_core_qbuf.
>>
>> v11: - minor doc/comments fixes (Hans Verkuil)
>>     - reviewed the in-fence path at __fill_v4l2_buffer()
>>
>> v10: - rename fence to in_fence in many places
>>     - handle fences signalling with error better (Hans Verkuil)
>>
>> v9: - improve comments and docs (Hans Verkuil)
>>    - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>>    - move in-fences code that was in the out-fences patch here (Alex)
>>
>> v8: - improve comments about fences with errors
>>
>> v7: - get rid of the fence array stuff for ordering and just use
>>      get_num_buffers_ready() (Hans)
>>    - fix issue of queuing the buffer twice (Hans)
>>    - avoid the dma_fence_wait() in core_qbuf() (Alex)
>>    - merge preparation commit in
>>
>> v6: - With fences always keep the order userspace queues the buffers.
>>    - Protect in_fence manipulation with a lock (Brian Starkey)
>>    - check if fences have the same context before adding a fence array
>>    - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>>    - Clean up fence if __set_in_fence() fails (Brian Starkey)
>>    - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
>>
>> v5: - use fence_array to keep buffers ordered in vb2 core when
>>      needed (Brian Starkey)
>>    - keep backward compat on the reserved2 field (Brian Starkey)
>>    - protect fence callback removal with lock (Brian Starkey)
>>
>> v4: - Add a comment about dma_fence_add_callback() not returning a
>>      error (Hans)
>>    - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>>    - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>>    - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>>    - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>>    - Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>>      vb2_start_streaming() (Hans)
>>    - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>>    - Queue buffers to the driver as soon as they are ready (Hans)
>>    - call fill_user_buffer() after queuing the buffer (Hans)
>>    - add err: label to clean up fence
>>    - add dma_fence_wait() before calling vb2_start_streaming()
>>
>> v3: - document fence parameter
>>    - remove ternary if at vb2_qbuf() return (Mauro)
>>    - do not change if conditions behaviour (Mauro)
>>
>> v2: - fix vb2_queue_or_prepare_buf() ret check
>>    - remove check for VB2_MEMORY_DMABUF only (Javier)
>>    - check num of ready buffers to start streaming
>>    - when queueing, start from the first ready buffer
>>    - handle queue cancel
>>
>> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>> ---
>> drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
>> drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
>> drivers/media/dvb-core/dvb_vb2.c                |   2 +-
>> drivers/media/v4l2-core/Kconfig                 |  33 ++++
>> include/media/videobuf2-core.h                  |  14 +-
>> 5 files changed, 249 insertions(+), 31 deletions(-)
>>
>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>> index 6b8e083893ad..996b99497a98 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-core.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
>> @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>> 		vb->index = q->num_buffers + buffer;
>> 		vb->type = q->type;
>> 		vb->memory = memory;
>> +		spin_lock_init(&vb->fence_cb_lock);
>> 		for (plane = 0; plane < num_planes; ++plane) {
>> 			vb->planes[plane].length = plane_sizes[plane];
>> 			vb->planes[plane].min_length = plane_sizes[plane];
>> @@ -905,20 +906,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>> }
>> EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>>
>> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> {
>> 	struct vb2_queue *q = vb->vb2_queue;
>> 	unsigned long flags;
>> 	unsigned int plane;
>>
>> -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>> -		return;
>> -
>> -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>> -		    state != VB2_BUF_STATE_ERROR &&
>> -		    state != VB2_BUF_STATE_QUEUED &&
>> -		    state != VB2_BUF_STATE_REQUEUEING))
>> -		state = VB2_BUF_STATE_ERROR;
>>
>> #ifdef CONFIG_VIDEO_ADV_DEBUG
>> 	/*
>> @@ -935,6 +928,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> 		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>>
>> 	spin_lock_irqsave(&q->done_lock, flags);
>> +	if (vb->state == VB2_BUF_STATE_ACTIVE)
>> +		atomic_dec(&q->owned_by_drv_count);
>> +
>> 	if (state == VB2_BUF_STATE_QUEUED ||
>> 	    state == VB2_BUF_STATE_REQUEUEING) {
>> 		vb->state = VB2_BUF_STATE_QUEUED;
>> @@ -943,7 +939,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> 		list_add_tail(&vb->done_entry, &q->done_list);
>> 		vb->state = state;
>> 	}
>> -	atomic_dec(&q->owned_by_drv_count);
>> +
>> 	spin_unlock_irqrestore(&q->done_lock, flags);
>>
>> 	trace_vb2_buf_done(q, vb);
>> @@ -960,6 +956,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> 		wake_up(&q->done_wq);
>> 		break;
>> 	}
>> +
>> +}
>> +
>> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> +{
>> +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>> +		return;
>> +
>> +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>> +		    state != VB2_BUF_STATE_ERROR &&
>> +		    state != VB2_BUF_STATE_QUEUED &&
>> +		    state != VB2_BUF_STATE_REQUEUEING))
>> +		state = VB2_BUF_STATE_ERROR;
>> +
>> +	vb2_process_buffer_done(vb, state);
>> +
>> +	/*
>> +	 * Check if there is any buffer with error in the next position of the queue,
>> +	 * buffers whose in-fence signaled with error are not queued to the driver
>> +	 * and kept on the queue until the buffer before them is done, so to not
>> +	 * deliver buffers back to userspace in the wrong order. Here we process
>> +	 * any existing buffers with errors and wake up userspace.
>> +	 */
>> +	for (;;) {
>> +		vb = list_next_entry(vb, queued_entry);
>> +		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
>> +			break;
>> +
>> +		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
>> +        }
>> }
>> EXPORT_SYMBOL_GPL(vb2_buffer_done);
>>
>> @@ -1244,6 +1270,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>> {
>> 	struct vb2_queue *q = vb->vb2_queue;
>>
>> +	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>> +		return;
> 
> Seems like this is not needed now. Either __enqueue_in_driver() is
> called when there's no in_fence; or it's called from the fence_cb in
> which case we're guaranteed to already be signaled.
> 
> Might be worth a loud error if these conditions are ever true - but
> the silent return is a leftover from old patches I think.
> 
>> +
>> 	vb->state = VB2_BUF_STATE_ACTIVE;
>> 	atomic_inc(&q->owned_by_drv_count);
>>
>> @@ -1295,6 +1324,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>> 	return 0;
>> }
>>
>> +static int __get_num_ready_buffers(struct vb2_queue *q)
>> +{
>> +	struct vb2_buffer *vb;
>> +	int ready_count = 0;
>> +	unsigned long flags;
>> +
>> +	/* count num of buffers ready in front of the queued_list */
>> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>> +			break;
>> +		ready_count++;
>> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +	}
>> +
>> +	return ready_count;
>> +}
>> +
>> int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>> {
>> 	struct vb2_buffer *vb;
>> @@ -1383,9 +1430,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
>> 	return ret;
>> }
>>
>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>> +{
>> +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
>> +	struct vb2_queue *q = vb->vb2_queue;
>> +	unsigned long flags;
>> +
>> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +	/*
>> +	 * If the fence signals with an error we mark the buffer as such
>> +	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
>> +	 * not queueing it to the driver. However we can't notify the error
>> +	 * to userspace right now because, at the time this callback run, QBUF
>> +	 * has returned already.
>> +	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
>> +	 * as well.
>> +	 */
>> +	if (vb->in_fence->error)
>> +		vb->state = VB2_BUF_STATE_ERROR;
>> +
>> +	dma_fence_put(vb->in_fence);
>> +	vb->in_fence = NULL;
>> +
>> +	if (vb->state == VB2_BUF_STATE_ERROR) {
>> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +		return;
>> +	}
>> +
>> +	if (q->start_streaming_called)
>> +		__enqueue_in_driver(vb);
>> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +}
>> +
>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>> +		  struct dma_fence *in_fence)
>> {
>> 	struct vb2_buffer *vb;
>> +	unsigned long flags;
>> 	int ret;
>>
>> 	vb = q->bufs[index];
>> @@ -1394,16 +1475,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> 	case VB2_BUF_STATE_DEQUEUED:
>> 		ret = __buf_prepare(vb, pb);
>> 		if (ret)
>> -			return ret;
>> +			goto err;
>> 		break;
>> 	case VB2_BUF_STATE_PREPARED:
>> 		break;
>> 	case VB2_BUF_STATE_PREPARING:
>> 		dprintk(1, "buffer still being prepared\n");
>> -		return -EINVAL;
>> +		ret = -EINVAL;
>> +		goto err;
>> 	default:
>> 		dprintk(1, "invalid buffer state %d\n", vb->state);
>> -		return -EINVAL;
>> +		ret = -EINVAL;
>> +		goto err;
>> 	}
>>
>> 	/*
>> @@ -1414,6 +1497,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> 	q->queued_count++;
>> 	q->waiting_for_buffers = false;
>> 	vb->state = VB2_BUF_STATE_QUEUED;
>> +	vb->in_fence = in_fence;
>>
>> 	if (pb)
>> 		call_void_bufop(q, copy_timestamp, vb, pb);
>> @@ -1421,15 +1505,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> 	trace_vb2_qbuf(q, vb);
>>
>> 	/*
>> -	 * If already streaming, give the buffer to driver for processing.
>> -	 * If not, the buffer will be given to driver on next streamon.
>> +	 * For explicit synchronization: If the fence didn't signal
>> +	 * yet we setup a callback to queue the buffer once the fence
>> +	 * signals and then return successfully. But if the fence
>> +	 * already signaled we lose the reference we held and queue the
>> +	 * buffer to the driver.
>> 	 */
>> -	if (q->start_streaming_called)
>> -		__enqueue_in_driver(vb);
>> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +	if (vb->in_fence) {
>> +		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
>> +					     vb2_qbuf_fence_cb);
>> +		/* is the fence signaled? */
>> +		if (ret == -ENOENT) {
>> +			dma_fence_put(vb->in_fence);
>> +			vb->in_fence = NULL;
>> +		} else if (ret) {
>> +			goto unlock;
>> +		}
>> +	}
>>
>> -	/* Fill buffer information for the userspace */
>> -	if (pb)
>> -		call_void_bufop(q, fill_user_buffer, vb, pb);
>> +	/*
>> +	 * If already streaming and there is no fence to wait on
>> +	 * give the buffer to driver for processing.
>> +	 */
>> +	if (q->start_streaming_called) {
>> +		struct vb2_buffer *b;
>> +
>> +		list_for_each_entry(b, &q->queued_list, queued_entry) {
>> +			if (b->state != VB2_BUF_STATE_QUEUED)
>> +				continue;
>> +			if (b->in_fence)
>> +				break;
>> +			__enqueue_in_driver(b);
>> +		}
>> +	}
>>
>> 	/*
>> 	 * If streamon has been called, and we haven't yet called
>> @@ -1438,14 +1547,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> 	 * then we can finally call start_streaming().
>> 	 */
>> 	if (q->streaming && !q->start_streaming_called &&
>> -	    q->queued_count >= q->min_buffers_needed) {
>> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>> 		ret = vb2_start_streaming(q);
> 
> If (q->streaming && !q->start_streaming_called), and all buffers have
> in_fences which aren't signalled yet, then who calls
> vb2_start_streaming()?
> 
> It seems like the fence path will always jump over here, and I don't
> see a call to vb2_start_streaming() in the fence_cb path. (Sorry if I
> missed something).

Good point, I think you are right about that.

This same sequence should be present in the fence_cb path. Question: can
fence_cb be called from interrupt context? If so, then that needs to start a
workqueue or something since vb2_start_streaming cannot run in interrupt
context. And in addition that workqueue needs to take and release q->lock.
We should discuss locking on irc, there are some subtleties there.

This would be a good test to add to v4l2-compliance, BTW.

> 
>> 		if (ret)
>> -			return ret;
>> +			goto unlock;
>> 	}
>>
>> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +
>> +	/* Fill buffer information for the userspace */
>> +	if (pb)
>> +		call_void_bufop(q, fill_user_buffer, vb, pb);
>> +
>> 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>> 	return 0;
>> +
>> +unlock:
>> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +
>> +err:
>> +	/* Fill buffer information for the userspace */
>> +	if (pb)
>> +		call_void_bufop(q, fill_user_buffer, vb, pb);
>> +
>> +	if (vb->in_fence) {
>> +		dma_fence_put(vb->in_fence);
>> +		vb->in_fence = NULL;
>> +	}
>> +
>> +	return ret;
>> +
>> }
>> EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>
>> @@ -1656,6 +1787,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
>> static void __vb2_queue_cancel(struct vb2_queue *q)
>> {
>> 	unsigned int i;
>> +	struct vb2_buffer *vb;
>> +	unsigned long flags;
>>
>> 	/*
>> 	 * Tell driver to stop all transactions and release all queued
>> @@ -1686,6 +1819,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>> 	q->queued_count = 0;
>> 	q->error = 0;
>>
>> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +		if (vb->in_fence) {
>> +			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
>> +			dma_fence_put(vb->in_fence);
>> +			vb->in_fence = NULL;
>> +		}
>> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +	}
>> +
>> 	/*
>> 	 * Remove all buffers from videobuf's list...
>> 	 */
>> @@ -1756,7 +1899,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>> 	 * Tell driver to start streaming provided sufficient buffers
>> 	 * are available.
>> 	 */
>> -	if (q->queued_count >= q->min_buffers_needed) {
>> +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>> 		ret = v4l_vb2q_enable_media_source(q);
>> 		if (ret)
>> 			return ret;
>> @@ -2278,7 +2421,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>> 		 * Queue all buffers.
>> 		 */
>> 		for (i = 0; i < q->num_buffers; i++) {
>> -			ret = vb2_core_qbuf(q, i, NULL);
>> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>> 			if (ret)
>> 				goto err_reqbufs;
>> 			fileio->bufs[i].queued = 1;
>> @@ -2457,7 +2600,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>>
>> 		if (copy_timestamp)
>> 			b->timestamp = ktime_get_ns();
>> -		ret = vb2_core_qbuf(q, index, NULL);
>> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>> 		dprintk(5, "vb2_dbuf result: %d\n", ret);
>> 		if (ret)
>> 			return ret;
>> @@ -2560,7 +2703,7 @@ static int vb2_thread(void *data)
>> 		if (copy_timestamp)
>> 			vb->timestamp = ktime_get_ns();
>> 		if (!threadio->stop)
>> -			ret = vb2_core_qbuf(q, vb->index, NULL);
>> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>> 		call_void_qop(q, wait_prepare, q);
>> 		if (ret || threadio->stop)
>> 			break;
>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> index b1c0fa2b0b88..74d7062e5285 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> @@ -23,6 +23,7 @@
>> #include <linux/sched.h>
>> #include <linux/freezer.h>
>> #include <linux/kthread.h>
>> +#include <linux/sync_file.h>
>>
>> #include <media/v4l2-dev.h>
>> #include <media/v4l2-fh.h>
>> @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>> 		return -EINVAL;
>> 	}
>>
>> +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
>> +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>> +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
>> +		return -EINVAL;
>> +	}
>> +
>> +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>> +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
>> +		return -EINVAL;
>> +	}
>> +
>> 	return __verify_planes_array(q->bufs[b->index], b);
>> }
>>
>> @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>> 	b->timestamp = ns_to_timeval(vb->timestamp);
>> 	b->timecode = vbuf->timecode;
>> 	b->sequence = vbuf->sequence;
>> -	b->fence_fd = 0;
>> 	b->reserved = 0;
>>
>> +	b->fence_fd = 0;
> 
> I didn't understand why we're returning 0 instead of -1. Actually the
> doc in patch 10 seems to say it will be -1 or 0 depending on whether
> we set one of the fence flags? I'm not sure:
> 
>     For all other ioctls V4L2 sets this field to -1 if
>     ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
>     otherwise this field is set to 0 for backward compatibility.

I'm not sure about this either. This also ties in with the Request API
series which has a similar issue. I think setting this to -1 when unused
is better.

Regards,

	Hans

> 
> Cheers,
> -Brian
> 
>> +	if (vb->in_fence)
>> +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>> +	else
>> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
>> +
>> 	if (q->is_multiplanar) {
>> 		/*
>> 		 * Fill in plane-related data if userspace provided an array
>> @@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>>
>> int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>> {
>> +	struct dma_fence *in_fence = NULL;
>> 	int ret;
>>
>> 	if (vb2_fileio_is_active(q)) {
>> @@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>> 	}
>>
>> 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
>> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
>> +	if (ret)
>> +		return ret;
>> +
>> +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
>> +		in_fence = sync_file_get_fence(b->fence_fd);
>> +		if (!in_fence) {
>> +			dprintk(1, "failed to get in-fence from fd %d\n",
>> +				b->fence_fd);
>> +			return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return vb2_core_qbuf(q, b->index, b, in_fence);
>> }
>> EXPORT_SYMBOL_GPL(vb2_qbuf);
>>
>> diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
>> index b811adf88afa..7da53f10db1a 100644
>> --- a/drivers/media/dvb-core/dvb_vb2.c
>> +++ b/drivers/media/dvb-core/dvb_vb2.c
>> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
>> {
>> 	int ret;
>>
>> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
>> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>> 	if (ret) {
>> 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>> 			b->index, ret);
>> diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
>> index 8e37e7c5e0f7..79dfb5dfd1fc 100644
>> --- a/drivers/media/v4l2-core/Kconfig
>> +++ b/drivers/media/v4l2-core/Kconfig
>> @@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
>> config VIDEOBUF_DVB
>> 	tristate
>> 	select VIDEOBUF_GEN
>> +
>> +# Used by drivers that need Videobuf2 modules
>> +config VIDEOBUF2_CORE
>> +	select DMA_SHARED_BUFFER
>> +	select SYNC_FILE
>> +	tristate
>> +
>> +config VIDEOBUF2_MEMOPS
>> +	tristate
>> +	select FRAME_VECTOR
>> +
>> +config VIDEOBUF2_DMA_CONTIG
>> +	tristate
>> +	depends on HAS_DMA
>> +	select VIDEOBUF2_CORE
>> +	select VIDEOBUF2_MEMOPS
>> +	select DMA_SHARED_BUFFER
>> +
>> +config VIDEOBUF2_VMALLOC
>> +	tristate
>> +	select VIDEOBUF2_CORE
>> +	select VIDEOBUF2_MEMOPS
>> +	select DMA_SHARED_BUFFER
>> +
>> +config VIDEOBUF2_DMA_SG
>> +	tristate
>> +	depends on HAS_DMA
>> +	select VIDEOBUF2_CORE
>> +	select VIDEOBUF2_MEMOPS
>> +
>> +config VIDEOBUF2_DVB
>> +	tristate
>> +	select VIDEOBUF2_CORE
>> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>> index 364e4cb41b10..28ce8f66882e 100644
>> --- a/include/media/videobuf2-core.h
>> +++ b/include/media/videobuf2-core.h
>> @@ -17,6 +17,7 @@
>> #include <linux/poll.h>
>> #include <linux/dma-buf.h>
>> #include <linux/bitops.h>
>> +#include <linux/dma-fence.h>
>>
>> #define VB2_MAX_FRAME	(32)
>> #define VB2_MAX_PLANES	(8)
>> @@ -255,12 +256,21 @@ struct vb2_buffer {
>> 	 * done_entry:		entry on the list that stores all buffers ready
>> 	 *			to be dequeued to userspace
>> 	 * vb2_plane:		per-plane information; do not change
>> +	 * in_fence:		fence received from vb2 client to wait on before
>> +	 *			using the buffer (queueing to the driver)
>> +	 * fence_cb:		fence callback information
>> +	 * fence_cb_lock:	protect callback signal/remove
>> 	 */
>> 	enum vb2_buffer_state	state;
>>
>> 	struct vb2_plane	planes[VB2_MAX_PLANES];
>> 	struct list_head	queued_entry;
>> 	struct list_head	done_entry;
>> +
>> +	struct dma_fence	*in_fence;
>> +	struct dma_fence_cb	fence_cb;
>> +	spinlock_t              fence_cb_lock;
>> +
>> #ifdef CONFIG_VIDEO_ADV_DEBUG
>> 	/*
>> 	 * Counters for how often these buffer-related ops are
>> @@ -773,6 +783,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>>  * @index:	id number of the buffer
>>  * @pb:		buffer structure passed from userspace to
>>  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
>> + * @in_fence:	in-fence to wait on before queueing the buffer
>>  *
>>  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>>  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
>> @@ -787,7 +798,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>>  *
>>  * Return: returns zero on success; an error code otherwise.
>>  */
>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>> +		  struct dma_fence *in_fence);
>>
>> /**
>>  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
>> -- 
>> 2.16.3
>>

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

On Wed, 2018-05-09 at 11:36 +0100, Brian Starkey wrote:

[..]
> > @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> > 	b->timestamp = ns_to_timeval(vb->timestamp);
> > 	b->timecode = vbuf->timecode;
> > 	b->sequence = vbuf->sequence;
> > -	b->fence_fd = 0;
> > 	b->reserved = 0;
> > 
> > +	b->fence_fd = 0;
> 
> I didn't understand why we're returning 0 instead of -1. Actually the
> doc in patch 10 seems to say it will be -1 or 0 depending on whether
> we set one of the fence flags? I'm not sure:
> 
>     For all other ioctls V4L2 sets this field to -1 if
>     ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
>     otherwise this field is set to 0 for backward compatibility.
> 

Well, I think that for backwards compatibility (userspace not knowing
about fence_fd field), we should return 0, unless the flags are explicitly
set.

That is what the doc says and it sounds sane.

The bits are implemented in patch 12, but as I mentioned in my reply to
patch 10, I will move it to patch 10, for consistency.

Thanks,
Eze

Re: [PATCH v9 11/15] vb2: add in-fence support to QBUF

[Brian Starkey]

On Wed, May 09, 2018 at 01:03:15PM -0300, Ezequiel Garcia wrote:
>On Wed, 2018-05-09 at 11:36 +0100, Brian Starkey wrote:
>
>[..]
>> > @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>> > 	b->timestamp = ns_to_timeval(vb->timestamp);
>> > 	b->timecode = vbuf->timecode;
>> > 	b->sequence = vbuf->sequence;
>> > -	b->fence_fd = 0;
>> > 	b->reserved = 0;
>> >
>> > +	b->fence_fd = 0;
>>
>> I didn't understand why we're returning 0 instead of -1. Actually the
>> doc in patch 10 seems to say it will be -1 or 0 depending on whether
>> we set one of the fence flags? I'm not sure:
>>
>>     For all other ioctls V4L2 sets this field to -1 if
>>     ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
>>     otherwise this field is set to 0 for backward compatibility.
>>
>
>Well, I think that for backwards compatibility (userspace not knowing
>about fence_fd field), we should return 0, unless the flags are explicitly
>set.
>
>That is what the doc says and it sounds sane.

On the line below where you snipped, is this:

+      if (vb->in_fence)
+              b->flags |= V4L2_BUF_FLAG_IN_FENCE;
+      else
+              b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;

If the "if (vb->in_fence)" condition is true, then the flag is set,
and the fence_fd field is 0. I think that's the opposite of what the
doc says:

    For all other ioctls V4L2 sets this field to -1 if
    ``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
    otherwise this field is set to 0 for backward compatibility.

V4L2_BUF_FLAG_IN_FENCE is set, therefore the doc says V4L2 will set
this field to -1. (Or at least the comment should be made less
ambiguous).

>
>The bits are implemented in patch 12, but as I mentioned in my reply to
>patch 10, I will move it to patch 10, for consistency.

Yeah as you say, it looks like you change this behaviour in path 12,
so I'm not totally sure which is right or expected. But consistency is
good :-)

-Brian

>
>Thanks,
>Eze

[PATCH v9 12/15] vb2: add out-fence support to QBUF

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace in the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v11: - Return fence_fd to userpace only in the QBUF ioctl.
     - Rework implementation to avoid storing the sync_file
       as state, which is not really needed.

v10: - use -EIO for fence error (Hans Verkuil)
     - add comment around fence context creation (Hans Verkuil)

v9: - remove in-fences changes from this patch (Alex Courbot)
    - improve fence context creation (Hans Verkuil)
    - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)

v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
    - fix crash when checking not using fences in vb2_buffer_done()

v7: - merge patch that add the infrastructure to out-fences into
      this one (Alex Courbot)
    - Do not install the fd if there is no fence. (Alex Courbot)
    - do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
      ordering in vb2 for queueing in the driver, so the event is not
      necessary anymore and the out_fence_fd is sent back to userspace
      on QBUF call return arg
    - do not allow requeueing with out-fences, instead mark the buffer
      with an error and wake up to userspace.
    - send the out_fence_fd back to userspace on the fence_fd field

v5: - delay fd_install to DQ_EVENT (Hans)
    - if queue is fully ordered send OUT_FENCE event right away
      (Brian)
    - rename 'q->ordered' to 'q->ordered_in_driver'
    - merge change to implement OUT_FENCE event here

v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)

v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
    - set the OUT_FENCE flag if there is a fence pending (Hans)
    - call fd_install() after vb2_core_qbuf() (Hans)
    - clean up fence if vb2_core_qbuf() fails (Hans)
    - add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 95 +++++++++++++++++++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 22 +++++-
 drivers/media/dvb-core/dvb_vb2.c                |  2 +-
 include/media/videobuf2-core.h                  | 16 ++++-
 4 files changed, 127 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 996b99497a98..0f7306a04db7 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/videobuf2-core.h>
 #include <media/v4l2-mc.h>
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
 		}
+		vb->out_fence_fd = -1;
 		q->bufs[vb->index] = vb;
 
 		/* Allocate video buffer memory for the MMAP type */
@@ -948,10 +950,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
 	case VB2_BUF_STATE_QUEUED:
 		return;
 	case VB2_BUF_STATE_REQUEUEING:
+		/* Requeuing with explicit synchronization, spit warning */
+		WARN_ON_ONCE(vb->out_fence);
+
 		if (q->start_streaming_called)
 			__enqueue_in_driver(vb);
 		return;
 	default:
+		if (vb->out_fence) {
+			if (state == VB2_BUF_STATE_ERROR)
+				dma_fence_set_error(vb->out_fence, -EIO);
+			dma_fence_signal(vb->out_fence);
+			dma_fence_put(vb->out_fence);
+			vb->out_fence = NULL;
+			vb->out_fence_fd = -1;
+		}
+
 		/* Inform any processes that may be waiting for buffers */
 		wake_up(&q->done_wq);
 		break;
@@ -1367,6 +1381,68 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+	return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+	return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+	return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+	.get_driver_name = vb2_fence_get_driver_name,
+	.get_timeline_name = vb2_fence_get_timeline_name,
+	.enable_signaling = vb2_fence_enable_signaling,
+	.wait = dma_fence_default_wait,
+};
+
+static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
+{
+	struct sync_file *sync_file;
+	int ret;
+
+	vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+	/*
+	 * The same context can be used only if the queue is ordered,
+	 * so if the queue is ordered create one when the queueing start,
+	 * otherwise create one for every buffer
+	 */
+	if (q->unordered || !q->queueing_started)
+		q->out_fence_context = dma_fence_context_alloc(1);
+
+	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
+	if (!vb->out_fence) {
+		ret = -ENOMEM;
+		goto err_put_fd;
+	}
+	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
+		       q->out_fence_context, 1);
+
+	sync_file = sync_file_create(vb->out_fence);
+	if (!sync_file) {
+		ret = -ENOMEM;
+		goto err_free_fence;
+	}
+	fd_install(vb->out_fence_fd, sync_file->file);
+	return 0;
+
+err_free_fence:
+	dma_fence_put(vb->out_fence);
+	vb->out_fence = NULL;
+err_put_fd:
+	put_unused_fd(vb->out_fence_fd);
+	vb->out_fence_fd = -1;
+	return ret;
+}
+
 /*
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1463,7 +1539,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
 }
 
 int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
-		  struct dma_fence *in_fence)
+		  struct dma_fence *in_fence, bool out_fence)
 {
 	struct vb2_buffer *vb;
 	unsigned long flags;
@@ -1496,6 +1572,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 	list_add_tail(&vb->queued_entry, &q->queued_list);
 	q->queued_count++;
 	q->waiting_for_buffers = false;
+	q->queueing_started = 1;
 	vb->state = VB2_BUF_STATE_QUEUED;
 	vb->in_fence = in_fence;
 
@@ -1555,6 +1632,14 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 
 	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
 
+	if (out_fence) {
+		ret = vb2_setup_out_fence(q, vb);
+		if (ret) {
+			dprintk(1, "failed to set up out-fence\n");
+			goto err;
+		}
+	}
+
 	/* Fill buffer information for the userspace */
 	if (pb)
 		call_void_bufop(q, fill_user_buffer, vb, pb);
@@ -1818,6 +1903,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	q->start_streaming_called = 0;
 	q->queued_count = 0;
 	q->error = 0;
+	q->queueing_started = 0;
 
 	list_for_each_entry(vb, &q->queued_list, queued_entry) {
 		spin_lock_irqsave(&vb->fence_cb_lock, flags);
@@ -2170,6 +2256,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
 	spin_lock_init(&q->done_lock);
 	mutex_init(&q->mmap_lock);
 	init_waitqueue_head(&q->done_wq);
+	spin_lock_init(&q->out_fence_lock);
 
 	q->memory = VB2_MEMORY_UNKNOWN;
 
@@ -2421,7 +2508,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL, false);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2600,7 +2687,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL, false);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2703,7 +2790,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, false);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 74d7062e5285..41a88839683e 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -217,7 +217,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->sequence = vbuf->sequence;
 	b->reserved = 0;
 
-	b->fence_fd = 0;
+	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE)
+		b->fence_fd = vb->out_fence_fd;
+	else
+		b->fence_fd = 0;
+
 	if (vb->in_fence)
 		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
 	else
@@ -496,6 +500,13 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	ret = __verify_planes_array(vb, b);
 	if (!ret)
 		vb2_core_querybuf(q, b->index, b);
+
+	/*
+	 * Can't return a fence fd, because it only
+	 * makes sense on the process that queued the buffer.
+	 */
+	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
+		b->fence_fd = -1;
 	return ret;
 }
 EXPORT_SYMBOL(vb2_querybuf);
@@ -600,7 +611,8 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		}
 	}
 
-	return vb2_core_qbuf(q, b->index, b, in_fence);
+	return vb2_core_qbuf(q, b->index, b, in_fence,
+			     b->flags & V4L2_BUF_FLAG_OUT_FENCE);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
@@ -626,6 +638,12 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
 	 */
 	b->flags &= ~V4L2_BUF_FLAG_DONE;
 
+	/*
+	 * Can't return a fence fd, because it only
+	 * makes sense on the process that queued the buffer.
+	 */
+	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
+		b->fence_fd = -1;
 	return ret;
 }
 EXPORT_SYMBOL_GPL(vb2_dqbuf);
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index 7da53f10db1a..053803c9ff45 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
 {
 	int ret;
 
-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, false);
 	if (ret) {
 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
 			b->index, ret);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 28ce8f66882e..794140fcb5cf 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -260,6 +260,9 @@ struct vb2_buffer {
 	 *			using the buffer (queueing to the driver)
 	 * fence_cb:		fence callback information
 	 * fence_cb_lock:	protect callback signal/remove
+	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
+	 * out_fence:		the out-fence associated with the buffer once
+	 *			it is queued to the driver.
 	 */
 	enum vb2_buffer_state	state;
 
@@ -271,6 +274,9 @@ struct vb2_buffer {
 	struct dma_fence_cb	fence_cb;
 	spinlock_t              fence_cb_lock;
 
+	int			out_fence_fd;
+	struct dma_fence	*out_fence;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -533,6 +539,9 @@ struct vb2_buf_ops {
  * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
  *		last decoded buffer was already dequeued. Set for capture queues
  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
+ * @queueing_started: if queueing has started. Currently used to determine
+ *		if an out_fence_context is needed.
+ * @out_fence_context: the fence context for the out fences
  * @fileio:	file io emulator internal data, used only if emulator is active
  * @threadio:	thread io internal data, used only if thread is active
  */
@@ -586,6 +595,10 @@ struct vb2_queue {
 	unsigned int			is_output:1;
 	unsigned int			copy_timestamp:1;
 	unsigned int			last_buffer_dequeued:1;
+	unsigned int			queueing_started:1;
+
+	u64				out_fence_context;
+	spinlock_t			out_fence_lock;
 
 	struct vb2_fileio_data		*fileio;
 	struct vb2_threadio_data	*threadio;
@@ -784,6 +797,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
  * @in_fence:	in-fence to wait on before queueing the buffer
+ * @out_fence:	create an out-fence as a part of the queue operation
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
@@ -799,7 +813,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * Return: returns zero on success; an error code otherwise.
  */
 int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
-		  struct dma_fence *in_fence);
+		  struct dma_fence *in_fence, bool out_fence);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.16.3

Re: [PATCH v9 12/15] vb2: add out-fence support to QBUF

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> an out_fence and send its fd to userspace in the fence_fd field as a
> return arg for the QBUF call.
> 
> The fence is signaled on buffer_done(), when the job on the buffer is
> finished.
> 
> v11: - Return fence_fd to userpace only in the QBUF ioctl.
>      - Rework implementation to avoid storing the sync_file
>        as state, which is not really needed.
> 
> v10: - use -EIO for fence error (Hans Verkuil)
>      - add comment around fence context creation (Hans Verkuil)
> 
> v9: - remove in-fences changes from this patch (Alex Courbot)
>     - improve fence context creation (Hans Verkuil)
>     - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
> 
> v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
>     - fix crash when checking not using fences in vb2_buffer_done()
> 
> v7: - merge patch that add the infrastructure to out-fences into
>       this one (Alex Courbot)
>     - Do not install the fd if there is no fence. (Alex Courbot)
>     - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
> 
> v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
>       ordering in vb2 for queueing in the driver, so the event is not
>       necessary anymore and the out_fence_fd is sent back to userspace
>       on QBUF call return arg
>     - do not allow requeueing with out-fences, instead mark the buffer
>       with an error and wake up to userspace.
>     - send the out_fence_fd back to userspace on the fence_fd field
> 
> v5: - delay fd_install to DQ_EVENT (Hans)
>     - if queue is fully ordered send OUT_FENCE event right away
>       (Brian)
>     - rename 'q->ordered' to 'q->ordered_in_driver'
>     - merge change to implement OUT_FENCE event here
> 
> v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)
> 
> v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
>     - set the OUT_FENCE flag if there is a fence pending (Hans)
>     - call fd_install() after vb2_core_qbuf() (Hans)
>     - clean up fence if vb2_core_qbuf() fails (Hans)
>     - add list to store sync_file and fence for the next queued buffer
> 
> v2: check if the queue is ordered.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 95 +++++++++++++++++++++++--
>  drivers/media/common/videobuf2/videobuf2-v4l2.c | 22 +++++-
>  drivers/media/dvb-core/dvb_vb2.c                |  2 +-
>  include/media/videobuf2-core.h                  | 16 ++++-
>  4 files changed, 127 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 996b99497a98..0f7306a04db7 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/videobuf2-core.h>
>  #include <media/v4l2-mc.h>
> @@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
>  		}
> +		vb->out_fence_fd = -1;
>  		q->bufs[vb->index] = vb;
>  
>  		/* Allocate video buffer memory for the MMAP type */
> @@ -948,10 +950,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
>  	case VB2_BUF_STATE_QUEUED:
>  		return;
>  	case VB2_BUF_STATE_REQUEUEING:

Off-topic: I really need to take a good look at this requeueing thing. It's
a pain for both fences and request API, and it is only used in the cobalt driver,
although I know of another out-of-tree driver that uses it as well.

I wonder if I should just get rid of this state.

> +		/* Requeuing with explicit synchronization, spit warning */
> +		WARN_ON_ONCE(vb->out_fence);
> +
>  		if (q->start_streaming_called)
>  			__enqueue_in_driver(vb);
>  		return;
>  	default:
> +		if (vb->out_fence) {
> +			if (state == VB2_BUF_STATE_ERROR)
> +				dma_fence_set_error(vb->out_fence, -EIO);
> +			dma_fence_signal(vb->out_fence);
> +			dma_fence_put(vb->out_fence);
> +			vb->out_fence = NULL;
> +			vb->out_fence_fd = -1;
> +		}
> +
>  		/* Inform any processes that may be waiting for buffers */
>  		wake_up(&q->done_wq);
>  		break;
> @@ -1367,6 +1381,68 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>  
> +static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence";
> +}
> +
> +static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence_timeline";
> +}
> +
> +static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
> +{
> +	return true;
> +}
> +
> +static const struct dma_fence_ops vb2_fence_ops = {
> +	.get_driver_name = vb2_fence_get_driver_name,
> +	.get_timeline_name = vb2_fence_get_timeline_name,
> +	.enable_signaling = vb2_fence_enable_signaling,
> +	.wait = dma_fence_default_wait,
> +};
> +
> +static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
> +{
> +	struct sync_file *sync_file;
> +	int ret;
> +
> +	vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
> +
> +	/*
> +	 * The same context can be used only if the queue is ordered,
> +	 * so if the queue is ordered create one when the queueing start,
> +	 * otherwise create one for every buffer
> +	 */
> +	if (q->unordered || !q->queueing_started)
> +		q->out_fence_context = dma_fence_context_alloc(1);
> +
> +	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
> +	if (!vb->out_fence) {
> +		ret = -ENOMEM;
> +		goto err_put_fd;
> +	}
> +	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
> +		       q->out_fence_context, 1);
> +
> +	sync_file = sync_file_create(vb->out_fence);
> +	if (!sync_file) {
> +		ret = -ENOMEM;
> +		goto err_free_fence;
> +	}
> +	fd_install(vb->out_fence_fd, sync_file->file);
> +	return 0;
> +
> +err_free_fence:
> +	dma_fence_put(vb->out_fence);
> +	vb->out_fence = NULL;
> +err_put_fd:
> +	put_unused_fd(vb->out_fence_fd);
> +	vb->out_fence_fd = -1;
> +	return ret;
> +}
> +
>  /*
>   * vb2_start_streaming() - Attempt to start streaming.
>   * @q:		videobuf2 queue
> @@ -1463,7 +1539,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>  }
>  
>  int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> -		  struct dma_fence *in_fence)
> +		  struct dma_fence *in_fence, bool out_fence)
>  {
>  	struct vb2_buffer *vb;
>  	unsigned long flags;
> @@ -1496,6 +1572,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  	list_add_tail(&vb->queued_entry, &q->queued_list);
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
> +	q->queueing_started = 1;
>  	vb->state = VB2_BUF_STATE_QUEUED;
>  	vb->in_fence = in_fence;
>  
> @@ -1555,6 +1632,14 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  
>  	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>  
> +	if (out_fence) {
> +		ret = vb2_setup_out_fence(q, vb);
> +		if (ret) {
> +			dprintk(1, "failed to set up out-fence\n");
> +			goto err;
> +		}
> +	}
> +
>  	/* Fill buffer information for the userspace */
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
> @@ -1818,6 +1903,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	q->start_streaming_called = 0;
>  	q->queued_count = 0;
>  	q->error = 0;
> +	q->queueing_started = 0;
>  
>  	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>  		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> @@ -2170,6 +2256,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
>  	spin_lock_init(&q->done_lock);
>  	mutex_init(&q->mmap_lock);
>  	init_waitqueue_head(&q->done_wq);
> +	spin_lock_init(&q->out_fence_lock);
>  
>  	q->memory = VB2_MEMORY_UNKNOWN;
>  
> @@ -2421,7 +2508,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL, false);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2600,7 +2687,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL, false);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2703,7 +2790,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, false);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 74d7062e5285..41a88839683e 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -217,7 +217,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->sequence = vbuf->sequence;
>  	b->reserved = 0;
>  
> -	b->fence_fd = 0;
> +	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE)
> +		b->fence_fd = vb->out_fence_fd;
> +	else
> +		b->fence_fd = 0;
> +
>  	if (vb->in_fence)
>  		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>  	else
> @@ -496,6 +500,13 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  	ret = __verify_planes_array(vb, b);
>  	if (!ret)
>  		vb2_core_querybuf(q, b->index, b);
> +
> +	/*
> +	 * Can't return a fence fd, because it only
> +	 * makes sense on the process that queued the buffer.
> +	 */
> +	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
> +		b->fence_fd = -1;
>  	return ret;
>  }
>  EXPORT_SYMBOL(vb2_querybuf);
> @@ -600,7 +611,8 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  		}
>  	}
>  
> -	return vb2_core_qbuf(q, b->index, b, in_fence);
> +	return vb2_core_qbuf(q, b->index, b, in_fence,
> +			     b->flags & V4L2_BUF_FLAG_OUT_FENCE);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> @@ -626,6 +638,12 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
>  	 */
>  	b->flags &= ~V4L2_BUF_FLAG_DONE;
>  
> +	/*
> +	 * Can't return a fence fd, because it only
> +	 * makes sense on the process that queued the buffer.

I'd rephrase this:

"it only makes sense for VIDIOC_QBUF."

It's not really related to the process.

> +	 */
> +	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
> +		b->fence_fd = -1;

Why not put this code in __fill_v4l2_buffer() and instead set b->fence_fd
in vb2_qbuf, since that's the only place that can return a valid out-fence.

>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(vb2_dqbuf);
> diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
> index 7da53f10db1a..053803c9ff45 100644
> --- a/drivers/media/dvb-core/dvb_vb2.c
> +++ b/drivers/media/dvb-core/dvb_vb2.c
> @@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
>  {
>  	int ret;
>  
> -	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
> +	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, false);
>  	if (ret) {
>  		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
>  			b->index, ret);
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 28ce8f66882e..794140fcb5cf 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -260,6 +260,9 @@ struct vb2_buffer {
>  	 *			using the buffer (queueing to the driver)
>  	 * fence_cb:		fence callback information
>  	 * fence_cb_lock:	protect callback signal/remove
> +	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
> +	 * out_fence:		the out-fence associated with the buffer once
> +	 *			it is queued to the driver.
>  	 */
>  	enum vb2_buffer_state	state;
>  
> @@ -271,6 +274,9 @@ struct vb2_buffer {
>  	struct dma_fence_cb	fence_cb;
>  	spinlock_t              fence_cb_lock;
>  
> +	int			out_fence_fd;
> +	struct dma_fence	*out_fence;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -533,6 +539,9 @@ struct vb2_buf_ops {
>   * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
>   *		last decoded buffer was already dequeued. Set for capture queues
>   *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
> + * @queueing_started: if queueing has started. Currently used to determine
> + *		if an out_fence_context is needed.
> + * @out_fence_context: the fence context for the out fences
>   * @fileio:	file io emulator internal data, used only if emulator is active
>   * @threadio:	thread io internal data, used only if thread is active
>   */
> @@ -586,6 +595,10 @@ struct vb2_queue {
>  	unsigned int			is_output:1;
>  	unsigned int			copy_timestamp:1;
>  	unsigned int			last_buffer_dequeued:1;
> +	unsigned int			queueing_started:1;
> +
> +	u64				out_fence_context;
> +	spinlock_t			out_fence_lock;
>  
>  	struct vb2_fileio_data		*fileio;
>  	struct vb2_threadio_data	*threadio;
> @@ -784,6 +797,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
>   * @in_fence:	in-fence to wait on before queueing the buffer
> + * @out_fence:	create an out-fence as a part of the queue operation
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> @@ -799,7 +813,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * Return: returns zero on success; an error code otherwise.
>   */
>  int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> -		  struct dma_fence *in_fence);
> +		  struct dma_fence *in_fence, bool out_fence);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Regards,

	Hans

Re: [PATCH v9 12/15] vb2: add out-fence support to QBUF

[Brian Starkey]

Hi,

On Fri, May 04, 2018 at 05:06:09PM -0300, Ezequiel Garcia wrote:
>From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
>If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
>an out_fence and send its fd to userspace in the fence_fd field as a
>return arg for the QBUF call.
>
>The fence is signaled on buffer_done(), when the job on the buffer is
>finished.
>
>v11: - Return fence_fd to userpace only in the QBUF ioctl.
>     - Rework implementation to avoid storing the sync_file
>       as state, which is not really needed.
>
>v10: - use -EIO for fence error (Hans Verkuil)
>     - add comment around fence context creation (Hans Verkuil)
>
>v9: - remove in-fences changes from this patch (Alex Courbot)
>    - improve fence context creation (Hans Verkuil)
>    - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
>
>v8: - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
>    - fix crash when checking not using fences in vb2_buffer_done()
>
>v7: - merge patch that add the infrastructure to out-fences into
>      this one (Alex Courbot)
>    - Do not install the fd if there is no fence. (Alex Courbot)
>    - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
>
>v6: - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
>      ordering in vb2 for queueing in the driver, so the event is not
>      necessary anymore and the out_fence_fd is sent back to userspace
>      on QBUF call return arg
>    - do not allow requeueing with out-fences, instead mark the buffer
>      with an error and wake up to userspace.
>    - send the out_fence_fd back to userspace on the fence_fd field
>
>v5: - delay fd_install to DQ_EVENT (Hans)
>    - if queue is fully ordered send OUT_FENCE event right away
>      (Brian)
>    - rename 'q->ordered' to 'q->ordered_in_driver'
>    - merge change to implement OUT_FENCE event here
>
>v4: - return the out_fence_fd in the BUF_QUEUED event(Hans)
>
>v3: - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
>    - set the OUT_FENCE flag if there is a fence pending (Hans)
>    - call fd_install() after vb2_core_qbuf() (Hans)
>    - clean up fence if vb2_core_qbuf() fails (Hans)
>    - add list to store sync_file and fence for the next queued buffer
>
>v2: check if the queue is ordered.
>
>Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
>---
> drivers/media/common/videobuf2/videobuf2-core.c | 95 +++++++++++++++++++++++--
> drivers/media/common/videobuf2/videobuf2-v4l2.c | 22 +++++-
> drivers/media/dvb-core/dvb_vb2.c                |  2 +-
> include/media/videobuf2-core.h                  | 16 ++++-
> 4 files changed, 127 insertions(+), 8 deletions(-)
>
>diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>index 996b99497a98..0f7306a04db7 100644
>--- a/drivers/media/common/videobuf2/videobuf2-core.c
>+++ b/drivers/media/common/videobuf2/videobuf2-core.c
>@@ -25,6 +25,7 @@
> #include <linux/sched.h>
> #include <linux/freezer.h>
> #include <linux/kthread.h>
>+#include <linux/sync_file.h>
>
> #include <media/videobuf2-core.h>
> #include <media/v4l2-mc.h>
>@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
> 			vb->planes[plane].length = plane_sizes[plane];
> 			vb->planes[plane].min_length = plane_sizes[plane];
> 		}
>+		vb->out_fence_fd = -1;
> 		q->bufs[vb->index] = vb;
>
> 		/* Allocate video buffer memory for the MMAP type */
>@@ -948,10 +950,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
> 	case VB2_BUF_STATE_QUEUED:
> 		return;
> 	case VB2_BUF_STATE_REQUEUEING:
>+		/* Requeuing with explicit synchronization, spit warning */
>+		WARN_ON_ONCE(vb->out_fence);
>+
> 		if (q->start_streaming_called)
> 			__enqueue_in_driver(vb);
> 		return;
> 	default:
>+		if (vb->out_fence) {
>+			if (state == VB2_BUF_STATE_ERROR)
>+				dma_fence_set_error(vb->out_fence, -EIO);
>+			dma_fence_signal(vb->out_fence);
>+			dma_fence_put(vb->out_fence);
>+			vb->out_fence = NULL;
>+			vb->out_fence_fd = -1;
>+		}
>+
> 		/* Inform any processes that may be waiting for buffers */
> 		wake_up(&q->done_wq);
> 		break;
>@@ -1367,6 +1381,68 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
> }
> EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>
>+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
>+{
>+	return "vb2_fence";
>+}
>+
>+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
>+{
>+	return "vb2_fence_timeline";
>+}
>+
>+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
>+{
>+	return true;
>+}
>+
>+static const struct dma_fence_ops vb2_fence_ops = {
>+	.get_driver_name = vb2_fence_get_driver_name,
>+	.get_timeline_name = vb2_fence_get_timeline_name,
>+	.enable_signaling = vb2_fence_enable_signaling,
>+	.wait = dma_fence_default_wait,
>+};
>+
>+static int vb2_setup_out_fence(struct vb2_queue *q, struct vb2_buffer *vb)
>+{
>+	struct sync_file *sync_file;
>+	int ret;
>+
>+	vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);

I think you should check the return value here. fd_install() doesn't
seem to do any validation. Best case you hit a BUG_ON(), worst case
you trash something random :-(

>+
>+	/*
>+	 * The same context can be used only if the queue is ordered,
>+	 * so if the queue is ordered create one when the queueing start,
>+	 * otherwise create one for every buffer
>+	 */
>+	if (q->unordered || !q->queueing_started)
>+		q->out_fence_context = dma_fence_context_alloc(1);

Isn't queueing_started always true here? Seems to be set in
vb2_core_qbuf() unconditionally, before vb2_setup_out_fence() is
called.

>+
>+	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
>+	if (!vb->out_fence) {
>+		ret = -ENOMEM;
>+		goto err_put_fd;
>+	}
>+	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
>+		       q->out_fence_context, 1);

If the context is shared, we should increment the seqno for every
fence.

Cheers,
-Brian

>+
>+	sync_file = sync_file_create(vb->out_fence);
>+	if (!sync_file) {
>+		ret = -ENOMEM;
>+		goto err_free_fence;
>+	}
>+	fd_install(vb->out_fence_fd, sync_file->file);
>+	return 0;
>+
>+err_free_fence:
>+	dma_fence_put(vb->out_fence);
>+	vb->out_fence = NULL;
>+err_put_fd:
>+	put_unused_fd(vb->out_fence_fd);
>+	vb->out_fence_fd = -1;
>+	return ret;
>+}
>+
> /*
>  * vb2_start_streaming() - Attempt to start streaming.
>  * @q:		videobuf2 queue
>@@ -1463,7 +1539,7 @@ static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> }
>
> int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>-		  struct dma_fence *in_fence)
>+		  struct dma_fence *in_fence, bool out_fence)
> {
> 	struct vb2_buffer *vb;
> 	unsigned long flags;
>@@ -1496,6 +1572,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> 	list_add_tail(&vb->queued_entry, &q->queued_list);
> 	q->queued_count++;
> 	q->waiting_for_buffers = false;
>+	q->queueing_started = 1;
> 	vb->state = VB2_BUF_STATE_QUEUED;
> 	vb->in_fence = in_fence;
>
>@@ -1555,6 +1632,14 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>
> 	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>
>+	if (out_fence) {
>+		ret = vb2_setup_out_fence(q, vb);
>+		if (ret) {
>+			dprintk(1, "failed to set up out-fence\n");
>+			goto err;
>+		}
>+	}
>+
> 	/* Fill buffer information for the userspace */
> 	if (pb)
> 		call_void_bufop(q, fill_user_buffer, vb, pb);
>@@ -1818,6 +1903,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
> 	q->start_streaming_called = 0;
> 	q->queued_count = 0;
> 	q->error = 0;
>+	q->queueing_started = 0;
>
> 	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> 		spin_lock_irqsave(&vb->fence_cb_lock, flags);
>@@ -2170,6 +2256,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
> 	spin_lock_init(&q->done_lock);
> 	mutex_init(&q->mmap_lock);
> 	init_waitqueue_head(&q->done_wq);
>+	spin_lock_init(&q->out_fence_lock);
>
> 	q->memory = VB2_MEMORY_UNKNOWN;
>
>@@ -2421,7 +2508,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
> 		 * Queue all buffers.
> 		 */
> 		for (i = 0; i < q->num_buffers; i++) {
>-			ret = vb2_core_qbuf(q, i, NULL, NULL);
>+			ret = vb2_core_qbuf(q, i, NULL, NULL, false);
> 			if (ret)
> 				goto err_reqbufs;
> 			fileio->bufs[i].queued = 1;
>@@ -2600,7 +2687,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>
> 		if (copy_timestamp)
> 			b->timestamp = ktime_get_ns();
>-		ret = vb2_core_qbuf(q, index, NULL, NULL);
>+		ret = vb2_core_qbuf(q, index, NULL, NULL, false);
> 		dprintk(5, "vb2_dbuf result: %d\n", ret);
> 		if (ret)
> 			return ret;
>@@ -2703,7 +2790,7 @@ static int vb2_thread(void *data)
> 		if (copy_timestamp)
> 			vb->timestamp = ktime_get_ns();
> 		if (!threadio->stop)
>-			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL, false);
> 		call_void_qop(q, wait_prepare, q);
> 		if (ret || threadio->stop)
> 			break;
>diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>index 74d7062e5285..41a88839683e 100644
>--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>@@ -217,7 +217,11 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
> 	b->sequence = vbuf->sequence;
> 	b->reserved = 0;
>
>-	b->fence_fd = 0;
>+	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE)
>+		b->fence_fd = vb->out_fence_fd;
>+	else
>+		b->fence_fd = 0;
>+
> 	if (vb->in_fence)
> 		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> 	else
>@@ -496,6 +500,13 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
> 	ret = __verify_planes_array(vb, b);
> 	if (!ret)
> 		vb2_core_querybuf(q, b->index, b);
>+
>+	/*
>+	 * Can't return a fence fd, because it only
>+	 * makes sense on the process that queued the buffer.
>+	 */
>+	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
>+		b->fence_fd = -1;
> 	return ret;
> }
> EXPORT_SYMBOL(vb2_querybuf);
>@@ -600,7 +611,8 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
> 		}
> 	}
>
>-	return vb2_core_qbuf(q, b->index, b, in_fence);
>+	return vb2_core_qbuf(q, b->index, b, in_fence,
>+			     b->flags & V4L2_BUF_FLAG_OUT_FENCE);
> }
> EXPORT_SYMBOL_GPL(vb2_qbuf);
>
>@@ -626,6 +638,12 @@ int vb2_dqbuf(struct vb2_queue *q, struct v4l2_buffer *b, bool nonblocking)
> 	 */
> 	b->flags &= ~V4L2_BUF_FLAG_DONE;
>
>+	/*
>+	 * Can't return a fence fd, because it only
>+	 * makes sense on the process that queued the buffer.
>+	 */
>+	if (b->flags & (V4L2_BUF_FLAG_IN_FENCE | V4L2_BUF_FLAG_OUT_FENCE))
>+		b->fence_fd = -1;
> 	return ret;
> }
> EXPORT_SYMBOL_GPL(vb2_dqbuf);
>diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
>index 7da53f10db1a..053803c9ff45 100644
>--- a/drivers/media/dvb-core/dvb_vb2.c
>+++ b/drivers/media/dvb-core/dvb_vb2.c
>@@ -385,7 +385,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
> {
> 	int ret;
>
>-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
>+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL, false);
> 	if (ret) {
> 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
> 			b->index, ret);
>diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>index 28ce8f66882e..794140fcb5cf 100644
>--- a/include/media/videobuf2-core.h
>+++ b/include/media/videobuf2-core.h
>@@ -260,6 +260,9 @@ struct vb2_buffer {
> 	 *			using the buffer (queueing to the driver)
> 	 * fence_cb:		fence callback information
> 	 * fence_cb_lock:	protect callback signal/remove
>+	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
>+	 * out_fence:		the out-fence associated with the buffer once
>+	 *			it is queued to the driver.
> 	 */
> 	enum vb2_buffer_state	state;
>
>@@ -271,6 +274,9 @@ struct vb2_buffer {
> 	struct dma_fence_cb	fence_cb;
> 	spinlock_t              fence_cb_lock;
>
>+	int			out_fence_fd;
>+	struct dma_fence	*out_fence;
>+
> #ifdef CONFIG_VIDEO_ADV_DEBUG
> 	/*
> 	 * Counters for how often these buffer-related ops are
>@@ -533,6 +539,9 @@ struct vb2_buf_ops {
>  * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
>  *		last decoded buffer was already dequeued. Set for capture queues
>  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
>+ * @queueing_started: if queueing has started. Currently used to determine
>+ *		if an out_fence_context is needed.
>+ * @out_fence_context: the fence context for the out fences
>  * @fileio:	file io emulator internal data, used only if emulator is active
>  * @threadio:	thread io internal data, used only if thread is active
>  */
>@@ -586,6 +595,10 @@ struct vb2_queue {
> 	unsigned int			is_output:1;
> 	unsigned int			copy_timestamp:1;
> 	unsigned int			last_buffer_dequeued:1;
>+	unsigned int			queueing_started:1;
>+
>+	u64				out_fence_context;
>+	spinlock_t			out_fence_lock;
>
> 	struct vb2_fileio_data		*fileio;
> 	struct vb2_threadio_data	*threadio;
>@@ -784,6 +797,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  * @pb:		buffer structure passed from userspace to
>  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
>  * @in_fence:	in-fence to wait on before queueing the buffer
>+ * @out_fence:	create an out-fence as a part of the queue operation
>  *
>  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
>@@ -799,7 +813,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  * Return: returns zero on success; an error code otherwise.
>  */
> int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>-		  struct dma_fence *in_fence);
>+		  struct dma_fence *in_fence, bool out_fence);
>
> /**
>  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
>-- 
>2.16.3
>

[PATCH v9 13/15] v4l: introduce the fences capability

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers capable of using fences (vb2 drivers) should report the
V4L2_CAP_FENCES to userspace, so add this flag to the uapi.

v2: minor doc/english fix (Hans Verkuil)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst b/Documentation/media/uapi/v4l/vidioc-querycap.rst
index 66fb1b3d6e6e..df3ad57f07a3 100644
--- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
@@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
     * - ``V4L2_CAP_TOUCH``
       - 0x10000000
       - This is a touch device.
+    * - ``V4L2_CAP_FENCES``
+      - 0x20000000
+      - The device supports explicit synchronization.
     * - ``V4L2_CAP_DEVICE_CAPS``
       - 0x80000000
       - The driver fills the ``device_caps`` field. This capability can
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1f18dc68ecab..cab35fca7c7f 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -460,6 +460,7 @@ struct v4l2_capability {
 #define V4L2_CAP_STREAMING              0x04000000  /* streaming I/O ioctls */
 
 #define V4L2_CAP_TOUCH                  0x10000000  /* Is a touch device */
+#define V4L2_CAP_FENCES                 0x20000000  /* Supports explicit synchronization */
 
 #define V4L2_CAP_DEVICE_CAPS            0x80000000  /* sets device capabilities field */
 
-- 
2.16.3

[PATCH v9 14/15] v4l: Add V4L2_CAP_FENCES to drivers

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers that use videobuf2 are capable of using fences and
should report that to userspace.

v9: Add in the core.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 8 ++++++++
 include/media/v4l2-fh.h              | 2 --
 2 files changed, 8 insertions(+), 2 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f75ad954a6f2..2ae527ef0bc7 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -1002,12 +1002,20 @@ static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
 {
 	struct v4l2_capability *cap = (struct v4l2_capability *)arg;
 	struct video_device *vfd = video_devdata(file);
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
 	int ret;
 
 	cap->version = LINUX_VERSION_CODE;
 	cap->device_caps = vfd->device_caps;
 	cap->capabilities = vfd->device_caps | V4L2_CAP_DEVICE_CAPS;
 
+	/* If it has a queue or a m2m context, then the
+	 * device supports fence synchronization.
+	 */
+	if (vfd->queue || (vfh && vfh->m2m_ctx))
+		cap->device_caps |= V4L2_CAP_FENCES;
+
 	ret = ops->vidioc_querycap(file, fh, cap);
 
 	cap->capabilities |= V4L2_CAP_EXT_PIX_FORMAT;
diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
index ea73fef8bdc0..e993ddc06991 100644
--- a/include/media/v4l2-fh.h
+++ b/include/media/v4l2-fh.h
@@ -57,9 +57,7 @@ struct v4l2_fh {
 	unsigned int		navailable;
 	u32			sequence;
 
-#if IS_ENABLED(CONFIG_V4L2_MEM2MEM_DEV)
 	struct v4l2_m2m_ctx	*m2m_ctx;
-#endif
 };
 
 /**
-- 
2.16.3

Re: [PATCH v9 14/15] v4l: Add V4L2_CAP_FENCES to drivers

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Drivers that use videobuf2 are capable of using fences and
> should report that to userspace.
> 
> v9: Add in the core.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 8 ++++++++
>  include/media/v4l2-fh.h              | 2 --
>  2 files changed, 8 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index f75ad954a6f2..2ae527ef0bc7 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -1002,12 +1002,20 @@ static int v4l_querycap(const struct v4l2_ioctl_ops *ops,
>  {
>  	struct v4l2_capability *cap = (struct v4l2_capability *)arg;
>  	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_fh *vfh =
> +		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
>  	int ret;
>  
>  	cap->version = LINUX_VERSION_CODE;
>  	cap->device_caps = vfd->device_caps;
>  	cap->capabilities = vfd->device_caps | V4L2_CAP_DEVICE_CAPS;
>  
> +	/* If it has a queue or a m2m context, then the
> +	 * device supports fence synchronization.
> +	 */
> +	if (vfd->queue || (vfh && vfh->m2m_ctx))
> +		cap->device_caps |= V4L2_CAP_FENCES;

This should also be set in cap->capabilities. I would actually do this
after the ops->vidioc_querycap call to avoid drivers overwriting this
capability flag. It also should only be set if device_caps has V4L2_CAP_STREAMING
set (not all drivers do!).

Note that not all m2m drivers use m2m_ctx, so some of them might have to
set the capability manually in the driver.

> +
>  	ret = ops->vidioc_querycap(file, fh, cap);
>  
>  	cap->capabilities |= V4L2_CAP_EXT_PIX_FORMAT;
> diff --git a/include/media/v4l2-fh.h b/include/media/v4l2-fh.h
> index ea73fef8bdc0..e993ddc06991 100644
> --- a/include/media/v4l2-fh.h
> +++ b/include/media/v4l2-fh.h
> @@ -57,9 +57,7 @@ struct v4l2_fh {
>  	unsigned int		navailable;
>  	u32			sequence;
>  
> -#if IS_ENABLED(CONFIG_V4L2_MEM2MEM_DEV)

I prefer is this #if was kept (so it needs to be added to v4l_querycap as well).

>  	struct v4l2_m2m_ctx	*m2m_ctx;
> -#endif
>  };
>  
>  /**
> 

Regards,

	Hans

[PATCH v9 15/15] v4l: Document explicit synchronization behavior

[Ezequiel Garcia]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it

v8: amend querybuf documentation.

v7: minor issues and English improvements (Hans Verkuil)

v6: Close some gaps in the docs (Hans)

v5: - Remove V4L2_CAP_ORDERED
    - Add doc about V4L2_FMT_FLAG_UNORDERED

v4: - Document ordering behavior for in-fences
    - Document V4L2_CAP_ORDERED capability
    - Remove doc about OUT_FENCE event
    - Document immediate return of out-fence in QBUF

v3: - make the out_fence refer to the current buffer (Hans)
    - Note what happens when the IN_FENCE is not set (Hans)

v2: - mention that fences are files (Hans)
    - rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 54 +++++++++++++++++++++++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
 2 files changed, 62 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..88a51b8bec7e 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,58 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+------------------------
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer. On the other
+side, the kernel can create out-fences for the buffers it queues to the
+drivers. Out-fences signal when the driver is finished with buffer, i.e., the
+buffer is ready. The fences are represented as a file and passed as a file
+descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the othe will cause ``VIDIOC_QBUF`` to return
+with an error.
+
+The videobuf2-core will guarantee that all buffers queued with an in-fence will
+be queued to the drivers in the same order. Fences may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering. So when
+waiting on a fence to signal all buffers queued afterwards will also be blocked
+until that fence signals.
+
+If the in-fence signals with an error the buffer will be marked with
+``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
+Even with the error the order of dequeueing the buffers is preserved.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument in the `fence_fd` field.
+
+Note the same `fence_fd` field is used for both sending the in-fence as
+at input argument and to receive the out-fence as a return argument. A buffer can
+have both an in-fence and an out-fence.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. They will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the  when using this format the order in which buffers are dequeued can
+be different from the order in which they were queued.
+
+Ordering is important to fences because it can optimize the pipeline with
+other drivers like a DRM/KMS display driver. For example, if a capture from the
+camera is happening in an orderly manner one can send the capture buffer
+out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on
+the screen at the correct order. If an ordered queue can not be set then such
+arrangements with other drivers may not be possible.
 
 Return Value
 ============
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..cda73b43c334 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
+one. The ``reserved`` field must be set to 0. When
 using the :ref:`multi-planar API <planar-apis>`, the ``m.planes``
 field must contain a userspace pointer to an array of struct
 :c:type:`v4l2_plane` and the ``length`` field has to be set
@@ -64,8 +64,14 @@ elements will be used instead and the ``length`` field of struct
 array elements. The driver may or may not set the remaining fields and
 flags, they are meaningless in this context.
 
-The struct :c:type:`v4l2_buffer` structure is specified in
-:ref:`buffer`.
+When using in-fences, the ``V4L2_BUF_FLAG_IN_FENCE`` will be set if the
+in-fence didn't signal at the time of the
+:ref:`VIDIOC_QUERYBUF`. Similarly, the ``V4L2_BUF_FLAG_OUT_FENCE`` will be
+set if there's a pending out-fence for the buffer. Note that the ``fence_fd``
+field is not set, because the file descriptor only makes sense
+for the process that received the out-fence in the :ref:`VIDIOC_QBUF` response.
+
+The struct :c:type:`v4l2_buffer` structure is specified in :ref:`buffer`.
 
 
 Return Value
-- 
2.16.3

Re: [PATCH v9 15/15] v4l: Document explicit synchronization behavior

[Hans Verkuil]

On 04/05/18 22:06, Ezequiel Garcia wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it
> 
> v8: amend querybuf documentation.
> 
> v7: minor issues and English improvements (Hans Verkuil)
> 
> v6: Close some gaps in the docs (Hans)
> 
> v5: - Remove V4L2_CAP_ORDERED
>     - Add doc about V4L2_FMT_FLAG_UNORDERED
> 
> v4: - Document ordering behavior for in-fences
>     - Document V4L2_CAP_ORDERED capability
>     - Remove doc about OUT_FENCE event
>     - Document immediate return of out-fence in QBUF
> 
> v3: - make the out_fence refer to the current buffer (Hans)
>     - Note what happens when the IN_FENCE is not set (Hans)
> 
> v2: - mention that fences are files (Hans)
>     - rework for the new API
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 54 +++++++++++++++++++++++-
>  Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
>  2 files changed, 62 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> index 9e448a4aa3aa..88a51b8bec7e 100644
> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> @@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
>  or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
>  ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
>  for details. Applications must also set ``flags`` to 0. The
> -``reserved2`` and ``reserved`` fields must be set to 0. When using the
> +``reserved`` field must be set to 0. When using the
>  :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
>  contain a userspace pointer to a filled-in array of struct
>  :c:type:`v4l2_plane` and the ``length`` field must be set
> @@ -118,6 +118,58 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
>  The struct :c:type:`v4l2_buffer` structure is specified in
>  :ref:`buffer`.
>  
> +Explicit Synchronization
> +------------------------
> +
> +Explicit Synchronization allows us to control the synchronization of
> +shared buffers from userspace by passing fences to the kernel and/or
> +receiving them from it. Fences passed to the kernel are named in-fences and
> +the kernel should wait on them to signal before using the buffer. On the other
> +side, the kernel can create out-fences for the buffers it queues to the
> +drivers. Out-fences signal when the driver is finished with buffer, i.e., the
> +buffer is ready. The fences are represented as a file and passed as a file
> +descriptor to userspace.
> +
> +The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
> +using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
> +an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
> +fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
> +as well. Setting one but not the othe will cause ``VIDIOC_QBUF`` to return

othe -> other

> +with an error.
> +
> +The videobuf2-core will guarantee that all buffers queued with an in-fence will

'It is guaranteed that'

No need to refer to internal frameworks in userspace documentation.

> +be queued to the drivers in the same order. Fences may signal out of order, so
> +this guarantee at videobuf2 is necessary to not change ordering. So when

s/at videobuf2//
s/when/while/

> +waiting on a fence to signal all buffers queued afterwards will also be blocked
> +until that fence signals.
> +
> +If the in-fence signals with an error the buffer will be marked with
> +``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
> +Even with the error the order of dequeueing the buffers is preserved.
> +
> +To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
> +be set to ask for a fence to be attached to the buffer. The out-fence fd is
> +sent to userspace as a ``VIDIOC_QBUF`` return argument in the `fence_fd` field.
> +
> +Note the same `fence_fd` field is used for both sending the in-fence as
> +at input argument and to receive the out-fence as a return argument. A buffer can
> +have both an in-fence and an out-fence.
> +
> +At streamoff the out-fences will either signal normally if the driver waits
> +for the operations on the buffers to finish or signal with an error if the
> +driver cancels the pending operations. Buffers with in-fences won't be queued
> +to the driver if their fences signal. They will be cleaned up.
> +
> +The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
> +that the  when using this format the order in which buffers are dequeued can

s/the//

> +be different from the order in which they were queued.
> +
> +Ordering is important to fences because it can optimize the pipeline with
> +other drivers like a DRM/KMS display driver. For example, if a capture from the
> +camera is happening in an orderly manner one can send the capture buffer

orderly -> ordered?

> +out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on

s/sure/assured/

> +the screen at the correct order. If an ordered queue can not be set then such

s/at/in/

'If the queue is not ordered, then...'

> +arrangements with other drivers may not be possible.
>  
>  Return Value
>  ============
> diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> index dd54747fabc9..cda73b43c334 100644
> --- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> @@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero to the
>  number of buffers allocated with
>  :ref:`VIDIOC_REQBUFS` (struct
>  :c:type:`v4l2_requestbuffers` ``count``) minus
> -one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
> +one. The ``reserved`` field must be set to 0. When
>  using the :ref:`multi-planar API <planar-apis>`, the ``m.planes``
>  field must contain a userspace pointer to an array of struct
>  :c:type:`v4l2_plane` and the ``length`` field has to be set
> @@ -64,8 +64,14 @@ elements will be used instead and the ``length`` field of struct
>  array elements. The driver may or may not set the remaining fields and
>  flags, they are meaningless in this context.
>  
> -The struct :c:type:`v4l2_buffer` structure is specified in
> -:ref:`buffer`.
> +When using in-fences, the ``V4L2_BUF_FLAG_IN_FENCE`` will be set if the
> +in-fence didn't signal at the time of the
> +:ref:`VIDIOC_QUERYBUF`. Similarly, the ``V4L2_BUF_FLAG_OUT_FENCE`` will be
> +set if there's a pending out-fence for the buffer. Note that the ``fence_fd``
> +field is not set, because the file descriptor only makes sense
> +for the process that received the out-fence in the :ref:`VIDIOC_QBUF` response.
> +
> +The struct :c:type:`v4l2_buffer` structure is specified in :ref:`buffer`.
>  
>  
>  Return Value
> 

Regards,

	Hans