[PATCH v8 00/13] V4L2 Explicit Synchronization

[PATCH v8 00/13] V4L2 Explicit Synchronization

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Hi,

So v8 is finally out addressing the comments from the previous version[1].
For more info see v5 cover letter[2]. The most important points I address
here is the handling of fences that signal with error to follow Hans
suggestion. I also added V4L2_CAP_FENCES to all vb2 drivers and marked all
codec and cobalt as unordered. More specific changelog are noted on the
patches itself.

The first 3 patches are just clean ups in preparation to add the new cap flag
and can go upstream earlier. Then there are patches to add unordered info,
the actual fences implementation and later the V4L2_CAP_FENCES flag.
The last patch contains the Documentation.

You can find the code at:

https://gitlab.collabora.com/padovan/linux/tree/v4l2-fences

The test tools I've been using are:
https://gitlab.collabora.com/padovan/drm-v4l2-test
https://gitlab.collabora.com/padovan/v4l2-fences-test

Please review,

Gustavo

[1] https://lkml.org/lkml/2018/1/10/644
[2] https://lkml.org/lkml/2017/11/15/550

Gustavo Padovan (13):
  [media] xilinx: regroup caps on querycap
  [media] hackrf: group device capabilities
  [media] omap3isp: group device capabilities
  [media] vb2: add is_unordered callback for drivers
  [media] v4l: add 'unordered' flag to format description ioctl
  [media] cobalt: add .is_unordered() for cobalt
  [media] vb2: mark codec drivers as unordered
  [media] vb2: add explicit fence user API
  [media] vb2: add in-fence support to QBUF
  [media] vb2: add out-fence support to QBUF
  [media] v4l: introduce the fences capability
  [media] v4l: Add V4L2_CAP_FENCES to drivers
  [media] v4l: Document explicit synchronization behavior

 Documentation/media/uapi/v4l/buffer.rst            |  45 +++-
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst   |   7 +
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  55 +++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst   |  12 +-
 Documentation/media/uapi/v4l/vidioc-querycap.rst   |   3 +
 drivers/media/common/videobuf2/videobuf2-core.c    | 289 ++++++++++++++++++---
 drivers/media/common/videobuf2/videobuf2-v4l2.c    |  58 ++++-
 drivers/media/pci/cobalt/cobalt-v4l2.c             |   4 +-
 drivers/media/pci/cx23885/cx23885-417.c            |   2 +-
 drivers/media/pci/cx23885/cx23885-video.c          |   3 +-
 drivers/media/pci/cx88/cx88-video.c                |   3 +-
 drivers/media/pci/dt3155/dt3155.c                  |   2 +-
 drivers/media/pci/saa7134/saa7134-video.c          |   2 +
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c     |   3 +-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c         |   3 +-
 drivers/media/pci/sta2x11/sta2x11_vip.c            |   2 +-
 drivers/media/pci/tw68/tw68-video.c                |   3 +-
 drivers/media/pci/tw686x/tw686x-video.c            |   2 +-
 drivers/media/platform/am437x/am437x-vpfe.c        |   2 +-
 drivers/media/platform/blackfin/bfin_capture.c     |   3 +-
 drivers/media/platform/coda/coda-common.c          |   4 +-
 drivers/media/platform/davinci/vpbe_display.c      |   3 +-
 drivers/media/platform/davinci/vpfe_capture.c      |   3 +-
 drivers/media/platform/davinci/vpif_capture.c      |   3 +-
 drivers/media/platform/davinci/vpif_display.c      |   3 +-
 drivers/media/platform/exynos-gsc/gsc-m2m.c        |   4 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   3 +-
 drivers/media/platform/exynos4-is/fimc-isp-video.c |   3 +-
 drivers/media/platform/exynos4-is/fimc-lite.c      |   2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c       |   4 +-
 drivers/media/platform/m2m-deinterlace.c           |   4 +-
 drivers/media/platform/marvell-ccic/mcam-core.c    |   2 +-
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c    |   1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c |   1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c |   1 +
 drivers/media/platform/mx2_emmaprp.c               |   4 +-
 drivers/media/platform/omap3isp/ispvideo.c         |   9 +-
 drivers/media/platform/pxa_camera.c                |   3 +-
 drivers/media/platform/qcom/venus/vdec.c           |   1 +
 drivers/media/platform/qcom/venus/venc.c           |   1 +
 drivers/media/platform/rcar_fdp1.c                 |   1 +
 drivers/media/platform/rcar_jpu.c                  |   4 +-
 drivers/media/platform/rockchip/rga/rga-buf.c      |   1 +
 drivers/media/platform/s3c-camif/camif-capture.c   |   3 +-
 drivers/media/platform/s5p-g2d/g2d.c               |   4 +-
 drivers/media/platform/s5p-jpeg/jpeg-core.c        |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       |   4 +-
 drivers/media/platform/sh_veu.c                    |   4 +-
 drivers/media/platform/sh_vou.c                    |   2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c      |   4 +-
 drivers/media/platform/ti-vpe/cal.c                |   2 +-
 drivers/media/platform/ti-vpe/vpe.c                |   4 +-
 drivers/media/platform/vim2m.c                     |   4 +-
 drivers/media/platform/vivid/vivid-core.c          |   2 +-
 drivers/media/platform/vsp1/vsp1_histo.c           |   2 +-
 drivers/media/platform/vsp1/vsp1_video.c           |   2 +-
 drivers/media/platform/xilinx/xilinx-dma.c         |   7 +-
 drivers/media/usb/airspy/airspy.c                  |   2 +-
 drivers/media/usb/au0828/au0828-video.c            |   3 +-
 drivers/media/usb/em28xx/em28xx-video.c            |   1 +
 drivers/media/usb/go7007/go7007-v4l2.c             |   2 +-
 drivers/media/usb/hackrf/hackrf.c                  |  12 +-
 drivers/media/usb/msi2500/msi2500.c                |   2 +-
 drivers/media/usb/pwc/pwc-v4l.c                    |   2 +-
 drivers/media/usb/s2255/s2255drv.c                 |   2 +-
 drivers/media/usb/stk1160/stk1160-v4l.c            |   3 +-
 drivers/media/usb/usbtv/usbtv-video.c              |   3 +-
 drivers/media/usb/uvc/uvc_driver.c                 |   1 +
 drivers/media/v4l2-core/Kconfig                    |  33 +++
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |   4 +-
 include/media/videobuf2-core.h                     |  45 +++-
 include/media/videobuf2-v4l2.h                     |  10 +
 include/uapi/linux/videodev2.h                     |   9 +-
 75 files changed, 650 insertions(+), 105 deletions(-)

-- 
2.14.3

[PATCH v8 01/13] [media] xilinx: regroup caps on querycap

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

To better organize the code we concentrate the setting of
V4L2_CAP_STREAMING in one place.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 7 ++++---
 1 file changed, 4 insertions(+), 3 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 522cdfdd3345..565e466ba4fa 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,13 +494,14 @@ xvip_dma_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	struct v4l2_fh *vfh = file->private_data;
 	struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
+	cap->device_caps = V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
 			  | dma->xdev->v4l2_caps;
 
 	if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
 	strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
 	strlcpy(cap->card, dma->video.name, sizeof(cap->card));
-- 
2.14.3

[PATCH v8 02/13] [media] hackrf: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING and V4L2_CAP_READWRITE
everywhere, set device_caps earlier with these values.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/usb/hackrf/hackrf.c | 11 ++++-------
 1 file changed, 4 insertions(+), 7 deletions(-)

diff --git a/drivers/media/usb/hackrf/hackrf.c b/drivers/media/usb/hackrf/hackrf.c
index 7eb53517a82f..6d692fb3e8dd 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,18 +909,15 @@ static int hackrf_querycap(struct file *file, void *fh,
 
 	dev_dbg(&intf->dev, "\n");
 
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
 	if (vdev->vfl_dir == VFL_DIR_RX)
-		cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
-
+		cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
 	else
-		cap->device_caps = V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-				   V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+		cap->device_caps |= V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR;
 
 	cap->capabilities = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER |
 			    V4L2_CAP_SDR_OUTPUT | V4L2_CAP_MODULATOR |
-			    V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
-			    V4L2_CAP_DEVICE_CAPS;
+			    V4L2_CAP_DEVICE_CAPS | cap->device_caps;
 	strlcpy(cap->driver, KBUILD_MODNAME, sizeof(cap->driver));
 	strlcpy(cap->card, dev->rx_vdev.name, sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
-- 
2.14.3

[PATCH v8 03/13] [media] omap3isp: group device capabilities

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
earlier with this value.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/omap3isp/ispvideo.c | 9 +++++----
 1 file changed, 5 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a751c89a3ea8..b4d4ef926749 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,13 +658,14 @@ isp_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	strlcpy(cap->card, video->video.name, sizeof(cap->card));
 	strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-	cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
-		| V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
+	cap->device_caps = V4L2_CAP_STREAMING;
+	cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
+		V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
 	if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
-		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
-		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 
 	return 0;
 }
-- 
2.14.3

[PATCH v8 04/13] [media] vb2: add is_unordered callback for drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Explicit synchronization benefits a lot from ordered queues, they fit
better in a pipeline with DRM for example so create a opt-in way for
drivers notify videobuf2 that the queue is unordered.

Drivers don't need implement it if the queue is ordered.

v2: 	- improve comments for is_unordered flag (Hans)

v3: 	- make it bool (Hans)
	- create vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 ++++++
 include/media/videobuf2-core.h                  |  6 ++++++
 include/media/videobuf2-v4l2.h                  | 10 ++++++++++
 3 files changed, 22 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..68291ba8632d 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -961,6 +961,12 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
 }
 EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
 
+bool vb2_ops_set_unordered(struct vb2_queue *q)
+{
+	return true;
+}
+EXPORT_SYMBOL_GPL(vb2_ops_set_unordered);
+
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak <pawel@osciak.com>, Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 5b6c541e4e1b..46a9e674f7e1 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -370,6 +370,10 @@ struct vb2_buffer {
  *			callback by calling vb2_buffer_done() with either
  *			%VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
  *			vb2_wait_for_all_buffers() function
+ * @is_unordered:	tell if the queue is unordered, i.e. buffers can be
+ *			dequeued in a different order from how they were queued.
+ *			The default is assumed to be ordered and this function
+ *			only needs to be implemented for unordered queues.
  * @buf_queue:		passes buffer vb to the driver; driver may start
  *			hardware operation on this buffer; driver should give
  *			the buffer back by calling vb2_buffer_done() function;
@@ -393,6 +397,7 @@ struct vb2_ops {
 
 	int (*start_streaming)(struct vb2_queue *q, unsigned int count);
 	void (*stop_streaming)(struct vb2_queue *q);
+	bool (*is_unordered)(struct vb2_queue *q);
 
 	void (*buf_queue)(struct vb2_buffer *vb);
 };
@@ -566,6 +571,7 @@ struct vb2_queue {
 	u32				cnt_wait_finish;
 	u32				cnt_start_streaming;
 	u32				cnt_stop_streaming;
+	u32				cnt_is_unordered;
 #endif
 };
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..9de3c887c875 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -291,4 +291,14 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
  */
 void vb2_ops_wait_finish(struct vb2_queue *vq);
 
+/**
+ * vb2_ops_set_unordered - helper function to mark queue as unordered
+ *
+ * @vq: pointer to &struct vb2_queue
+ *
+ * This helper just return true to notify that the driver can't deal with
+ * ordered queues.
+ */
+bool vb2_ops_set_unordered(struct vb2_queue *q);
+
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.14.3

[PATCH v8 05/13] [media] v4l: add 'unordered' flag to format description ioctl

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

For explicit synchronization it important for userspace to know if the
format being used by the driver can deliver the buffers back to userspace
in the same order they were queued with QBUF.

Ordered streams fits nicely in a pipeline with DRM for example, where
ordered buffer are expected.

v2	- Improve documentation (Hans)

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-enum-fmt.rst | 7 +++++++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 8 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
index 019c513df217..df8e039b9ac2 100644
--- a/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
+++ b/Documentation/media/uapi/v4l/vidioc-enum-fmt.rst
@@ -116,6 +116,13 @@ one until ``EINVAL`` is returned.
       - This format is not native to the device but emulated through
 	software (usually libv4l2), where possible try to use a native
 	format instead for better performance.
+    * - ``V4L2_FMT_FLAG_UNORDERED``
+      - 0x0004
+      - This format doesn't guarantee ordered buffer handling. I.e. the order
+	in which buffers are dequeued with
+	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` may be different
+	from the order in which they were queued with
+	:ref:`VIDIOC_QBUF <VIDIOC_QBUF>`.
 
 
 Return Value
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..58894cfe9479 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -716,6 +716,7 @@ struct v4l2_fmtdesc {
 
 #define V4L2_FMT_FLAG_COMPRESSED 0x0001
 #define V4L2_FMT_FLAG_EMULATED   0x0002
+#define V4L2_FMT_FLAG_UNORDERED  0x0004
 
 	/* Frame Size and frame rate enumeration */
 /*
-- 
2.14.3

[PATCH v8 06/13] [media] cobalt: add .is_unordered() for cobalt

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

The cobalt driver may reorder the capture buffers so we need to report
it as such.

v2: - use vb2_ops_set_unordered() helper

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
 1 file changed, 1 insertion(+)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
index e2a4c705d353..6b6611a0e190 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
 	.stop_streaming = cobalt_stop_streaming,
 	.wait_prepare = vb2_ops_wait_prepare,
 	.wait_finish = vb2_ops_wait_finish,
+	.is_unordered = vb2_ops_set_unordered,
 };
 
 /* V4L2 ioctls */
-- 
2.14.3

[PATCH v8 07/13] [media] vb2: mark codec drivers as unordered

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

In preparation to have full support to explicit fence we are
marking codec as non-ordered preventively. It is easier and safer from an
uAPI point of view to move from unordered to ordered than the opposite.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/platform/coda/coda-common.c          | 1 +
 drivers/media/platform/exynos-gsc/gsc-m2m.c        | 1 +
 drivers/media/platform/exynos4-is/fimc-m2m.c       | 1 +
 drivers/media/platform/m2m-deinterlace.c           | 1 +
 drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c    | 1 +
 drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
 drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
 drivers/media/platform/mx2_emmaprp.c               | 1 +
 drivers/media/platform/qcom/venus/vdec.c           | 1 +
 drivers/media/platform/qcom/venus/venc.c           | 1 +
 drivers/media/platform/rcar_fdp1.c                 | 1 +
 drivers/media/platform/rcar_jpu.c                  | 1 +
 drivers/media/platform/rockchip/rga/rga-buf.c      | 1 +
 drivers/media/platform/s5p-g2d/g2d.c               | 1 +
 drivers/media/platform/s5p-jpeg/jpeg-core.c        | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       | 1 +
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       | 1 +
 drivers/media/platform/sh_veu.c                    | 1 +
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c      | 1 +
 drivers/media/platform/ti-vpe/vpe.c                | 1 +
 drivers/media/platform/vim2m.c                     | 1 +
 22 files changed, 22 insertions(+)

diff --git a/drivers/media/platform/coda/coda-common.c b/drivers/media/platform/coda/coda-common.c
index 04e35d70ce2e..6deb29fe6eb7 100644
--- a/drivers/media/platform/coda/coda-common.c
+++ b/drivers/media/platform/coda/coda-common.c
@@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
 	.stop_streaming		= coda_stop_streaming,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c b/drivers/media/platform/exynos-gsc/gsc-m2m.c
index e9ff27949a91..10c3e4659d38 100644
--- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
+++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
@@ -286,6 +286,7 @@ static const struct vb2_ops gsc_m2m_qops = {
 	.wait_finish	 = vb2_ops_wait_finish,
 	.stop_streaming	 = gsc_m2m_stop_streaming,
 	.start_streaming = gsc_m2m_start_streaming,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int gsc_m2m_querycap(struct file *file, void *fh,
diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c b/drivers/media/platform/exynos4-is/fimc-m2m.c
index a19f8b164a47..dfc487a582c0 100644
--- a/drivers/media/platform/exynos4-is/fimc-m2m.c
+++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
@@ -227,6 +227,7 @@ static const struct vb2_ops fimc_qops = {
 	.wait_finish	 = vb2_ops_wait_finish,
 	.stop_streaming	 = stop_streaming,
 	.start_streaming = start_streaming,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 /*
diff --git a/drivers/media/platform/m2m-deinterlace.c b/drivers/media/platform/m2m-deinterlace.c
index 1e4195144f39..35a0f45d2a51 100644
--- a/drivers/media/platform/m2m-deinterlace.c
+++ b/drivers/media/platform/m2m-deinterlace.c
@@ -856,6 +856,7 @@ static const struct vb2_ops deinterlace_qops = {
 	.queue_setup	 = deinterlace_queue_setup,
 	.buf_prepare	 = deinterlace_buf_prepare,
 	.buf_queue	 = deinterlace_buf_queue,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
index 226f90886484..34a4b5b2e1b5 100644
--- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
+++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
@@ -764,6 +764,7 @@ static const struct vb2_ops mtk_jpeg_qops = {
 	.wait_finish        = vb2_ops_wait_finish,
 	.start_streaming    = mtk_jpeg_start_streaming,
 	.stop_streaming     = mtk_jpeg_stop_streaming,
+	.is_unordered       = vb2_ops_set_unordered,
 };
 
 static void mtk_jpeg_set_dec_src(struct mtk_jpeg_ctx *ctx,
diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
index 583d47724ee8..f3bb9f277f55 100644
--- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
+++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
@@ -629,6 +629,7 @@ static const struct vb2_ops mtk_mdp_m2m_qops = {
 	.wait_finish	 = mtk_mdp_ctx_lock,
 	.stop_streaming	 = mtk_mdp_m2m_stop_streaming,
 	.start_streaming = mtk_mdp_m2m_start_streaming,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int mtk_mdp_m2m_querycap(struct file *file, void *fh,
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
index 86f0a7134365..4f33e9741248 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
@@ -1445,6 +1445,7 @@ static const struct vb2_ops mtk_vdec_vb2_ops = {
 	.buf_finish	= vb2ops_vdec_buf_finish,
 	.start_streaming	= vb2ops_vdec_start_streaming,
 	.stop_streaming	= vb2ops_vdec_stop_streaming,
+	.is_unordered	= vb2_ops_set_unordered,
 };
 
 const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops = {
diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
index 1b1a28abbf1f..fd763249d7bd 100644
--- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
+++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
@@ -931,6 +931,7 @@ static const struct vb2_ops mtk_venc_vb2_ops = {
 	.wait_finish		= vb2_ops_wait_finish,
 	.start_streaming	= vb2ops_venc_start_streaming,
 	.stop_streaming		= vb2ops_venc_stop_streaming,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 static int mtk_venc_encode_header(void *priv)
diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c
index 5a8eff60e95f..d03becff66cf 100644
--- a/drivers/media/platform/mx2_emmaprp.c
+++ b/drivers/media/platform/mx2_emmaprp.c
@@ -747,6 +747,7 @@ static const struct vb2_ops emmaprp_qops = {
 	.queue_setup	 = emmaprp_queue_setup,
 	.buf_prepare	 = emmaprp_buf_prepare,
 	.buf_queue	 = emmaprp_buf_queue,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/qcom/venus/vdec.c b/drivers/media/platform/qcom/venus/vdec.c
index c9e9576bb08a..20acbfe2150d 100644
--- a/drivers/media/platform/qcom/venus/vdec.c
+++ b/drivers/media/platform/qcom/venus/vdec.c
@@ -793,6 +793,7 @@ static const struct vb2_ops vdec_vb2_ops = {
 	.start_streaming = vdec_start_streaming,
 	.stop_streaming = venus_helper_vb2_stop_streaming,
 	.buf_queue = venus_helper_vb2_buf_queue,
+	.is_unordered = vb2_ops_set_unordered,
 };
 
 static void vdec_buf_done(struct venus_inst *inst, unsigned int buf_type,
diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
index e3a10a852cad..abefae68ce5a 100644
--- a/drivers/media/platform/qcom/venus/venc.c
+++ b/drivers/media/platform/qcom/venus/venc.c
@@ -982,6 +982,7 @@ static const struct vb2_ops venc_vb2_ops = {
 	.start_streaming = venc_start_streaming,
 	.stop_streaming = venus_helper_vb2_stop_streaming,
 	.buf_queue = venus_helper_vb2_buf_queue,
+	.is_unordered = vb2_ops_set_unordered,
 };
 
 static void venc_buf_done(struct venus_inst *inst, unsigned int buf_type,
diff --git a/drivers/media/platform/rcar_fdp1.c b/drivers/media/platform/rcar_fdp1.c
index b13dec3081e5..6a744a9c1738 100644
--- a/drivers/media/platform/rcar_fdp1.c
+++ b/drivers/media/platform/rcar_fdp1.c
@@ -2040,6 +2040,7 @@ static const struct vb2_ops fdp1_qops = {
 	.stop_streaming  = fdp1_stop_streaming,
 	.wait_prepare	 = vb2_ops_wait_prepare,
 	.wait_finish	 = vb2_ops_wait_finish,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c
index f6092ae45912..b4b2e2cf5d1a 100644
--- a/drivers/media/platform/rcar_jpu.c
+++ b/drivers/media/platform/rcar_jpu.c
@@ -1192,6 +1192,7 @@ static const struct vb2_ops jpu_qops = {
 	.stop_streaming		= jpu_stop_streaming,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 static int jpu_queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/rockchip/rga/rga-buf.c b/drivers/media/platform/rockchip/rga/rga-buf.c
index fa1ba98c96dc..48932f34144d 100644
--- a/drivers/media/platform/rockchip/rga/rga-buf.c
+++ b/drivers/media/platform/rockchip/rga/rga-buf.c
@@ -112,6 +112,7 @@ const struct vb2_ops rga_qops = {
 	.wait_finish = vb2_ops_wait_finish,
 	.start_streaming = rga_buf_start_streaming,
 	.stop_streaming = rga_buf_stop_streaming,
+	.is_unordered = vb2_ops_set_unordered,
 };
 
 /* RGA MMU is a 1-Level MMU, so it can't be used through the IOMMU API.
diff --git a/drivers/media/platform/s5p-g2d/g2d.c b/drivers/media/platform/s5p-g2d/g2d.c
index 66aa8cf1d048..cb7d916bfc8b 100644
--- a/drivers/media/platform/s5p-g2d/g2d.c
+++ b/drivers/media/platform/s5p-g2d/g2d.c
@@ -142,6 +142,7 @@ static const struct vb2_ops g2d_qops = {
 	.queue_setup	= g2d_queue_setup,
 	.buf_prepare	= g2d_buf_prepare,
 	.buf_queue	= g2d_buf_queue,
+	.is_unordered	= vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c
index 79b63da27f53..28485e6b9cc8 100644
--- a/drivers/media/platform/s5p-jpeg/jpeg-core.c
+++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c
@@ -2648,6 +2648,7 @@ static const struct vb2_ops s5p_jpeg_qops = {
 	.wait_finish		= vb2_ops_wait_finish,
 	.start_streaming	= s5p_jpeg_start_streaming,
 	.stop_streaming		= s5p_jpeg_stop_streaming,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
index 8937b0af7cb3..369db08dbcae 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
@@ -1099,6 +1099,7 @@ static struct vb2_ops s5p_mfc_dec_qops = {
 	.start_streaming	= s5p_mfc_start_streaming,
 	.stop_streaming		= s5p_mfc_stop_streaming,
 	.buf_queue		= s5p_mfc_buf_queue,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 const struct s5p_mfc_codec_ops *get_dec_codec_ops(void)
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
index 0d5d465561be..fece496c2a8e 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
@@ -2036,6 +2036,7 @@ static struct vb2_ops s5p_mfc_enc_qops = {
 	.start_streaming	= s5p_mfc_start_streaming,
 	.stop_streaming		= s5p_mfc_stop_streaming,
 	.buf_queue		= s5p_mfc_buf_queue,
+	.is_unordered		= vb2_ops_set_unordered,
 };
 
 const struct s5p_mfc_codec_ops *get_enc_codec_ops(void)
diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c
index 1a0cde017fdf..0682b50a67fc 100644
--- a/drivers/media/platform/sh_veu.c
+++ b/drivers/media/platform/sh_veu.c
@@ -927,6 +927,7 @@ static const struct vb2_ops sh_veu_qops = {
 	.buf_queue	 = sh_veu_buf_queue,
 	.wait_prepare	 = vb2_ops_wait_prepare,
 	.wait_finish	 = vb2_ops_wait_finish,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int sh_veu_queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
index bf4ca16db440..0cfdc5a67855 100644
--- a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
+++ b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
@@ -535,6 +535,7 @@ static const struct vb2_ops bdisp_qops = {
 	.wait_finish     = vb2_ops_wait_finish,
 	.stop_streaming  = bdisp_stop_streaming,
 	.start_streaming = bdisp_start_streaming,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv,
diff --git a/drivers/media/platform/ti-vpe/vpe.c b/drivers/media/platform/ti-vpe/vpe.c
index e395aa85c8ad..c2d838884e1c 100644
--- a/drivers/media/platform/ti-vpe/vpe.c
+++ b/drivers/media/platform/ti-vpe/vpe.c
@@ -2202,6 +2202,7 @@ static const struct vb2_ops vpe_qops = {
 	.wait_finish	 = vb2_ops_wait_finish,
 	.start_streaming = vpe_start_streaming,
 	.stop_streaming  = vpe_stop_streaming,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq,
diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..e1a54a28b082 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -823,6 +823,7 @@ static const struct vb2_ops vim2m_qops = {
 	.stop_streaming  = vim2m_stop_streaming,
 	.wait_prepare	 = vb2_ops_wait_prepare,
 	.wait_finish	 = vb2_ops_wait_finish,
+	.is_unordered	 = vb2_ops_set_unordered,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
-- 
2.14.3

[PATCH v8 08/13] [media] vb2: add explicit fence user API

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Turn the reserved2 field into fence_fd that we will use to send
an in-fence to the kernel or return an out-fence from the kernel to
userspace.

Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
when sending a fence to the kernel to be waited on, and
V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.

v6:	- big improvement on doc (Hans Verkuil)

v5:
	- keep using reserved2 field for cpia2
	- set fence_fd to 0 for now, for compat with userspace(Mauro)

v4:
	- make it a union with reserved2 and fence_fd (Hans Verkuil)

v3:
	- make the out_fence refer to the current buffer (Hans Verkuil)

v2: add documentation

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
 include/uapi/linux/videodev2.h                  |  7 +++-
 4 files changed, 51 insertions(+), 7 deletions(-)

diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..49273026740f 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,22 @@ struct v4l2_buffer
 	elements in the ``planes`` array. The driver will fill in the
 	actual number of valid elements in that array.
     * - __u32
-      - ``reserved2``
+      - ``fence_fd``
       -
-      - A place holder for future extensions. Drivers and applications
-	must set this to 0.
+      - Used to communicate fences file descriptors from userspace to kernel
+	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
+	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
+	be used and this field set to the fence file descriptor of the in-fence
+	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.
+
+        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
+	must be set, the kernel will return the out-fence file descriptor on
+	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns
+        an error.
+
+	In all other ioctls V4L2 sets this field to -1 if
+	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
+	otherwise this field is set to 0 for backward compatibility.
     * - __u32
       - ``reserved``
       -
@@ -648,6 +660,33 @@ Buffer Flags
       - Start Of Exposure. The buffer timestamp has been taken when the
 	exposure of the frame has begun. This is only valid for the
 	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
+    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
+
+      - ``V4L2_BUF_FLAG_IN_FENCE``
+      - 0x00200000
+      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
+	buffer won't be queued to the driver until the fence signals. The order
+	in which buffers are queued is guaranteed to be preserved, so any
+	buffers queued after this buffer will also be blocked until this fence
+	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
+	other ioctls the driver just report the value of the flag.
+
+        If the fence signals the flag is cleared and not reported anymore.
+	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
+
+
+    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
+
+      - ``V4L2_BUF_FLAG_OUT_FENCE``
+      - 0x00400000
+      - Request for a fence to be attached to the buffer. The driver will fill
+	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
+	<VIDIOC_QBUF>` returns. This flag must be set before calling
+	``VIDIOC_QBUF``. For other ioctls the driver just report the value of
+	the flag.
+
+        If the creation of the  out-fence  fails ``VIDIOC_QBUF`` returns an
+	error.
 
 
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 68291ba8632d..ad1e032c3bf5 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
+	b->fence_fd = 0;
 	b->reserved = 0;
 
 	if (q->is_multiplanar) {
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..3de2252e3632 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__s32			fence_fd;
 	__u32			reserved;
 };
 
@@ -604,7 +604,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
 	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
 	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
 	    assign_in_user(&up->sequence, &kp->sequence) ||
-	    assign_in_user(&up->reserved2, &kp->reserved2) ||
+	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
 	    assign_in_user(&up->reserved, &kp->reserved) ||
 	    get_user(length, &kp->length) ||
 	    put_user(length, &up->length))
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 58894cfe9479..2d424aebdd1e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -933,7 +933,10 @@ struct v4l2_buffer {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	union {
+		__s32		fence_fd;
+		__u32		reserved2;
+	};
 	__u32			reserved;
 };
 
@@ -970,6 +973,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
+#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.14.3

[PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Receive in-fence from userspace and add support for waiting on them
before queueing the buffer to the driver. Buffers can't be queued to the
driver before its fences signal. And a buffer can't be queue to the driver
out of the order they were queued from userspace. That means that even if
it fence signal it must wait all other buffers, ahead of it in the queue,
to signal first.

If the fence for some buffer fails we do not queue it to the driver,
instead we mark it as error and wait until the previous buffer is done
to notify userspace of the error. We wait here to deliver the buffers back
to userspace in order.

v9:	- rename fence to in_fence in many places
	- handle fences signalling with error better (Hans Verkuil)

v8:	- improve comments and docs (Hans Verkuil)
	- fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
	- move in-fences code that was in the out-fences patch here (Alex)

v8:	- improve comments about fences with errors

v7:
	- get rid of the fence array stuff for ordering and just use
	get_num_buffers_ready() (Hans)
	- fix issue of queuing the buffer twice (Hans)
	- avoid the dma_fence_wait() in core_qbuf() (Alex)
	- merge preparation commit in

v6:
	- With fences always keep the order userspace queues the buffers.
	- Protect in_fence manipulation with a lock (Brian Starkey)
	- check if fences have the same context before adding a fence array
	- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
	- Clean up fence if __set_in_fence() fails (Brian Starkey)
	- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)

v5:	- use fence_array to keep buffers ordered in vb2 core when
	needed (Brian Starkey)
	- keep backward compat on the reserved2 field (Brian Starkey)
	- protect fence callback removal with lock (Brian Starkey)

v4:
	- Add a comment about dma_fence_add_callback() not returning a
	error (Hans)
	- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
	- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
	- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
	- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
	-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
	vb2_start_streaming() (Hans)
	- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
	- Queue buffers to the driver as soon as they are ready (Hans)
	- call fill_user_buffer() after queuing the buffer (Hans)
	- add err: label to clean up fence
	- add dma_fence_wait() before calling vb2_start_streaming()

v3:	- document fence parameter
	- remove ternary if at vb2_qbuf() return (Mauro)
	- do not change if conditions behaviour (Mauro)

v2:
	- fix vb2_queue_or_prepare_buf() ret check
	- remove check for VB2_MEMORY_DMABUF only (Javier)
	- check num of ready buffers to start streaming
	- when queueing, start from the first ready buffer
	- handle queue cancel

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
 drivers/media/v4l2-core/Kconfig                 |  33 ++++
 include/media/videobuf2-core.h                  |  14 +-
 4 files changed, 248 insertions(+), 30 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..5de5e35cfc40 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 		vb->index = q->num_buffers + buffer;
 		vb->type = q->type;
 		vb->memory = memory;
+		spin_lock_init(&vb->fence_cb_lock);
 		for (plane = 0; plane < num_planes; ++plane) {
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
@@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
 }
 EXPORT_SYMBOL_GPL(vb2_plane_cookie);
 
-void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned long flags;
 	unsigned int plane;
 
-	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
-		return;
-
-	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
-		    state != VB2_BUF_STATE_ERROR &&
-		    state != VB2_BUF_STATE_QUEUED &&
-		    state != VB2_BUF_STATE_REQUEUEING))
-		state = VB2_BUF_STATE_ERROR;
 
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
@@ -921,6 +914,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
 
 	spin_lock_irqsave(&q->done_lock, flags);
+	if (vb->state == VB2_BUF_STATE_ACTIVE)
+		atomic_dec(&q->owned_by_drv_count);
+
 	if (state == VB2_BUF_STATE_QUEUED ||
 	    state == VB2_BUF_STATE_REQUEUEING) {
 		vb->state = VB2_BUF_STATE_QUEUED;
@@ -929,7 +925,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		list_add_tail(&vb->done_entry, &q->done_list);
 		vb->state = state;
 	}
-	atomic_dec(&q->owned_by_drv_count);
+
 	spin_unlock_irqrestore(&q->done_lock, flags);
 
 	trace_vb2_buf_done(q, vb);
@@ -946,6 +942,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		wake_up(&q->done_wq);
 		break;
 	}
+
+}
+
+void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
+{
+	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
+		return;
+
+	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
+		    state != VB2_BUF_STATE_ERROR &&
+		    state != VB2_BUF_STATE_QUEUED &&
+		    state != VB2_BUF_STATE_REQUEUEING))
+		state = VB2_BUF_STATE_ERROR;
+
+	vb2_process_buffer_done(vb, state);
+
+	/*
+	 * Check if there is any buffer with error in the next position of the queue,
+	 * buffers whose in-fence signaled with error are not queued to the driver
+	 * and kept on the queue until the buffer before them is done, so to not
+	 * delivery buffers back to userspace in the wrong order. Here we process
+	 * any existing buffers with errors and wake up userspace.
+	 */
+	for (;;) {
+		vb = list_next_entry(vb, queued_entry);
+		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
+			break;
+
+		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
+        }
 }
 EXPORT_SYMBOL_GPL(vb2_buffer_done);
 
@@ -1230,6 +1256,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 
+	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
+		return;
+
 	vb->state = VB2_BUF_STATE_ACTIVE;
 	atomic_inc(&q->owned_by_drv_count);
 
@@ -1281,6 +1310,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
 	return 0;
 }
 
+static int __get_num_ready_buffers(struct vb2_queue *q)
+{
+	struct vb2_buffer *vb;
+	int ready_count = 0;
+	unsigned long flags;
+
+	/* count num of buffers ready in front of the queued_list */
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
+		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
+			break;
+		ready_count++;
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+	}
+
+	return ready_count;
+}
+
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 {
 	struct vb2_buffer *vb;
@@ -1369,9 +1416,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
+{
+	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
+	struct vb2_queue *q = vb->vb2_queue;
+	unsigned long flags;
+
+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
+	/*
+	 * If the fence signals with an error we mark the buffer as such
+	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
+	 * not queueing it to the driver. However we can't notify the error
+	 * to userspace right now because, at the time this callback run, QBUF
+	 * returned already.
+	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
+	 * as well.
+	 */
+	if (vb->in_fence->error)
+		vb->state = VB2_BUF_STATE_ERROR;
+
+	dma_fence_put(vb->in_fence);
+	vb->in_fence = NULL;
+
+	if (vb->state == VB2_BUF_STATE_ERROR) {
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+		return;
+	}
+
+	if (q->start_streaming_called)
+		__enqueue_in_driver(vb);
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+}
+
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence)
 {
 	struct vb2_buffer *vb;
+	unsigned long flags;
 	int ret;
 
 	vb = q->bufs[index];
@@ -1380,16 +1461,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	case VB2_BUF_STATE_DEQUEUED:
 		ret = __buf_prepare(vb, pb);
 		if (ret)
-			return ret;
+			goto err;
 		break;
 	case VB2_BUF_STATE_PREPARED:
 		break;
 	case VB2_BUF_STATE_PREPARING:
 		dprintk(1, "buffer still being prepared\n");
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err;
 	default:
 		dprintk(1, "invalid buffer state %d\n", vb->state);
-		return -EINVAL;
+		ret = -EINVAL;
+		goto err;
 	}
 
 	/*
@@ -1400,6 +1483,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	q->queued_count++;
 	q->waiting_for_buffers = false;
 	vb->state = VB2_BUF_STATE_QUEUED;
+	vb->in_fence = in_fence;
 
 	if (pb)
 		call_void_bufop(q, copy_timestamp, vb, pb);
@@ -1407,15 +1491,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	trace_vb2_qbuf(q, vb);
 
 	/*
-	 * If already streaming, give the buffer to driver for processing.
-	 * If not, the buffer will be given to driver on next streamon.
+	 * For explicit synchronization: If the fence didn't signal
+	 * yet we setup a callback to queue the buffer once the fence
+	 * signals, and then, return successfully. But if the fence
+	 * already signaled we lose the reference we held and queue the
+	 * buffer to the driver.
 	 */
-	if (q->start_streaming_called)
-		__enqueue_in_driver(vb);
+	spin_lock_irqsave(&vb->fence_cb_lock, flags);
+	if (vb->in_fence) {
+		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
+					     vb2_qbuf_fence_cb);
+		/* is the fence signaled? */
+		if (ret == -ENOENT) {
+			dma_fence_put(vb->in_fence);
+			vb->in_fence = NULL;
+		} else if (ret) {
+			goto unlock;
+		}
+	}
 
-	/* Fill buffer information for the userspace */
-	if (pb)
-		call_void_bufop(q, fill_user_buffer, vb, pb);
+	/*
+	 * If already streaming and there is no fence to wait on
+	 * give the buffer to driver for processing.
+	 */
+	if (q->start_streaming_called) {
+		struct vb2_buffer *b;
+
+		list_for_each_entry(b, &q->queued_list, queued_entry) {
+			if (b->state != VB2_BUF_STATE_QUEUED)
+				continue;
+			if (b->in_fence)
+				break;
+			__enqueue_in_driver(b);
+		}
+	}
 
 	/*
 	 * If streamon has been called, and we haven't yet called
@@ -1424,14 +1533,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	 * then we can finally call start_streaming().
 	 */
 	if (q->streaming && !q->start_streaming_called &&
-	    q->queued_count >= q->min_buffers_needed) {
+	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = vb2_start_streaming(q);
 		if (ret)
-			return ret;
+			goto unlock;
 	}
 
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+
+	/* Fill buffer information for the userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
 	return 0;
+
+unlock:
+	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+
+err:
+	/* Fill buffer information for the userspace */
+	if (pb)
+		call_void_bufop(q, fill_user_buffer, vb, pb);
+
+	if (vb->in_fence) {
+		dma_fence_put(vb->in_fence);
+		vb->in_fence = NULL;
+	}
+
+	return ret;
+
 }
 EXPORT_SYMBOL_GPL(vb2_core_qbuf);
 
@@ -1642,6 +1773,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
 static void __vb2_queue_cancel(struct vb2_queue *q)
 {
 	unsigned int i;
+	struct vb2_buffer *vb;
+	unsigned long flags;
 
 	/*
 	 * Tell driver to stop all transactions and release all queued
@@ -1672,6 +1805,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	q->queued_count = 0;
 	q->error = 0;
 
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		spin_lock_irqsave(&vb->fence_cb_lock, flags);
+		if (vb->in_fence) {
+			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
+			dma_fence_put(vb->in_fence);
+			vb->in_fence = NULL;
+		}
+		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
+	}
+
 	/*
 	 * Remove all buffers from videobuf's list...
 	 */
@@ -1742,7 +1885,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
 	 * Tell driver to start streaming provided sufficient buffers
 	 * are available.
 	 */
-	if (q->queued_count >= q->min_buffers_needed) {
+	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
 		ret = v4l_vb2q_enable_media_source(q);
 		if (ret)
 			return ret;
@@ -2264,7 +2407,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2443,7 +2586,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2546,7 +2689,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index ad1e032c3bf5..1df5dd01c0cd 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -23,6 +23,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/v4l2-dev.h>
 #include <media/v4l2-fh.h>
@@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 		return -EINVAL;
 	}
 
+	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
+	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
+		return -EINVAL;
+	}
+
+	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
+		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
+		return -EINVAL;
+	}
+
 	return __verify_planes_array(q->bufs[b->index], b);
 }
 
@@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->fence_fd = 0;
 	b->reserved = 0;
 
+	b->fence_fd = 0;
+	if (vb->in_fence)
+		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
+	else
+		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
+
 	if (q->is_multiplanar) {
 		/*
 		 * Fill in plane-related data if userspace provided an array
@@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	struct dma_fence *in_fence = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	}
 
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	if (ret)
+		return ret;
+
+	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
+		in_fence = sync_file_get_fence(b->fence_fd);
+		if (!in_fence) {
+			dprintk(1, "failed to get in-fence from fd %d\n",
+				b->fence_fd);
+			return -EINVAL;
+		}
+	}
+
+	return vb2_core_qbuf(q, b->index, b, in_fence);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
index 8e37e7c5e0f7..79dfb5dfd1fc 100644
--- a/drivers/media/v4l2-core/Kconfig
+++ b/drivers/media/v4l2-core/Kconfig
@@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
 config VIDEOBUF_DVB
 	tristate
 	select VIDEOBUF_GEN
+
+# Used by drivers that need Videobuf2 modules
+config VIDEOBUF2_CORE
+	select DMA_SHARED_BUFFER
+	select SYNC_FILE
+	tristate
+
+config VIDEOBUF2_MEMOPS
+	tristate
+	select FRAME_VECTOR
+
+config VIDEOBUF2_DMA_CONTIG
+	tristate
+	depends on HAS_DMA
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+	select DMA_SHARED_BUFFER
+
+config VIDEOBUF2_VMALLOC
+	tristate
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+	select DMA_SHARED_BUFFER
+
+config VIDEOBUF2_DMA_SG
+	tristate
+	depends on HAS_DMA
+	select VIDEOBUF2_CORE
+	select VIDEOBUF2_MEMOPS
+
+config VIDEOBUF2_DVB
+	tristate
+	select VIDEOBUF2_CORE
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 46a9e674f7e1..59cd9a6ac168 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -17,6 +17,7 @@
 #include <linux/poll.h>
 #include <linux/dma-buf.h>
 #include <linux/bitops.h>
+#include <linux/dma-fence.h>
 
 #define VB2_MAX_FRAME	(32)
 #define VB2_MAX_PLANES	(8)
@@ -255,12 +256,21 @@ struct vb2_buffer {
 	 * done_entry:		entry on the list that stores all buffers ready
 	 *			to be dequeued to userspace
 	 * vb2_plane:		per-plane information; do not change
+	 * in_fence:		fence received from vb2 client to wait on before
+	 *			using the buffer (queueing to the driver)
+	 * fence_cb:		fence callback information
+	 * fence_cb_lock:	protect callback signal/remove
 	 */
 	enum vb2_buffer_state	state;
 
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 	struct list_head	queued_entry;
 	struct list_head	done_entry;
+
+	struct dma_fence	*in_fence;
+	struct dma_fence_cb	fence_cb;
+	spinlock_t              fence_cb_lock;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -751,6 +761,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @index:	id number of the buffer
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
+ * @in_fence:	in-fence to wait on before queueing the buffer
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
@@ -765,7 +776,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * Return: returns zero on success; an error code otherwise.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct dma_fence *in_fence);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.14.3

[PATCH v8 10/13] [media] vb2: add out-fence support to QBUF

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
an out_fence and send its fd to userspace on the fence_fd field as a
return arg for the QBUF call.

The fence is signaled on buffer_done(), when the job on the buffer is
finished.

v9:	- remove in-fences changes from this patch (Alex Courbot)
	- improve fence context creation (Hans Verkuil)
	- clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)

v8:
	- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
	- fix crash when checking not using fences in vb2_buffer_done()

v7:
	- merge patch that add the infrastructure to out-fences into
	this one (Alex Courbot)
	- Do not install the fd if there is no fence. (Alex Courbot)
	- do not report error on requeueing, just WARN_ON_ONCE() (Hans)

v6
	- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
	ordering in vb2 for queueing in the driver, so the event is not
	necessary anymore and the out_fence_fd is sent back to userspace
	on QBUF call return arg
	- do not allow requeueing with out-fences, instead mark the buffer
	with an error and wake up to userspace.
	- send the out_fence_fd back to userspace on the fence_fd field

v5:
	- delay fd_install to DQ_EVENT (Hans)
	- if queue is fully ordered send OUT_FENCE event right away
	(Brian)
	- rename 'q->ordered' to 'q->ordered_in_driver'
	- merge change to implement OUT_FENCE event here

v4:
	- return the out_fence_fd in the BUF_QUEUED event(Hans)

v3:	- add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
	- set the OUT_FENCE flag if there is a fence pending (Hans)
	- call fd_install() after vb2_core_qbuf() (Hans)
	- clean up fence if vb2_core_qbuf() fails (Hans)
	- add list to store sync_file and fence for the next queued buffer

v2: check if the queue is ordered.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 88 +++++++++++++++++++++++++
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 20 +++++-
 include/media/videobuf2-core.h                  | 25 +++++++
 3 files changed, 132 insertions(+), 1 deletion(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 5de5e35cfc40..dd18a9f345c7 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -25,6 +25,7 @@
 #include <linux/sched.h>
 #include <linux/freezer.h>
 #include <linux/kthread.h>
+#include <linux/sync_file.h>
 
 #include <media/videobuf2-core.h>
 #include <media/v4l2-mc.h>
@@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
 			vb->planes[plane].length = plane_sizes[plane];
 			vb->planes[plane].min_length = plane_sizes[plane];
 		}
+		vb->out_fence_fd = -1;
 		q->bufs[vb->index] = vb;
 
 		/* Allocate video buffer memory for the MMAP type */
@@ -934,10 +936,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
 	case VB2_BUF_STATE_QUEUED:
 		return;
 	case VB2_BUF_STATE_REQUEUEING:
+		/* Requeuing with explicit synchronization, spit warning */
+		WARN_ON_ONCE(vb->out_fence);
+
 		if (q->start_streaming_called)
 			__enqueue_in_driver(vb);
 		return;
 	default:
+		if (vb->out_fence) {
+			if (state == VB2_BUF_STATE_ERROR)
+				dma_fence_set_error(vb->out_fence, -EFAULT);
+			dma_fence_signal(vb->out_fence);
+			dma_fence_put(vb->out_fence);
+			vb->out_fence = NULL;
+			vb->out_fence_fd = -1;
+		}
+
 		/* Inform any processes that may be waiting for buffers */
 		wake_up(&q->done_wq);
 		break;
@@ -1353,6 +1367,62 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
+{
+	return "vb2_fence";
+}
+
+static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
+{
+	return "vb2_fence_timeline";
+}
+
+static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
+{
+	return true;
+}
+
+static const struct dma_fence_ops vb2_fence_ops = {
+	.get_driver_name = vb2_fence_get_driver_name,
+	.get_timeline_name = vb2_fence_get_timeline_name,
+	.enable_signaling = vb2_fence_enable_signaling,
+	.wait = dma_fence_default_wait,
+};
+
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
+{
+	struct vb2_buffer *vb;
+
+	vb = q->bufs[index];
+
+	vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
+
+	if (call_qop(q, is_unordered, q) || !q->queueing_started)
+		q->out_fence_context = dma_fence_context_alloc(1);
+
+	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
+	if (!vb->out_fence)
+		return -ENOMEM;
+
+	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
+		       q->out_fence_context, 1);
+	if (!vb->out_fence) {
+		put_unused_fd(vb->out_fence_fd);
+		return -ENOMEM;
+	}
+
+	vb->sync_file = sync_file_create(vb->out_fence);
+	if (!vb->sync_file) {
+		put_unused_fd(vb->out_fence_fd);
+		dma_fence_put(vb->out_fence);
+		vb->out_fence = NULL;
+		return -ENOMEM;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vb2_setup_out_fence);
+
 /*
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1482,6 +1552,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 	list_add_tail(&vb->queued_entry, &q->queued_list);
 	q->queued_count++;
 	q->waiting_for_buffers = false;
+	q->queueing_started = 1;
 	vb->state = VB2_BUF_STATE_QUEUED;
 	vb->in_fence = in_fence;
 
@@ -1545,6 +1616,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 	if (pb)
 		call_void_bufop(q, fill_user_buffer, vb, pb);
 
+	if (vb->out_fence) {
+		fd_install(vb->out_fence_fd, vb->sync_file->file);
+		vb->sync_file = NULL;
+	}
+
 	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
 	return 0;
 
@@ -1552,6 +1628,16 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
 	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
 
 err:
+	if (vb->sync_file) {
+		put_unused_fd(vb->out_fence_fd);
+		vb->out_fence_fd = -1;
+
+		dma_fence_put(vb->out_fence);
+
+		fput(vb->sync_file->file);
+		vb->sync_file = NULL;
+	}
+
 	/* Fill buffer information for the userspace */
 	if (pb)
 		call_void_bufop(q, fill_user_buffer, vb, pb);
@@ -1804,6 +1890,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	q->start_streaming_called = 0;
 	q->queued_count = 0;
 	q->error = 0;
+	q->queueing_started = 0;
 
 	list_for_each_entry(vb, &q->queued_list, queued_entry) {
 		spin_lock_irqsave(&vb->fence_cb_lock, flags);
@@ -2156,6 +2243,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
 	spin_lock_init(&q->done_lock);
 	mutex_init(&q->mmap_lock);
 	init_waitqueue_head(&q->done_wq);
+	spin_lock_init(&q->out_fence_lock);
 
 	q->memory = VB2_MEMORY_UNKNOWN;
 
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 1df5dd01c0cd..ab5b2b71d784 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -217,7 +217,12 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->sequence = vbuf->sequence;
 	b->reserved = 0;
 
-	b->fence_fd = 0;
+	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
+		b->fence_fd = vb->out_fence_fd;
+	} else {
+		b->fence_fd = 0;
+	}
+
 	if (vb->in_fence)
 		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
 	else
@@ -496,6 +501,10 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	ret = __verify_planes_array(vb, b);
 	if (!ret)
 		vb2_core_querybuf(q, b->index, b);
+
+	/* Do not return the out-fence fd on querybuf */
+	if (vb->out_fence)
+		b->fence_fd = -1;
 	return ret;
 }
 EXPORT_SYMBOL(vb2_querybuf);
@@ -600,6 +609,15 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		}
 	}
 
+	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
+		ret = vb2_setup_out_fence(q, b->index);
+		if (ret) {
+			dprintk(1, "failed to set up out-fence\n");
+			dma_fence_put(in_fence);
+			return ret;
+		}
+	}
+
 	return vb2_core_qbuf(q, b->index, b, in_fence);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 59cd9a6ac168..2b9d3433e960 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -260,6 +260,10 @@ struct vb2_buffer {
 	 *			using the buffer (queueing to the driver)
 	 * fence_cb:		fence callback information
 	 * fence_cb_lock:	protect callback signal/remove
+	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
+	 * out_fence:		the out-fence associated with the buffer once
+	 *			it is queued to the driver.
+	 * sync_file:		the sync file to wrap the out fence
 	 */
 	enum vb2_buffer_state	state;
 
@@ -271,6 +275,10 @@ struct vb2_buffer {
 	struct dma_fence_cb	fence_cb;
 	spinlock_t              fence_cb_lock;
 
+	int			out_fence_fd;
+	struct dma_fence	*out_fence;
+	struct sync_file	*sync_file;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these buffer-related ops are
@@ -515,6 +523,9 @@ struct vb2_buf_ops {
  * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
  *		last decoded buffer was already dequeued. Set for capture queues
  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
+ * @queueing_started: if queueing has started. Currently used to determine
+ *		if an out_fence_context is needed.
+ * @out_fence_context: the fence context for the out fences
  * @fileio:	file io emulator internal data, used only if emulator is active
  * @threadio:	thread io internal data, used only if thread is active
  */
@@ -567,6 +578,10 @@ struct vb2_queue {
 	unsigned int			is_output:1;
 	unsigned int			copy_timestamp:1;
 	unsigned int			last_buffer_dequeued:1;
+	unsigned int			queueing_started:1;
+
+	u64				out_fence_context;
+	spinlock_t			out_fence_lock;
 
 	struct vb2_fileio_data		*fileio;
 	struct vb2_threadio_data	*threadio;
@@ -754,6 +769,16 @@ int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
  */
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
 
+/**
+ * vb2_setup_out_fence() - setup new out-fence
+ * @q:		The vb2_queue where to setup it
+ * @index:	index of the buffer
+ *
+ * Setup the file descriptor, the fence and the sync_file for the next
+ * buffer to be queued and add everything to the tail of the q->out_fence_list.
+ */
+int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index);
+
 /**
  * vb2_core_qbuf() - Queue a buffer from userspace
  *
-- 
2.14.3

[PATCH v8 11/13] [media] v4l: introduce the fences capability

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers capable of using fences (vb2 drivers) should report the
V4L2_CAP_FENCES to userspace, so add this flag to the uapi.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
 include/uapi/linux/videodev2.h                   | 1 +
 2 files changed, 4 insertions(+)

diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst b/Documentation/media/uapi/v4l/vidioc-querycap.rst
index 66fb1b3d6e6e..414016065309 100644
--- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
@@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
     * - ``V4L2_CAP_TOUCH``
       - 0x10000000
       - This is a touch device.
+    * - ``V4L2_CAP_FENCES``
+      - 0x20000000
+      - The device support explicit synchronization.
     * - ``V4L2_CAP_DEVICE_CAPS``
       - 0x80000000
       - The driver fills the ``device_caps`` field. This capability can
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 2d424aebdd1e..db58204e346e 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -460,6 +460,7 @@ struct v4l2_capability {
 #define V4L2_CAP_STREAMING              0x04000000  /* streaming I/O ioctls */
 
 #define V4L2_CAP_TOUCH                  0x10000000  /* Is a touch device */
+#define V4L2_CAP_FENCES                 0x20000000  /* Supports explicit synchronization */
 
 #define V4L2_CAP_DEVICE_CAPS            0x80000000  /* sets device capabilities field */
 
-- 
2.14.3

[PATCH v8 12/13] [media] v4l: Add V4L2_CAP_FENCES to drivers

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Drivers that use videobuf2 are capable of using fences and
should report that to userspace.

The coding style is following what each drivers was already
doing.

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 drivers/media/pci/cobalt/cobalt-v4l2.c             | 3 ++-
 drivers/media/pci/cx23885/cx23885-417.c            | 2 +-
 drivers/media/pci/cx23885/cx23885-video.c          | 3 ++-
 drivers/media/pci/cx88/cx88-video.c                | 3 ++-
 drivers/media/pci/dt3155/dt3155.c                  | 2 +-
 drivers/media/pci/saa7134/saa7134-video.c          | 2 ++
 drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c     | 3 ++-
 drivers/media/pci/solo6x10/solo6x10-v4l2.c         | 3 ++-
 drivers/media/pci/sta2x11/sta2x11_vip.c            | 2 +-
 drivers/media/pci/tw68/tw68-video.c                | 3 ++-
 drivers/media/pci/tw686x/tw686x-video.c            | 2 +-
 drivers/media/platform/am437x/am437x-vpfe.c        | 2 +-
 drivers/media/platform/blackfin/bfin_capture.c     | 3 ++-
 drivers/media/platform/coda/coda-common.c          | 3 ++-
 drivers/media/platform/davinci/vpbe_display.c      | 3 ++-
 drivers/media/platform/davinci/vpfe_capture.c      | 3 ++-
 drivers/media/platform/davinci/vpif_capture.c      | 3 ++-
 drivers/media/platform/davinci/vpif_display.c      | 3 ++-
 drivers/media/platform/exynos-gsc/gsc-m2m.c        | 3 ++-
 drivers/media/platform/exynos4-is/fimc-capture.c   | 3 ++-
 drivers/media/platform/exynos4-is/fimc-isp-video.c | 3 ++-
 drivers/media/platform/exynos4-is/fimc-lite.c      | 2 +-
 drivers/media/platform/exynos4-is/fimc-m2m.c       | 3 ++-
 drivers/media/platform/m2m-deinterlace.c           | 3 ++-
 drivers/media/platform/marvell-ccic/mcam-core.c    | 2 +-
 drivers/media/platform/mx2_emmaprp.c               | 3 ++-
 drivers/media/platform/omap3isp/ispvideo.c         | 2 +-
 drivers/media/platform/pxa_camera.c                | 3 ++-
 drivers/media/platform/rcar_jpu.c                  | 3 ++-
 drivers/media/platform/s3c-camif/camif-capture.c   | 3 ++-
 drivers/media/platform/s5p-g2d/g2d.c               | 3 ++-
 drivers/media/platform/s5p-jpeg/jpeg-core.c        | 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       | 3 ++-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       | 3 ++-
 drivers/media/platform/sh_veu.c                    | 3 ++-
 drivers/media/platform/sh_vou.c                    | 2 +-
 drivers/media/platform/sti/bdisp/bdisp-v4l2.c      | 3 ++-
 drivers/media/platform/ti-vpe/cal.c                | 2 +-
 drivers/media/platform/ti-vpe/vpe.c                | 3 ++-
 drivers/media/platform/vim2m.c                     | 3 ++-
 drivers/media/platform/vivid/vivid-core.c          | 2 +-
 drivers/media/platform/vsp1/vsp1_histo.c           | 2 +-
 drivers/media/platform/vsp1/vsp1_video.c           | 2 +-
 drivers/media/platform/xilinx/xilinx-dma.c         | 2 +-
 drivers/media/usb/airspy/airspy.c                  | 2 +-
 drivers/media/usb/au0828/au0828-video.c            | 3 ++-
 drivers/media/usb/em28xx/em28xx-video.c            | 1 +
 drivers/media/usb/go7007/go7007-v4l2.c             | 2 +-
 drivers/media/usb/hackrf/hackrf.c                  | 3 ++-
 drivers/media/usb/msi2500/msi2500.c                | 2 +-
 drivers/media/usb/pwc/pwc-v4l.c                    | 2 +-
 drivers/media/usb/s2255/s2255drv.c                 | 2 +-
 drivers/media/usb/stk1160/stk1160-v4l.c            | 3 ++-
 drivers/media/usb/usbtv/usbtv-video.c              | 3 ++-
 drivers/media/usb/uvc/uvc_driver.c                 | 1 +
 55 files changed, 89 insertions(+), 52 deletions(-)

diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
index 6b6611a0e190..ef1014b5d4a7 100644
--- a/drivers/media/pci/cobalt/cobalt-v4l2.c
+++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
@@ -484,7 +484,8 @@ static int cobalt_querycap(struct file *file, void *priv_fh,
 	strlcpy(vcap->card, "cobalt", sizeof(vcap->card));
 	snprintf(vcap->bus_info, sizeof(vcap->bus_info),
 		 "PCIe:%s", pci_name(cobalt->pci_dev));
-	vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+	vcap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
+		V4L2_CAP_FENCES;
 	if (s->is_output)
 		vcap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
 	else
diff --git a/drivers/media/pci/cx23885/cx23885-417.c b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..56bf7ec4e25f 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1334,7 +1334,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
 		sizeof(cap->card));
 	sprintf(cap->bus_info, "PCIe:%s", pci_name(dev->pci));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE |
-			   V4L2_CAP_STREAMING;
+			   V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	if (dev->tuner_type != TUNER_ABSENT)
 		cap->device_caps |= V4L2_CAP_TUNER;
 	cap->capabilities = cap->device_caps | V4L2_CAP_VBI_CAPTURE |
diff --git a/drivers/media/pci/cx23885/cx23885-video.c b/drivers/media/pci/cx23885/cx23885-video.c
index f8a3deadc77a..dd54e7f7074a 100644
--- a/drivers/media/pci/cx23885/cx23885-video.c
+++ b/drivers/media/pci/cx23885/cx23885-video.c
@@ -643,7 +643,8 @@ static int vidioc_querycap(struct file *file, void  *priv,
 	strlcpy(cap->card, cx23885_boards[dev->board].name,
 		sizeof(cap->card));
 	sprintf(cap->bus_info, "PCIe:%s", pci_name(dev->pci));
-	cap->device_caps = V4L2_CAP_READWRITE | V4L2_CAP_STREAMING | V4L2_CAP_AUDIO;
+	cap->device_caps = V4L2_CAP_READWRITE | V4L2_CAP_STREAMING |
+		V4L2_CAP_AUDIO | V4L2_CAP_FENCES;
 	if (dev->tuner_type != TUNER_ABSENT)
 		cap->device_caps |= V4L2_CAP_TUNER;
 	if (vdev->vfl_type == VFL_TYPE_VBI)
diff --git a/drivers/media/pci/cx88/cx88-video.c b/drivers/media/pci/cx88/cx88-video.c
index 9be682cdb644..acc74f402a43 100644
--- a/drivers/media/pci/cx88/cx88-video.c
+++ b/drivers/media/pci/cx88/cx88-video.c
@@ -812,7 +812,8 @@ int cx88_querycap(struct file *file, struct cx88_core *core,
 	struct video_device *vdev = video_devdata(file);
 
 	strlcpy(cap->card, core->board.name, sizeof(cap->card));
-	cap->device_caps = V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_READWRITE | V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 	if (core->board.tuner_type != UNSET)
 		cap->device_caps |= V4L2_CAP_TUNER;
 	switch (vdev->vfl_type) {
diff --git a/drivers/media/pci/dt3155/dt3155.c b/drivers/media/pci/dt3155/dt3155.c
index 1775c36891ae..b677096c7a14 100644
--- a/drivers/media/pci/dt3155/dt3155.c
+++ b/drivers/media/pci/dt3155/dt3155.c
@@ -311,7 +311,7 @@ static int dt3155_querycap(struct file *filp, void *p,
 	strcpy(cap->card, DT3155_NAME " frame grabber");
 	sprintf(cap->bus_info, "PCI:%s", pci_name(pd->pdev));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE |
-		V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+		V4L2_CAP_STREAMING | V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c
index 1ca6a32ad10e..aa0bc52633c3 100644
--- a/drivers/media/pci/saa7134/saa7134-video.c
+++ b/drivers/media/pci/saa7134/saa7134-video.c
@@ -1534,6 +1534,8 @@ int saa7134_querycap(struct file *file, void *priv,
 	default:
 		return -EINVAL;
 	}
+
+	cap->device_caps |= V4L2_CAP_FENCES;
 	cap->capabilities = radio_caps | video_caps | vbi_caps |
 		cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	if (vdev->vfl_type == VFL_TYPE_RADIO) {
diff --git a/drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c b/drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c
index 25f9f2ebff1d..2900cde5d4a0 100644
--- a/drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c
+++ b/drivers/media/pci/solo6x10/solo6x10-v4l2-enc.c
@@ -781,7 +781,8 @@ static int solo_enc_querycap(struct file *file, void  *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s",
 		 pci_name(solo_dev->pdev));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE |
-			V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+			V4L2_CAP_READWRITE | V4L2_CAP_STREAMING |
+			V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/pci/solo6x10/solo6x10-v4l2.c b/drivers/media/pci/solo6x10/solo6x10-v4l2.c
index 99ffd1ed4a73..acb86823313e 100644
--- a/drivers/media/pci/solo6x10/solo6x10-v4l2.c
+++ b/drivers/media/pci/solo6x10/solo6x10-v4l2.c
@@ -388,7 +388,8 @@ static int solo_querycap(struct file *file, void  *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s",
 		 pci_name(solo_dev->pdev));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE |
-			V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+			V4L2_CAP_READWRITE | V4L2_CAP_STREAMING |
+			V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/pci/sta2x11/sta2x11_vip.c b/drivers/media/pci/sta2x11/sta2x11_vip.c
index dd199bfc1d45..4113d1279d72 100644
--- a/drivers/media/pci/sta2x11/sta2x11_vip.c
+++ b/drivers/media/pci/sta2x11/sta2x11_vip.c
@@ -420,7 +420,7 @@ static int vidioc_querycap(struct file *file, void *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "PCI:%s",
 		 pci_name(vip->pdev));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE |
-			   V4L2_CAP_STREAMING;
+			   V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/pci/tw68/tw68-video.c b/drivers/media/pci/tw68/tw68-video.c
index 8c1f4a049764..6a11838a96ce 100644
--- a/drivers/media/pci/tw68/tw68-video.c
+++ b/drivers/media/pci/tw68/tw68-video.c
@@ -741,7 +741,8 @@ static int tw68_querycap(struct file *file, void  *priv,
 	cap->device_caps =
 		V4L2_CAP_VIDEO_CAPTURE |
 		V4L2_CAP_READWRITE |
-		V4L2_CAP_STREAMING;
+		V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
diff --git a/drivers/media/pci/tw686x/tw686x-video.c b/drivers/media/pci/tw686x/tw686x-video.c
index c3fafa97b2d0..2a527b6a83e3 100644
--- a/drivers/media/pci/tw686x/tw686x-video.c
+++ b/drivers/media/pci/tw686x/tw686x-video.c
@@ -770,7 +770,7 @@ static int tw686x_querycap(struct file *file, void *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info),
 		 "PCI:%s", pci_name(dev->pci_dev));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
-			   V4L2_CAP_READWRITE;
+			   V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/am437x/am437x-vpfe.c b/drivers/media/platform/am437x/am437x-vpfe.c
index 601ae6487617..b376c60ade04 100644
--- a/drivers/media/platform/am437x/am437x-vpfe.c
+++ b/drivers/media/platform/am437x/am437x-vpfe.c
@@ -1413,7 +1413,7 @@ static int vpfe_querycap(struct file *file, void  *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info),
 			"platform:%s", vpfe->v4l2_dev.name);
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
-			    V4L2_CAP_READWRITE;
+			    V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/blackfin/bfin_capture.c b/drivers/media/platform/blackfin/bfin_capture.c
index b7660b1000fd..cfbca15db6ac 100644
--- a/drivers/media/platform/blackfin/bfin_capture.c
+++ b/drivers/media/platform/blackfin/bfin_capture.c
@@ -704,7 +704,8 @@ static int bcap_querycap(struct file *file, void  *priv,
 {
 	struct bcap_device *bcap_dev = video_drvdata(file);
 
-	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	strlcpy(cap->driver, CAPTURE_DRV_NAME, sizeof(cap->driver));
 	strlcpy(cap->bus_info, "Blackfin Platform", sizeof(cap->bus_info));
diff --git a/drivers/media/platform/coda/coda-common.c b/drivers/media/platform/coda/coda-common.c
index 6deb29fe6eb7..0b7ec013ff32 100644
--- a/drivers/media/platform/coda/coda-common.c
+++ b/drivers/media/platform/coda/coda-common.c
@@ -394,7 +394,8 @@ static int coda_querycap(struct file *file, void *priv,
 	strlcpy(cap->card, coda_product_name(ctx->dev->devtype->product),
 		sizeof(cap->card));
 	strlcpy(cap->bus_info, "platform:" CODA_NAME, sizeof(cap->bus_info));
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/davinci/vpbe_display.c b/drivers/media/platform/davinci/vpbe_display.c
index 6aabd21fe69f..7ec13e50b4bb 100644
--- a/drivers/media/platform/davinci/vpbe_display.c
+++ b/drivers/media/platform/davinci/vpbe_display.c
@@ -638,7 +638,8 @@ static int vpbe_display_querycap(struct file *file, void  *priv,
 	struct vpbe_layer *layer = video_drvdata(file);
 	struct vpbe_device *vpbe_dev = layer->disp_dev->vpbe_dev;
 
-	cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	snprintf(cap->driver, sizeof(cap->driver), "%s",
 		dev_name(vpbe_dev->pdev));
diff --git a/drivers/media/platform/davinci/vpfe_capture.c b/drivers/media/platform/davinci/vpfe_capture.c
index 6f44abf7fa31..93defad26f63 100644
--- a/drivers/media/platform/davinci/vpfe_capture.c
+++ b/drivers/media/platform/davinci/vpfe_capture.c
@@ -887,7 +887,8 @@ static int vpfe_querycap(struct file *file, void  *priv,
 
 	v4l2_dbg(1, debug, &vpfe_dev->v4l2_dev, "vpfe_querycap\n");
 
-	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	strlcpy(cap->driver, CAPTURE_DRV_NAME, sizeof(cap->driver));
 	strlcpy(cap->bus_info, "VPFE", sizeof(cap->bus_info));
diff --git a/drivers/media/platform/davinci/vpif_capture.c b/drivers/media/platform/davinci/vpif_capture.c
index 9364cdf62f54..ca13c8894078 100644
--- a/drivers/media/platform/davinci/vpif_capture.c
+++ b/drivers/media/platform/davinci/vpif_capture.c
@@ -1092,7 +1092,8 @@ static int vpif_querycap(struct file *file, void  *priv,
 {
 	struct vpif_capture_config *config = vpif_dev->platform_data;
 
-	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	strlcpy(cap->driver, VPIF_DRIVER_NAME, sizeof(cap->driver));
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
diff --git a/drivers/media/platform/davinci/vpif_display.c b/drivers/media/platform/davinci/vpif_display.c
index 7be636237acf..2c39b72d466f 100644
--- a/drivers/media/platform/davinci/vpif_display.c
+++ b/drivers/media/platform/davinci/vpif_display.c
@@ -584,7 +584,8 @@ static int vpif_querycap(struct file *file, void  *priv,
 {
 	struct vpif_display_config *config = vpif_dev->platform_data;
 
-	cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	strlcpy(cap->driver, VPIF_DRIVER_NAME, sizeof(cap->driver));
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c b/drivers/media/platform/exynos-gsc/gsc-m2m.c
index 10c3e4659d38..a6465a22f5a8 100644
--- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
+++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
@@ -299,7 +299,8 @@ static int gsc_m2m_querycap(struct file *file, void *fh,
 	strlcpy(cap->card, GSC_MODULE_NAME " gscaler", sizeof(cap->card));
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 		 dev_name(&gsc->pdev->dev));
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index ed9302caa004..d14e22f3c910 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -726,7 +726,8 @@ static int fimc_cap_querycap(struct file *file, void *priv,
 	struct fimc_dev *fimc = video_drvdata(file);
 
 	__fimc_vidioc_querycap(&fimc->pdev->dev, cap, V4L2_CAP_STREAMING |
-					V4L2_CAP_VIDEO_CAPTURE_MPLANE);
+					V4L2_CAP_VIDEO_CAPTURE_MPLANE |
+					V4L2_CAP_FENCES);
 	return 0;
 }
 
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 55ba696b8cf4..d3c8d8a8428c 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -349,7 +349,8 @@ static int isp_video_querycap(struct file *file, void *priv,
 {
 	struct fimc_isp *isp = video_drvdata(file);
 
-	__fimc_vidioc_querycap(&isp->pdev->dev, cap, V4L2_CAP_STREAMING);
+	__fimc_vidioc_querycap(&isp->pdev->dev, cap, V4L2_CAP_STREAMING |
+			       V4L2_CAP_FENCES);
 	return 0;
 }
 
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index 70d5f5586a5d..5519854ef728 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -659,7 +659,7 @@ static int fimc_lite_querycap(struct file *file, void *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 					dev_name(&fimc->pdev->dev));
 
-	cap->device_caps = V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c b/drivers/media/platform/exynos4-is/fimc-m2m.c
index dfc487a582c0..fb64abdc80a0 100644
--- a/drivers/media/platform/exynos4-is/fimc-m2m.c
+++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
@@ -237,7 +237,8 @@ static int fimc_m2m_querycap(struct file *file, void *fh,
 				     struct v4l2_capability *cap)
 {
 	struct fimc_dev *fimc = video_drvdata(file);
-	unsigned int caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE;
+	unsigned int caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE |
+			    V4L2_CAP_FENCES;
 
 	__fimc_vidioc_querycap(&fimc->pdev->dev, cap, caps);
 	return 0;
diff --git a/drivers/media/platform/m2m-deinterlace.c b/drivers/media/platform/m2m-deinterlace.c
index 35a0f45d2a51..2d5a096adc8b 100644
--- a/drivers/media/platform/m2m-deinterlace.c
+++ b/drivers/media/platform/m2m-deinterlace.c
@@ -461,7 +461,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	 * and are scheduled for removal.
 	 */
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT |
-			   V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+			   V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/marvell-ccic/mcam-core.c b/drivers/media/platform/marvell-ccic/mcam-core.c
index 80670eeee142..4ac0edbfc194 100644
--- a/drivers/media/platform/marvell-ccic/mcam-core.c
+++ b/drivers/media/platform/marvell-ccic/mcam-core.c
@@ -1307,7 +1307,7 @@ static int mcam_vidioc_querycap(struct file *file, void *priv,
 	strcpy(cap->card, "marvell_ccic");
 	strlcpy(cap->bus_info, cam->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE |
-		V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+		V4L2_CAP_READWRITE | V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c
index d03becff66cf..b25d492e8d77 100644
--- a/drivers/media/platform/mx2_emmaprp.c
+++ b/drivers/media/platform/mx2_emmaprp.c
@@ -401,7 +401,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 {
 	strncpy(cap->driver, MEM2MEM_NAME, sizeof(cap->driver) - 1);
 	strncpy(cap->card, MEM2MEM_NAME, sizeof(cap->card) - 1);
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index b4d4ef926749..cefcd1195a9d 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -658,7 +658,7 @@ isp_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	strlcpy(cap->card, video->video.name, sizeof(cap->card));
 	strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
 
-	cap->device_caps = V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
 		V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;
 
diff --git a/drivers/media/platform/pxa_camera.c b/drivers/media/platform/pxa_camera.c
index c71a00736541..402649257f2b 100644
--- a/drivers/media/platform/pxa_camera.c
+++ b/drivers/media/platform/pxa_camera.c
@@ -1997,7 +1997,8 @@ static int pxac_vidioc_querycap(struct file *file, void *priv,
 	strlcpy(cap->bus_info, "platform:pxa-camera", sizeof(cap->bus_info));
 	strlcpy(cap->driver, PXA_CAM_DRV_NAME, sizeof(cap->driver));
 	strlcpy(cap->card, pxa_cam_driver_description, sizeof(cap->card));
-	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c
index b4b2e2cf5d1a..29d8e30edc8f 100644
--- a/drivers/media/platform/rcar_jpu.c
+++ b/drivers/media/platform/rcar_jpu.c
@@ -676,7 +676,8 @@ static int jpu_querycap(struct file *file, void *priv,
 	strlcpy(cap->driver, DRV_NAME, sizeof(cap->driver));
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 		 dev_name(ctx->jpu->dev));
-	cap->device_caps |= V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE;
+	cap->device_caps |= V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M_MPLANE |
+		V4L2_CAP_FENCES;
 	cap->capabilities = V4L2_CAP_DEVICE_CAPS | cap->device_caps;
 	memset(cap->reserved, 0, sizeof(cap->reserved));
 
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 9ab8e7ee2e1e..1cec90e5e159 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -643,7 +643,8 @@ static int s3c_camif_vidioc_querycap(struct file *file, void *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s.%d",
 		 dev_name(vp->camif->dev), vp->id);
 
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_CAPTURE;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_CAPTURE |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/platform/s5p-g2d/g2d.c b/drivers/media/platform/s5p-g2d/g2d.c
index cb7d916bfc8b..225a765fda7c 100644
--- a/drivers/media/platform/s5p-g2d/g2d.c
+++ b/drivers/media/platform/s5p-g2d/g2d.c
@@ -299,7 +299,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	strncpy(cap->driver, G2D_NAME, sizeof(cap->driver) - 1);
 	strncpy(cap->card, G2D_NAME, sizeof(cap->card) - 1);
 	cap->bus_info[0] = 0;
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c
index 28485e6b9cc8..bcd43b33eb3e 100644
--- a/drivers/media/platform/s5p-jpeg/jpeg-core.c
+++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c
@@ -1288,7 +1288,8 @@ static int s5p_jpeg_querycap(struct file *file, void *priv,
 	}
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 		 dev_name(ctx->jpeg->dev));
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M |
+			   V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
index 369db08dbcae..ba2f79c928bc 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
@@ -274,7 +274,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	 * device capability flags are left only for backward compatibility
 	 * and are scheduled for removal.
 	 */
-	cap->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
index fece496c2a8e..b00415ae644b 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
@@ -952,7 +952,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	 * device capability flags are left only for backward compatibility
 	 * and are scheduled for removal.
 	 */
-	cap->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c
index 0682b50a67fc..fc0695292393 100644
--- a/drivers/media/platform/sh_veu.c
+++ b/drivers/media/platform/sh_veu.c
@@ -352,7 +352,8 @@ static int sh_veu_querycap(struct file *file, void *priv,
 	strlcpy(cap->card, "sh-mobile VEU", sizeof(cap->card));
 	strlcpy(cap->bus_info, "platform:sh-veu", sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
-	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
+	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS |
+			    V4L2_CAP_FENCES;
 
 	return 0;
 }
diff --git a/drivers/media/platform/sh_vou.c b/drivers/media/platform/sh_vou.c
index 4dccf29e9d78..8affb8703825 100644
--- a/drivers/media/platform/sh_vou.c
+++ b/drivers/media/platform/sh_vou.c
@@ -385,7 +385,7 @@ static int sh_vou_querycap(struct file *file, void  *priv,
 	strlcpy(cap->driver, "sh-vou", sizeof(cap->driver));
 	strlcpy(cap->bus_info, "platform:sh-vou", sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_READWRITE |
-			   V4L2_CAP_STREAMING;
+			   V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
index 0cfdc5a67855..f3745b74724c 100644
--- a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
+++ b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
@@ -694,7 +694,8 @@ static int bdisp_querycap(struct file *file, void *fh,
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s%d",
 		 BDISP_NAME, bdisp->id);
 
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_VIDEO_M2M |
+		V4L2_CAP_FENCES;
 
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
diff --git a/drivers/media/platform/ti-vpe/cal.c b/drivers/media/platform/ti-vpe/cal.c
index d1febe5baa6d..83f59b6408c0 100644
--- a/drivers/media/platform/ti-vpe/cal.c
+++ b/drivers/media/platform/ti-vpe/cal.c
@@ -918,7 +918,7 @@ static int cal_querycap(struct file *file, void *priv,
 	snprintf(cap->bus_info, sizeof(cap->bus_info),
 		 "platform:%s", ctx->v4l2_dev.name);
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
-			    V4L2_CAP_READWRITE;
+			    V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/ti-vpe/vpe.c b/drivers/media/platform/ti-vpe/vpe.c
index c2d838884e1c..75840b21fff1 100644
--- a/drivers/media/platform/ti-vpe/vpe.c
+++ b/drivers/media/platform/ti-vpe/vpe.c
@@ -1512,7 +1512,8 @@ static int vpe_querycap(struct file *file, void *priv,
 	strncpy(cap->card, VPE_MODULE_NAME, sizeof(cap->card) - 1);
 	snprintf(cap->bus_info, sizeof(cap->bus_info), "platform:%s",
 		VPE_MODULE_NAME);
-	cap->device_caps  = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING;
+	cap->device_caps  = V4L2_CAP_VIDEO_M2M_MPLANE | V4L2_CAP_STREAMING |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index e1a54a28b082..13711d203fd6 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -432,7 +432,8 @@ static int vidioc_querycap(struct file *file, void *priv,
 	strncpy(cap->card, MEM2MEM_NAME, sizeof(cap->card) - 1);
 	snprintf(cap->bus_info, sizeof(cap->bus_info),
 			"platform:%s", MEM2MEM_NAME);
-	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_VIDEO_M2M | V4L2_CAP_STREAMING |
+			V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/platform/vivid/vivid-core.c b/drivers/media/platform/vivid/vivid-core.c
index 82ec216f2ad8..39cd4aedd968 100644
--- a/drivers/media/platform/vivid/vivid-core.c
+++ b/drivers/media/platform/vivid/vivid-core.c
@@ -205,7 +205,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
 	cap->capabilities = dev->vid_cap_caps | dev->vid_out_caps |
 		dev->vbi_cap_caps | dev->vbi_out_caps |
 		dev->radio_rx_caps | dev->radio_tx_caps |
-		dev->sdr_cap_caps | V4L2_CAP_DEVICE_CAPS;
+		dev->sdr_cap_caps | V4L2_CAP_DEVICE_CAPS | V4L2_CAP_FENCES;
 	return 0;
 }
 
diff --git a/drivers/media/platform/vsp1/vsp1_histo.c b/drivers/media/platform/vsp1/vsp1_histo.c
index afab77cf4fa5..647b0c4d6f39 100644
--- a/drivers/media/platform/vsp1/vsp1_histo.c
+++ b/drivers/media/platform/vsp1/vsp1_histo.c
@@ -480,7 +480,7 @@ static int histo_v4l2_querycap(struct file *file, void *fh,
 	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
 			  | V4L2_CAP_VIDEO_CAPTURE_MPLANE
 			  | V4L2_CAP_VIDEO_OUTPUT_MPLANE
-			  | V4L2_CAP_META_CAPTURE;
+			  | V4L2_CAP_META_CAPTURE | V4L2_CAP_FENCES;
 	cap->device_caps = V4L2_CAP_META_CAPTURE
 			 | V4L2_CAP_STREAMING;
 
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index c2d3b8f0f487..67e44a76a4f8 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -964,7 +964,7 @@ vsp1_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 
 	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
 			  | V4L2_CAP_VIDEO_CAPTURE_MPLANE
-			  | V4L2_CAP_VIDEO_OUTPUT_MPLANE;
+			  | V4L2_CAP_VIDEO_OUTPUT_MPLANE | V4L2_CAP_FENCES;
 
 	if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
 		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE_MPLANE
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 565e466ba4fa..5652ca23ec7d 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -494,7 +494,7 @@ xvip_dma_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
 	struct v4l2_fh *vfh = file->private_data;
 	struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
 
-	cap->device_caps = V4L2_CAP_STREAMING;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
 			  | dma->xdev->v4l2_caps;
 
diff --git a/drivers/media/usb/airspy/airspy.c b/drivers/media/usb/airspy/airspy.c
index e70c9e2f3798..64a3192413e3 100644
--- a/drivers/media/usb/airspy/airspy.c
+++ b/drivers/media/usb/airspy/airspy.c
@@ -623,7 +623,7 @@ static int airspy_querycap(struct file *file, void *fh,
 	strlcpy(cap->card, s->vdev.name, sizeof(cap->card));
 	usb_make_path(s->udev, cap->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_STREAMING |
-			V4L2_CAP_READWRITE | V4L2_CAP_TUNER;
+			V4L2_CAP_READWRITE | V4L2_CAP_TUNER | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 
 	return 0;
diff --git a/drivers/media/usb/au0828/au0828-video.c b/drivers/media/usb/au0828/au0828-video.c
index c765d546114d..29db7799ecb6 100644
--- a/drivers/media/usb/au0828/au0828-video.c
+++ b/drivers/media/usb/au0828/au0828-video.c
@@ -1199,7 +1199,8 @@ static int vidioc_querycap(struct file *file, void  *priv,
 	cap->device_caps = V4L2_CAP_AUDIO |
 		V4L2_CAP_READWRITE |
 		V4L2_CAP_STREAMING |
-		V4L2_CAP_TUNER;
+		V4L2_CAP_TUNER |
+		V4L2_CAP_FENCES;
 	if (vdev->vfl_type == VFL_TYPE_GRABBER)
 		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
 	else
diff --git a/drivers/media/usb/em28xx/em28xx-video.c b/drivers/media/usb/em28xx/em28xx-video.c
index f31339727d3b..32a58f1da4e7 100644
--- a/drivers/media/usb/em28xx/em28xx-video.c
+++ b/drivers/media/usb/em28xx/em28xx-video.c
@@ -1938,6 +1938,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
 	if (dev->tuner_type != TUNER_ABSENT)
 		cap->device_caps |= V4L2_CAP_TUNER;
 
+	cap->device_caps |= V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps |
 			    V4L2_CAP_DEVICE_CAPS | V4L2_CAP_READWRITE |
 			    V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
diff --git a/drivers/media/usb/go7007/go7007-v4l2.c b/drivers/media/usb/go7007/go7007-v4l2.c
index 98cd57eaf36a..0ef0587eb136 100644
--- a/drivers/media/usb/go7007/go7007-v4l2.c
+++ b/drivers/media/usb/go7007/go7007-v4l2.c
@@ -289,7 +289,7 @@ static int vidioc_querycap(struct file *file, void  *priv,
 	strlcpy(cap->bus_info, go->bus_info, sizeof(cap->bus_info));
 
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_READWRITE |
-				V4L2_CAP_STREAMING;
+				V4L2_CAP_STREAMING | V4L2_CAP_FENCES;
 
 	if (go->board_info->num_aud_inputs)
 		cap->device_caps |= V4L2_CAP_AUDIO;
diff --git a/drivers/media/usb/hackrf/hackrf.c b/drivers/media/usb/hackrf/hackrf.c
index 6d692fb3e8dd..5912df138394 100644
--- a/drivers/media/usb/hackrf/hackrf.c
+++ b/drivers/media/usb/hackrf/hackrf.c
@@ -909,7 +909,8 @@ static int hackrf_querycap(struct file *file, void *fh,
 
 	dev_dbg(&intf->dev, "\n");
 
-	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE;
+	cap->device_caps = V4L2_CAP_STREAMING | V4L2_CAP_READWRITE |
+			   V4L2_CAP_FENCES;
 	if (vdev->vfl_dir == VFL_DIR_RX)
 		cap->device_caps |= V4L2_CAP_SDR_CAPTURE | V4L2_CAP_TUNER;
 	else
diff --git a/drivers/media/usb/msi2500/msi2500.c b/drivers/media/usb/msi2500/msi2500.c
index 65ef755adfdc..737b10761869 100644
--- a/drivers/media/usb/msi2500/msi2500.c
+++ b/drivers/media/usb/msi2500/msi2500.c
@@ -608,7 +608,7 @@ static int msi2500_querycap(struct file *file, void *fh,
 	strlcpy(cap->card, dev->vdev.name, sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_SDR_CAPTURE | V4L2_CAP_STREAMING |
-			V4L2_CAP_READWRITE | V4L2_CAP_TUNER;
+			V4L2_CAP_READWRITE | V4L2_CAP_TUNER | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/usb/pwc/pwc-v4l.c b/drivers/media/usb/pwc/pwc-v4l.c
index 043b2b97cee6..571a8c4661c5 100644
--- a/drivers/media/usb/pwc/pwc-v4l.c
+++ b/drivers/media/usb/pwc/pwc-v4l.c
@@ -496,7 +496,7 @@ static int pwc_querycap(struct file *file, void *fh, struct v4l2_capability *cap
 	strlcpy(cap->card, pdev->vdev.name, sizeof(cap->card));
 	usb_make_path(pdev->udev, cap->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
-					V4L2_CAP_READWRITE;
+					V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/usb/s2255/s2255drv.c b/drivers/media/usb/s2255/s2255drv.c
index a00a15f55d37..fe4c4f23adaa 100644
--- a/drivers/media/usb/s2255/s2255drv.c
+++ b/drivers/media/usb/s2255/s2255drv.c
@@ -734,7 +734,7 @@ static int vidioc_querycap(struct file *file, void *priv,
 	strlcpy(cap->card, "s2255", sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING |
-		V4L2_CAP_READWRITE;
+		V4L2_CAP_READWRITE | V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/usb/stk1160/stk1160-v4l.c b/drivers/media/usb/stk1160/stk1160-v4l.c
index 77b759a0bcd9..00c8ff034880 100644
--- a/drivers/media/usb/stk1160/stk1160-v4l.c
+++ b/drivers/media/usb/stk1160/stk1160-v4l.c
@@ -350,7 +350,8 @@ static int vidioc_querycap(struct file *file,
 	cap->device_caps =
 		V4L2_CAP_VIDEO_CAPTURE |
 		V4L2_CAP_STREAMING |
-		V4L2_CAP_READWRITE;
+		V4L2_CAP_READWRITE |
+		V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/usb/usbtv/usbtv-video.c b/drivers/media/usb/usbtv/usbtv-video.c
index 3668a04359e8..4b889dbff6a9 100644
--- a/drivers/media/usb/usbtv/usbtv-video.c
+++ b/drivers/media/usb/usbtv/usbtv-video.c
@@ -521,7 +521,8 @@ static int usbtv_querycap(struct file *file, void *priv,
 	strlcpy(cap->card, "usbtv", sizeof(cap->card));
 	usb_make_path(dev->udev, cap->bus_info, sizeof(cap->bus_info));
 	cap->device_caps = V4L2_CAP_VIDEO_CAPTURE;
-	cap->device_caps |= V4L2_CAP_READWRITE | V4L2_CAP_STREAMING;
+	cap->device_caps |= V4L2_CAP_READWRITE | V4L2_CAP_STREAMING |
+			    V4L2_CAP_FENCES;
 	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS;
 	return 0;
 }
diff --git a/drivers/media/usb/uvc/uvc_driver.c b/drivers/media/usb/uvc/uvc_driver.c
index 2469b49b2b30..745f69cbf8e0 100644
--- a/drivers/media/usb/uvc/uvc_driver.c
+++ b/drivers/media/usb/uvc/uvc_driver.c
@@ -1968,6 +1968,7 @@ static int uvc_register_video(struct uvc_device *dev,
 		return ret;
 	}
 
+	stream->chain->caps |= V4L2_CAP_FENCES;
 	if (stream->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
 		stream->chain->caps |= V4L2_CAP_VIDEO_CAPTURE
 			| V4L2_CAP_META_CAPTURE;
-- 
2.14.3

[PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[Gustavo Padovan]

From: Gustavo Padovan <gustavo.padovan@collabora.com>

Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it

v6:	- Close some gaps in the docs (Hans)

v5:
	- Remove V4L2_CAP_ORDERED
	- Add doc about V4L2_FMT_FLAG_UNORDERED

v4:
	- Document ordering behavior for in-fences
	- Document V4L2_CAP_ORDERED capability
	- Remove doc about OUT_FENCE event
	- Document immediate return of out-fence in QBUF

v3:
	- make the out_fence refer to the current buffer (Hans)
	- Note what happens when the IN_FENCE is not set (Hans)

v2:
	- mention that fences are files (Hans)
	- rework for the new API

Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
---
 Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 55 +++++++++++++++++++++++-
 Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
 2 files changed, 63 insertions(+), 4 deletions(-)

diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..371d84966e34 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
 or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
 ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
 for details. Applications must also set ``flags`` to 0. The
-``reserved2`` and ``reserved`` fields must be set to 0. When using the
+``reserved`` field must be set to 0. When using the
 :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
 contain a userspace pointer to a filled-in array of struct
 :c:type:`v4l2_plane` and the ``length`` field must be set
@@ -118,6 +118,59 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
+Explicit Synchronization
+------------------------
+
+Explicit Synchronization allows us to control the synchronization of
+shared buffers from userspace by passing fences to the kernel and/or
+receiving them from it. Fences passed to the kernel are named in-fences and
+the kernel should wait on them to signal before using the buffer. On the other
+side, the kernel can create out-fences for the buffers it queues to the
+drivers. Out-fences signal when the driver is finished with buffer, i.e., the
+buffer is ready. The fences are represented as a file and passed as a file
+descriptor to userspace.
+
+The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
+using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
+an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
+fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
+as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
+with an error. The fence_fd field will be ignored if the
+``V4L2_BUF_FLAG_IN_FENCE`` is not set.
+
+The videobuf2-core will guarantee that all buffers queued with an in-fence will
+be queued to the drivers in the same order. Fences may signal out of order, so
+this guarantee at videobuf2 is necessary to not change ordering. So when
+waiting on a fence to signal all buffers queued after will be also block until
+that fence signal.
+
+If the in-fence signals with an error the buffer will be marked with
+``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
+Even with the error the order of dequeueing the buffers are preserved.
+
+To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
+be set to ask for a fence to be attached to the buffer. The out-fence fd is
+sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.
+
+Note the the same `fence_fd` field is used for both sending the in-fence as
+input argument to receive the out-fence as a return argument. A buffer can
+have both in-fence ond out-fence.
+
+At streamoff the out-fences will either signal normally if the driver waits
+for the operations on the buffers to finish or signal with an error if the
+driver cancels the pending operations. Buffers with in-fences won't be queued
+to the driver if their fences signal. They will be cleaned up.
+
+The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
+that the  when using this format the order in which buffers are dequeued can
+be different from the order in which they were queued.
+
+Ordering is important to fences because it can optimize the pipeline with
+other drivers like a DRM/KMS display driver. For example, if a capture from the
+camera is happening in an orderly manner one can send the capture buffer
+out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on
+the screen at the correct order. If an ordered queue can not be set then such
+arrangements with other drivers may not be possible.
 
 Return Value
 ============
diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
index dd54747fabc9..9aa3358bee0b 100644
--- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
@@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero to the
 number of buffers allocated with
 :ref:`VIDIOC_REQBUFS` (struct
 :c:type:`v4l2_requestbuffers` ``count``) minus
-one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
+one. The ``reserved`` field must be set to 0. When
 using the :ref:`multi-planar API <planar-apis>`, the ``m.planes``
 field must contain a userspace pointer to an array of struct
 :c:type:`v4l2_plane` and the ``length`` field has to be set
@@ -64,8 +64,14 @@ elements will be used instead and the ``length`` field of struct
 array elements. The driver may or may not set the remaining fields and
 flags, they are meaningless in this context.
 
-The struct :c:type:`v4l2_buffer` structure is specified in
-:ref:`buffer`.
+When using in-fences, the ``V4L2_BUF_FLAG_IN_FENCE`` will be set if the
+in-fence didn't signal at the time of the
+:ref:`VIDIOC_QUERYBUF`. The ``V4L2_BUF_FLAG_OUT_FENCE`` will be set if
+the user asked for an out-fence for the buffer and the ``fence_fd``
+field will be set to the out-fence fd. In case ``V4L2_BUF_FLAG_OUT_FENCE`` is
+not set ``fence_fd`` will be set to 0 for backward compatibility.
+
+The struct :c:type:`v4l2_buffer` structure is specified in :ref:`buffer`.
 
 
 Return Value
-- 
2.14.3

Re: [PATCH v8 07/13] [media] vb2: mark codec drivers as unordered

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 14532 bytes --]

Le vendredi 09 mars 2018 à 14:49 -0300, Gustavo Padovan a écrit :
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> In preparation to have full support to explicit fence we are
> marking codec as non-ordered preventively. It is easier and safer from an

The usage of "codec" is soso ....

> uAPI point of view to move from unordered to ordered than the opposite.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/platform/coda/coda-common.c          | 1 +
>  drivers/media/platform/exynos-gsc/gsc-m2m.c        | 1 +
>  drivers/media/platform/exynos4-is/fimc-m2m.c       | 1 +
>  drivers/media/platform/m2m-deinterlace.c           | 1 +

... these tree are not codecs. Did you just set all M2M drivers are
unordered ?

>  drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c    | 1 +
>  drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       | 1 +
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
>  drivers/media/platform/mx2_emmaprp.c               | 1 +
>  drivers/media/platform/qcom/venus/vdec.c           | 1 +
>  drivers/media/platform/qcom/venus/venc.c           | 1 +
>  drivers/media/platform/rcar_fdp1.c                 | 1 +
>  drivers/media/platform/rcar_jpu.c                  | 1 +
>  drivers/media/platform/rockchip/rga/rga-buf.c      | 1 +
>  drivers/media/platform/s5p-g2d/g2d.c               | 1 +

If this 2D blitter driver picks input buffers in random order, we have
a serious problem.

>  drivers/media/platform/s5p-jpeg/jpeg-core.c        | 1 +
>  drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       | 1 +
>  drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       | 1 +
>  drivers/media/platform/sh_veu.c                    | 1 +
>  drivers/media/platform/sti/bdisp/bdisp-v4l2.c      | 1 +
>  drivers/media/platform/ti-vpe/vpe.c                | 1 +
>  drivers/media/platform/vim2m.c                     | 1 +
>  22 files changed, 22 insertions(+)
> 
> diff --git a/drivers/media/platform/coda/coda-common.c b/drivers/media/platform/coda/coda-common.c
> index 04e35d70ce2e..6deb29fe6eb7 100644
> --- a/drivers/media/platform/coda/coda-common.c
> +++ b/drivers/media/platform/coda/coda-common.c
> @@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
>  	.stop_streaming		= coda_stop_streaming,
>  	.wait_prepare		= vb2_ops_wait_prepare,
>  	.wait_finish		= vb2_ops_wait_finish,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
> diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c b/drivers/media/platform/exynos-gsc/gsc-m2m.c
> index e9ff27949a91..10c3e4659d38 100644
> --- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
> +++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
> @@ -286,6 +286,7 @@ static const struct vb2_ops gsc_m2m_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.stop_streaming	 = gsc_m2m_stop_streaming,
>  	.start_streaming = gsc_m2m_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int gsc_m2m_querycap(struct file *file, void *fh,
> diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c b/drivers/media/platform/exynos4-is/fimc-m2m.c
> index a19f8b164a47..dfc487a582c0 100644
> --- a/drivers/media/platform/exynos4-is/fimc-m2m.c
> +++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
> @@ -227,6 +227,7 @@ static const struct vb2_ops fimc_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.stop_streaming	 = stop_streaming,
>  	.start_streaming = start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  /*
> diff --git a/drivers/media/platform/m2m-deinterlace.c b/drivers/media/platform/m2m-deinterlace.c
> index 1e4195144f39..35a0f45d2a51 100644
> --- a/drivers/media/platform/m2m-deinterlace.c
> +++ b/drivers/media/platform/m2m-deinterlace.c
> @@ -856,6 +856,7 @@ static const struct vb2_ops deinterlace_qops = {
>  	.queue_setup	 = deinterlace_queue_setup,
>  	.buf_prepare	 = deinterlace_buf_prepare,
>  	.buf_queue	 = deinterlace_buf_queue,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> index 226f90886484..34a4b5b2e1b5 100644
> --- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> +++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> @@ -764,6 +764,7 @@ static const struct vb2_ops mtk_jpeg_qops = {
>  	.wait_finish        = vb2_ops_wait_finish,
>  	.start_streaming    = mtk_jpeg_start_streaming,
>  	.stop_streaming     = mtk_jpeg_stop_streaming,
> +	.is_unordered       = vb2_ops_set_unordered,
>  };
>  
>  static void mtk_jpeg_set_dec_src(struct mtk_jpeg_ctx *ctx,
> diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> index 583d47724ee8..f3bb9f277f55 100644
> --- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> +++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> @@ -629,6 +629,7 @@ static const struct vb2_ops mtk_mdp_m2m_qops = {
>  	.wait_finish	 = mtk_mdp_ctx_lock,
>  	.stop_streaming	 = mtk_mdp_m2m_stop_streaming,
>  	.start_streaming = mtk_mdp_m2m_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int mtk_mdp_m2m_querycap(struct file *file, void *fh,
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> index 86f0a7134365..4f33e9741248 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> @@ -1445,6 +1445,7 @@ static const struct vb2_ops mtk_vdec_vb2_ops = {
>  	.buf_finish	= vb2ops_vdec_buf_finish,
>  	.start_streaming	= vb2ops_vdec_start_streaming,
>  	.stop_streaming	= vb2ops_vdec_stop_streaming,
> +	.is_unordered	= vb2_ops_set_unordered,
>  };
>  
>  const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops = {
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> index 1b1a28abbf1f..fd763249d7bd 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> @@ -931,6 +931,7 @@ static const struct vb2_ops mtk_venc_vb2_ops = {
>  	.wait_finish		= vb2_ops_wait_finish,
>  	.start_streaming	= vb2ops_venc_start_streaming,
>  	.stop_streaming		= vb2ops_venc_stop_streaming,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int mtk_venc_encode_header(void *priv)
> diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c
> index 5a8eff60e95f..d03becff66cf 100644
> --- a/drivers/media/platform/mx2_emmaprp.c
> +++ b/drivers/media/platform/mx2_emmaprp.c
> @@ -747,6 +747,7 @@ static const struct vb2_ops emmaprp_qops = {
>  	.queue_setup	 = emmaprp_queue_setup,
>  	.buf_prepare	 = emmaprp_buf_prepare,
>  	.buf_queue	 = emmaprp_buf_queue,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/qcom/venus/vdec.c b/drivers/media/platform/qcom/venus/vdec.c
> index c9e9576bb08a..20acbfe2150d 100644
> --- a/drivers/media/platform/qcom/venus/vdec.c
> +++ b/drivers/media/platform/qcom/venus/vdec.c
> @@ -793,6 +793,7 @@ static const struct vb2_ops vdec_vb2_ops = {
>  	.start_streaming = vdec_start_streaming,
>  	.stop_streaming = venus_helper_vb2_stop_streaming,
>  	.buf_queue = venus_helper_vb2_buf_queue,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  static void vdec_buf_done(struct venus_inst *inst, unsigned int buf_type,
> diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
> index e3a10a852cad..abefae68ce5a 100644
> --- a/drivers/media/platform/qcom/venus/venc.c
> +++ b/drivers/media/platform/qcom/venus/venc.c
> @@ -982,6 +982,7 @@ static const struct vb2_ops venc_vb2_ops = {
>  	.start_streaming = venc_start_streaming,
>  	.stop_streaming = venus_helper_vb2_stop_streaming,
>  	.buf_queue = venus_helper_vb2_buf_queue,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  static void venc_buf_done(struct venus_inst *inst, unsigned int buf_type,
> diff --git a/drivers/media/platform/rcar_fdp1.c b/drivers/media/platform/rcar_fdp1.c
> index b13dec3081e5..6a744a9c1738 100644
> --- a/drivers/media/platform/rcar_fdp1.c
> +++ b/drivers/media/platform/rcar_fdp1.c
> @@ -2040,6 +2040,7 @@ static const struct vb2_ops fdp1_qops = {
>  	.stop_streaming  = fdp1_stop_streaming,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c
> index f6092ae45912..b4b2e2cf5d1a 100644
> --- a/drivers/media/platform/rcar_jpu.c
> +++ b/drivers/media/platform/rcar_jpu.c
> @@ -1192,6 +1192,7 @@ static const struct vb2_ops jpu_qops = {
>  	.stop_streaming		= jpu_stop_streaming,
>  	.wait_prepare		= vb2_ops_wait_prepare,
>  	.wait_finish		= vb2_ops_wait_finish,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int jpu_queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/rockchip/rga/rga-buf.c b/drivers/media/platform/rockchip/rga/rga-buf.c
> index fa1ba98c96dc..48932f34144d 100644
> --- a/drivers/media/platform/rockchip/rga/rga-buf.c
> +++ b/drivers/media/platform/rockchip/rga/rga-buf.c
> @@ -112,6 +112,7 @@ const struct vb2_ops rga_qops = {
>  	.wait_finish = vb2_ops_wait_finish,
>  	.start_streaming = rga_buf_start_streaming,
>  	.stop_streaming = rga_buf_stop_streaming,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  /* RGA MMU is a 1-Level MMU, so it can't be used through the IOMMU API.
> diff --git a/drivers/media/platform/s5p-g2d/g2d.c b/drivers/media/platform/s5p-g2d/g2d.c
> index 66aa8cf1d048..cb7d916bfc8b 100644
> --- a/drivers/media/platform/s5p-g2d/g2d.c
> +++ b/drivers/media/platform/s5p-g2d/g2d.c
> @@ -142,6 +142,7 @@ static const struct vb2_ops g2d_qops = {
>  	.queue_setup	= g2d_queue_setup,
>  	.buf_prepare	= g2d_buf_prepare,
>  	.buf_queue	= g2d_buf_queue,
> +	.is_unordered	= vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c
> index 79b63da27f53..28485e6b9cc8 100644
> --- a/drivers/media/platform/s5p-jpeg/jpeg-core.c
> +++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c
> @@ -2648,6 +2648,7 @@ static const struct vb2_ops s5p_jpeg_qops = {
>  	.wait_finish		= vb2_ops_wait_finish,
>  	.start_streaming	= s5p_jpeg_start_streaming,
>  	.stop_streaming		= s5p_jpeg_stop_streaming,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> index 8937b0af7cb3..369db08dbcae 100644
> --- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> +++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> @@ -1099,6 +1099,7 @@ static struct vb2_ops s5p_mfc_dec_qops = {
>  	.start_streaming	= s5p_mfc_start_streaming,
>  	.stop_streaming		= s5p_mfc_stop_streaming,
>  	.buf_queue		= s5p_mfc_buf_queue,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  const struct s5p_mfc_codec_ops *get_dec_codec_ops(void)
> diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> index 0d5d465561be..fece496c2a8e 100644
> --- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> +++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> @@ -2036,6 +2036,7 @@ static struct vb2_ops s5p_mfc_enc_qops = {
>  	.start_streaming	= s5p_mfc_start_streaming,
>  	.stop_streaming		= s5p_mfc_stop_streaming,
>  	.buf_queue		= s5p_mfc_buf_queue,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  const struct s5p_mfc_codec_ops *get_enc_codec_ops(void)
> diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c
> index 1a0cde017fdf..0682b50a67fc 100644
> --- a/drivers/media/platform/sh_veu.c
> +++ b/drivers/media/platform/sh_veu.c
> @@ -927,6 +927,7 @@ static const struct vb2_ops sh_veu_qops = {
>  	.buf_queue	 = sh_veu_buf_queue,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int sh_veu_queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> index bf4ca16db440..0cfdc5a67855 100644
> --- a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> +++ b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> @@ -535,6 +535,7 @@ static const struct vb2_ops bdisp_qops = {
>  	.wait_finish     = vb2_ops_wait_finish,
>  	.stop_streaming  = bdisp_stop_streaming,
>  	.start_streaming = bdisp_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv,
> diff --git a/drivers/media/platform/ti-vpe/vpe.c b/drivers/media/platform/ti-vpe/vpe.c
> index e395aa85c8ad..c2d838884e1c 100644
> --- a/drivers/media/platform/ti-vpe/vpe.c
> +++ b/drivers/media/platform/ti-vpe/vpe.c
> @@ -2202,6 +2202,7 @@ static const struct vb2_ops vpe_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.start_streaming = vpe_start_streaming,
>  	.stop_streaming  = vpe_stop_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index 065483e62db4..e1a54a28b082 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -823,6 +823,7 @@ static const struct vb2_ops vim2m_qops = {
>  	.stop_streaming  = vim2m_stop_streaming,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
> -- 
> 2.14.3
> 
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[jacopo mondi]

[-- Attachment #1: Type: text/plain, Size: 1590 bytes --]

Hi Gustavo,
  a very small comment below

On Fri, Mar 09, 2018 at 02:49:20PM -0300, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
> Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it
>
> v6:	- Close some gaps in the docs (Hans)
>
> v5:
> 	- Remove V4L2_CAP_ORDERED
> 	- Add doc about V4L2_FMT_FLAG_UNORDERED
>
> v4:
> 	- Document ordering behavior for in-fences
> 	- Document V4L2_CAP_ORDERED capability
> 	- Remove doc about OUT_FENCE event
> 	- Document immediate return of out-fence in QBUF
>
> v3:
> 	- make the out_fence refer to the current buffer (Hans)
> 	- Note what happens when the IN_FENCE is not set (Hans)
>
> v2:
> 	- mention that fences are files (Hans)
> 	- rework for the new API
>
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 55 +++++++++++++++++++++++-
>  Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
>  2 files changed, 63 insertions(+), 4 deletions(-)
>
[snip]
> +Note the the same `fence_fd` field is used for both sending the in-fence as
> +input argument to receive the out-fence as a return argument. A buffer can
> +have both in-fence ond out-fence.

I feel like an "and" is missing here...

 the same `fence_fd` field is used for both sending the in-fence as
 input argument to receive the out-fence as a return argument

 the same `fence_fd` field is used for both sending the in-fence as
 input argument *and* to receive the out-fence as a return argument

I'm not a native speaker so I might be wrong though.

Thanks
   j

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 819 bytes --]

Re: [PATCH v8 01/13] [media] xilinx: regroup caps on querycap

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> To better organize the code we concentrate the setting of
> V4L2_CAP_STREAMING in one place.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/platform/xilinx/xilinx-dma.c | 7 ++++---
>  1 file changed, 4 insertions(+), 3 deletions(-)
> 
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 522cdfdd3345..565e466ba4fa 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -494,13 +494,14 @@ xvip_dma_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
>  	struct v4l2_fh *vfh = file->private_data;
>  	struct xvip_dma *dma = to_xvip_dma(vfh->vdev);
>  
> -	cap->capabilities = V4L2_CAP_DEVICE_CAPS | V4L2_CAP_STREAMING
> +	cap->device_caps = V4L2_CAP_STREAMING;
> +	cap->capabilities = cap->device_caps | V4L2_CAP_DEVICE_CAPS
>  			  | dma->xdev->v4l2_caps;

Shouldn't this cap->capabilities assignment be moved down to after the
if-else? Otherwise cap->device_caps isn't fully initialized yet.

>  
>  	if (dma->queue.type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> -		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
> +		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
>  	else
> -		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
> +		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
>  
>  	strlcpy(cap->driver, "xilinx-vipp", sizeof(cap->driver));
>  	strlcpy(cap->card, dma->video.name, sizeof(cap->card));
> 

Regards,

	Hans

Re: [PATCH v8 03/13] [media] omap3isp: group device capabilities

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Instead of putting V4L2_CAP_STREAMING everywhere, set device_caps
> earlier with this value.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/platform/omap3isp/ispvideo.c | 9 +++++----
>  1 file changed, 5 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index a751c89a3ea8..b4d4ef926749 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -658,13 +658,14 @@ isp_video_querycap(struct file *file, void *fh, struct v4l2_capability *cap)
>  	strlcpy(cap->card, video->video.name, sizeof(cap->card));
>  	strlcpy(cap->bus_info, "media", sizeof(cap->bus_info));
>  
> -	cap->capabilities = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_VIDEO_OUTPUT
> -		| V4L2_CAP_STREAMING | V4L2_CAP_DEVICE_CAPS;
> +	cap->device_caps = V4L2_CAP_STREAMING;
> +	cap->capabilities = cap->device_caps | V4L2_CAP_VIDEO_CAPTURE |
> +		V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_DEVICE_CAPS;

Same as in patch 1: I'd move this down to after the if-else. It makes more
sense that way.

>  
>  	if (video->type == V4L2_BUF_TYPE_VIDEO_CAPTURE)
> -		cap->device_caps = V4L2_CAP_VIDEO_CAPTURE | V4L2_CAP_STREAMING;
> +		cap->device_caps |= V4L2_CAP_VIDEO_CAPTURE;
>  	else
> -		cap->device_caps = V4L2_CAP_VIDEO_OUTPUT | V4L2_CAP_STREAMING;
> +		cap->device_caps |= V4L2_CAP_VIDEO_OUTPUT;
>  
>  	return 0;
>  }
> 

Regards,

	Hans

Re: [PATCH v8 04/13] [media] vb2: add is_unordered callback for drivers

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Explicit synchronization benefits a lot from ordered queues, they fit
> better in a pipeline with DRM for example so create a opt-in way for
> drivers notify videobuf2 that the queue is unordered.
> 
> Drivers don't need implement it if the queue is ordered.
> 
> v2: 	- improve comments for is_unordered flag (Hans)
> 
> v3: 	- make it bool (Hans)
> 	- create vb2_ops_set_unordered() helper
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  6 ++++++
>  include/media/videobuf2-core.h                  |  6 ++++++
>  include/media/videobuf2-v4l2.h                  | 10 ++++++++++
>  3 files changed, 22 insertions(+)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 886a2d8d5c6c..68291ba8632d 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -961,6 +961,12 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
>  }
>  EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
>  
> +bool vb2_ops_set_unordered(struct vb2_queue *q)
> +{
> +	return true;
> +}
> +EXPORT_SYMBOL_GPL(vb2_ops_set_unordered);

This should be called vb2_ops_is_unordered. The name after the vb2_ops_
prefix should match the op name (is_unordered).

I also would add this to videobuf2-core.c since there is nothing V4L2
specific about this.

> +
>  MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
>  MODULE_AUTHOR("Pawel Osciak <pawel@osciak.com>, Marek Szyprowski");
>  MODULE_LICENSE("GPL");
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 5b6c541e4e1b..46a9e674f7e1 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -370,6 +370,10 @@ struct vb2_buffer {
>   *			callback by calling vb2_buffer_done() with either
>   *			%VB2_BUF_STATE_DONE or %VB2_BUF_STATE_ERROR; may use
>   *			vb2_wait_for_all_buffers() function
> + * @is_unordered:	tell if the queue is unordered, i.e. buffers can be
> + *			dequeued in a different order from how they were queued.
> + *			The default is assumed to be ordered and this function
> + *			only needs to be implemented for unordered queues.
>   * @buf_queue:		passes buffer vb to the driver; driver may start
>   *			hardware operation on this buffer; driver should give
>   *			the buffer back by calling vb2_buffer_done() function;
> @@ -393,6 +397,7 @@ struct vb2_ops {
>  
>  	int (*start_streaming)(struct vb2_queue *q, unsigned int count);
>  	void (*stop_streaming)(struct vb2_queue *q);
> +	bool (*is_unordered)(struct vb2_queue *q);
>  
>  	void (*buf_queue)(struct vb2_buffer *vb);
>  };
> @@ -566,6 +571,7 @@ struct vb2_queue {
>  	u32				cnt_wait_finish;
>  	u32				cnt_start_streaming;
>  	u32				cnt_stop_streaming;
> +	u32				cnt_is_unordered;
>  #endif
>  };
>  
> diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
> index 3d5e2d739f05..9de3c887c875 100644
> --- a/include/media/videobuf2-v4l2.h
> +++ b/include/media/videobuf2-v4l2.h
> @@ -291,4 +291,14 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
>   */
>  void vb2_ops_wait_finish(struct vb2_queue *vq);
>  
> +/**
> + * vb2_ops_set_unordered - helper function to mark queue as unordered
> + *
> + * @vq: pointer to &struct vb2_queue
> + *
> + * This helper just return true to notify that the driver can't deal with

return -> returns

> + * ordered queues.
> + */
> +bool vb2_ops_set_unordered(struct vb2_queue *q);
> +
>  #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
> 

Regards,

	Hans

Re: [PATCH v8 06/13] [media] cobalt: add .is_unordered() for cobalt

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> The cobalt driver may reorder the capture buffers so we need to report
> it as such.
> 
> v2: - use vb2_ops_set_unordered() helper
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/pci/cobalt/cobalt-v4l2.c | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/drivers/media/pci/cobalt/cobalt-v4l2.c b/drivers/media/pci/cobalt/cobalt-v4l2.c
> index e2a4c705d353..6b6611a0e190 100644
> --- a/drivers/media/pci/cobalt/cobalt-v4l2.c
> +++ b/drivers/media/pci/cobalt/cobalt-v4l2.c
> @@ -430,6 +430,7 @@ static const struct vb2_ops cobalt_qops = {
>  	.stop_streaming = cobalt_stop_streaming,
>  	.wait_prepare = vb2_ops_wait_prepare,
>  	.wait_finish = vb2_ops_wait_finish,
> +	.is_unordered = vb2_ops_set_unordered,
>  };

If you do this, then you should also set the UNORDERED format flag for
all formats used by cobalt.

Regards,

	Hans

>  
>  /* V4L2 ioctls */
> 

Re: [PATCH v8 07/13] [media] vb2: mark codec drivers as unordered

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> In preparation to have full support to explicit fence we are
> marking codec as non-ordered preventively. It is easier and safer from an
> uAPI point of view to move from unordered to ordered than the opposite.

Same comment as for the cobalt driver: if you mark these drivers as
unordered, shouldn't you mark the compressed formats as UNORDERED
as well? Should we perhaps do this by default for compressed formats
(except MJPEG) in v4l2-ioctl.c? Thus requiring drivers to clear the
flag if they are actually ordered.

> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/platform/coda/coda-common.c          | 1 +
>  drivers/media/platform/exynos-gsc/gsc-m2m.c        | 1 +
>  drivers/media/platform/exynos4-is/fimc-m2m.c       | 1 +
>  drivers/media/platform/m2m-deinterlace.c           | 1 +

This is a deinterlaced, so this should be ordered.

>  drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c    | 1 +

JPEG is ordered.

>  drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c       | 1 +
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c | 1 +
>  drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c | 1 +
>  drivers/media/platform/mx2_emmaprp.c               | 1 +

I believe this is also not a codec, so ordered,

>  drivers/media/platform/qcom/venus/vdec.c           | 1 +
>  drivers/media/platform/qcom/venus/venc.c           | 1 +
>  drivers/media/platform/rcar_fdp1.c                 | 1 +
>  drivers/media/platform/rcar_jpu.c                  | 1 +

I don't think these two R-Car drivers are codecs, so these would be
ordered (the 2nd is a JPEG codec, not sure about the first one).

>  drivers/media/platform/rockchip/rga/rga-buf.c      | 1 +

Not a codec driver, this is ordered,

>  drivers/media/platform/s5p-g2d/g2d.c               | 1 +

Not sure about this one, I don't think it is a codec driver,

>  drivers/media/platform/s5p-jpeg/jpeg-core.c        | 1 +

MJPEG, so ordered,

>  drivers/media/platform/s5p-mfc/s5p_mfc_dec.c       | 1 +
>  drivers/media/platform/s5p-mfc/s5p_mfc_enc.c       | 1 +
>  drivers/media/platform/sh_veu.c                    | 1 +

Not a codec, so ordered,

>  drivers/media/platform/sti/bdisp/bdisp-v4l2.c      | 1 +
>  drivers/media/platform/ti-vpe/vpe.c                | 1 +
>  drivers/media/platform/vim2m.c                     | 1 +

Same for these three.

Regards,

	Hans

>  22 files changed, 22 insertions(+)
> 
> diff --git a/drivers/media/platform/coda/coda-common.c b/drivers/media/platform/coda/coda-common.c
> index 04e35d70ce2e..6deb29fe6eb7 100644
> --- a/drivers/media/platform/coda/coda-common.c
> +++ b/drivers/media/platform/coda/coda-common.c
> @@ -1649,6 +1649,7 @@ static const struct vb2_ops coda_qops = {
>  	.stop_streaming		= coda_stop_streaming,
>  	.wait_prepare		= vb2_ops_wait_prepare,
>  	.wait_finish		= vb2_ops_wait_finish,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int coda_s_ctrl(struct v4l2_ctrl *ctrl)
> diff --git a/drivers/media/platform/exynos-gsc/gsc-m2m.c b/drivers/media/platform/exynos-gsc/gsc-m2m.c
> index e9ff27949a91..10c3e4659d38 100644
> --- a/drivers/media/platform/exynos-gsc/gsc-m2m.c
> +++ b/drivers/media/platform/exynos-gsc/gsc-m2m.c
> @@ -286,6 +286,7 @@ static const struct vb2_ops gsc_m2m_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.stop_streaming	 = gsc_m2m_stop_streaming,
>  	.start_streaming = gsc_m2m_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int gsc_m2m_querycap(struct file *file, void *fh,
> diff --git a/drivers/media/platform/exynos4-is/fimc-m2m.c b/drivers/media/platform/exynos4-is/fimc-m2m.c
> index a19f8b164a47..dfc487a582c0 100644
> --- a/drivers/media/platform/exynos4-is/fimc-m2m.c
> +++ b/drivers/media/platform/exynos4-is/fimc-m2m.c
> @@ -227,6 +227,7 @@ static const struct vb2_ops fimc_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.stop_streaming	 = stop_streaming,
>  	.start_streaming = start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  /*
> diff --git a/drivers/media/platform/m2m-deinterlace.c b/drivers/media/platform/m2m-deinterlace.c
> index 1e4195144f39..35a0f45d2a51 100644
> --- a/drivers/media/platform/m2m-deinterlace.c
> +++ b/drivers/media/platform/m2m-deinterlace.c
> @@ -856,6 +856,7 @@ static const struct vb2_ops deinterlace_qops = {
>  	.queue_setup	 = deinterlace_queue_setup,
>  	.buf_prepare	 = deinterlace_buf_prepare,
>  	.buf_queue	 = deinterlace_buf_queue,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> index 226f90886484..34a4b5b2e1b5 100644
> --- a/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> +++ b/drivers/media/platform/mtk-jpeg/mtk_jpeg_core.c
> @@ -764,6 +764,7 @@ static const struct vb2_ops mtk_jpeg_qops = {
>  	.wait_finish        = vb2_ops_wait_finish,
>  	.start_streaming    = mtk_jpeg_start_streaming,
>  	.stop_streaming     = mtk_jpeg_stop_streaming,
> +	.is_unordered       = vb2_ops_set_unordered,
>  };
>  
>  static void mtk_jpeg_set_dec_src(struct mtk_jpeg_ctx *ctx,
> diff --git a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> index 583d47724ee8..f3bb9f277f55 100644
> --- a/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> +++ b/drivers/media/platform/mtk-mdp/mtk_mdp_m2m.c
> @@ -629,6 +629,7 @@ static const struct vb2_ops mtk_mdp_m2m_qops = {
>  	.wait_finish	 = mtk_mdp_ctx_lock,
>  	.stop_streaming	 = mtk_mdp_m2m_stop_streaming,
>  	.start_streaming = mtk_mdp_m2m_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int mtk_mdp_m2m_querycap(struct file *file, void *fh,
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> index 86f0a7134365..4f33e9741248 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_dec.c
> @@ -1445,6 +1445,7 @@ static const struct vb2_ops mtk_vdec_vb2_ops = {
>  	.buf_finish	= vb2ops_vdec_buf_finish,
>  	.start_streaming	= vb2ops_vdec_start_streaming,
>  	.stop_streaming	= vb2ops_vdec_stop_streaming,
> +	.is_unordered	= vb2_ops_set_unordered,
>  };
>  
>  const struct v4l2_ioctl_ops mtk_vdec_ioctl_ops = {
> diff --git a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> index 1b1a28abbf1f..fd763249d7bd 100644
> --- a/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> +++ b/drivers/media/platform/mtk-vcodec/mtk_vcodec_enc.c
> @@ -931,6 +931,7 @@ static const struct vb2_ops mtk_venc_vb2_ops = {
>  	.wait_finish		= vb2_ops_wait_finish,
>  	.start_streaming	= vb2ops_venc_start_streaming,
>  	.stop_streaming		= vb2ops_venc_stop_streaming,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int mtk_venc_encode_header(void *priv)
> diff --git a/drivers/media/platform/mx2_emmaprp.c b/drivers/media/platform/mx2_emmaprp.c
> index 5a8eff60e95f..d03becff66cf 100644
> --- a/drivers/media/platform/mx2_emmaprp.c
> +++ b/drivers/media/platform/mx2_emmaprp.c
> @@ -747,6 +747,7 @@ static const struct vb2_ops emmaprp_qops = {
>  	.queue_setup	 = emmaprp_queue_setup,
>  	.buf_prepare	 = emmaprp_buf_prepare,
>  	.buf_queue	 = emmaprp_buf_queue,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/qcom/venus/vdec.c b/drivers/media/platform/qcom/venus/vdec.c
> index c9e9576bb08a..20acbfe2150d 100644
> --- a/drivers/media/platform/qcom/venus/vdec.c
> +++ b/drivers/media/platform/qcom/venus/vdec.c
> @@ -793,6 +793,7 @@ static const struct vb2_ops vdec_vb2_ops = {
>  	.start_streaming = vdec_start_streaming,
>  	.stop_streaming = venus_helper_vb2_stop_streaming,
>  	.buf_queue = venus_helper_vb2_buf_queue,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  static void vdec_buf_done(struct venus_inst *inst, unsigned int buf_type,
> diff --git a/drivers/media/platform/qcom/venus/venc.c b/drivers/media/platform/qcom/venus/venc.c
> index e3a10a852cad..abefae68ce5a 100644
> --- a/drivers/media/platform/qcom/venus/venc.c
> +++ b/drivers/media/platform/qcom/venus/venc.c
> @@ -982,6 +982,7 @@ static const struct vb2_ops venc_vb2_ops = {
>  	.start_streaming = venc_start_streaming,
>  	.stop_streaming = venus_helper_vb2_stop_streaming,
>  	.buf_queue = venus_helper_vb2_buf_queue,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  static void venc_buf_done(struct venus_inst *inst, unsigned int buf_type,
> diff --git a/drivers/media/platform/rcar_fdp1.c b/drivers/media/platform/rcar_fdp1.c
> index b13dec3081e5..6a744a9c1738 100644
> --- a/drivers/media/platform/rcar_fdp1.c
> +++ b/drivers/media/platform/rcar_fdp1.c
> @@ -2040,6 +2040,7 @@ static const struct vb2_ops fdp1_qops = {
>  	.stop_streaming  = fdp1_stop_streaming,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/rcar_jpu.c b/drivers/media/platform/rcar_jpu.c
> index f6092ae45912..b4b2e2cf5d1a 100644
> --- a/drivers/media/platform/rcar_jpu.c
> +++ b/drivers/media/platform/rcar_jpu.c
> @@ -1192,6 +1192,7 @@ static const struct vb2_ops jpu_qops = {
>  	.stop_streaming		= jpu_stop_streaming,
>  	.wait_prepare		= vb2_ops_wait_prepare,
>  	.wait_finish		= vb2_ops_wait_finish,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int jpu_queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/rockchip/rga/rga-buf.c b/drivers/media/platform/rockchip/rga/rga-buf.c
> index fa1ba98c96dc..48932f34144d 100644
> --- a/drivers/media/platform/rockchip/rga/rga-buf.c
> +++ b/drivers/media/platform/rockchip/rga/rga-buf.c
> @@ -112,6 +112,7 @@ const struct vb2_ops rga_qops = {
>  	.wait_finish = vb2_ops_wait_finish,
>  	.start_streaming = rga_buf_start_streaming,
>  	.stop_streaming = rga_buf_stop_streaming,
> +	.is_unordered = vb2_ops_set_unordered,
>  };
>  
>  /* RGA MMU is a 1-Level MMU, so it can't be used through the IOMMU API.
> diff --git a/drivers/media/platform/s5p-g2d/g2d.c b/drivers/media/platform/s5p-g2d/g2d.c
> index 66aa8cf1d048..cb7d916bfc8b 100644
> --- a/drivers/media/platform/s5p-g2d/g2d.c
> +++ b/drivers/media/platform/s5p-g2d/g2d.c
> @@ -142,6 +142,7 @@ static const struct vb2_ops g2d_qops = {
>  	.queue_setup	= g2d_queue_setup,
>  	.buf_prepare	= g2d_buf_prepare,
>  	.buf_queue	= g2d_buf_queue,
> +	.is_unordered	= vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/s5p-jpeg/jpeg-core.c b/drivers/media/platform/s5p-jpeg/jpeg-core.c
> index 79b63da27f53..28485e6b9cc8 100644
> --- a/drivers/media/platform/s5p-jpeg/jpeg-core.c
> +++ b/drivers/media/platform/s5p-jpeg/jpeg-core.c
> @@ -2648,6 +2648,7 @@ static const struct vb2_ops s5p_jpeg_qops = {
>  	.wait_finish		= vb2_ops_wait_finish,
>  	.start_streaming	= s5p_jpeg_start_streaming,
>  	.stop_streaming		= s5p_jpeg_stop_streaming,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> index 8937b0af7cb3..369db08dbcae 100644
> --- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> +++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
> @@ -1099,6 +1099,7 @@ static struct vb2_ops s5p_mfc_dec_qops = {
>  	.start_streaming	= s5p_mfc_start_streaming,
>  	.stop_streaming		= s5p_mfc_stop_streaming,
>  	.buf_queue		= s5p_mfc_buf_queue,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  const struct s5p_mfc_codec_ops *get_dec_codec_ops(void)
> diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> index 0d5d465561be..fece496c2a8e 100644
> --- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> +++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
> @@ -2036,6 +2036,7 @@ static struct vb2_ops s5p_mfc_enc_qops = {
>  	.start_streaming	= s5p_mfc_start_streaming,
>  	.stop_streaming		= s5p_mfc_stop_streaming,
>  	.buf_queue		= s5p_mfc_buf_queue,
> +	.is_unordered		= vb2_ops_set_unordered,
>  };
>  
>  const struct s5p_mfc_codec_ops *get_enc_codec_ops(void)
> diff --git a/drivers/media/platform/sh_veu.c b/drivers/media/platform/sh_veu.c
> index 1a0cde017fdf..0682b50a67fc 100644
> --- a/drivers/media/platform/sh_veu.c
> +++ b/drivers/media/platform/sh_veu.c
> @@ -927,6 +927,7 @@ static const struct vb2_ops sh_veu_qops = {
>  	.buf_queue	 = sh_veu_buf_queue,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int sh_veu_queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> index bf4ca16db440..0cfdc5a67855 100644
> --- a/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> +++ b/drivers/media/platform/sti/bdisp/bdisp-v4l2.c
> @@ -535,6 +535,7 @@ static const struct vb2_ops bdisp_qops = {
>  	.wait_finish     = vb2_ops_wait_finish,
>  	.stop_streaming  = bdisp_stop_streaming,
>  	.start_streaming = bdisp_start_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv,
> diff --git a/drivers/media/platform/ti-vpe/vpe.c b/drivers/media/platform/ti-vpe/vpe.c
> index e395aa85c8ad..c2d838884e1c 100644
> --- a/drivers/media/platform/ti-vpe/vpe.c
> +++ b/drivers/media/platform/ti-vpe/vpe.c
> @@ -2202,6 +2202,7 @@ static const struct vb2_ops vpe_qops = {
>  	.wait_finish	 = vb2_ops_wait_finish,
>  	.start_streaming = vpe_start_streaming,
>  	.stop_streaming  = vpe_stop_streaming,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq,
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index 065483e62db4..e1a54a28b082 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -823,6 +823,7 @@ static const struct vb2_ops vim2m_qops = {
>  	.stop_streaming  = vim2m_stop_streaming,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.is_unordered	 = vb2_ops_set_unordered,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
> 

Re: [PATCH v8 08/13] [media] vb2: add explicit fence user API

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Turn the reserved2 field into fence_fd that we will use to send
> an in-fence to the kernel or return an out-fence from the kernel to
> userspace.
> 
> Two new flags were added, V4L2_BUF_FLAG_IN_FENCE, that should be used
> when sending a fence to the kernel to be waited on, and
> V4L2_BUF_FLAG_OUT_FENCE, to ask the kernel to give back an out-fence.
> 
> v6:	- big improvement on doc (Hans Verkuil)
> 
> v5:
> 	- keep using reserved2 field for cpia2
> 	- set fence_fd to 0 for now, for compat with userspace(Mauro)
> 
> v4:
> 	- make it a union with reserved2 and fence_fd (Hans Verkuil)
> 
> v3:
> 	- make the out_fence refer to the current buffer (Hans Verkuil)
> 
> v2: add documentation
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/buffer.rst         | 45 +++++++++++++++++++++++--
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  4 +--
>  include/uapi/linux/videodev2.h                  |  7 +++-
>  4 files changed, 51 insertions(+), 7 deletions(-)
> 
> diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
> index e2c85ddc990b..49273026740f 100644
> --- a/Documentation/media/uapi/v4l/buffer.rst
> +++ b/Documentation/media/uapi/v4l/buffer.rst
> @@ -301,10 +301,22 @@ struct v4l2_buffer
>  	elements in the ``planes`` array. The driver will fill in the
>  	actual number of valid elements in that array.
>      * - __u32
> -      - ``reserved2``
> +      - ``fence_fd``
>        -
> -      - A place holder for future extensions. Drivers and applications
> -	must set this to 0.
> +      - Used to communicate fences file descriptors from userspace to kernel

fences file descriptors -> a fence file descriptor

> +	and vice-versa. On :ref:`VIDIOC_QBUF <VIDIOC_QBUF>` when sending
> +	an in-fence for V4L2 to wait on, the ``V4L2_BUF_FLAG_IN_FENCE`` flag must
> +	be used and this field set to the fence file descriptor of the in-fence

Missing period at the end of this sentence.

> +	If the in-fence is not valid ` VIDIOC_QBUF`` returns an error.

` -> ``

> +
> +        To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE``
> +	must be set, the kernel will return the out-fence file descriptor on

on -> in

> +	this field. If it fails to create the out-fence ``VIDIOC_QBUF` returns

` -> ``

> +        an error.
> +
> +	In all other ioctls V4L2 sets this field to -1 if

In all other -> For all other buffer

> +	``V4L2_BUF_FLAG_IN_FENCE`` and/or ``V4L2_BUF_FLAG_OUT_FENCE`` are set,
> +	otherwise this field is set to 0 for backward compatibility.
>      * - __u32
>        - ``reserved``
>        -
> @@ -648,6 +660,33 @@ Buffer Flags
>        - Start Of Exposure. The buffer timestamp has been taken when the
>  	exposure of the frame has begun. This is only valid for the
>  	``V4L2_BUF_TYPE_VIDEO_CAPTURE`` buffer type.
> +    * .. _`V4L2-BUF-FLAG-IN-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_IN_FENCE``
> +      - 0x00200000
> +      - Ask V4L2 to wait on the fence passed in the ``fence_fd`` field. The
> +	buffer won't be queued to the driver until the fence signals. The order
> +	in which buffers are queued is guaranteed to be preserved, so any
> +	buffers queued after this buffer will also be blocked until this fence
> +	signals. This flag must be set before calling ``VIDIOC_QBUF``. For
> +	other ioctls the driver just report the value of the flag.

report -> reports

> +
> +        If the fence signals the flag is cleared and not reported anymore.
> +	If the fence is not valid ``VIDIOC_QBUF`` returns an error.
> +
> +
> +    * .. _`V4L2-BUF-FLAG-OUT-FENCE`:
> +
> +      - ``V4L2_BUF_FLAG_OUT_FENCE``
> +      - 0x00400000
> +      - Request for a fence to be attached to the buffer. The driver will fill
> +	in the out-fence fd in the ``fence_fd`` field when :ref:`VIDIOC_QBUF
> +	<VIDIOC_QBUF>` returns. This flag must be set before calling
> +	``VIDIOC_QBUF``. For other ioctls the driver just report the value of

report -> reports

> +	the flag.
> +
> +        If the creation of the  out-fence  fails ``VIDIOC_QBUF`` returns an

double spaces before and after 'out-fence'.

> +	error.
>  
>  
>  
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 68291ba8632d..ad1e032c3bf5 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->reserved2 = 0;
> +	b->fence_fd = 0;
>  	b->reserved = 0;
>  
>  	if (q->is_multiplanar) {
> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> index 5198c9eeb348..3de2252e3632 100644
> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> @@ -386,7 +386,7 @@ struct v4l2_buffer32 {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	__s32			fence_fd;
>  	__u32			reserved;
>  };
>  
> @@ -604,7 +604,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
>  	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
>  	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
>  	    assign_in_user(&up->sequence, &kp->sequence) ||
> -	    assign_in_user(&up->reserved2, &kp->reserved2) ||
> +	    assign_in_user(&up->fence_fd, &kp->fence_fd) ||
>  	    assign_in_user(&up->reserved, &kp->reserved) ||
>  	    get_user(length, &kp->length) ||
>  	    put_user(length, &up->length))
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 58894cfe9479..2d424aebdd1e 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -933,7 +933,10 @@ struct v4l2_buffer {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	union {
> +		__s32		fence_fd;
> +		__u32		reserved2;
> +	};
>  	__u32			reserved;
>  };
>  
> @@ -970,6 +973,8 @@ struct v4l2_buffer {
>  #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
>  /* mem2mem encoder/decoder */
>  #define V4L2_BUF_FLAG_LAST			0x00100000
> +#define V4L2_BUF_FLAG_IN_FENCE			0x00200000
> +#define V4L2_BUF_FLAG_OUT_FENCE			0x00400000
>  
>  /**
>   * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
> 

Regards,

	Hans

Re: [PATCH v8 11/13] [media] v4l: introduce the fences capability

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Drivers capable of using fences (vb2 drivers) should report the
> V4L2_CAP_FENCES to userspace, so add this flag to the uapi.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/vidioc-querycap.rst | 3 +++
>  include/uapi/linux/videodev2.h                   | 1 +
>  2 files changed, 4 insertions(+)
> 
> diff --git a/Documentation/media/uapi/v4l/vidioc-querycap.rst b/Documentation/media/uapi/v4l/vidioc-querycap.rst
> index 66fb1b3d6e6e..414016065309 100644
> --- a/Documentation/media/uapi/v4l/vidioc-querycap.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-querycap.rst
> @@ -254,6 +254,9 @@ specification the ioctl returns an ``EINVAL`` error code.
>      * - ``V4L2_CAP_TOUCH``
>        - 0x10000000
>        - This is a touch device.
> +    * - ``V4L2_CAP_FENCES``
> +      - 0x20000000
> +      - The device support explicit synchronization.

support -> supports

>      * - ``V4L2_CAP_DEVICE_CAPS``
>        - 0x80000000
>        - The driver fills the ``device_caps`` field. This capability can
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 2d424aebdd1e..db58204e346e 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -460,6 +460,7 @@ struct v4l2_capability {
>  #define V4L2_CAP_STREAMING              0x04000000  /* streaming I/O ioctls */
>  
>  #define V4L2_CAP_TOUCH                  0x10000000  /* Is a touch device */
> +#define V4L2_CAP_FENCES                 0x20000000  /* Supports explicit synchronization */
>  
>  #define V4L2_CAP_DEVICE_CAPS            0x80000000  /* sets device capabilities field */
>  
> 

Regards,

	Hans

Re: [PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it
> 
> v6:	- Close some gaps in the docs (Hans)
> 
> v5:
> 	- Remove V4L2_CAP_ORDERED
> 	- Add doc about V4L2_FMT_FLAG_UNORDERED
> 
> v4:
> 	- Document ordering behavior for in-fences
> 	- Document V4L2_CAP_ORDERED capability
> 	- Remove doc about OUT_FENCE event
> 	- Document immediate return of out-fence in QBUF
> 
> v3:
> 	- make the out_fence refer to the current buffer (Hans)
> 	- Note what happens when the IN_FENCE is not set (Hans)
> 
> v2:
> 	- mention that fences are files (Hans)
> 	- rework for the new API
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 55 +++++++++++++++++++++++-
>  Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
>  2 files changed, 63 insertions(+), 4 deletions(-)
> 
> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> index 9e448a4aa3aa..371d84966e34 100644
> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> @@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
>  or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
>  ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
>  for details. Applications must also set ``flags`` to 0. The
> -``reserved2`` and ``reserved`` fields must be set to 0. When using the
> +``reserved`` field must be set to 0. When using the
>  :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
>  contain a userspace pointer to a filled-in array of struct
>  :c:type:`v4l2_plane` and the ``length`` field must be set
> @@ -118,6 +118,59 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
>  The struct :c:type:`v4l2_buffer` structure is specified in
>  :ref:`buffer`.
>  
> +Explicit Synchronization
> +------------------------
> +
> +Explicit Synchronization allows us to control the synchronization of
> +shared buffers from userspace by passing fences to the kernel and/or
> +receiving them from it. Fences passed to the kernel are named in-fences and
> +the kernel should wait on them to signal before using the buffer. On the other
> +side, the kernel can create out-fences for the buffers it queues to the
> +drivers. Out-fences signal when the driver is finished with buffer, i.e., the
> +buffer is ready. The fences are represented as a file and passed as a file
> +descriptor to userspace.
> +
> +The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
> +using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
> +an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
> +fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
> +as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
> +with an error.

This sentence is confusing since it is not clear what 'one' and 'the other' refer
to. Be specific here. I think it should be 'Setting V4L2_BUF_FLAG_IN_FENCE but not
fence_fd'.

 The fence_fd field will be ignored if the
> +``V4L2_BUF_FLAG_IN_FENCE`` is not set.
> +
> +The videobuf2-core will guarantee that all buffers queued with an in-fence will
> +be queued to the drivers in the same order. Fences may signal out of order, so
> +this guarantee at videobuf2 is necessary to not change ordering. So when
> +waiting on a fence to signal all buffers queued after will be also block until

after -> afterwards
will be also block -> will also be blocked

> +that fence signal.

signal -> signals

> +
> +If the in-fence signals with an error the buffer will be marked with
> +``V4L2_BUF_FLAG_ERROR`` when returned to userspace at ``VIDIOC_DQBUF``.
> +Even with the error the order of dequeueing the buffers are preserved.

are -> is

> +
> +To get an out-fence back from V4L2 the ``V4L2_BUF_FLAG_OUT_FENCE`` flag should
> +be set to ask for a fence to be attached to the buffer. The out-fence fd is
> +sent to userspace as a ``VIDIOC_QBUF`` return argument on the `fence_fd` field.

on -> in

> +
> +Note the the same `fence_fd` field is used for both sending the in-fence as
> +input argument to receive the out-fence as a return argument. A buffer can

input argument -> an input argument and

> +have both in-fence ond out-fence.

have both an in-fence and an out-fence.

> +
> +At streamoff the out-fences will either signal normally if the driver waits
> +for the operations on the buffers to finish or signal with an error if the
> +driver cancels the pending operations. Buffers with in-fences won't be queued
> +to the driver if their fences signal. They will be cleaned up.
> +
> +The ``V4L2_FMT_FLAG_UNORDERED`` flag in ``VIDIOC_ENUM_FMT`` tells userspace
> +that the  when using this format the order in which buffers are dequeued can

remove 'the'

> +be different from the order in which they were queued.
> +
> +Ordering is important to fences because it can optimize the pipeline with
> +other drivers like a DRM/KMS display driver. For example, if a capture from the
> +camera is happening in an orderly manner one can send the capture buffer
> +out-fence to the DRM/KMS driver and rest sure that the buffers will be shown on

rest sure -> rest assured

But I'd rephrase it as: be certain

> +the screen at the correct order. If an ordered queue can not be set then such
> +arrangements with other drivers may not be possible.
>  
>  Return Value
>  ============
> diff --git a/Documentation/media/uapi/v4l/vidioc-querybuf.rst b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> index dd54747fabc9..9aa3358bee0b 100644
> --- a/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-querybuf.rst
> @@ -44,7 +44,7 @@ and the ``index`` field. Valid index numbers range from zero to the
>  number of buffers allocated with
>  :ref:`VIDIOC_REQBUFS` (struct
>  :c:type:`v4l2_requestbuffers` ``count``) minus
> -one. The ``reserved`` and ``reserved2`` fields must be set to 0. When
> +one. The ``reserved`` field must be set to 0. When
>  using the :ref:`multi-planar API <planar-apis>`, the ``m.planes``
>  field must contain a userspace pointer to an array of struct
>  :c:type:`v4l2_plane` and the ``length`` field has to be set
> @@ -64,8 +64,14 @@ elements will be used instead and the ``length`` field of struct
>  array elements. The driver may or may not set the remaining fields and
>  flags, they are meaningless in this context.
>  
> -The struct :c:type:`v4l2_buffer` structure is specified in
> -:ref:`buffer`.
> +When using in-fences, the ``V4L2_BUF_FLAG_IN_FENCE`` will be set if the
> +in-fence didn't signal at the time of the
> +:ref:`VIDIOC_QUERYBUF`. The ``V4L2_BUF_FLAG_OUT_FENCE`` will be set if
> +the user asked for an out-fence for the buffer and the ``fence_fd``
> +field will be set to the out-fence fd. In case ``V4L2_BUF_FLAG_OUT_FENCE`` is
> +not set ``fence_fd`` will be set to 0 for backward compatibility.
> +
> +The struct :c:type:`v4l2_buffer` structure is specified in :ref:`buffer`.
>  
>  
>  Return Value
> 

Regards,

	Hans

Re: [PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Receive in-fence from userspace and add support for waiting on them
> before queueing the buffer to the driver. Buffers can't be queued to the
> driver before its fences signal. And a buffer can't be queue to the driver

queue -> queued

> out of the order they were queued from userspace. That means that even if
> it fence signal it must wait all other buffers, ahead of it in the queue,

it fence signal -> its fence signals
wait all -> wait for all

> to signal first.
> 
> If the fence for some buffer fails we do not queue it to the driver,
> instead we mark it as error and wait until the previous buffer is done
> to notify userspace of the error. We wait here to deliver the buffers back
> to userspace in order.
> 
> v9:	- rename fence to in_fence in many places
> 	- handle fences signalling with error better (Hans Verkuil)
> 
> v8:	- improve comments and docs (Hans Verkuil)
> 	- fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
> 	- move in-fences code that was in the out-fences patch here (Alex)
> 
> v8:	- improve comments about fences with errors

v9? Two v8 entries?

> 
> v7:
> 	- get rid of the fence array stuff for ordering and just use
> 	get_num_buffers_ready() (Hans)
> 	- fix issue of queuing the buffer twice (Hans)
> 	- avoid the dma_fence_wait() in core_qbuf() (Alex)
> 	- merge preparation commit in
> 
> v6:
> 	- With fences always keep the order userspace queues the buffers.
> 	- Protect in_fence manipulation with a lock (Brian Starkey)
> 	- check if fences have the same context before adding a fence array
> 	- Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
> 	- Clean up fence if __set_in_fence() fails (Brian Starkey)
> 	- treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
> 
> v5:	- use fence_array to keep buffers ordered in vb2 core when
> 	needed (Brian Starkey)
> 	- keep backward compat on the reserved2 field (Brian Starkey)
> 	- protect fence callback removal with lock (Brian Starkey)
> 
> v4:
> 	- Add a comment about dma_fence_add_callback() not returning a
> 	error (Hans)
> 	- Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
> 	- select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
> 	- Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
> 	- Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
> 	-  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
> 	vb2_start_streaming() (Hans)
> 	- set IN_FENCE flags on __fill_v4l2_buffer (Hans)
> 	- Queue buffers to the driver as soon as they are ready (Hans)
> 	- call fill_user_buffer() after queuing the buffer (Hans)
> 	- add err: label to clean up fence
> 	- add dma_fence_wait() before calling vb2_start_streaming()
> 
> v3:	- document fence parameter
> 	- remove ternary if at vb2_qbuf() return (Mauro)
> 	- do not change if conditions behaviour (Mauro)
> 
> v2:
> 	- fix vb2_queue_or_prepare_buf() ret check
> 	- remove check for VB2_MEMORY_DMABUF only (Javier)
> 	- check num of ready buffers to start streaming
> 	- when queueing, start from the first ready buffer
> 	- handle queue cancel
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
>  drivers/media/v4l2-core/Kconfig                 |  33 ++++
>  include/media/videobuf2-core.h                  |  14 +-
>  4 files changed, 248 insertions(+), 30 deletions(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index d3f7bb33a54d..5de5e35cfc40 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  		vb->index = q->num_buffers + buffer;
>  		vb->type = q->type;
>  		vb->memory = memory;
> +		spin_lock_init(&vb->fence_cb_lock);
>  		for (plane = 0; plane < num_planes; ++plane) {
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
> @@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>  }
>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>  
> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  	unsigned long flags;
>  	unsigned int plane;
>  
> -	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> -		return;
> -
> -	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> -		    state != VB2_BUF_STATE_ERROR &&
> -		    state != VB2_BUF_STATE_QUEUED &&
> -		    state != VB2_BUF_STATE_REQUEUEING))
> -		state = VB2_BUF_STATE_ERROR;
>  
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
> @@ -921,6 +914,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>  
>  	spin_lock_irqsave(&q->done_lock, flags);
> +	if (vb->state == VB2_BUF_STATE_ACTIVE)
> +		atomic_dec(&q->owned_by_drv_count);
> +
>  	if (state == VB2_BUF_STATE_QUEUED ||
>  	    state == VB2_BUF_STATE_REQUEUEING) {
>  		vb->state = VB2_BUF_STATE_QUEUED;
> @@ -929,7 +925,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		list_add_tail(&vb->done_entry, &q->done_list);
>  		vb->state = state;
>  	}
> -	atomic_dec(&q->owned_by_drv_count);
> +
>  	spin_unlock_irqrestore(&q->done_lock, flags);
>  
>  	trace_vb2_buf_done(q, vb);
> @@ -946,6 +942,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		wake_up(&q->done_wq);
>  		break;
>  	}
> +
> +}
> +
> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
> +{
> +	if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
> +		return;
> +
> +	if (WARN_ON(state != VB2_BUF_STATE_DONE &&
> +		    state != VB2_BUF_STATE_ERROR &&
> +		    state != VB2_BUF_STATE_QUEUED &&
> +		    state != VB2_BUF_STATE_REQUEUEING))
> +		state = VB2_BUF_STATE_ERROR;
> +
> +	vb2_process_buffer_done(vb, state);
> +
> +	/*
> +	 * Check if there is any buffer with error in the next position of the queue,
> +	 * buffers whose in-fence signaled with error are not queued to the driver
> +	 * and kept on the queue until the buffer before them is done, so to not
> +	 * delivery buffers back to userspace in the wrong order. Here we process

delivery -> deliver

> +	 * any existing buffers with errors and wake up userspace.
> +	 */
> +	for (;;) {
> +		vb = list_next_entry(vb, queued_entry);
> +		if (!vb || vb->state != VB2_BUF_STATE_ERROR)
> +			break;
> +
> +		vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
> +        }
>  }
>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>  
> @@ -1230,6 +1256,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>  {
>  	struct vb2_queue *q = vb->vb2_queue;
>  
> +	if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> +		return;
> +
>  	vb->state = VB2_BUF_STATE_ACTIVE;
>  	atomic_inc(&q->owned_by_drv_count);
>  
> @@ -1281,6 +1310,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>  	return 0;
>  }
>  
> +static int __get_num_ready_buffers(struct vb2_queue *q)
> +{
> +	struct vb2_buffer *vb;
> +	int ready_count = 0;
> +	unsigned long flags;
> +
> +	/* count num of buffers ready in front of the queued_list */
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +		if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
> +			break;
> +		ready_count++;
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +	}
> +
> +	return ready_count;
> +}
> +
>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  {
>  	struct vb2_buffer *vb;
> @@ -1369,9 +1416,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	return ret;
>  }
>  
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
> +{
> +	struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
> +	struct vb2_queue *q = vb->vb2_queue;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +	/*
> +	 * If the fence signals with an error we mark the buffer as such
> +	 * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
> +	 * not queueing it to the driver. However we can't notify the error
> +	 * to userspace right now because, at the time this callback run, QBUF
> +	 * returned already.

returned -> has returned

> +	 * So we delay that to DQBUF time. See comments in vb2_buffer_done()
> +	 * as well.
> +	 */
> +	if (vb->in_fence->error)
> +		vb->state = VB2_BUF_STATE_ERROR;
> +
> +	dma_fence_put(vb->in_fence);
> +	vb->in_fence = NULL;
> +
> +	if (vb->state == VB2_BUF_STATE_ERROR) {
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +		return;
> +	}
> +
> +	if (q->start_streaming_called)
> +		__enqueue_in_driver(vb);
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +}
> +
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence)
>  {
>  	struct vb2_buffer *vb;
> +	unsigned long flags;
>  	int ret;
>  
>  	vb = q->bufs[index];
> @@ -1380,16 +1461,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	case VB2_BUF_STATE_DEQUEUED:
>  		ret = __buf_prepare(vb, pb);
>  		if (ret)
> -			return ret;
> +			goto err;
>  		break;
>  	case VB2_BUF_STATE_PREPARED:
>  		break;
>  	case VB2_BUF_STATE_PREPARING:
>  		dprintk(1, "buffer still being prepared\n");
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err;
>  	default:
>  		dprintk(1, "invalid buffer state %d\n", vb->state);
> -		return -EINVAL;
> +		ret = -EINVAL;
> +		goto err;
>  	}
>  
>  	/*
> @@ -1400,6 +1483,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
>  	vb->state = VB2_BUF_STATE_QUEUED;
> +	vb->in_fence = in_fence;
>  
>  	if (pb)
>  		call_void_bufop(q, copy_timestamp, vb, pb);
> @@ -1407,15 +1491,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	trace_vb2_qbuf(q, vb);
>  
>  	/*
> -	 * If already streaming, give the buffer to driver for processing.
> -	 * If not, the buffer will be given to driver on next streamon.
> +	 * For explicit synchronization: If the fence didn't signal
> +	 * yet we setup a callback to queue the buffer once the fence
> +	 * signals, and then, return successfully. But if the fence

, and then, -> and then (no commas)

> +	 * already signaled we lose the reference we held and queue the
> +	 * buffer to the driver.
>  	 */
> -	if (q->start_streaming_called)
> -		__enqueue_in_driver(vb);
> +	spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +	if (vb->in_fence) {
> +		ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
> +					     vb2_qbuf_fence_cb);
> +		/* is the fence signaled? */
> +		if (ret == -ENOENT) {
> +			dma_fence_put(vb->in_fence);
> +			vb->in_fence = NULL;
> +		} else if (ret) {
> +			goto unlock;
> +		}
> +	}
>  
> -	/* Fill buffer information for the userspace */
> -	if (pb)
> -		call_void_bufop(q, fill_user_buffer, vb, pb);
> +	/*
> +	 * If already streaming and there is no fence to wait on
> +	 * give the buffer to driver for processing.
> +	 */
> +	if (q->start_streaming_called) {
> +		struct vb2_buffer *b;
> +
> +		list_for_each_entry(b, &q->queued_list, queued_entry) {
> +			if (b->state != VB2_BUF_STATE_QUEUED)
> +				continue;
> +			if (b->in_fence)
> +				break;
> +			__enqueue_in_driver(b);
> +		}
> +	}
>  
>  	/*
>  	 * If streamon has been called, and we haven't yet called
> @@ -1424,14 +1533,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	 * then we can finally call start_streaming().
>  	 */
>  	if (q->streaming && !q->start_streaming_called &&
> -	    q->queued_count >= q->min_buffers_needed) {
> +	    __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = vb2_start_streaming(q);
>  		if (ret)
> -			return ret;
> +			goto unlock;
>  	}
>  
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +
> +	/* Fill buffer information for the userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
>  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>  	return 0;
> +
> +unlock:
> +	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +
> +err:
> +	/* Fill buffer information for the userspace */
> +	if (pb)
> +		call_void_bufop(q, fill_user_buffer, vb, pb);
> +
> +	if (vb->in_fence) {
> +		dma_fence_put(vb->in_fence);
> +		vb->in_fence = NULL;
> +	}
> +
> +	return ret;
> +
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>  
> @@ -1642,6 +1773,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
>  static void __vb2_queue_cancel(struct vb2_queue *q)
>  {
>  	unsigned int i;
> +	struct vb2_buffer *vb;
> +	unsigned long flags;
>  
>  	/*
>  	 * Tell driver to stop all transactions and release all queued
> @@ -1672,6 +1805,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	q->queued_count = 0;
>  	q->error = 0;
>  
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> +		if (vb->in_fence) {
> +			dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
> +			dma_fence_put(vb->in_fence);
> +			vb->in_fence = NULL;
> +		}
> +		spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
> +	}
> +
>  	/*
>  	 * Remove all buffers from videobuf's list...
>  	 */
> @@ -1742,7 +1885,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>  	 * Tell driver to start streaming provided sufficient buffers
>  	 * are available.
>  	 */
> -	if (q->queued_count >= q->min_buffers_needed) {
> +	if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>  		ret = v4l_vb2q_enable_media_source(q);
>  		if (ret)
>  			return ret;
> @@ -2264,7 +2407,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2443,7 +2586,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2546,7 +2689,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index ad1e032c3bf5..1df5dd01c0cd 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -23,6 +23,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/v4l2-dev.h>
>  #include <media/v4l2-fh.h>
> @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>  		return -EINVAL;
>  	}
>  
> +	if ((b->fence_fd != 0 && b->fence_fd != -1) &&
> +	    !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
> +		return -EINVAL;
> +	}
> +
> +	if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
> +		dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
> +		return -EINVAL;
> +	}
> +
>  	return __verify_planes_array(q->bufs[b->index], b);
>  }
>  
> @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->fence_fd = 0;
>  	b->reserved = 0;
>  
> +	b->fence_fd = 0;
> +	if (vb->in_fence)
> +		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
> +	else
> +		b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
> +

I was wondering: if the in-fence was already triggered this flag is not set because
vb->in_fence will be NULL. Would it be useful to add a vb2 flag that is set when the
in-fence triggered and report that here somehow? It might be useful for debugging.

Just wondering.

>  	if (q->is_multiplanar) {
>  		/*
>  		 * Fill in plane-related data if userspace provided an array
> @@ -562,6 +579,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	struct dma_fence *in_fence = NULL;
>  	int ret;
>  
>  	if (vb2_fileio_is_active(q)) {
> @@ -570,7 +588,19 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  	}
>  
>  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> +	if (ret)
> +		return ret;
> +
> +	if (b->flags & V4L2_BUF_FLAG_IN_FENCE) {
> +		in_fence = sync_file_get_fence(b->fence_fd);
> +		if (!in_fence) {
> +			dprintk(1, "failed to get in-fence from fd %d\n",
> +				b->fence_fd);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	return vb2_core_qbuf(q, b->index, b, in_fence);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> diff --git a/drivers/media/v4l2-core/Kconfig b/drivers/media/v4l2-core/Kconfig
> index 8e37e7c5e0f7..79dfb5dfd1fc 100644
> --- a/drivers/media/v4l2-core/Kconfig
> +++ b/drivers/media/v4l2-core/Kconfig
> @@ -80,3 +80,36 @@ config VIDEOBUF_DMA_CONTIG
>  config VIDEOBUF_DVB
>  	tristate
>  	select VIDEOBUF_GEN
> +
> +# Used by drivers that need Videobuf2 modules
> +config VIDEOBUF2_CORE
> +	select DMA_SHARED_BUFFER
> +	select SYNC_FILE
> +	tristate
> +
> +config VIDEOBUF2_MEMOPS
> +	tristate
> +	select FRAME_VECTOR
> +
> +config VIDEOBUF2_DMA_CONTIG
> +	tristate
> +	depends on HAS_DMA
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +	select DMA_SHARED_BUFFER
> +
> +config VIDEOBUF2_VMALLOC
> +	tristate
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +	select DMA_SHARED_BUFFER
> +
> +config VIDEOBUF2_DMA_SG
> +	tristate
> +	depends on HAS_DMA
> +	select VIDEOBUF2_CORE
> +	select VIDEOBUF2_MEMOPS
> +
> +config VIDEOBUF2_DVB
> +	tristate
> +	select VIDEOBUF2_CORE
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 46a9e674f7e1..59cd9a6ac168 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -17,6 +17,7 @@
>  #include <linux/poll.h>
>  #include <linux/dma-buf.h>
>  #include <linux/bitops.h>
> +#include <linux/dma-fence.h>
>  
>  #define VB2_MAX_FRAME	(32)
>  #define VB2_MAX_PLANES	(8)
> @@ -255,12 +256,21 @@ struct vb2_buffer {
>  	 * done_entry:		entry on the list that stores all buffers ready
>  	 *			to be dequeued to userspace
>  	 * vb2_plane:		per-plane information; do not change
> +	 * in_fence:		fence received from vb2 client to wait on before
> +	 *			using the buffer (queueing to the driver)
> +	 * fence_cb:		fence callback information
> +	 * fence_cb_lock:	protect callback signal/remove
>  	 */
>  	enum vb2_buffer_state	state;
>  
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>  	struct list_head	queued_entry;
>  	struct list_head	done_entry;
> +
> +	struct dma_fence	*in_fence;
> +	struct dma_fence_cb	fence_cb;
> +	spinlock_t              fence_cb_lock;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -751,6 +761,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * @index:	id number of the buffer
>   * @pb:		buffer structure passed from userspace to
>   *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
> + * @in_fence:	in-fence to wait on before queueing the buffer
>   *
>   * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
>   * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
> @@ -765,7 +776,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   *
>   * Return: returns zero on success; an error code otherwise.
>   */
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
> +		  struct dma_fence *in_fence);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Looks good!

Regards,

	Hans

Re: [PATCH v8 13/13] [media] v4l: Document explicit synchronization behavior

[Hans Verkuil]

On 03/13/2018 08:33 PM, Hans Verkuil wrote:
> On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
>> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>>
>> Add section to VIDIOC_QBUF and VIDIOC_QUERY_BUF about it
>>
>> v6:	- Close some gaps in the docs (Hans)
>>
>> v5:
>> 	- Remove V4L2_CAP_ORDERED
>> 	- Add doc about V4L2_FMT_FLAG_UNORDERED
>>
>> v4:
>> 	- Document ordering behavior for in-fences
>> 	- Document V4L2_CAP_ORDERED capability
>> 	- Remove doc about OUT_FENCE event
>> 	- Document immediate return of out-fence in QBUF
>>
>> v3:
>> 	- make the out_fence refer to the current buffer (Hans)
>> 	- Note what happens when the IN_FENCE is not set (Hans)
>>
>> v2:
>> 	- mention that fences are files (Hans)
>> 	- rework for the new API
>>
>> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>> ---
>>  Documentation/media/uapi/v4l/vidioc-qbuf.rst     | 55 +++++++++++++++++++++++-
>>  Documentation/media/uapi/v4l/vidioc-querybuf.rst | 12 ++++--
>>  2 files changed, 63 insertions(+), 4 deletions(-)
>>
>> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> index 9e448a4aa3aa..371d84966e34 100644
>> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> @@ -54,7 +54,7 @@ When the buffer is intended for output (``type`` is
>>  or ``V4L2_BUF_TYPE_VBI_OUTPUT``) applications must also initialize the
>>  ``bytesused``, ``field`` and ``timestamp`` fields, see :ref:`buffer`
>>  for details. Applications must also set ``flags`` to 0. The
>> -``reserved2`` and ``reserved`` fields must be set to 0. When using the
>> +``reserved`` field must be set to 0. When using the
>>  :ref:`multi-planar API <planar-apis>`, the ``m.planes`` field must
>>  contain a userspace pointer to a filled-in array of struct
>>  :c:type:`v4l2_plane` and the ``length`` field must be set
>> @@ -118,6 +118,59 @@ immediately with an ``EAGAIN`` error code when no buffer is available.
>>  The struct :c:type:`v4l2_buffer` structure is specified in
>>  :ref:`buffer`.
>>  
>> +Explicit Synchronization
>> +------------------------
>> +
>> +Explicit Synchronization allows us to control the synchronization of
>> +shared buffers from userspace by passing fences to the kernel and/or
>> +receiving them from it. Fences passed to the kernel are named in-fences and
>> +the kernel should wait on them to signal before using the buffer. On the other
>> +side, the kernel can create out-fences for the buffers it queues to the
>> +drivers. Out-fences signal when the driver is finished with buffer, i.e., the
>> +buffer is ready. The fences are represented as a file and passed as a file
>> +descriptor to userspace.
>> +
>> +The in-fences are communicated to the kernel at the ``VIDIOC_QBUF`` ioctl
>> +using the ``V4L2_BUF_FLAG_IN_FENCE`` buffer flag and the `fence_fd` field. If
>> +an in-fence needs to be passed to the kernel, `fence_fd` should be set to the
>> +fence file descriptor number and the ``V4L2_BUF_FLAG_IN_FENCE`` should be set
>> +as well. Setting one but not the other will cause ``VIDIOC_QBUF`` to return
>> +with an error.
> 
> This sentence is confusing since it is not clear what 'one' and 'the other' refer
> to. Be specific here. I think it should be 'Setting V4L2_BUF_FLAG_IN_FENCE but not
> fence_fd'.

Ignore this comment.

> 
>  The fence_fd field will be ignored if the
>> +``V4L2_BUF_FLAG_IN_FENCE`` is not set.

Looking at the code, I don't think this is correct. You get an error if IN_FENCE is
not set but fence_fd is. Removing this sentence would fix this and the previous
sentence would make a lot more sense.

>> +
>> +The videobuf2-core will guarantee that all buffers queued with an in-fence will
>> +be queued to the drivers in the same order. Fences may signal out of order, so
>> +this guarantee at videobuf2 is necessary to not change ordering. So when
>> +waiting on a fence to signal all buffers queued after will be also block until
> 
> after -> afterwards
> will be also block -> will also be blocked
> 
>> +that fence signal.
> 
> signal -> signals

What is missing in the documentation is what happens when you mix in-fence buffers
and buffers without an in-fence.

Regards,

	Hans

Re: [PATCH v8 12/13] [media] v4l: Add V4L2_CAP_FENCES to drivers

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> Drivers that use videobuf2 are capable of using fences and
> should report that to userspace.
> 
> The coding style is following what each drivers was already
> doing.

I think this can be simplified for most drivers: you can set this
flag in the v4l_querycap function if vdev->queue is not NULL or if
m2m_ctx is set in struct v4l2_fh.

I believe all non-m2m drivers that use vb2 set vdev->queue. But not
all m2m drivers will set m2m_ctx, so that will need to be checked.

In other words, this way you only need to modify m2m drivers that
do not set m2m_ctx.

Regards,

	Hans

Re: [PATCH v8 10/13] [media] vb2: add out-fence support to QBUF

[Hans Verkuil]

On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
> 
> If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> an out_fence and send its fd to userspace on the fence_fd field as a

on the -> in the

> return arg for the QBUF call.
> 
> The fence is signaled on buffer_done(), when the job on the buffer is
> finished.
> 
> v9:	- remove in-fences changes from this patch (Alex Courbot)
> 	- improve fence context creation (Hans Verkuil)
> 	- clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
> 
> v8:
> 	- return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
> 	- fix crash when checking not using fences in vb2_buffer_done()
> 
> v7:
> 	- merge patch that add the infrastructure to out-fences into
> 	this one (Alex Courbot)
> 	- Do not install the fd if there is no fence. (Alex Courbot)
> 	- do not report error on requeueing, just WARN_ON_ONCE() (Hans)
> 
> v6
> 	- get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
> 	ordering in vb2 for queueing in the driver, so the event is not
> 	necessary anymore and the out_fence_fd is sent back to userspace
> 	on QBUF call return arg
> 	- do not allow requeueing with out-fences, instead mark the buffer
> 	with an error and wake up to userspace.
> 	- send the out_fence_fd back to userspace on the fence_fd field
> 
> v5:
> 	- delay fd_install to DQ_EVENT (Hans)
> 	- if queue is fully ordered send OUT_FENCE event right away
> 	(Brian)
> 	- rename 'q->ordered' to 'q->ordered_in_driver'
> 	- merge change to implement OUT_FENCE event here
> 
> v4:
> 	- return the out_fence_fd in the BUF_QUEUED event(Hans)
> 
> v3:	- add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
> 	- set the OUT_FENCE flag if there is a fence pending (Hans)
> 	- call fd_install() after vb2_core_qbuf() (Hans)
> 	- clean up fence if vb2_core_qbuf() fails (Hans)
> 	- add list to store sync_file and fence for the next queued buffer
> 
> v2: check if the queue is ordered.
> 
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 88 +++++++++++++++++++++++++
>  drivers/media/common/videobuf2/videobuf2-v4l2.c | 20 +++++-
>  include/media/videobuf2-core.h                  | 25 +++++++
>  3 files changed, 132 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 5de5e35cfc40..dd18a9f345c7 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>  
>  #include <media/videobuf2-core.h>
>  #include <media/v4l2-mc.h>
> @@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>  			vb->planes[plane].length = plane_sizes[plane];
>  			vb->planes[plane].min_length = plane_sizes[plane];
>  		}
> +		vb->out_fence_fd = -1;
>  		q->bufs[vb->index] = vb;
>  
>  		/* Allocate video buffer memory for the MMAP type */
> @@ -934,10 +936,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
>  	case VB2_BUF_STATE_QUEUED:
>  		return;
>  	case VB2_BUF_STATE_REQUEUEING:
> +		/* Requeuing with explicit synchronization, spit warning */
> +		WARN_ON_ONCE(vb->out_fence);
> +
>  		if (q->start_streaming_called)
>  			__enqueue_in_driver(vb);
>  		return;
>  	default:
> +		if (vb->out_fence) {
> +			if (state == VB2_BUF_STATE_ERROR)
> +				dma_fence_set_error(vb->out_fence, -EFAULT);

-EFAULT doesn't sound right. -EIO seems more appropriate.

> +			dma_fence_signal(vb->out_fence);
> +			dma_fence_put(vb->out_fence);
> +			vb->out_fence = NULL;
> +			vb->out_fence_fd = -1;
> +		}
> +
>  		/* Inform any processes that may be waiting for buffers */
>  		wake_up(&q->done_wq);
>  		break;
> @@ -1353,6 +1367,62 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>  
> +static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence";
> +}
> +
> +static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
> +{
> +	return "vb2_fence_timeline";
> +}
> +
> +static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
> +{
> +	return true;
> +}
> +
> +static const struct dma_fence_ops vb2_fence_ops = {
> +	.get_driver_name = vb2_fence_get_driver_name,
> +	.get_timeline_name = vb2_fence_get_timeline_name,
> +	.enable_signaling = vb2_fence_enable_signaling,
> +	.wait = dma_fence_default_wait,
> +};
> +
> +int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
> +{
> +	struct vb2_buffer *vb;
> +
> +	vb = q->bufs[index];
> +
> +	vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
> +
> +	if (call_qop(q, is_unordered, q) || !q->queueing_started)
> +		q->out_fence_context = dma_fence_context_alloc(1);

This needs a comment explaining why out_fence_context is only needed
in this case.

> +
> +	vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
> +	if (!vb->out_fence)
> +		return -ENOMEM;
> +
> +	dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
> +		       q->out_fence_context, 1);
> +	if (!vb->out_fence) {
> +		put_unused_fd(vb->out_fence_fd);
> +		return -ENOMEM;
> +	}
> +
> +	vb->sync_file = sync_file_create(vb->out_fence);
> +	if (!vb->sync_file) {
> +		put_unused_fd(vb->out_fence_fd);
> +		dma_fence_put(vb->out_fence);
> +		vb->out_fence = NULL;
> +		return -ENOMEM;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(vb2_setup_out_fence);
> +
>  /*
>   * vb2_start_streaming() - Attempt to start streaming.
>   * @q:		videobuf2 queue
> @@ -1482,6 +1552,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  	list_add_tail(&vb->queued_entry, &q->queued_list);
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
> +	q->queueing_started = 1;
>  	vb->state = VB2_BUF_STATE_QUEUED;
>  	vb->in_fence = in_fence;
>  
> @@ -1545,6 +1616,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
>  
> +	if (vb->out_fence) {
> +		fd_install(vb->out_fence_fd, vb->sync_file->file);
> +		vb->sync_file = NULL;
> +	}
> +
>  	dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>  	return 0;
>  
> @@ -1552,6 +1628,16 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>  	spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>  
>  err:
> +	if (vb->sync_file) {
> +		put_unused_fd(vb->out_fence_fd);
> +		vb->out_fence_fd = -1;
> +
> +		dma_fence_put(vb->out_fence);
> +
> +		fput(vb->sync_file->file);
> +		vb->sync_file = NULL;
> +	}
> +
>  	/* Fill buffer information for the userspace */
>  	if (pb)
>  		call_void_bufop(q, fill_user_buffer, vb, pb);
> @@ -1804,6 +1890,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>  	q->start_streaming_called = 0;
>  	q->queued_count = 0;
>  	q->error = 0;
> +	q->queueing_started = 0;
>  
>  	list_for_each_entry(vb, &q->queued_list, queued_entry) {
>  		spin_lock_irqsave(&vb->fence_cb_lock, flags);
> @@ -2156,6 +2243,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
>  	spin_lock_init(&q->done_lock);
>  	mutex_init(&q->mmap_lock);
>  	init_waitqueue_head(&q->done_wq);
> +	spin_lock_init(&q->out_fence_lock);
>  
>  	q->memory = VB2_MEMORY_UNKNOWN;
>  
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 1df5dd01c0cd..ab5b2b71d784 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -217,7 +217,12 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->sequence = vbuf->sequence;
>  	b->reserved = 0;
>  
> -	b->fence_fd = 0;
> +	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
> +		b->fence_fd = vb->out_fence_fd;
> +	} else {
> +		b->fence_fd = 0;
> +	}
> +
>  	if (vb->in_fence)
>  		b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>  	else
> @@ -496,6 +501,10 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  	ret = __verify_planes_array(vb, b);
>  	if (!ret)
>  		vb2_core_querybuf(q, b->index, b);
> +
> +	/* Do not return the out-fence fd on querybuf */
> +	if (vb->out_fence)
> +		b->fence_fd = -1;
>  	return ret;
>  }
>  EXPORT_SYMBOL(vb2_querybuf);
> @@ -600,6 +609,15 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  		}
>  	}
>  
> +	if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
> +		ret = vb2_setup_out_fence(q, b->index);
> +		if (ret) {
> +			dprintk(1, "failed to set up out-fence\n");
> +			dma_fence_put(in_fence);
> +			return ret;
> +		}
> +	}
> +
>  	return vb2_core_qbuf(q, b->index, b, in_fence);
>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 59cd9a6ac168..2b9d3433e960 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -260,6 +260,10 @@ struct vb2_buffer {
>  	 *			using the buffer (queueing to the driver)
>  	 * fence_cb:		fence callback information
>  	 * fence_cb_lock:	protect callback signal/remove
> +	 * out_fence_fd:	the out_fence_fd to be shared with userspace.
> +	 * out_fence:		the out-fence associated with the buffer once
> +	 *			it is queued to the driver.
> +	 * sync_file:		the sync file to wrap the out fence
>  	 */
>  	enum vb2_buffer_state	state;
>  
> @@ -271,6 +275,10 @@ struct vb2_buffer {
>  	struct dma_fence_cb	fence_cb;
>  	spinlock_t              fence_cb_lock;
>  
> +	int			out_fence_fd;
> +	struct dma_fence	*out_fence;
> +	struct sync_file	*sync_file;
> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these buffer-related ops are
> @@ -515,6 +523,9 @@ struct vb2_buf_ops {
>   * @last_buffer_dequeued: used in poll() and DQBUF to immediately return if the
>   *		last decoded buffer was already dequeued. Set for capture queues
>   *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
> + * @queueing_started: if queueing has started. Currently used to determine
> + *		if an out_fence_context is needed.
> + * @out_fence_context: the fence context for the out fences
>   * @fileio:	file io emulator internal data, used only if emulator is active
>   * @threadio:	thread io internal data, used only if thread is active
>   */
> @@ -567,6 +578,10 @@ struct vb2_queue {
>  	unsigned int			is_output:1;
>  	unsigned int			copy_timestamp:1;
>  	unsigned int			last_buffer_dequeued:1;
> +	unsigned int			queueing_started:1;
> +
> +	u64				out_fence_context;
> +	spinlock_t			out_fence_lock;
>  
>  	struct vb2_fileio_data		*fileio;
>  	struct vb2_threadio_data	*threadio;
> @@ -754,6 +769,16 @@ int vb2_core_create_bufs(struct vb2_queue *q, enum vb2_memory memory,
>   */
>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>  
> +/**
> + * vb2_setup_out_fence() - setup new out-fence
> + * @q:		The vb2_queue where to setup it
> + * @index:	index of the buffer
> + *
> + * Setup the file descriptor, the fence and the sync_file for the next
> + * buffer to be queued and add everything to the tail of the q->out_fence_list.
> + */
> +int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index);
> +
>  /**
>   * vb2_core_qbuf() - Queue a buffer from userspace
>   *
> 

Looks good!

Regards,

	Hans

Re: [PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Ezequiel Garcia]

On 14 March 2018 at 12:55, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
>> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>>
>> Receive in-fence from userspace and add support for waiting on them
>> before queueing the buffer to the driver. Buffers can't be queued to the
>> driver before its fences signal. And a buffer can't be queue to the driver
>
> queue -> queued
>
>> out of the order they were queued from userspace. That means that even if
>> it fence signal it must wait all other buffers, ahead of it in the queue,
>
> it fence signal -> its fence signals
> wait all -> wait for all
>
>> to signal first.
>>
>> If the fence for some buffer fails we do not queue it to the driver,
>> instead we mark it as error and wait until the previous buffer is done
>> to notify userspace of the error. We wait here to deliver the buffers back
>> to userspace in order.
>>
>> v9:   - rename fence to in_fence in many places
>>       - handle fences signalling with error better (Hans Verkuil)
>>
>> v8:   - improve comments and docs (Hans Verkuil)
>>       - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>>       - move in-fences code that was in the out-fences patch here (Alex)
>>
>> v8:   - improve comments about fences with errors
>
> v9? Two v8 entries?
>
>>
>> v7:
>>       - get rid of the fence array stuff for ordering and just use
>>       get_num_buffers_ready() (Hans)
>>       - fix issue of queuing the buffer twice (Hans)
>>       - avoid the dma_fence_wait() in core_qbuf() (Alex)
>>       - merge preparation commit in
>>
>> v6:
>>       - With fences always keep the order userspace queues the buffers.
>>       - Protect in_fence manipulation with a lock (Brian Starkey)
>>       - check if fences have the same context before adding a fence array
>>       - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>>       - Clean up fence if __set_in_fence() fails (Brian Starkey)
>>       - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
>>
>> v5:   - use fence_array to keep buffers ordered in vb2 core when
>>       needed (Brian Starkey)
>>       - keep backward compat on the reserved2 field (Brian Starkey)
>>       - protect fence callback removal with lock (Brian Starkey)
>>
>> v4:
>>       - Add a comment about dma_fence_add_callback() not returning a
>>       error (Hans)
>>       - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>>       - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>>       - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>>       - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>>       -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>>       vb2_start_streaming() (Hans)
>>       - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>>       - Queue buffers to the driver as soon as they are ready (Hans)
>>       - call fill_user_buffer() after queuing the buffer (Hans)
>>       - add err: label to clean up fence
>>       - add dma_fence_wait() before calling vb2_start_streaming()
>>
>> v3:   - document fence parameter
>>       - remove ternary if at vb2_qbuf() return (Mauro)
>>       - do not change if conditions behaviour (Mauro)
>>
>> v2:
>>       - fix vb2_queue_or_prepare_buf() ret check
>>       - remove check for VB2_MEMORY_DMABUF only (Javier)
>>       - check num of ready buffers to start streaming
>>       - when queueing, start from the first ready buffer
>>       - handle queue cancel
>>
>> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>> ---
>>  drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
>>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
>>  drivers/media/v4l2-core/Kconfig                 |  33 ++++
>>  include/media/videobuf2-core.h                  |  14 +-
>>  4 files changed, 248 insertions(+), 30 deletions(-)
>>
>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>> index d3f7bb33a54d..5de5e35cfc40 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-core.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
>> @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>>               vb->index = q->num_buffers + buffer;
>>               vb->type = q->type;
>>               vb->memory = memory;
>> +             spin_lock_init(&vb->fence_cb_lock);
>>               for (plane = 0; plane < num_planes; ++plane) {
>>                       vb->planes[plane].length = plane_sizes[plane];
>>                       vb->planes[plane].min_length = plane_sizes[plane];
>> @@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>>
>> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>  {
>>       struct vb2_queue *q = vb->vb2_queue;
>>       unsigned long flags;
>>       unsigned int plane;
>>
>> -     if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>> -             return;
>> -
>> -     if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>> -                 state != VB2_BUF_STATE_ERROR &&
>> -                 state != VB2_BUF_STATE_QUEUED &&
>> -                 state != VB2_BUF_STATE_REQUEUEING))
>> -             state = VB2_BUF_STATE_ERROR;
>>
>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>       /*
>> @@ -921,6 +914,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>               call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>>
>>       spin_lock_irqsave(&q->done_lock, flags);
>> +     if (vb->state == VB2_BUF_STATE_ACTIVE)
>> +             atomic_dec(&q->owned_by_drv_count);
>> +
>>       if (state == VB2_BUF_STATE_QUEUED ||
>>           state == VB2_BUF_STATE_REQUEUEING) {
>>               vb->state = VB2_BUF_STATE_QUEUED;
>> @@ -929,7 +925,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>               list_add_tail(&vb->done_entry, &q->done_list);
>>               vb->state = state;
>>       }
>> -     atomic_dec(&q->owned_by_drv_count);
>> +
>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>
>>       trace_vb2_buf_done(q, vb);
>> @@ -946,6 +942,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>               wake_up(&q->done_wq);
>>               break;
>>       }
>> +
>> +}
>> +
>> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>> +{
>> +     if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>> +             return;
>> +
>> +     if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>> +                 state != VB2_BUF_STATE_ERROR &&
>> +                 state != VB2_BUF_STATE_QUEUED &&
>> +                 state != VB2_BUF_STATE_REQUEUEING))
>> +             state = VB2_BUF_STATE_ERROR;
>> +
>> +     vb2_process_buffer_done(vb, state);
>> +
>> +     /*
>> +      * Check if there is any buffer with error in the next position of the queue,
>> +      * buffers whose in-fence signaled with error are not queued to the driver
>> +      * and kept on the queue until the buffer before them is done, so to not
>> +      * delivery buffers back to userspace in the wrong order. Here we process
>
> delivery -> deliver
>
>> +      * any existing buffers with errors and wake up userspace.
>> +      */
>> +     for (;;) {
>> +             vb = list_next_entry(vb, queued_entry);
>> +             if (!vb || vb->state != VB2_BUF_STATE_ERROR)
>> +                     break;
>> +
>> +             vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
>> +        }
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>>
>> @@ -1230,6 +1256,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>>  {
>>       struct vb2_queue *q = vb->vb2_queue;
>>
>> +     if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>> +             return;
>> +
>>       vb->state = VB2_BUF_STATE_ACTIVE;
>>       atomic_inc(&q->owned_by_drv_count);
>>
>> @@ -1281,6 +1310,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>>       return 0;
>>  }
>>
>> +static int __get_num_ready_buffers(struct vb2_queue *q)
>> +{
>> +     struct vb2_buffer *vb;
>> +     int ready_count = 0;
>> +     unsigned long flags;
>> +
>> +     /* count num of buffers ready in front of the queued_list */
>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +             spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +             if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>> +                     break;
>> +             ready_count++;
>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +     }
>> +
>> +     return ready_count;
>> +}
>> +
>>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>  {
>>       struct vb2_buffer *vb;
>> @@ -1369,9 +1416,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>       return ret;
>>  }
>>
>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>> +{
>> +     struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
>> +     struct vb2_queue *q = vb->vb2_queue;
>> +     unsigned long flags;
>> +
>> +     spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +     /*
>> +      * If the fence signals with an error we mark the buffer as such
>> +      * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
>> +      * not queueing it to the driver. However we can't notify the error
>> +      * to userspace right now because, at the time this callback run, QBUF
>> +      * returned already.
>
> returned -> has returned
>
>> +      * So we delay that to DQBUF time. See comments in vb2_buffer_done()
>> +      * as well.
>> +      */
>> +     if (vb->in_fence->error)
>> +             vb->state = VB2_BUF_STATE_ERROR;
>> +
>> +     dma_fence_put(vb->in_fence);
>> +     vb->in_fence = NULL;
>> +
>> +     if (vb->state == VB2_BUF_STATE_ERROR) {
>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +             return;
>> +     }
>> +
>> +     if (q->start_streaming_called)
>> +             __enqueue_in_driver(vb);
>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +}
>> +
>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>> +               struct dma_fence *in_fence)
>>  {
>>       struct vb2_buffer *vb;
>> +     unsigned long flags;
>>       int ret;
>>
>>       vb = q->bufs[index];
>> @@ -1380,16 +1461,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>       case VB2_BUF_STATE_DEQUEUED:
>>               ret = __buf_prepare(vb, pb);
>>               if (ret)
>> -                     return ret;
>> +                     goto err;
>>               break;
>>       case VB2_BUF_STATE_PREPARED:
>>               break;
>>       case VB2_BUF_STATE_PREPARING:
>>               dprintk(1, "buffer still being prepared\n");
>> -             return -EINVAL;
>> +             ret = -EINVAL;
>> +             goto err;
>>       default:
>>               dprintk(1, "invalid buffer state %d\n", vb->state);
>> -             return -EINVAL;
>> +             ret = -EINVAL;
>> +             goto err;
>>       }
>>
>>       /*
>> @@ -1400,6 +1483,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>       q->queued_count++;
>>       q->waiting_for_buffers = false;
>>       vb->state = VB2_BUF_STATE_QUEUED;
>> +     vb->in_fence = in_fence;
>>
>>       if (pb)
>>               call_void_bufop(q, copy_timestamp, vb, pb);
>> @@ -1407,15 +1491,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>       trace_vb2_qbuf(q, vb);
>>
>>       /*
>> -      * If already streaming, give the buffer to driver for processing.
>> -      * If not, the buffer will be given to driver on next streamon.
>> +      * For explicit synchronization: If the fence didn't signal
>> +      * yet we setup a callback to queue the buffer once the fence
>> +      * signals, and then, return successfully. But if the fence
>
> , and then, -> and then (no commas)
>
>> +      * already signaled we lose the reference we held and queue the
>> +      * buffer to the driver.
>>        */
>> -     if (q->start_streaming_called)
>> -             __enqueue_in_driver(vb);
>> +     spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +     if (vb->in_fence) {
>> +             ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
>> +                                          vb2_qbuf_fence_cb);
>> +             /* is the fence signaled? */
>> +             if (ret == -ENOENT) {
>> +                     dma_fence_put(vb->in_fence);
>> +                     vb->in_fence = NULL;
>> +             } else if (ret) {
>> +                     goto unlock;
>> +             }
>> +     }
>>
>> -     /* Fill buffer information for the userspace */
>> -     if (pb)
>> -             call_void_bufop(q, fill_user_buffer, vb, pb);
>> +     /*
>> +      * If already streaming and there is no fence to wait on
>> +      * give the buffer to driver for processing.
>> +      */
>> +     if (q->start_streaming_called) {
>> +             struct vb2_buffer *b;
>> +
>> +             list_for_each_entry(b, &q->queued_list, queued_entry) {
>> +                     if (b->state != VB2_BUF_STATE_QUEUED)
>> +                             continue;
>> +                     if (b->in_fence)
>> +                             break;
>> +                     __enqueue_in_driver(b);
>> +             }
>> +     }
>>
>>       /*
>>        * If streamon has been called, and we haven't yet called
>> @@ -1424,14 +1533,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>        * then we can finally call start_streaming().
>>        */
>>       if (q->streaming && !q->start_streaming_called &&
>> -         q->queued_count >= q->min_buffers_needed) {
>> +         __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>>               ret = vb2_start_streaming(q);
>>               if (ret)
>> -                     return ret;
>> +                     goto unlock;
>>       }
>>
>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +
>> +     /* Fill buffer information for the userspace */
>> +     if (pb)
>> +             call_void_bufop(q, fill_user_buffer, vb, pb);
>> +
>>       dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>>       return 0;
>> +
>> +unlock:
>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +
>> +err:
>> +     /* Fill buffer information for the userspace */
>> +     if (pb)
>> +             call_void_bufop(q, fill_user_buffer, vb, pb);
>> +
>> +     if (vb->in_fence) {
>> +             dma_fence_put(vb->in_fence);
>> +             vb->in_fence = NULL;
>> +     }
>> +
>> +     return ret;
>> +
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>
>> @@ -1642,6 +1773,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
>>  static void __vb2_queue_cancel(struct vb2_queue *q)
>>  {
>>       unsigned int i;
>> +     struct vb2_buffer *vb;
>> +     unsigned long flags;
>>
>>       /*
>>        * Tell driver to stop all transactions and release all queued
>> @@ -1672,6 +1805,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>>       q->queued_count = 0;
>>       q->error = 0;
>>
>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +             spin_lock_irqsave(&vb->fence_cb_lock, flags);
>> +             if (vb->in_fence) {
>> +                     dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
>> +                     dma_fence_put(vb->in_fence);
>> +                     vb->in_fence = NULL;
>> +             }
>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>> +     }
>> +
>>       /*
>>        * Remove all buffers from videobuf's list...
>>        */
>> @@ -1742,7 +1885,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>>        * Tell driver to start streaming provided sufficient buffers
>>        * are available.
>>        */
>> -     if (q->queued_count >= q->min_buffers_needed) {
>> +     if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>>               ret = v4l_vb2q_enable_media_source(q);
>>               if (ret)
>>                       return ret;
>> @@ -2264,7 +2407,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>>                * Queue all buffers.
>>                */
>>               for (i = 0; i < q->num_buffers; i++) {
>> -                     ret = vb2_core_qbuf(q, i, NULL);
>> +                     ret = vb2_core_qbuf(q, i, NULL, NULL);
>>                       if (ret)
>>                               goto err_reqbufs;
>>                       fileio->bufs[i].queued = 1;
>> @@ -2443,7 +2586,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>>
>>               if (copy_timestamp)
>>                       b->timestamp = ktime_get_ns();
>> -             ret = vb2_core_qbuf(q, index, NULL);
>> +             ret = vb2_core_qbuf(q, index, NULL, NULL);
>>               dprintk(5, "vb2_dbuf result: %d\n", ret);
>>               if (ret)
>>                       return ret;
>> @@ -2546,7 +2689,7 @@ static int vb2_thread(void *data)
>>               if (copy_timestamp)
>>                       vb->timestamp = ktime_get_ns();
>>               if (!threadio->stop)
>> -                     ret = vb2_core_qbuf(q, vb->index, NULL);
>> +                     ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>>               call_void_qop(q, wait_prepare, q);
>>               if (ret || threadio->stop)
>>                       break;
>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> index ad1e032c3bf5..1df5dd01c0cd 100644
>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>> @@ -23,6 +23,7 @@
>>  #include <linux/sched.h>
>>  #include <linux/freezer.h>
>>  #include <linux/kthread.h>
>> +#include <linux/sync_file.h>
>>
>>  #include <media/v4l2-dev.h>
>>  #include <media/v4l2-fh.h>
>> @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>>               return -EINVAL;
>>       }
>>
>> +     if ((b->fence_fd != 0 && b->fence_fd != -1) &&
>> +         !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>> +             dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
>> +             return -EINVAL;
>> +     }
>> +
>> +     if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>> +             dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
>> +             return -EINVAL;
>> +     }
>> +
>>       return __verify_planes_array(q->bufs[b->index], b);
>>  }
>>
>> @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>>       b->timestamp = ns_to_timeval(vb->timestamp);
>>       b->timecode = vbuf->timecode;
>>       b->sequence = vbuf->sequence;
>> -     b->fence_fd = 0;
>>       b->reserved = 0;
>>
>> +     b->fence_fd = 0;
>> +     if (vb->in_fence)
>> +             b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>> +     else
>> +             b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
>> +
>
> I was wondering: if the in-fence was already triggered this flag is not set because
> vb->in_fence will be NULL. Would it be useful to add a vb2 flag that is set when the
> in-fence triggered and report that here somehow? It might be useful for debugging.
>
> Just wondering.
>

I'm also wondering :-) ... does it make sense to use a flag for this?

#define V4L2_BUF_FLAG_IN_FENCE                  0x00200000
#define V4L2_BUF_FLAG_OUT_FENCE                 0x00400000
#define V4L2_BUF_FLAG_IN_FENCE_SIGNALED         0x00800000

Aren't we using a very limited resource? Is it worry?

Thanks,
-- 
Ezequiel García, VanguardiaSur
www.vanguardiasur.com.ar

Re: [PATCH v8 09/13] [media] vb2: add in-fence support to QBUF

[Hans Verkuil]

On 04/25/2018 10:11 PM, Ezequiel Garcia wrote:
> On 14 March 2018 at 12:55, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 03/09/2018 09:49 AM, Gustavo Padovan wrote:
>>> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>>>
>>> Receive in-fence from userspace and add support for waiting on them
>>> before queueing the buffer to the driver. Buffers can't be queued to the
>>> driver before its fences signal. And a buffer can't be queue to the driver
>>
>> queue -> queued
>>
>>> out of the order they were queued from userspace. That means that even if
>>> it fence signal it must wait all other buffers, ahead of it in the queue,
>>
>> it fence signal -> its fence signals
>> wait all -> wait for all
>>
>>> to signal first.
>>>
>>> If the fence for some buffer fails we do not queue it to the driver,
>>> instead we mark it as error and wait until the previous buffer is done
>>> to notify userspace of the error. We wait here to deliver the buffers back
>>> to userspace in order.
>>>
>>> v9:   - rename fence to in_fence in many places
>>>       - handle fences signalling with error better (Hans Verkuil)
>>>
>>> v8:   - improve comments and docs (Hans Verkuil)
>>>       - fix unlocking of vb->fence_cb_lock on vb2_core_qbuf (Hans Verkuil)
>>>       - move in-fences code that was in the out-fences patch here (Alex)
>>>
>>> v8:   - improve comments about fences with errors
>>
>> v9? Two v8 entries?
>>
>>>
>>> v7:
>>>       - get rid of the fence array stuff for ordering and just use
>>>       get_num_buffers_ready() (Hans)
>>>       - fix issue of queuing the buffer twice (Hans)
>>>       - avoid the dma_fence_wait() in core_qbuf() (Alex)
>>>       - merge preparation commit in
>>>
>>> v6:
>>>       - With fences always keep the order userspace queues the buffers.
>>>       - Protect in_fence manipulation with a lock (Brian Starkey)
>>>       - check if fences have the same context before adding a fence array
>>>       - Fix last_fence ref unbalance in __set_in_fence() (Brian Starkey)
>>>       - Clean up fence if __set_in_fence() fails (Brian Starkey)
>>>       - treat -EINVAL from dma_fence_add_callback() (Brian Starkey)
>>>
>>> v5:   - use fence_array to keep buffers ordered in vb2 core when
>>>       needed (Brian Starkey)
>>>       - keep backward compat on the reserved2 field (Brian Starkey)
>>>       - protect fence callback removal with lock (Brian Starkey)
>>>
>>> v4:
>>>       - Add a comment about dma_fence_add_callback() not returning a
>>>       error (Hans)
>>>       - Call dma_fence_put(vb->in_fence) if fence signaled (Hans)
>>>       - select SYNC_FILE under config VIDEOBUF2_CORE (Hans)
>>>       - Move dma_fence_is_signaled() check to __enqueue_in_driver() (Hans)
>>>       - Remove list_for_each_entry() in __vb2_core_qbuf() (Hans)
>>>       -  Remove if (vb->state != VB2_BUF_STATE_QUEUED) from
>>>       vb2_start_streaming() (Hans)
>>>       - set IN_FENCE flags on __fill_v4l2_buffer (Hans)
>>>       - Queue buffers to the driver as soon as they are ready (Hans)
>>>       - call fill_user_buffer() after queuing the buffer (Hans)
>>>       - add err: label to clean up fence
>>>       - add dma_fence_wait() before calling vb2_start_streaming()
>>>
>>> v3:   - document fence parameter
>>>       - remove ternary if at vb2_qbuf() return (Mauro)
>>>       - do not change if conditions behaviour (Mauro)
>>>
>>> v2:
>>>       - fix vb2_queue_or_prepare_buf() ret check
>>>       - remove check for VB2_MEMORY_DMABUF only (Javier)
>>>       - check num of ready buffers to start streaming
>>>       - when queueing, start from the first ready buffer
>>>       - handle queue cancel
>>>
>>> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
>>> ---
>>>  drivers/media/common/videobuf2/videobuf2-core.c | 197 ++++++++++++++++++++----
>>>  drivers/media/common/videobuf2/videobuf2-v4l2.c |  34 +++-
>>>  drivers/media/v4l2-core/Kconfig                 |  33 ++++
>>>  include/media/videobuf2-core.h                  |  14 +-
>>>  4 files changed, 248 insertions(+), 30 deletions(-)
>>>
>>> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
>>> index d3f7bb33a54d..5de5e35cfc40 100644
>>> --- a/drivers/media/common/videobuf2/videobuf2-core.c
>>> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
>>> @@ -352,6 +352,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>>>               vb->index = q->num_buffers + buffer;
>>>               vb->type = q->type;
>>>               vb->memory = memory;
>>> +             spin_lock_init(&vb->fence_cb_lock);
>>>               for (plane = 0; plane < num_planes; ++plane) {
>>>                       vb->planes[plane].length = plane_sizes[plane];
>>>                       vb->planes[plane].min_length = plane_sizes[plane];
>>> @@ -891,20 +892,12 @@ void *vb2_plane_cookie(struct vb2_buffer *vb, unsigned int plane_no)
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_plane_cookie);
>>>
>>> -void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>> +static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>  {
>>>       struct vb2_queue *q = vb->vb2_queue;
>>>       unsigned long flags;
>>>       unsigned int plane;
>>>
>>> -     if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>>> -             return;
>>> -
>>> -     if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>>> -                 state != VB2_BUF_STATE_ERROR &&
>>> -                 state != VB2_BUF_STATE_QUEUED &&
>>> -                 state != VB2_BUF_STATE_REQUEUEING))
>>> -             state = VB2_BUF_STATE_ERROR;
>>>
>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>       /*
>>> @@ -921,6 +914,9 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>               call_void_memop(vb, finish, vb->planes[plane].mem_priv);
>>>
>>>       spin_lock_irqsave(&q->done_lock, flags);
>>> +     if (vb->state == VB2_BUF_STATE_ACTIVE)
>>> +             atomic_dec(&q->owned_by_drv_count);
>>> +
>>>       if (state == VB2_BUF_STATE_QUEUED ||
>>>           state == VB2_BUF_STATE_REQUEUEING) {
>>>               vb->state = VB2_BUF_STATE_QUEUED;
>>> @@ -929,7 +925,7 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>               vb->state = state;
>>>       }
>>> -     atomic_dec(&q->owned_by_drv_count);
>>> +
>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>>
>>>       trace_vb2_buf_done(q, vb);
>>> @@ -946,6 +942,36 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>               wake_up(&q->done_wq);
>>>               break;
>>>       }
>>> +
>>> +}
>>> +
>>> +void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>> +{
>>> +     if (WARN_ON(vb->state != VB2_BUF_STATE_ACTIVE))
>>> +             return;
>>> +
>>> +     if (WARN_ON(state != VB2_BUF_STATE_DONE &&
>>> +                 state != VB2_BUF_STATE_ERROR &&
>>> +                 state != VB2_BUF_STATE_QUEUED &&
>>> +                 state != VB2_BUF_STATE_REQUEUEING))
>>> +             state = VB2_BUF_STATE_ERROR;
>>> +
>>> +     vb2_process_buffer_done(vb, state);
>>> +
>>> +     /*
>>> +      * Check if there is any buffer with error in the next position of the queue,
>>> +      * buffers whose in-fence signaled with error are not queued to the driver
>>> +      * and kept on the queue until the buffer before them is done, so to not
>>> +      * delivery buffers back to userspace in the wrong order. Here we process
>>
>> delivery -> deliver
>>
>>> +      * any existing buffers with errors and wake up userspace.
>>> +      */
>>> +     for (;;) {
>>> +             vb = list_next_entry(vb, queued_entry);
>>> +             if (!vb || vb->state != VB2_BUF_STATE_ERROR)
>>> +                     break;
>>> +
>>> +             vb2_process_buffer_done(vb, VB2_BUF_STATE_ERROR);
>>> +        }
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_buffer_done);
>>>
>>> @@ -1230,6 +1256,9 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
>>>  {
>>>       struct vb2_queue *q = vb->vb2_queue;
>>>
>>> +     if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>>> +             return;
>>> +
>>>       vb->state = VB2_BUF_STATE_ACTIVE;
>>>       atomic_inc(&q->owned_by_drv_count);
>>>
>>> @@ -1281,6 +1310,24 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
>>>       return 0;
>>>  }
>>>
>>> +static int __get_num_ready_buffers(struct vb2_queue *q)
>>> +{
>>> +     struct vb2_buffer *vb;
>>> +     int ready_count = 0;
>>> +     unsigned long flags;
>>> +
>>> +     /* count num of buffers ready in front of the queued_list */
>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>> +             spin_lock_irqsave(&vb->fence_cb_lock, flags);
>>> +             if (vb->in_fence && !dma_fence_is_signaled(vb->in_fence))
>>> +                     break;
>>> +             ready_count++;
>>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +     }
>>> +
>>> +     return ready_count;
>>> +}
>>> +
>>>  int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>  {
>>>       struct vb2_buffer *vb;
>>> @@ -1369,9 +1416,43 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>       return ret;
>>>  }
>>>
>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>> +static void vb2_qbuf_fence_cb(struct dma_fence *f, struct dma_fence_cb *cb)
>>> +{
>>> +     struct vb2_buffer *vb = container_of(cb, struct vb2_buffer, fence_cb);
>>> +     struct vb2_queue *q = vb->vb2_queue;
>>> +     unsigned long flags;
>>> +
>>> +     spin_lock_irqsave(&vb->fence_cb_lock, flags);
>>> +     /*
>>> +      * If the fence signals with an error we mark the buffer as such
>>> +      * and avoid using it by setting it to VB2_BUF_STATE_ERROR and
>>> +      * not queueing it to the driver. However we can't notify the error
>>> +      * to userspace right now because, at the time this callback run, QBUF
>>> +      * returned already.
>>
>> returned -> has returned
>>
>>> +      * So we delay that to DQBUF time. See comments in vb2_buffer_done()
>>> +      * as well.
>>> +      */
>>> +     if (vb->in_fence->error)
>>> +             vb->state = VB2_BUF_STATE_ERROR;
>>> +
>>> +     dma_fence_put(vb->in_fence);
>>> +     vb->in_fence = NULL;
>>> +
>>> +     if (vb->state == VB2_BUF_STATE_ERROR) {
>>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +             return;
>>> +     }
>>> +
>>> +     if (q->start_streaming_called)
>>> +             __enqueue_in_driver(vb);
>>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +}
>>> +
>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>>> +               struct dma_fence *in_fence)
>>>  {
>>>       struct vb2_buffer *vb;
>>> +     unsigned long flags;
>>>       int ret;
>>>
>>>       vb = q->bufs[index];
>>> @@ -1380,16 +1461,18 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>       case VB2_BUF_STATE_DEQUEUED:
>>>               ret = __buf_prepare(vb, pb);
>>>               if (ret)
>>> -                     return ret;
>>> +                     goto err;
>>>               break;
>>>       case VB2_BUF_STATE_PREPARED:
>>>               break;
>>>       case VB2_BUF_STATE_PREPARING:
>>>               dprintk(1, "buffer still being prepared\n");
>>> -             return -EINVAL;
>>> +             ret = -EINVAL;
>>> +             goto err;
>>>       default:
>>>               dprintk(1, "invalid buffer state %d\n", vb->state);
>>> -             return -EINVAL;
>>> +             ret = -EINVAL;
>>> +             goto err;
>>>       }
>>>
>>>       /*
>>> @@ -1400,6 +1483,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>       q->queued_count++;
>>>       q->waiting_for_buffers = false;
>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>> +     vb->in_fence = in_fence;
>>>
>>>       if (pb)
>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>> @@ -1407,15 +1491,40 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>       trace_vb2_qbuf(q, vb);
>>>
>>>       /*
>>> -      * If already streaming, give the buffer to driver for processing.
>>> -      * If not, the buffer will be given to driver on next streamon.
>>> +      * For explicit synchronization: If the fence didn't signal
>>> +      * yet we setup a callback to queue the buffer once the fence
>>> +      * signals, and then, return successfully. But if the fence
>>
>> , and then, -> and then (no commas)
>>
>>> +      * already signaled we lose the reference we held and queue the
>>> +      * buffer to the driver.
>>>        */
>>> -     if (q->start_streaming_called)
>>> -             __enqueue_in_driver(vb);
>>> +     spin_lock_irqsave(&vb->fence_cb_lock, flags);
>>> +     if (vb->in_fence) {
>>> +             ret = dma_fence_add_callback(vb->in_fence, &vb->fence_cb,
>>> +                                          vb2_qbuf_fence_cb);
>>> +             /* is the fence signaled? */
>>> +             if (ret == -ENOENT) {
>>> +                     dma_fence_put(vb->in_fence);
>>> +                     vb->in_fence = NULL;
>>> +             } else if (ret) {
>>> +                     goto unlock;
>>> +             }
>>> +     }
>>>
>>> -     /* Fill buffer information for the userspace */
>>> -     if (pb)
>>> -             call_void_bufop(q, fill_user_buffer, vb, pb);
>>> +     /*
>>> +      * If already streaming and there is no fence to wait on
>>> +      * give the buffer to driver for processing.
>>> +      */
>>> +     if (q->start_streaming_called) {
>>> +             struct vb2_buffer *b;
>>> +
>>> +             list_for_each_entry(b, &q->queued_list, queued_entry) {
>>> +                     if (b->state != VB2_BUF_STATE_QUEUED)
>>> +                             continue;
>>> +                     if (b->in_fence)
>>> +                             break;
>>> +                     __enqueue_in_driver(b);
>>> +             }
>>> +     }
>>>
>>>       /*
>>>        * If streamon has been called, and we haven't yet called
>>> @@ -1424,14 +1533,36 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>        * then we can finally call start_streaming().
>>>        */
>>>       if (q->streaming && !q->start_streaming_called &&
>>> -         q->queued_count >= q->min_buffers_needed) {
>>> +         __get_num_ready_buffers(q) >= q->min_buffers_needed) {
>>>               ret = vb2_start_streaming(q);
>>>               if (ret)
>>> -                     return ret;
>>> +                     goto unlock;
>>>       }
>>>
>>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +
>>> +     /* Fill buffer information for the userspace */
>>> +     if (pb)
>>> +             call_void_bufop(q, fill_user_buffer, vb, pb);
>>> +
>>>       dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>>>       return 0;
>>> +
>>> +unlock:
>>> +     spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +
>>> +err:
>>> +     /* Fill buffer information for the userspace */
>>> +     if (pb)
>>> +             call_void_bufop(q, fill_user_buffer, vb, pb);
>>> +
>>> +     if (vb->in_fence) {
>>> +             dma_fence_put(vb->in_fence);
>>> +             vb->in_fence = NULL;
>>> +     }
>>> +
>>> +     return ret;
>>> +
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>
>>> @@ -1642,6 +1773,8 @@ EXPORT_SYMBOL_GPL(vb2_core_dqbuf);
>>>  static void __vb2_queue_cancel(struct vb2_queue *q)
>>>  {
>>>       unsigned int i;
>>> +     struct vb2_buffer *vb;
>>> +     unsigned long flags;
>>>
>>>       /*
>>>        * Tell driver to stop all transactions and release all queued
>>> @@ -1672,6 +1805,16 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>>>       q->queued_count = 0;
>>>       q->error = 0;
>>>
>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>> +             spin_lock_irqsave(&vb->fence_cb_lock, flags);
>>> +             if (vb->in_fence) {
>>> +                     dma_fence_remove_callback(vb->in_fence, &vb->fence_cb);
>>> +                     dma_fence_put(vb->in_fence);
>>> +                     vb->in_fence = NULL;
>>> +             }
>>> +             spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>>> +     }
>>> +
>>>       /*
>>>        * Remove all buffers from videobuf's list...
>>>        */
>>> @@ -1742,7 +1885,7 @@ int vb2_core_streamon(struct vb2_queue *q, unsigned int type)
>>>        * Tell driver to start streaming provided sufficient buffers
>>>        * are available.
>>>        */
>>> -     if (q->queued_count >= q->min_buffers_needed) {
>>> +     if (__get_num_ready_buffers(q) >= q->min_buffers_needed) {
>>>               ret = v4l_vb2q_enable_media_source(q);
>>>               if (ret)
>>>                       return ret;
>>> @@ -2264,7 +2407,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>>>                * Queue all buffers.
>>>                */
>>>               for (i = 0; i < q->num_buffers; i++) {
>>> -                     ret = vb2_core_qbuf(q, i, NULL);
>>> +                     ret = vb2_core_qbuf(q, i, NULL, NULL);
>>>                       if (ret)
>>>                               goto err_reqbufs;
>>>                       fileio->bufs[i].queued = 1;
>>> @@ -2443,7 +2586,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>>>
>>>               if (copy_timestamp)
>>>                       b->timestamp = ktime_get_ns();
>>> -             ret = vb2_core_qbuf(q, index, NULL);
>>> +             ret = vb2_core_qbuf(q, index, NULL, NULL);
>>>               dprintk(5, "vb2_dbuf result: %d\n", ret);
>>>               if (ret)
>>>                       return ret;
>>> @@ -2546,7 +2689,7 @@ static int vb2_thread(void *data)
>>>               if (copy_timestamp)
>>>                       vb->timestamp = ktime_get_ns();
>>>               if (!threadio->stop)
>>> -                     ret = vb2_core_qbuf(q, vb->index, NULL);
>>> +                     ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>>>               call_void_qop(q, wait_prepare, q);
>>>               if (ret || threadio->stop)
>>>                       break;
>>> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> index ad1e032c3bf5..1df5dd01c0cd 100644
>>> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
>>> @@ -23,6 +23,7 @@
>>>  #include <linux/sched.h>
>>>  #include <linux/freezer.h>
>>>  #include <linux/kthread.h>
>>> +#include <linux/sync_file.h>
>>>
>>>  #include <media/v4l2-dev.h>
>>>  #include <media/v4l2-fh.h>
>>> @@ -178,6 +179,17 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
>>>               return -EINVAL;
>>>       }
>>>
>>> +     if ((b->fence_fd != 0 && b->fence_fd != -1) &&
>>> +         !(b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>>> +             dprintk(1, "%s: fence_fd set without IN_FENCE flag\n", opname);
>>> +             return -EINVAL;
>>> +     }
>>> +
>>> +     if (b->fence_fd < 0 && (b->flags & V4L2_BUF_FLAG_IN_FENCE)) {
>>> +             dprintk(1, "%s: IN_FENCE flag set but no fence_fd\n", opname);
>>> +             return -EINVAL;
>>> +     }
>>> +
>>>       return __verify_planes_array(q->bufs[b->index], b);
>>>  }
>>>
>>> @@ -203,9 +215,14 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>>>       b->timestamp = ns_to_timeval(vb->timestamp);
>>>       b->timecode = vbuf->timecode;
>>>       b->sequence = vbuf->sequence;
>>> -     b->fence_fd = 0;
>>>       b->reserved = 0;
>>>
>>> +     b->fence_fd = 0;
>>> +     if (vb->in_fence)
>>> +             b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>>> +     else
>>> +             b->flags &= ~V4L2_BUF_FLAG_IN_FENCE;
>>> +
>>
>> I was wondering: if the in-fence was already triggered this flag is not set because
>> vb->in_fence will be NULL. Would it be useful to add a vb2 flag that is set when the
>> in-fence triggered and report that here somehow? It might be useful for debugging.
>>
>> Just wondering.
>>
> 
> I'm also wondering :-) ... does it make sense to use a flag for this?
> 
> #define V4L2_BUF_FLAG_IN_FENCE                  0x00200000
> #define V4L2_BUF_FLAG_OUT_FENCE                 0x00400000
> #define V4L2_BUF_FLAG_IN_FENCE_SIGNALED         0x00800000
> 
> Aren't we using a very limited resource? Is it worry?

No, not a worry. It is very, very likely that there will be a new set of streaming ioctls
anyway (I have patches pending and will continue with that once fences and request API
are in). At that point there will be a new v4l2_buffer struct with more room. And besides,
there are still 8 bits left after this.

Regards,

	Hans

> 
> Thanks,
> 

Re: [PATCH v8 10/13] [media] vb2: add out-fence support to QBUF

[Ezequiel Garcia]

Hi guys,

I've a couple questions.

On 9 March 2018 at 14:49, Gustavo Padovan <gustavo@padovan.org> wrote:
> From: Gustavo Padovan <gustavo.padovan@collabora.com>
>
> If V4L2_BUF_FLAG_OUT_FENCE flag is present on the QBUF call we create
> an out_fence and send its fd to userspace on the fence_fd field as a
> return arg for the QBUF call.
>
> The fence is signaled on buffer_done(), when the job on the buffer is
> finished.
>
> v9:     - remove in-fences changes from this patch (Alex Courbot)
>         - improve fence context creation (Hans Verkuil)
>         - clean up out fences if vb2_core_qbuf() fails (Hans Verkuil)
>
> v8:
>         - return 0 as fence_fd if OUT_FENCE flag not used (Mauro)
>         - fix crash when checking not using fences in vb2_buffer_done()
>
> v7:
>         - merge patch that add the infrastructure to out-fences into
>         this one (Alex Courbot)
>         - Do not install the fd if there is no fence. (Alex Courbot)
>         - do not report error on requeueing, just WARN_ON_ONCE() (Hans)
>
> v6
>         - get rid of the V4L2_EVENT_OUT_FENCE event. We always keep the
>         ordering in vb2 for queueing in the driver, so the event is not
>         necessary anymore and the out_fence_fd is sent back to userspace
>         on QBUF call return arg
>         - do not allow requeueing with out-fences, instead mark the buffer
>         with an error and wake up to userspace.
>         - send the out_fence_fd back to userspace on the fence_fd field
>
> v5:
>         - delay fd_install to DQ_EVENT (Hans)
>         - if queue is fully ordered send OUT_FENCE event right away
>         (Brian)
>         - rename 'q->ordered' to 'q->ordered_in_driver'
>         - merge change to implement OUT_FENCE event here
>
> v4:
>         - return the out_fence_fd in the BUF_QUEUED event(Hans)
>
> v3:     - add WARN_ON_ONCE(q->ordered) on requeueing (Hans)
>         - set the OUT_FENCE flag if there is a fence pending (Hans)
>         - call fd_install() after vb2_core_qbuf() (Hans)
>         - clean up fence if vb2_core_qbuf() fails (Hans)
>         - add list to store sync_file and fence for the next queued buffer
>
> v2: check if the queue is ordered.
>
> Signed-off-by: Gustavo Padovan <gustavo.padovan@collabora.com>
> ---
>  drivers/media/common/videobuf2/videobuf2-core.c | 88 +++++++++++++++++++++++++
>  drivers/media/common/videobuf2/videobuf2-v4l2.c | 20 +++++-
>  include/media/videobuf2-core.h                  | 25 +++++++
>  3 files changed, 132 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
> index 5de5e35cfc40..dd18a9f345c7 100644
> --- a/drivers/media/common/videobuf2/videobuf2-core.c
> +++ b/drivers/media/common/videobuf2/videobuf2-core.c
> @@ -25,6 +25,7 @@
>  #include <linux/sched.h>
>  #include <linux/freezer.h>
>  #include <linux/kthread.h>
> +#include <linux/sync_file.h>
>
>  #include <media/videobuf2-core.h>
>  #include <media/v4l2-mc.h>
> @@ -357,6 +358,7 @@ static int __vb2_queue_alloc(struct vb2_queue *q, enum vb2_memory memory,
>                         vb->planes[plane].length = plane_sizes[plane];
>                         vb->planes[plane].min_length = plane_sizes[plane];
>                 }
> +               vb->out_fence_fd = -1;
>                 q->bufs[vb->index] = vb;
>
>                 /* Allocate video buffer memory for the MMAP type */
> @@ -934,10 +936,22 @@ static void vb2_process_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state
>         case VB2_BUF_STATE_QUEUED:
>                 return;
>         case VB2_BUF_STATE_REQUEUEING:
> +               /* Requeuing with explicit synchronization, spit warning */
> +               WARN_ON_ONCE(vb->out_fence);
> +
>                 if (q->start_streaming_called)
>                         __enqueue_in_driver(vb);
>                 return;
>         default:
> +               if (vb->out_fence) {
> +                       if (state == VB2_BUF_STATE_ERROR)
> +                               dma_fence_set_error(vb->out_fence, -EFAULT);
> +                       dma_fence_signal(vb->out_fence);
> +                       dma_fence_put(vb->out_fence);
> +                       vb->out_fence = NULL;
> +                       vb->out_fence_fd = -1;
> +               }
> +
>                 /* Inform any processes that may be waiting for buffers */
>                 wake_up(&q->done_wq);
>                 break;
> @@ -1353,6 +1367,62 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>
> +static inline const char *vb2_fence_get_driver_name(struct dma_fence *fence)
> +{
> +       return "vb2_fence";
> +}
> +
> +static inline const char *vb2_fence_get_timeline_name(struct dma_fence *fence)
> +{
> +       return "vb2_fence_timeline";
> +}
> +
> +static inline bool vb2_fence_enable_signaling(struct dma_fence *fence)
> +{
> +       return true;
> +}
> +
> +static const struct dma_fence_ops vb2_fence_ops = {
> +       .get_driver_name = vb2_fence_get_driver_name,
> +       .get_timeline_name = vb2_fence_get_timeline_name,
> +       .enable_signaling = vb2_fence_enable_signaling,
> +       .wait = dma_fence_default_wait,
> +};
> +
> +int vb2_setup_out_fence(struct vb2_queue *q, unsigned int index)
> +{
> +       struct vb2_buffer *vb;
> +
> +       vb = q->bufs[index];
> +
> +       vb->out_fence_fd = get_unused_fd_flags(O_CLOEXEC);
> +
> +       if (call_qop(q, is_unordered, q) || !q->queueing_started)
> +               q->out_fence_context = dma_fence_context_alloc(1);
> +
> +       vb->out_fence = kzalloc(sizeof(*vb->out_fence), GFP_KERNEL);
> +       if (!vb->out_fence)
> +               return -ENOMEM;
> +
> +       dma_fence_init(vb->out_fence, &vb2_fence_ops, &q->out_fence_lock,
> +                      q->out_fence_context, 1);
> +       if (!vb->out_fence) {
> +               put_unused_fd(vb->out_fence_fd);
> +               return -ENOMEM;
> +       }
> +
> +       vb->sync_file = sync_file_create(vb->out_fence);
> +       if (!vb->sync_file) {
> +               put_unused_fd(vb->out_fence_fd);
> +               dma_fence_put(vb->out_fence);
> +               vb->out_fence = NULL;
> +               return -ENOMEM;
> +       }
> +
> +       return 0;
> +}
> +EXPORT_SYMBOL_GPL(vb2_setup_out_fence);
> +
>  /*
>   * vb2_start_streaming() - Attempt to start streaming.
>   * @q:         videobuf2 queue
> @@ -1482,6 +1552,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>         list_add_tail(&vb->queued_entry, &q->queued_list);
>         q->queued_count++;
>         q->waiting_for_buffers = false;
> +       q->queueing_started = 1;
>         vb->state = VB2_BUF_STATE_QUEUED;
>         vb->in_fence = in_fence;
>
> @@ -1545,6 +1616,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>         if (pb)
>                 call_void_bufop(q, fill_user_buffer, vb, pb);
>
> +       if (vb->out_fence) {
> +               fd_install(vb->out_fence_fd, vb->sync_file->file);
> +               vb->sync_file = NULL;
> +       }
> +
>         dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
>         return 0;
>
> @@ -1552,6 +1628,16 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
>         spin_unlock_irqrestore(&vb->fence_cb_lock, flags);
>
>  err:
> +       if (vb->sync_file) {
> +               put_unused_fd(vb->out_fence_fd);
> +               vb->out_fence_fd = -1;
> +
> +               dma_fence_put(vb->out_fence);
> +
> +               fput(vb->sync_file->file);
> +               vb->sync_file = NULL;
> +       }
> +
>         /* Fill buffer information for the userspace */
>         if (pb)
>                 call_void_bufop(q, fill_user_buffer, vb, pb);
> @@ -1804,6 +1890,7 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
>         q->start_streaming_called = 0;
>         q->queued_count = 0;
>         q->error = 0;
> +       q->queueing_started = 0;
>
>         list_for_each_entry(vb, &q->queued_list, queued_entry) {
>                 spin_lock_irqsave(&vb->fence_cb_lock, flags);
> @@ -2156,6 +2243,7 @@ int vb2_core_queue_init(struct vb2_queue *q)
>         spin_lock_init(&q->done_lock);
>         mutex_init(&q->mmap_lock);
>         init_waitqueue_head(&q->done_wq);
> +       spin_lock_init(&q->out_fence_lock);
>
>         q->memory = VB2_MEMORY_UNKNOWN;
>
> diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> index 1df5dd01c0cd..ab5b2b71d784 100644
> --- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
> +++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
> @@ -217,7 +217,12 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>         b->sequence = vbuf->sequence;
>         b->reserved = 0;
>
> -       b->fence_fd = 0;
> +       if (b->flags & V4L2_BUF_FLAG_OUT_FENCE) {
> +               b->fence_fd = vb->out_fence_fd;
> +       } else {
> +               b->fence_fd = 0;
> +       }
> +
>         if (vb->in_fence)
>                 b->flags |= V4L2_BUF_FLAG_IN_FENCE;
>         else
> @@ -496,6 +501,10 @@ int vb2_querybuf(struct vb2_queue *q, struct v4l2_buffer *b)
>         ret = __verify_planes_array(vb, b);
>         if (!ret)
>                 vb2_core_querybuf(q, b->index, b);
> +
> +       /* Do not return the out-fence fd on querybuf */
> +       if (vb->out_fence)
> +               b->fence_fd = -1;

We aren't returning the fence_fd but as per the uapi
documentation we should. Which should we correct?

Also, any reason we don't recycle the out fence?

IOW, each time we queue a buffer, a new out fence is
created and returned to user, and has to be closed.

Thanks!
-- 
Ezequiel García, VanguardiaSur
www.vanguardiasur.com.ar