[PATCHv13 00/28] Request API

[PATCHv13 00/28] Request API

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Hi all,

This is version 13 of the Request API series.

The main changes compared to v12 are:

- Replaced media_request_cancel with a new vb2 op buf_request_complete
- No longer allow adding a prepared buffer to a request (see TODO below)
- Updated and reviewed the documentation
- Incorporated comments from Sakari
- Added missing kerneldoc documentation

This series has been tested with vim2m and vivid.

Regarding locking:

There are two request locks used: req_queue_mutex (part of media_device)
ensures high-level serialization of queueing/reiniting and canceling
requests. It serializes STREAMON/OFF, TRY/S_EXT_CTRLS, close() and
MEDIA_REQUEST_IOC_QUEUE. This is the top-level lock and should be taken
before any others.

The lock spin_lock in struct media_request protects that structure and
should be held for a short time only.

Note that VIDIOC_QBUF does not take this mutex: when
a buffer is queued for a request it will add it to the
request by calling media_request_object_bind(), and that returns an
error if the request is in the wrong state. It is serialized via the
spin_lock which in this specific case is sufficient.

When MEDIA_REQUEST_IOC_QUEUE is called it will validate and queue both
control handlers and vb2 buffers. For control handlers the control handler
lock will be taken, for vb2 buffers the vb2_queue lock will be taken.
This requires that the lock field in vb2_queue is set, which is checked in
videobuf2-v4l2.c.

TODO/Remarks:

1) vim2m: the media topology is a bit bogus, this needs to be fixed
   (i.e. a proper HW entity should be added). But for now it is
   good enough for testing. In addition as suggested by Tomasz the
   timer can be replaced by delayed_work.

2) It should be possible to queue an already prepared buffer to a
   request (VIDIOC_PREPARE_BUF, then VIDIOC_QBUF to a request).
   I ran out of time for this as it got a bit hairy in vb2. For
   now this is simply not allowed, which is fine for stateless
   codecs. I plan on adding support for this when I have a bit more
   time to see what the correct approach is. It's not impossible
   that this has to be postponed until the fence code is in as well.

Everything seemed to slip nicely into place while working on this,
so I hope this is finally an implementation that we can proceed to
upstream and build upon for complex camera pipelines in the future.

This patch series is also available here:

https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=reqv13

Regards,

	Hans


Alexandre Courbot (2):
  videodev2.h: add request_fd field to v4l2_ext_controls
  Documentation: v4l: document request API

Hans Verkuil (26):
  v4l2-device.h: always expose mdev
  uapi/linux/media.h: add request API
  media-request: implement media requests
  media-request: add media_request_get_by_fd
  media-request: add media_request_object_find
  v4l2-dev: lock req_queue_mutex
  v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
  v4l2-ctrls: prepare internal structs for request API
  v4l2-ctrls: alloc memory for p_req
  v4l2-ctrls: use ref in helper instead of ctrl
  v4l2-ctrls: add core request support
  v4l2-ctrls: support g/s_ext_ctrls for requests
  videodev2.h: Add request_fd field to v4l2_buffer
  vb2: store userspace data in vb2_v4l2_buffer
  videobuf2-core: embed media_request_object
  videobuf2-core: integrate with media requests
  videobuf2-v4l2: integrate with media requests
  videobuf2-core: add request helper functions
  videobuf2-v4l2: add vb2_request_queue/validate helpers
  v4l2-mem2mem: add vb2_m2m_request_queue
  media: vim2m: add media device
  vim2m: use workqueue
  vim2m: support requests
  vivid: add mc
  vivid: add request support
  RFC: media-requests: add debugfs node

 .../media/uapi/mediactl/media-funcs.rst       |   3 +
 .../uapi/mediactl/media-ioc-request-alloc.rst |  71 +++
 .../uapi/mediactl/media-request-ioc-queue.rst |  46 ++
 .../mediactl/media-request-ioc-reinit.rst     |  51 ++
 Documentation/media/uapi/v4l/buffer.rst       |  18 +-
 Documentation/media/uapi/v4l/common.rst       |   1 +
 Documentation/media/uapi/v4l/request-api.rst  | 211 ++++++++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst     |  25 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
 .../media/videodev2.h.rst.exceptions          |   1 +
 drivers/media/Makefile                        |   3 +-
 .../media/common/videobuf2/videobuf2-core.c   | 184 +++++--
 .../media/common/videobuf2/videobuf2-v4l2.c   | 457 ++++++++++------
 drivers/media/dvb-core/dvb_vb2.c              |   5 +-
 drivers/media/dvb-frontends/rtl2832_sdr.c     |   5 +-
 drivers/media/media-device.c                  |  55 ++
 drivers/media/media-devnode.c                 |  17 +
 drivers/media/media-request.c                 | 469 +++++++++++++++++
 drivers/media/pci/bt8xx/bttv-driver.c         |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c       |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c       |   2 +-
 drivers/media/pci/cx88/cx88-video.c           |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c   |   4 +-
 drivers/media/pci/saa7134/saa7134-video.c     |   2 +-
 .../media/platform/exynos4-is/fimc-capture.c  |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |   4 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |   2 +-
 drivers/media/platform/rcar_drif.c            |   2 +-
 .../media/platform/s3c-camif/camif-capture.c  |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c  |   4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c  |   4 +-
 .../media/platform/soc_camera/soc_camera.c    |   7 +-
 drivers/media/platform/vim2m.c                |  91 +++-
 drivers/media/platform/vivid/vivid-core.c     |  69 +++
 drivers/media/platform/vivid/vivid-core.h     |   8 +
 drivers/media/platform/vivid/vivid-ctrls.c    |  46 +-
 .../media/platform/vivid/vivid-kthread-cap.c  |  12 +
 .../media/platform/vivid/vivid-kthread-out.c  |  12 +
 drivers/media/platform/vivid/vivid-sdr-cap.c  |  16 +
 drivers/media/platform/vivid/vivid-vbi-cap.c  |  10 +
 drivers/media/platform/vivid/vivid-vbi-out.c  |  10 +
 drivers/media/platform/vivid/vivid-vid-cap.c  |  10 +
 drivers/media/platform/vivid/vivid-vid-out.c  |  10 +
 drivers/media/usb/cpia2/cpia2_v4l.c           |   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c       |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c     |   4 +-
 drivers/media/usb/msi2500/msi2500.c           |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c       |   2 +-
 drivers/media/usb/uvc/uvc_queue.c             |   5 +-
 drivers/media/usb/uvc/uvc_v4l2.c              |   3 +-
 drivers/media/usb/uvc/uvcvideo.h              |   1 +
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |  14 +-
 drivers/media/v4l2-core/v4l2-ctrls.c          | 486 +++++++++++++++++-
 drivers/media/v4l2-core/v4l2-dev.c            |  37 +-
 drivers/media/v4l2-core/v4l2-device.c         |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  22 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c        |  40 +-
 drivers/media/v4l2-core/v4l2-subdev.c         |   9 +-
 .../staging/media/davinci_vpfe/vpfe_video.c   |   3 +-
 drivers/staging/media/imx/imx-media-dev.c     |   2 +-
 drivers/staging/media/imx/imx-media-fim.c     |   2 +-
 drivers/staging/media/omap4iss/iss_video.c    |   3 +-
 drivers/usb/gadget/function/uvc_queue.c       |   2 +-
 include/media/media-device.h                  |  27 +
 include/media/media-devnode.h                 |   4 +
 include/media/media-request.h                 | 293 +++++++++++
 include/media/v4l2-ctrls.h                    |  45 +-
 include/media/v4l2-device.h                   |   4 +-
 include/media/v4l2-mem2mem.h                  |   4 +
 include/media/videobuf2-core.h                |  39 +-
 include/media/videobuf2-v4l2.h                |  20 +-
 include/uapi/linux/media.h                    |  12 +
 include/uapi/linux/videodev2.h                |  14 +-
 73 files changed, 2747 insertions(+), 325 deletions(-)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request.h

-- 
2.17.0

[PATCHv13 01/28] v4l2-device.h: always expose mdev

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
But since we will need to pass the media_device to vb2 and the
control framework it is very convenient to just make this field
available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
then it will just be NULL.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 include/media/v4l2-device.h | 4 +---
 1 file changed, 1 insertion(+), 3 deletions(-)

diff --git a/include/media/v4l2-device.h b/include/media/v4l2-device.h
index 0c9e4da55499..b330e4a08a6b 100644
--- a/include/media/v4l2-device.h
+++ b/include/media/v4l2-device.h
@@ -33,7 +33,7 @@ struct v4l2_ctrl_handler;
  * struct v4l2_device - main struct to for V4L2 device drivers
  *
  * @dev: pointer to struct device.
- * @mdev: pointer to struct media_device
+ * @mdev: pointer to struct media_device, may be NULL.
  * @subdevs: used to keep track of the registered subdevs
  * @lock: lock this struct; can be used by the driver as well
  *	if this struct is embedded into a larger struct.
@@ -58,9 +58,7 @@ struct v4l2_ctrl_handler;
  */
 struct v4l2_device {
 	struct device *dev;
-#if defined(CONFIG_MEDIA_CONTROLLER)
 	struct media_device *mdev;
-#endif
 	struct list_head subdevs;
 	spinlock_t lock;
 	char name[V4L2_DEVICE_NAME_SIZE];
-- 
2.17.0

[PATCHv13 02/28] uapi/linux/media.h: add request API

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Define the public request API.

This adds the new MEDIA_IOC_REQUEST_ALLOC ioctl to allocate a request
and two ioctls that operate on a request in order to queue the
contents of the request to the driver and to re-initialize the
request.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 include/uapi/linux/media.h | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index c7e9a5cba24e..32883d4d22b2 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -342,11 +342,23 @@ struct media_v2_topology {
 
 /* ioctls */
 
+struct __attribute__ ((packed)) media_request_alloc {
+	__s32 fd;
+};
+
 #define MEDIA_IOC_DEVICE_INFO	_IOWR('|', 0x00, struct media_device_info)
 #define MEDIA_IOC_ENUM_ENTITIES	_IOWR('|', 0x01, struct media_entity_desc)
 #define MEDIA_IOC_ENUM_LINKS	_IOWR('|', 0x02, struct media_links_enum)
 #define MEDIA_IOC_SETUP_LINK	_IOWR('|', 0x03, struct media_link_desc)
 #define MEDIA_IOC_G_TOPOLOGY	_IOWR('|', 0x04, struct media_v2_topology)
+#define MEDIA_IOC_REQUEST_ALLOC	_IOWR('|', 0x05, struct media_request_alloc)
+
+/*
+ * These ioctls are called from the request file descriptor as returned
+ * by MEDIA_IOC_REQUEST_ALLOC.
+ */
+#define MEDIA_REQUEST_IOC_QUEUE		_IO('|',  0x80)
+#define MEDIA_REQUEST_IOC_REINIT	_IO('|',  0x81)
 
 #if !defined(__KERNEL__) || defined(__NEED_MEDIA_LEGACY_API)
 
-- 
2.17.0

[PATCHv13 03/28] media-request: implement media requests

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add initial media request support:

1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
2) Add struct media_request to store request objects.
3) Add struct media_request_object to represent a request object.
4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.

Basic lifecycle: the application allocates a request, adds
objects to it, queues the request, polls until it is completed
and can then read the final values of the objects at the time
of completion. When it closes the file descriptor the request
memory will be freed (actually, when the last user of that request
releases the request).

Drivers will bind an object to a request (the 'adds objects to it'
phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
validated (req_validate op), then queued (the req_queue op).

When done with an object it can either be unbound from the request
(e.g. when the driver has finished with a vb2 buffer) or marked as
completed (e.g. for controls associated with a buffer). When all
objects in the request are completed (or unbound), then the request
fd will signal an exception (poll).

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/Makefile        |   3 +-
 drivers/media/media-device.c  |  14 ++
 drivers/media/media-request.c | 407 ++++++++++++++++++++++++++++++++++
 include/media/media-device.h  |  16 ++
 include/media/media-request.h | 244 ++++++++++++++++++++
 5 files changed, 683 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request.h

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..985d35ec6b29 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -3,7 +3,8 @@
 # Makefile for the kernel multimedia device drivers.
 #
 
-media-objs	:= media-device.o media-devnode.o media-entity.o
+media-objs	:= media-device.o media-devnode.o media-entity.o \
+		   media-request.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index 35e81f7c0d2f..bb6a64acd3f0 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,7 @@
 #include <media/media-device.h>
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
+#include <media/media-request.h>
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -366,6 +367,15 @@ static long media_device_get_topology(struct media_device *mdev,
 	return ret;
 }
 
+static long media_device_request_alloc(struct media_device *mdev,
+				       struct media_request_alloc *alloc)
+{
+	if (!mdev->ops || !mdev->ops->req_validate || !mdev->ops->req_queue)
+		return -ENOTTY;
+
+	return media_request_alloc(mdev, alloc);
+}
+
 static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
 {
 	/* All media IOCTLs are _IOWR() */
@@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
 	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
+	MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
@@ -686,6 +697,8 @@ void media_device_init(struct media_device *mdev)
 	INIT_LIST_HEAD(&mdev->pads);
 	INIT_LIST_HEAD(&mdev->links);
 	INIT_LIST_HEAD(&mdev->entity_notify);
+
+	mutex_init(&mdev->req_queue_mutex);
 	mutex_init(&mdev->graph_mutex);
 	ida_init(&mdev->entity_internal_idx);
 
@@ -699,6 +712,7 @@ void media_device_cleanup(struct media_device *mdev)
 	mdev->entity_internal_idx_max = 0;
 	media_graph_walk_cleanup(&mdev->pm_count_walk);
 	mutex_destroy(&mdev->graph_mutex);
+	mutex_destroy(&mdev->req_queue_mutex);
 }
 EXPORT_SYMBOL_GPL(media_device_cleanup);
 
diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
new file mode 100644
index 000000000000..c216c4ab628b
--- /dev/null
+++ b/drivers/media/media-request.c
@@ -0,0 +1,407 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Media device request objects
+ *
+ * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
+ * Copyright (C) 2018 Intel Corporation
+ * Copyright (C) 2018 Google, Inc.
+ *
+ * Author: Hans Verkuil <hans.verkuil@cisco.com>
+ * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
+ */
+
+#include <linux/anon_inodes.h>
+#include <linux/file.h>
+
+#include <media/media-device.h>
+#include <media/media-request.h>
+
+static const char * const request_state[] = {
+	[MEDIA_REQUEST_STATE_IDLE]	 = "idle",
+	[MEDIA_REQUEST_STATE_VALIDATING] = "validating",
+	[MEDIA_REQUEST_STATE_QUEUED]	 = "queued",
+	[MEDIA_REQUEST_STATE_COMPLETE]	 = "complete",
+	[MEDIA_REQUEST_STATE_CLEANING]	 = "cleaning",
+};
+
+static const char *
+media_request_state_str(enum media_request_state state)
+{
+	if (WARN_ON(state >= ARRAY_SIZE(request_state)))
+		return "invalid";
+	return request_state[state];
+}
+
+static void media_request_clean(struct media_request *req)
+{
+	struct media_request_object *obj, *obj_safe;
+
+	WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_CLEANING);
+
+	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
+		media_request_object_unbind(obj);
+		media_request_object_put(obj);
+	}
+
+	req->num_incomplete_objects = 0;
+	wake_up_interruptible_all(&req->poll_wait);
+}
+
+static void media_request_release(struct kref *kref)
+{
+	struct media_request *req =
+		container_of(kref, struct media_request, kref);
+	struct media_device *mdev = req->mdev;
+
+	dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
+
+	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
+
+	media_request_clean(req);
+
+	if (mdev->ops->req_free)
+		mdev->ops->req_free(req);
+	else
+		kfree(req);
+}
+
+void media_request_put(struct media_request *req)
+{
+	kref_put(&req->kref, media_request_release);
+}
+EXPORT_SYMBOL_GPL(media_request_put);
+
+static int media_request_close(struct inode *inode, struct file *filp)
+{
+	struct media_request *req = filp->private_data;
+
+	media_request_put(req);
+	return 0;
+}
+
+static unsigned int media_request_poll(struct file *filp,
+				       struct poll_table_struct *wait)
+{
+	struct media_request *req = filp->private_data;
+	unsigned long flags;
+	unsigned int ret = 0;
+	enum media_request_state state;
+
+	if (!(poll_requested_events(wait) & POLLPRI))
+		return 0;
+
+	spin_lock_irqsave(&req->lock, flags);
+	state = atomic_read(&req->state);
+
+	if (state == MEDIA_REQUEST_STATE_COMPLETE) {
+		ret = POLLPRI;
+		goto unlock;
+	}
+	if (state != MEDIA_REQUEST_STATE_QUEUED) {
+		ret = POLLERR;
+		goto unlock;
+	}
+
+	poll_wait(filp, &req->poll_wait, wait);
+
+unlock:
+	spin_unlock_irqrestore(&req->lock, flags);
+	return ret;
+}
+
+static long media_request_ioctl_queue(struct media_request *req)
+{
+	struct media_device *mdev = req->mdev;
+	enum media_request_state state;
+	unsigned long flags;
+	int ret = 0;
+
+	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
+
+	/*
+	 * Ensure the request that is validated will be the one that gets queued
+	 * next by serialising the queueing process. This mutex is also used
+	 * to serialize with canceling a vb2 queue and with setting values such
+	 * as controls in a request.
+	 */
+	mutex_lock(&mdev->req_queue_mutex);
+
+	spin_lock_irqsave(&req->lock, flags);
+	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
+			       MEDIA_REQUEST_STATE_VALIDATING);
+	spin_unlock_irqrestore(&req->lock, flags);
+	if (state != MEDIA_REQUEST_STATE_IDLE) {
+		dev_dbg(mdev->dev,
+			"request: unable to queue %s, request in state %s\n",
+			req->debug_str, media_request_state_str(state));
+		mutex_unlock(&mdev->req_queue_mutex);
+		return -EBUSY;
+	}
+
+	ret = mdev->ops->req_validate(req);
+
+	/*
+	 * If the req_validate was successful, then we mark the state as QUEUED
+	 * and call req_queue. The reason we set the state first is that this
+	 * allows req_queue to unbind or complete the queued objects in case
+	 * they are immediately 'consumed'. State changes from QUEUED to another
+	 * state can only happen if either the driver changes the state or if
+	 * the user cancels the vb2 queue. The driver can only change the state
+	 * after each object is queued through the req_queue op (and note that
+	 * that op cannot fail), so setting the state to QUEUED up front is
+	 * safe.
+	 *
+	 * The other reason for changing the state is if the vb2 queue is
+	 * canceled, and that uses the req_queue_mutex which is still locked
+	 * while req_queue is called, so that's safe as well.
+	 */
+	atomic_set(&req->state,
+		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);
+
+	if (!ret)
+		mdev->ops->req_queue(req);
+
+	mutex_unlock(&mdev->req_queue_mutex);
+
+	if (ret)
+		dev_dbg(mdev->dev, "request: can't queue %s (%d)\n",
+			req->debug_str, ret);
+	else
+		media_request_get(req);
+
+	return ret;
+}
+
+static long media_request_ioctl_reinit(struct media_request *req)
+{
+	struct media_device *mdev = req->mdev;
+	unsigned long flags;
+
+	spin_lock_irqsave(&req->lock, flags);
+	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE &&
+	    atomic_read(&req->state) != MEDIA_REQUEST_STATE_COMPLETE) {
+		dev_dbg(mdev->dev,
+			"request: %s not in idle or complete state, cannot reinit\n",
+			req->debug_str);
+		spin_unlock_irqrestore(&req->lock, flags);
+		return -EBUSY;
+	}
+	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
+	spin_unlock_irqrestore(&req->lock, flags);
+
+	media_request_clean(req);
+
+	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
+
+	return 0;
+}
+
+static long media_request_ioctl(struct file *filp, unsigned int cmd,
+				unsigned long arg)
+{
+	struct media_request *req = filp->private_data;
+
+	switch (cmd) {
+	case MEDIA_REQUEST_IOC_QUEUE:
+		return media_request_ioctl_queue(req);
+	case MEDIA_REQUEST_IOC_REINIT:
+		return media_request_ioctl_reinit(req);
+	default:
+		return -ENOIOCTLCMD;
+	}
+}
+
+static const struct file_operations request_fops = {
+	.owner = THIS_MODULE,
+	.poll = media_request_poll,
+	.unlocked_ioctl = media_request_ioctl,
+	.release = media_request_close,
+};
+
+int media_request_alloc(struct media_device *mdev,
+			struct media_request_alloc *alloc)
+{
+	struct media_request *req;
+	struct file *filp;
+	char comm[TASK_COMM_LEN];
+	int fd;
+	int ret;
+
+	/* Either both are NULL or both are non-NULL */
+	if (WARN_ON(!mdev->ops->req_alloc ^ !mdev->ops->req_free))
+		return -ENOMEM;
+
+	fd = get_unused_fd_flags(O_CLOEXEC);
+	if (fd < 0)
+		return fd;
+
+	filp = anon_inode_getfile("request", &request_fops, NULL, O_CLOEXEC);
+	if (IS_ERR(filp)) {
+		ret = PTR_ERR(filp);
+		goto err_put_fd;
+	}
+
+	if (mdev->ops->req_alloc)
+		req = mdev->ops->req_alloc(mdev);
+	else
+		req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req) {
+		ret = -ENOMEM;
+		goto err_fput;
+	}
+
+	filp->private_data = req;
+	req->mdev = mdev;
+	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
+	req->num_incomplete_objects = 0;
+	kref_init(&req->kref);
+	INIT_LIST_HEAD(&req->objects);
+	spin_lock_init(&req->lock);
+	init_waitqueue_head(&req->poll_wait);
+
+	alloc->fd = fd;
+
+	get_task_comm(comm, current);
+	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
+		 comm, fd);
+	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
+
+	fd_install(fd, filp);
+
+	return 0;
+
+err_fput:
+	fput(filp);
+
+err_put_fd:
+	put_unused_fd(fd);
+
+	return ret;
+}
+
+static void media_request_object_release(struct kref *kref)
+{
+	struct media_request_object *obj =
+		container_of(kref, struct media_request_object, kref);
+	struct media_request *req = obj->req;
+
+	if (req)
+		media_request_object_unbind(obj);
+	obj->ops->release(obj);
+}
+
+void media_request_object_put(struct media_request_object *obj)
+{
+	kref_put(&obj->kref, media_request_object_release);
+}
+EXPORT_SYMBOL_GPL(media_request_object_put);
+
+void media_request_object_init(struct media_request_object *obj)
+{
+	obj->ops = NULL;
+	obj->req = NULL;
+	obj->priv = NULL;
+	obj->completed = false;
+	INIT_LIST_HEAD(&obj->list);
+	kref_init(&obj->kref);
+}
+EXPORT_SYMBOL_GPL(media_request_object_init);
+
+int media_request_object_bind(struct media_request *req,
+			      const struct media_request_object_ops *ops,
+			      void *priv,
+			      struct media_request_object *obj)
+{
+	unsigned long flags;
+	int ret = -EBUSY;
+
+	if (WARN_ON(!ops->release))
+		return -EPERM;
+
+	obj->req = req;
+	obj->ops = ops;
+	obj->priv = priv;
+
+	spin_lock_irqsave(&req->lock, flags);
+
+	if (WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE))
+		goto unlock;
+
+	list_add_tail(&obj->list, &req->objects);
+	req->num_incomplete_objects++;
+	ret = 0;
+
+unlock:
+	spin_unlock_irqrestore(&req->lock, flags);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_request_object_bind);
+
+void media_request_object_unbind(struct media_request_object *obj)
+{
+	struct media_request *req = obj->req;
+	enum media_request_state state;
+	unsigned long flags;
+	bool completed = false;
+
+	if (WARN_ON(!req))
+		return;
+
+	spin_lock_irqsave(&req->lock, flags);
+	list_del(&obj->list);
+	obj->req = NULL;
+
+	state = atomic_read(&req->state);
+
+	if (state == MEDIA_REQUEST_STATE_COMPLETE ||
+	    state == MEDIA_REQUEST_STATE_CLEANING)
+		goto unlock;
+
+	if (WARN_ON(state == MEDIA_REQUEST_STATE_VALIDATING))
+		goto unlock;
+
+	if (WARN_ON(!req->num_incomplete_objects))
+		goto unlock;
+
+	req->num_incomplete_objects--;
+	if (state == MEDIA_REQUEST_STATE_QUEUED &&
+	    !req->num_incomplete_objects) {
+		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
+		completed = true;
+		wake_up_interruptible_all(&req->poll_wait);
+	}
+
+unlock:
+	spin_unlock_irqrestore(&req->lock, flags);
+	if (obj->ops->unbind)
+		obj->ops->unbind(obj);
+	if (completed)
+		media_request_put(req);
+}
+EXPORT_SYMBOL_GPL(media_request_object_unbind);
+
+void media_request_object_complete(struct media_request_object *obj)
+{
+	struct media_request *req = obj->req;
+	unsigned long flags;
+	bool completed = false;
+
+	spin_lock_irqsave(&req->lock, flags);
+	if (obj->completed)
+		goto unlock;
+	obj->completed = true;
+	if (WARN_ON(!req->num_incomplete_objects) ||
+	    WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_QUEUED))
+		goto unlock;
+
+	if (!--req->num_incomplete_objects) {
+		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
+		wake_up_interruptible_all(&req->poll_wait);
+		completed = true;
+	}
+unlock:
+	spin_unlock_irqrestore(&req->lock, flags);
+	if (completed)
+		media_request_put(req);
+}
+EXPORT_SYMBOL_GPL(media_request_object_complete);
diff --git a/include/media/media-device.h b/include/media/media-device.h
index bcc6ec434f1f..7d855823341c 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -27,6 +27,7 @@
 
 struct ida;
 struct device;
+struct media_device;
 
 /**
  * struct media_entity_notify - Media Entity Notify
@@ -50,10 +51,21 @@ struct media_entity_notify {
  * struct media_device_ops - Media device operations
  * @link_notify: Link state change notification callback. This callback is
  *		 called with the graph_mutex held.
+ * @req_alloc: Allocate a request
+ * @req_free: Free a request
+ * @req_validate: Validate a request, but do not queue yet
+ * @req_queue: Queue a validated request, cannot fail. If something goes
+ *	       wrong when queueing this request then it should be marked
+ *	       as such internally in the driver and any related buffers
+ *	       must eventually return to vb2 with state VB2_BUF_STATE_ERROR.
  */
 struct media_device_ops {
 	int (*link_notify)(struct media_link *link, u32 flags,
 			   unsigned int notification);
+	struct media_request *(*req_alloc)(struct media_device *mdev);
+	void (*req_free)(struct media_request *req);
+	int (*req_validate)(struct media_request *req);
+	void (*req_queue)(struct media_request *req);
 };
 
 /**
@@ -88,6 +100,8 @@ struct media_device_ops {
  * @disable_source: Disable Source Handler function pointer
  *
  * @ops:	Operation handler callbacks
+ * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
+ *		     media device.
  *
  * This structure represents an abstract high-level media device. It allows easy
  * access to entities and provides basic media device-level support. The
@@ -158,6 +172,8 @@ struct media_device {
 	void (*disable_source)(struct media_entity *entity);
 
 	const struct media_device_ops *ops;
+
+	struct mutex req_queue_mutex;
 };
 
 /* We don't need to include pci.h or usb.h here */
diff --git a/include/media/media-request.h b/include/media/media-request.h
new file mode 100644
index 000000000000..e39122dfd717
--- /dev/null
+++ b/include/media/media-request.h
@@ -0,0 +1,244 @@
+// SPDX-License-Identifier: GPL-2.0-only
+/*
+ * Media device request objects
+ *
+ * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
+ * Copyright (C) 2018 Intel Corporation
+ *
+ * Author: Hans Verkuil <hans.verkuil@cisco.com>
+ * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
+ */
+
+#ifndef MEDIA_REQUEST_H
+#define MEDIA_REQUEST_H
+
+#include <linux/list.h>
+#include <linux/slab.h>
+#include <linux/spinlock.h>
+#include <linux/atomic.h>
+
+#include <media/media-device.h>
+
+/**
+ * enum media_request_state - media request state
+ *
+ * @MEDIA_REQUEST_STATE_IDLE:		Idle
+ * @MEDIA_REQUEST_STATE_VALIDATING:	Validating the request, no state changes
+ *					allowed
+ * @MEDIA_REQUEST_STATE_QUEUED:		Queued
+ * @MEDIA_REQUEST_STATE_COMPLETE:	Completed, the request is done
+ * @MEDIA_REQUEST_STATE_CLEANING:	Cleaning, the request is being re-inited
+ */
+enum media_request_state {
+	MEDIA_REQUEST_STATE_IDLE,
+	MEDIA_REQUEST_STATE_VALIDATING,
+	MEDIA_REQUEST_STATE_QUEUED,
+	MEDIA_REQUEST_STATE_COMPLETE,
+	MEDIA_REQUEST_STATE_CLEANING,
+};
+
+struct media_request_object;
+
+/**
+ * struct media_request - Media device request
+ * @mdev: Media device this request belongs to
+ * @kref: Reference count
+ * @debug_str: Prefix for debug messages (process name:fd)
+ * @state: The state of the request
+ * @objects: List of @struct media_request_object request objects
+ * @num_objects: The number of objects in the request
+ * @num_incompleted_objects: The number of incomplete objects in the request
+ * @poll_wait: Wait queue for poll
+ * @lock: Serializes access to this struct
+ */
+struct media_request {
+	struct media_device *mdev;
+	struct kref kref;
+	char debug_str[TASK_COMM_LEN + 11];
+	atomic_t state;
+	struct list_head objects;
+	unsigned int num_incomplete_objects;
+	struct wait_queue_head poll_wait;
+	spinlock_t lock;
+};
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+
+/**
+ * media_request_get - Get the media request
+ *
+ * @req: The request
+ *
+ * Get the media request.
+ */
+static inline void media_request_get(struct media_request *req)
+{
+	kref_get(&req->kref);
+}
+
+/**
+ * media_request_put - Put the media request
+ *
+ * @req: The request
+ *
+ * Put the media request. The media request will be released
+ * when the refcount reaches 0.
+ */
+void media_request_put(struct media_request *req);
+
+/**
+ * media_request_alloc - Allocate the media request
+ *
+ * @mdev: Media device this request belongs to
+ * @alloc: Store the request's file descriptor in this struct
+ *
+ * Allocated the media request and put the fd in @alloc->fd.
+ */
+int media_request_alloc(struct media_device *mdev,
+			struct media_request_alloc *alloc);
+
+#else
+
+static inline void media_request_get(struct media_request *req)
+{
+}
+
+static inline void media_request_put(struct media_request *req)
+{
+}
+
+#endif
+
+/**
+ * struct media_request_object_ops - Media request object operations
+ * @prepare: Validate and prepare the request object, optional.
+ * @unprepare: Unprepare the request object, optional.
+ * @queue: Queue the request object, optional.
+ * @unbind: Unbind the request object, optional.
+ * @release: Release the request object, required.
+ */
+struct media_request_object_ops {
+	int (*prepare)(struct media_request_object *object);
+	void (*unprepare)(struct media_request_object *object);
+	void (*queue)(struct media_request_object *object);
+	void (*unbind)(struct media_request_object *object);
+	void (*release)(struct media_request_object *object);
+};
+
+/**
+ * struct media_request_object - An opaque object that belongs to a media
+ *				 request
+ *
+ * @ops: object's operations
+ * @priv: object's priv pointer
+ * @req: the request this object belongs to (can be NULL)
+ * @list: List entry of the object for @struct media_request
+ * @kref: Reference count of the object, acquire before releasing req->lock
+ * @completed: If true, then this object was completed.
+ *
+ * An object related to the request. This struct is embedded in the
+ * larger object data.
+ */
+struct media_request_object {
+	const struct media_request_object_ops *ops;
+	void *priv;
+	struct media_request *req;
+	struct list_head list;
+	struct kref kref;
+	bool completed;
+};
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+
+/**
+ * media_request_object_get - Get a media request object
+ *
+ * @obj: The object
+ *
+ * Get a media request object.
+ */
+static inline void media_request_object_get(struct media_request_object *obj)
+{
+	kref_get(&obj->kref);
+}
+
+/**
+ * media_request_object_put - Put a media request object
+ *
+ * @obj: The object
+ *
+ * Put a media request object. Once all references are gone, the
+ * object's memory is released.
+ */
+void media_request_object_put(struct media_request_object *obj);
+
+/**
+ * media_request_object_init - Initialise a media request object
+ *
+ * Initialise a media request object. The object will be released using the
+ * release callback of the ops once it has no references (this function
+ * initialises references to one).
+ */
+void media_request_object_init(struct media_request_object *obj);
+
+/**
+ * media_request_object_bind - Bind a media request object to a request
+ */
+int media_request_object_bind(struct media_request *req,
+			      const struct media_request_object_ops *ops,
+			      void *priv,
+			      struct media_request_object *obj);
+
+/**
+ * media_request_object_unbind - Unbind a media request object
+ *
+ * @obj: The object
+ *
+ * Unbind the media request object from the request.
+ */
+void media_request_object_unbind(struct media_request_object *obj);
+
+/**
+ * media_request_object_complete - Mark the media request object as complete
+ *
+ * @obj: The object
+ *
+ * Mark the media request object as complete.
+ */
+void media_request_object_complete(struct media_request_object *obj);
+
+#else
+
+static inline void media_request_object_get(struct media_request_object *obj)
+{
+}
+
+static inline void media_request_object_put(struct media_request_object *obj)
+{
+}
+
+static inline void media_request_object_init(struct media_request_object *obj)
+{
+	obj->ops = NULL;
+	obj->req = NULL;
+}
+
+static inline int media_request_object_bind(struct media_request *req,
+			       const struct media_request_object_ops *ops,
+			       void *priv,
+			       struct media_request_object *obj)
+{
+	return 0;
+}
+
+static inline void media_request_object_unbind(struct media_request_object *obj)
+{
+}
+
+static inline void media_request_object_complete(struct media_request_object *obj)
+{
+}
+
+#endif
+
+#endif
-- 
2.17.0

[PATCHv13 04/28] media-request: add media_request_get_by_fd

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add media_request_get_by_fd() to find a request based on the file
descriptor.

The caller has to call media_request_put() for the returned
request since this function increments the refcount.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/media-request.c | 32 ++++++++++++++++++++++++++++++++
 include/media/media-request.h | 24 ++++++++++++++++++++++++
 2 files changed, 56 insertions(+)

diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
index c216c4ab628b..edc1c3af1959 100644
--- a/drivers/media/media-request.c
+++ b/drivers/media/media-request.c
@@ -218,6 +218,38 @@ static const struct file_operations request_fops = {
 	.release = media_request_close,
 };
 
+struct media_request *
+media_request_get_by_fd(struct media_device *mdev, int request_fd)
+{
+	struct file *filp;
+	struct media_request *req;
+
+	if (!mdev || !mdev->ops ||
+	    !mdev->ops->req_validate || !mdev->ops->req_queue)
+		return ERR_PTR(-EPERM);
+
+	filp = fget(request_fd);
+	if (!filp)
+		return ERR_PTR(-ENOENT);
+
+	if (filp->f_op != &request_fops)
+		goto err_fput;
+	req = filp->private_data;
+	if (req->mdev != mdev)
+		goto err_fput;
+
+	media_request_get(req);
+	fput(filp);
+
+	return req;
+
+err_fput:
+	fput(filp);
+
+	return ERR_PTR(-ENOENT);
+}
+EXPORT_SYMBOL_GPL(media_request_get_by_fd);
+
 int media_request_alloc(struct media_device *mdev,
 			struct media_request_alloc *alloc)
 {
diff --git a/include/media/media-request.h b/include/media/media-request.h
index e39122dfd717..997e096d7128 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -86,6 +86,24 @@ static inline void media_request_get(struct media_request *req)
  */
 void media_request_put(struct media_request *req);
 
+/**
+ * media_request_get_by_fd - Get a media request by fd
+ *
+ * @mdev: Media device this request belongs to
+ * @request_fd: The file descriptor of the request
+ *
+ * Get the request represented by @request_fd that is owned
+ * by the media device.
+ *
+ * Return a -EPERM error pointer if requests are not supported
+ * by this driver. Return -ENOENT if the request was not found.
+ * Return the pointer to the request if found: the caller will
+ * have to call @media_request_put when it finished using the
+ * request.
+ */
+struct media_request *
+media_request_get_by_fd(struct media_device *mdev, int request_fd);
+
 /**
  * media_request_alloc - Allocate the media request
  *
@@ -107,6 +125,12 @@ static inline void media_request_put(struct media_request *req)
 {
 }
 
+static inline struct media_request *
+media_request_get_by_fd(struct media_device *mdev, int request_fd)
+{
+	return ERR_PTR(-EPERM);
+}
+
 #endif
 
 /**
-- 
2.17.0

[PATCHv13 05/28] media-request: add media_request_object_find

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add media_request_object_find to find a request object inside a
request based on ops and/or priv values.

Objects of the same type (vb2 buffer, control handler) will have
the same ops value. And objects that refer to the same 'parent'
object (e.g. the v4l2_ctrl_handler that has the current driver
state) will have the same priv value.

The caller has to call media_request_object_put() for the returned
object since this function increments the refcount.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/media-request.c | 25 +++++++++++++++++++++++++
 include/media/media-request.h | 24 ++++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
index edc1c3af1959..c7e11e816e27 100644
--- a/drivers/media/media-request.c
+++ b/drivers/media/media-request.c
@@ -322,6 +322,31 @@ static void media_request_object_release(struct kref *kref)
 	obj->ops->release(obj);
 }
 
+struct media_request_object *
+media_request_object_find(struct media_request *req,
+			  const struct media_request_object_ops *ops,
+			  void *priv)
+{
+	struct media_request_object *obj;
+	struct media_request_object *found = NULL;
+	unsigned long flags;
+
+	if (WARN_ON(!ops || !priv))
+		return NULL;
+
+	spin_lock_irqsave(&req->lock, flags);
+	list_for_each_entry(obj, &req->objects, list) {
+		if (obj->ops == ops && obj->priv == priv) {
+			media_request_object_get(obj);
+			found = obj;
+			break;
+		}
+	}
+	spin_unlock_irqrestore(&req->lock, flags);
+	return found;
+}
+EXPORT_SYMBOL_GPL(media_request_object_find);
+
 void media_request_object_put(struct media_request_object *obj)
 {
 	kref_put(&obj->kref, media_request_object_release);
diff --git a/include/media/media-request.h b/include/media/media-request.h
index 997e096d7128..5367b4a2f91c 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
  */
 void media_request_object_put(struct media_request_object *obj);
 
+/**
+ * media_request_object_find - Find an object in a request
+ *
+ * @ops: Find an object with this ops value
+ * @priv: Find an object with this priv value
+ *
+ * Both @ops and @priv must be non-NULL.
+ *
+ * Returns NULL if not found or the object pointer. The caller must
+ * call media_request_object_put() once it finished using the object.
+ */
+struct media_request_object *
+media_request_object_find(struct media_request *req,
+			  const struct media_request_object_ops *ops,
+			  void *priv);
+
 /**
  * media_request_object_init - Initialise a media request object
  *
@@ -241,6 +257,14 @@ static inline void media_request_object_put(struct media_request_object *obj)
 {
 }
 
+static inline struct media_request_object *
+media_request_object_find(struct media_request *req,
+			  const struct media_request_object_ops *ops,
+			  void *priv)
+{
+	return NULL;
+}
+
 static inline void media_request_object_init(struct media_request_object *obj)
 {
 	obj->ops = NULL;
-- 
2.17.0

[PATCHv13 06/28] v4l2-dev: lock req_queue_mutex

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

We need to serialize streamon/off with queueing new requests.
These ioctls may trigger the cancellation of a streaming
operation, and that should not be mixed with queuing a new
request at the same time.

Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
with MEDIA_REQUEST_IOC_QUEUE.

Finally close() needs this lock since that too can trigger the
cancellation of a streaming operation.

We take the req_queue_mutex here before any other locks since
it is a very high-level lock.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-dev.c | 37 +++++++++++++++++++++++++++++-
 1 file changed, 36 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 1d0b2208e8fb..b1c9efc0ecc4 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -353,13 +353,36 @@ static long v4l2_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
 
 	if (vdev->fops->unlocked_ioctl) {
 		struct mutex *lock = v4l2_ioctl_get_lock(vdev, cmd);
+		struct mutex *queue_lock = NULL;
 
-		if (lock && mutex_lock_interruptible(lock))
+		/*
+		 * We need to serialize streamon/off with queueing new requests.
+		 * These ioctls may trigger the cancellation of a streaming
+		 * operation, and that should not be mixed with queueing a new
+		 * request at the same time.
+		 *
+		 * Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
+		 * with MEDIA_REQUEST_IOC_QUEUE.
+		 */
+		if (vdev->v4l2_dev->mdev &&
+		    (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF ||
+		     cmd == VIDIOC_S_EXT_CTRLS || cmd == VIDIOC_TRY_EXT_CTRLS))
+			queue_lock = &vdev->v4l2_dev->mdev->req_queue_mutex;
+
+		if (queue_lock && mutex_lock_interruptible(queue_lock))
+			return -ERESTARTSYS;
+
+		if (lock && mutex_lock_interruptible(lock)) {
+			if (queue_lock)
+				mutex_unlock(queue_lock);
 			return -ERESTARTSYS;
+		}
 		if (video_is_registered(vdev))
 			ret = vdev->fops->unlocked_ioctl(filp, cmd, arg);
 		if (lock)
 			mutex_unlock(lock);
+		if (queue_lock)
+			mutex_unlock(queue_lock);
 	} else
 		ret = -ENOTTY;
 
@@ -442,8 +465,20 @@ static int v4l2_release(struct inode *inode, struct file *filp)
 	struct video_device *vdev = video_devdata(filp);
 	int ret = 0;
 
+	/*
+	 * We need to serialize the release() with queueing new requests.
+	 * The release() may trigger the cancellation of a streaming
+	 * operation, and that should not be mixed with queueing a new
+	 * request at the same time.
+	 */
+	if (vdev->v4l2_dev->mdev)
+		mutex_lock(&vdev->v4l2_dev->mdev->req_queue_mutex);
+
 	if (vdev->fops->release)
 		ret = vdev->fops->release(filp);
+
+	if (vdev->v4l2_dev->mdev)
+		mutex_unlock(&vdev->v4l2_dev->mdev->req_queue_mutex);
 	if (vdev->dev_debug & V4L2_DEV_DEBUG_FOP)
 		printk(KERN_DEBUG "%s: release\n",
 			video_device_node_name(vdev));
-- 
2.17.0

[PATCHv13 07/28] videodev2.h: add request_fd field to v4l2_ext_controls

[Hans Verkuil]

From: Alexandre Courbot <acourbot@chromium.org>

If 'which' is V4L2_CTRL_WHICH_REQUEST_VAL, then the 'request_fd' field
can be used to specify a request for the G/S/TRY_EXT_CTRLS ioctls.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 5 ++++-
 drivers/media/v4l2-core/v4l2-ioctl.c          | 6 +++---
 include/uapi/linux/videodev2.h                | 4 +++-
 3 files changed, 10 insertions(+), 5 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 6481212fda77..dcce86c1fe40 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -834,7 +834,8 @@ struct v4l2_ext_controls32 {
 	__u32 which;
 	__u32 count;
 	__u32 error_idx;
-	__u32 reserved[2];
+	__s32 request_fd;
+	__u32 reserved[1];
 	compat_caddr_t controls; /* actually struct v4l2_ext_control32 * */
 };
 
@@ -909,6 +910,7 @@ static int get_v4l2_ext_controls32(struct file *file,
 	    get_user(count, &p32->count) ||
 	    put_user(count, &p64->count) ||
 	    assign_in_user(&p64->error_idx, &p32->error_idx) ||
+	    assign_in_user(&p64->request_fd, &p32->request_fd) ||
 	    copy_in_user(p64->reserved, p32->reserved, sizeof(p64->reserved)))
 		return -EFAULT;
 
@@ -974,6 +976,7 @@ static int put_v4l2_ext_controls32(struct file *file,
 	    get_user(count, &p64->count) ||
 	    put_user(count, &p32->count) ||
 	    assign_in_user(&p32->error_idx, &p64->error_idx) ||
+	    assign_in_user(&p32->request_fd, &p64->request_fd) ||
 	    copy_in_user(p32->reserved, p64->reserved, sizeof(p32->reserved)) ||
 	    get_user(kcontrols, &p64->controls))
 		return -EFAULT;
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index f48c505550e0..9ce23e23c5bf 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -553,8 +553,8 @@ static void v4l_print_ext_controls(const void *arg, bool write_only)
 	const struct v4l2_ext_controls *p = arg;
 	int i;
 
-	pr_cont("which=0x%x, count=%d, error_idx=%d",
-			p->which, p->count, p->error_idx);
+	pr_cont("which=0x%x, count=%d, error_idx=%d, request_fd=%d",
+			p->which, p->count, p->error_idx, p->request_fd);
 	for (i = 0; i < p->count; i++) {
 		if (!p->controls[i].size)
 			pr_cont(", id/val=0x%x/0x%x",
@@ -870,7 +870,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
 	__u32 i;
 
 	/* zero the reserved fields */
-	c->reserved[0] = c->reserved[1] = 0;
+	c->reserved[0] = 0;
 	for (i = 0; i < c->count; i++)
 		c->controls[i].reserved2[0] = 0;
 
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 600877be5c22..16b53b82496c 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1592,7 +1592,8 @@ struct v4l2_ext_controls {
 	};
 	__u32 count;
 	__u32 error_idx;
-	__u32 reserved[2];
+	__s32 request_fd;
+	__u32 reserved[1];
 	struct v4l2_ext_control *controls;
 };
 
@@ -1605,6 +1606,7 @@ struct v4l2_ext_controls {
 #define V4L2_CTRL_MAX_DIMS	  (4)
 #define V4L2_CTRL_WHICH_CUR_VAL   0
 #define V4L2_CTRL_WHICH_DEF_VAL   0x0f000000
+#define V4L2_CTRL_WHICH_REQUEST_VAL 0x0f010000
 
 enum v4l2_ctrl_type {
 	V4L2_CTRL_TYPE_INTEGER	     = 1,
-- 
2.17.0

[PATCHv13 08/28] v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add a 'bool from_other_dev' argument: set to true if the two
handlers refer to different devices (e.g. it is true when
inheriting controls from a subdev into a main v4l2 bridge
driver).

This will be used later when implementing support for the
request API since we need to skip such controls.

TODO: check drivers/staging/media/imx/imx-media-fim.c change.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/dvb-frontends/rtl2832_sdr.c     |  5 +-
 drivers/media/pci/bt8xx/bttv-driver.c         |  2 +-
 drivers/media/pci/cx23885/cx23885-417.c       |  2 +-
 drivers/media/pci/cx88/cx88-blackbird.c       |  2 +-
 drivers/media/pci/cx88/cx88-video.c           |  2 +-
 drivers/media/pci/saa7134/saa7134-empress.c   |  4 +-
 drivers/media/pci/saa7134/saa7134-video.c     |  2 +-
 .../media/platform/exynos4-is/fimc-capture.c  |  2 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |  2 +-
 drivers/media/platform/rcar_drif.c            |  2 +-
 .../media/platform/soc_camera/soc_camera.c    |  3 +-
 drivers/media/platform/vivid/vivid-ctrls.c    | 46 +++++++++----------
 drivers/media/usb/cx231xx/cx231xx-417.c       |  2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c     |  4 +-
 drivers/media/usb/msi2500/msi2500.c           |  2 +-
 drivers/media/usb/tm6000/tm6000-video.c       |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c          | 11 +++--
 drivers/media/v4l2-core/v4l2-device.c         |  3 +-
 drivers/staging/media/imx/imx-media-dev.c     |  2 +-
 drivers/staging/media/imx/imx-media-fim.c     |  2 +-
 include/media/v4l2-ctrls.h                    |  8 +++-
 21 files changed, 61 insertions(+), 49 deletions(-)

diff --git a/drivers/media/dvb-frontends/rtl2832_sdr.c b/drivers/media/dvb-frontends/rtl2832_sdr.c
index c6e78d870ccd..6064d28224e8 100644
--- a/drivers/media/dvb-frontends/rtl2832_sdr.c
+++ b/drivers/media/dvb-frontends/rtl2832_sdr.c
@@ -1394,7 +1394,8 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
 	case RTL2832_SDR_TUNER_E4000:
 		v4l2_ctrl_handler_init(&dev->hdl, 9);
 		if (subdev)
-			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler, NULL);
+			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler,
+					      NULL, true);
 		break;
 	case RTL2832_SDR_TUNER_R820T:
 	case RTL2832_SDR_TUNER_R828D:
@@ -1423,7 +1424,7 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
 		v4l2_ctrl_handler_init(&dev->hdl, 2);
 		if (subdev)
 			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler,
-					      NULL);
+					      NULL, true);
 		break;
 	default:
 		v4l2_ctrl_handler_init(&dev->hdl, 0);
diff --git a/drivers/media/pci/bt8xx/bttv-driver.c b/drivers/media/pci/bt8xx/bttv-driver.c
index 707f57a9f940..802c248c8d65 100644
--- a/drivers/media/pci/bt8xx/bttv-driver.c
+++ b/drivers/media/pci/bt8xx/bttv-driver.c
@@ -4211,7 +4211,7 @@ static int bttv_probe(struct pci_dev *dev, const struct pci_device_id *pci_id)
 	/* register video4linux + input */
 	if (!bttv_tvcards[btv->c.type].no_video) {
 		v4l2_ctrl_add_handler(&btv->radio_ctrl_handler, hdl,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, false);
 		if (btv->radio_ctrl_handler.error) {
 			result = btv->radio_ctrl_handler.error;
 			goto fail2;
diff --git a/drivers/media/pci/cx23885/cx23885-417.c b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..762823871c78 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1527,7 +1527,7 @@ int cx23885_417_register(struct cx23885_dev *dev)
 	dev->cxhdl.priv = dev;
 	dev->cxhdl.func = cx23885_api_func;
 	cx2341x_handler_set_50hz(&dev->cxhdl, tsport->height == 576);
-	v4l2_ctrl_add_handler(&dev->ctrl_handler, &dev->cxhdl.hdl, NULL);
+	v4l2_ctrl_add_handler(&dev->ctrl_handler, &dev->cxhdl.hdl, NULL, false);
 
 	/* Allocate and initialize V4L video device */
 	dev->v4l_device = cx23885_video_dev_alloc(tsport,
diff --git a/drivers/media/pci/cx88/cx88-blackbird.c b/drivers/media/pci/cx88/cx88-blackbird.c
index 0e0952e60795..39f69d89a663 100644
--- a/drivers/media/pci/cx88/cx88-blackbird.c
+++ b/drivers/media/pci/cx88/cx88-blackbird.c
@@ -1183,7 +1183,7 @@ static int cx8802_blackbird_probe(struct cx8802_driver *drv)
 	err = cx2341x_handler_init(&dev->cxhdl, 36);
 	if (err)
 		goto fail_core;
-	v4l2_ctrl_add_handler(&dev->cxhdl.hdl, &core->video_hdl, NULL);
+	v4l2_ctrl_add_handler(&dev->cxhdl.hdl, &core->video_hdl, NULL, false);
 
 	/* blackbird stuff */
 	pr_info("cx23416 based mpeg encoder (blackbird reference design)\n");
diff --git a/drivers/media/pci/cx88/cx88-video.c b/drivers/media/pci/cx88/cx88-video.c
index 9be682cdb644..e35bfa03a1e2 100644
--- a/drivers/media/pci/cx88/cx88-video.c
+++ b/drivers/media/pci/cx88/cx88-video.c
@@ -1378,7 +1378,7 @@ static int cx8800_initdev(struct pci_dev *pci_dev,
 		if (vc->id == V4L2_CID_CHROMA_AGC)
 			core->chroma_agc = vc;
 	}
-	v4l2_ctrl_add_handler(&core->video_hdl, &core->audio_hdl, NULL);
+	v4l2_ctrl_add_handler(&core->video_hdl, &core->audio_hdl, NULL, false);
 
 	/* load and configure helper modules */
 
diff --git a/drivers/media/pci/saa7134/saa7134-empress.c b/drivers/media/pci/saa7134/saa7134-empress.c
index 66acfd35ffc6..fc75ce00dbf8 100644
--- a/drivers/media/pci/saa7134/saa7134-empress.c
+++ b/drivers/media/pci/saa7134/saa7134-empress.c
@@ -265,9 +265,9 @@ static int empress_init(struct saa7134_dev *dev)
 		 "%s empress (%s)", dev->name,
 		 saa7134_boards[dev->board].name);
 	v4l2_ctrl_handler_init(hdl, 21);
-	v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler, empress_ctrl_filter);
+	v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler, empress_ctrl_filter, false);
 	if (dev->empress_sd)
-		v4l2_ctrl_add_handler(hdl, dev->empress_sd->ctrl_handler, NULL);
+		v4l2_ctrl_add_handler(hdl, dev->empress_sd->ctrl_handler, NULL, true);
 	if (hdl->error) {
 		video_device_release(dev->empress_dev);
 		return hdl->error;
diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c
index 1a50ec9d084f..41d46488d22e 100644
--- a/drivers/media/pci/saa7134/saa7134-video.c
+++ b/drivers/media/pci/saa7134/saa7134-video.c
@@ -2136,7 +2136,7 @@ int saa7134_video_init1(struct saa7134_dev *dev)
 		hdl = &dev->radio_ctrl_handler;
 		v4l2_ctrl_handler_init(hdl, 2);
 		v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, false);
 		if (hdl->error)
 			return hdl->error;
 	}
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index a3cdac188190..2164375f0ee0 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -1424,7 +1424,7 @@ static int fimc_link_setup(struct media_entity *entity,
 		return 0;
 
 	return v4l2_ctrl_add_handler(&vc->ctx->ctrls.handler,
-				     sensor->ctrl_handler, NULL);
+				     sensor->ctrl_handler, NULL, true);
 }
 
 static const struct media_entity_operations fimc_sd_media_ops = {
diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
index 55b745ac86a5..4fd6cbae065d 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -439,7 +439,7 @@ static int rvin_digital_subdevice_attach(struct rvin_dev *vin,
 		return ret;
 
 	ret = v4l2_ctrl_add_handler(&vin->ctrl_handler, subdev->ctrl_handler,
-				    NULL);
+				    NULL, true);
 	if (ret < 0) {
 		v4l2_ctrl_handler_free(&vin->ctrl_handler);
 		return ret;
diff --git a/drivers/media/platform/rcar_drif.c b/drivers/media/platform/rcar_drif.c
index dc7e280c91b4..159c7d2c2066 100644
--- a/drivers/media/platform/rcar_drif.c
+++ b/drivers/media/platform/rcar_drif.c
@@ -1168,7 +1168,7 @@ static int rcar_drif_notify_complete(struct v4l2_async_notifier *notifier)
 	}
 
 	ret = v4l2_ctrl_add_handler(&sdr->ctrl_hdl,
-				    sdr->ep.subdev->ctrl_handler, NULL);
+				    sdr->ep.subdev->ctrl_handler, NULL, true);
 	if (ret) {
 		rdrif_err(sdr, "failed: ctrl add hdlr ret %d\n", ret);
 		goto error;
diff --git a/drivers/media/platform/soc_camera/soc_camera.c b/drivers/media/platform/soc_camera/soc_camera.c
index 69f0d8e80bd8..e6787abc34ae 100644
--- a/drivers/media/platform/soc_camera/soc_camera.c
+++ b/drivers/media/platform/soc_camera/soc_camera.c
@@ -1180,7 +1180,8 @@ static int soc_camera_probe_finish(struct soc_camera_device *icd)
 
 	v4l2_subdev_call(sd, video, g_tvnorms, &icd->vdev->tvnorms);
 
-	ret = v4l2_ctrl_add_handler(&icd->ctrl_handler, sd->ctrl_handler, NULL);
+	ret = v4l2_ctrl_add_handler(&icd->ctrl_handler, sd->ctrl_handler,
+				    NULL, true);
 	if (ret < 0)
 		return ret;
 
diff --git a/drivers/media/platform/vivid/vivid-ctrls.c b/drivers/media/platform/vivid/vivid-ctrls.c
index 6b0bfa091592..f369b94ad7ff 100644
--- a/drivers/media/platform/vivid/vivid-ctrls.c
+++ b/drivers/media/platform/vivid/vivid-ctrls.c
@@ -1662,59 +1662,59 @@ int vivid_create_controls(struct vivid_dev *dev, bool show_ccs_cap,
 		v4l2_ctrl_auto_cluster(2, &dev->autogain, 0, true);
 
 	if (dev->has_vid_cap) {
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_vid, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_aud, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_sdtv_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_loop_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_fb, NULL);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_vid, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_aud, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_sdtv_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_loop_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_fb, NULL, false);
 		if (hdl_vid_cap->error)
 			return hdl_vid_cap->error;
 		dev->vid_cap_dev.ctrl_handler = hdl_vid_cap;
 	}
 	if (dev->has_vid_out) {
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_aud, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_fb, NULL);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_aud, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_fb, NULL, false);
 		if (hdl_vid_out->error)
 			return hdl_vid_out->error;
 		dev->vid_out_dev.ctrl_handler = hdl_vid_out;
 	}
 	if (dev->has_vbi_cap) {
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_sdtv_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_loop_cap, NULL);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_sdtv_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_loop_cap, NULL, false);
 		if (hdl_vbi_cap->error)
 			return hdl_vbi_cap->error;
 		dev->vbi_cap_dev.ctrl_handler = hdl_vbi_cap;
 	}
 	if (dev->has_vbi_out) {
-		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_streaming, NULL);
+		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_streaming, NULL, false);
 		if (hdl_vbi_out->error)
 			return hdl_vbi_out->error;
 		dev->vbi_out_dev.ctrl_handler = hdl_vbi_out;
 	}
 	if (dev->has_radio_rx) {
-		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_aud, NULL);
+		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_aud, NULL, false);
 		if (hdl_radio_rx->error)
 			return hdl_radio_rx->error;
 		dev->radio_rx_dev.ctrl_handler = hdl_radio_rx;
 	}
 	if (dev->has_radio_tx) {
-		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_aud, NULL);
+		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_aud, NULL, false);
 		if (hdl_radio_tx->error)
 			return hdl_radio_tx->error;
 		dev->radio_tx_dev.ctrl_handler = hdl_radio_tx;
 	}
 	if (dev->has_sdr_cap) {
-		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_streaming, NULL);
+		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_streaming, NULL, false);
 		if (hdl_sdr_cap->error)
 			return hdl_sdr_cap->error;
 		dev->sdr_cap_dev.ctrl_handler = hdl_sdr_cap;
diff --git a/drivers/media/usb/cx231xx/cx231xx-417.c b/drivers/media/usb/cx231xx/cx231xx-417.c
index b80e6857e2eb..fca16cf8b3bf 100644
--- a/drivers/media/usb/cx231xx/cx231xx-417.c
+++ b/drivers/media/usb/cx231xx/cx231xx-417.c
@@ -1991,7 +1991,7 @@ int cx231xx_417_register(struct cx231xx *dev)
 	dev->mpeg_ctrl_handler.ops = &cx231xx_ops;
 	if (dev->sd_cx25840)
 		v4l2_ctrl_add_handler(&dev->mpeg_ctrl_handler.hdl,
-				dev->sd_cx25840->ctrl_handler, NULL);
+				dev->sd_cx25840->ctrl_handler, NULL, false);
 	if (dev->mpeg_ctrl_handler.hdl.error) {
 		err = dev->mpeg_ctrl_handler.hdl.error;
 		dprintk(3, "%s: can't add cx25840 controls\n", dev->name);
diff --git a/drivers/media/usb/cx231xx/cx231xx-video.c b/drivers/media/usb/cx231xx/cx231xx-video.c
index f7fcd733a2ca..2dedb18f63a0 100644
--- a/drivers/media/usb/cx231xx/cx231xx-video.c
+++ b/drivers/media/usb/cx231xx/cx231xx-video.c
@@ -2204,10 +2204,10 @@ int cx231xx_register_analog_devices(struct cx231xx *dev)
 
 	if (dev->sd_cx25840) {
 		v4l2_ctrl_add_handler(&dev->ctrl_handler,
-				dev->sd_cx25840->ctrl_handler, NULL);
+				dev->sd_cx25840->ctrl_handler, NULL, true);
 		v4l2_ctrl_add_handler(&dev->radio_ctrl_handler,
 				dev->sd_cx25840->ctrl_handler,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, true);
 	}
 
 	if (dev->ctrl_handler.error)
diff --git a/drivers/media/usb/msi2500/msi2500.c b/drivers/media/usb/msi2500/msi2500.c
index 65ef755adfdc..4aacd77a5d58 100644
--- a/drivers/media/usb/msi2500/msi2500.c
+++ b/drivers/media/usb/msi2500/msi2500.c
@@ -1278,7 +1278,7 @@ static int msi2500_probe(struct usb_interface *intf,
 	}
 
 	/* currently all controls are from subdev */
-	v4l2_ctrl_add_handler(&dev->hdl, sd->ctrl_handler, NULL);
+	v4l2_ctrl_add_handler(&dev->hdl, sd->ctrl_handler, NULL, true);
 
 	dev->v4l2_dev.ctrl_handler = &dev->hdl;
 	dev->vdev.v4l2_dev = &dev->v4l2_dev;
diff --git a/drivers/media/usb/tm6000/tm6000-video.c b/drivers/media/usb/tm6000/tm6000-video.c
index b2399d4266da..99d6b0da8b3d 100644
--- a/drivers/media/usb/tm6000/tm6000-video.c
+++ b/drivers/media/usb/tm6000/tm6000-video.c
@@ -1622,7 +1622,7 @@ int tm6000_v4l2_register(struct tm6000_core *dev)
 	v4l2_ctrl_new_std(&dev->ctrl_handler, &tm6000_ctrl_ops,
 			V4L2_CID_HUE, -128, 127, 1, 0);
 	v4l2_ctrl_add_handler(&dev->ctrl_handler,
-			&dev->radio_ctrl_handler, NULL);
+			&dev->radio_ctrl_handler, NULL, false);
 
 	if (dev->radio_ctrl_handler.error)
 		ret = dev->radio_ctrl_handler.error;
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index d29e45516eb7..aa1dd2015e84 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1995,7 +1995,8 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
-			   struct v4l2_ctrl *ctrl)
+			   struct v4l2_ctrl *ctrl,
+			   bool from_other_dev)
 {
 	struct v4l2_ctrl_ref *ref;
 	struct v4l2_ctrl_ref *new_ref;
@@ -2019,6 +2020,7 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	if (!new_ref)
 		return handler_set_err(hdl, -ENOMEM);
 	new_ref->ctrl = ctrl;
+	new_ref->from_other_dev = from_other_dev;
 	if (ctrl->handler == hdl) {
 		/* By default each control starts in a cluster of its own.
 		   new_ref->ctrl is basically a cluster array with one
@@ -2199,7 +2201,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
 		ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
 	}
 
-	if (handler_new_ref(hdl, ctrl)) {
+	if (handler_new_ref(hdl, ctrl, false)) {
 		kvfree(ctrl);
 		return NULL;
 	}
@@ -2368,7 +2370,8 @@ EXPORT_SYMBOL(v4l2_ctrl_new_int_menu);
 /* Add the controls from another handler to our own. */
 int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 			  struct v4l2_ctrl_handler *add,
-			  bool (*filter)(const struct v4l2_ctrl *ctrl))
+			  bool (*filter)(const struct v4l2_ctrl *ctrl),
+			  bool from_other_dev)
 {
 	struct v4l2_ctrl_ref *ref;
 	int ret = 0;
@@ -2391,7 +2394,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 		/* Filter any unwanted controls */
 		if (filter && !filter(ctrl))
 			continue;
-		ret = handler_new_ref(hdl, ctrl);
+		ret = handler_new_ref(hdl, ctrl, from_other_dev);
 		if (ret)
 			break;
 	}
diff --git a/drivers/media/v4l2-core/v4l2-device.c b/drivers/media/v4l2-core/v4l2-device.c
index 937c6de85606..8391a7f0895b 100644
--- a/drivers/media/v4l2-core/v4l2-device.c
+++ b/drivers/media/v4l2-core/v4l2-device.c
@@ -178,7 +178,8 @@ int v4l2_device_register_subdev(struct v4l2_device *v4l2_dev,
 
 	sd->v4l2_dev = v4l2_dev;
 	/* This just returns 0 if either of the two args is NULL */
-	err = v4l2_ctrl_add_handler(v4l2_dev->ctrl_handler, sd->ctrl_handler, NULL);
+	err = v4l2_ctrl_add_handler(v4l2_dev->ctrl_handler, sd->ctrl_handler,
+				    NULL, true);
 	if (err)
 		goto error_module;
 
diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
index 289d775c4820..08799beaea42 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -391,7 +391,7 @@ static int imx_media_inherit_controls(struct imx_media_dev *imxmd,
 
 		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
 					    sd->ctrl_handler,
-					    NULL);
+					    NULL, true);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
index 6df189135db8..8cf773eef9da 100644
--- a/drivers/staging/media/imx/imx-media-fim.c
+++ b/drivers/staging/media/imx/imx-media-fim.c
@@ -463,7 +463,7 @@ int imx_media_fim_add_controls(struct imx_media_fim *fim)
 {
 	/* add the FIM controls to the calling subdev ctrl handler */
 	return v4l2_ctrl_add_handler(fim->sd->ctrl_handler,
-				     &fim->ctrl_handler, NULL);
+				     &fim->ctrl_handler, NULL, false);
 }
 EXPORT_SYMBOL_GPL(imx_media_fim_add_controls);
 
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 5b445b5654f7..d26b8ddebb56 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -247,6 +247,8 @@ struct v4l2_ctrl {
  * @ctrl:	The actual control information.
  * @helper:	Pointer to helper struct. Used internally in
  *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
+ * @from_other_dev: If true, then @ctrl was defined in another
+ *		device than the &struct v4l2_ctrl_handler.
  *
  * Each control handler has a list of these refs. The list_head is used to
  * keep a sorted-by-control-ID list of all controls, while the next pointer
@@ -257,6 +259,7 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl_ref *next;
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
+	bool from_other_dev;
 };
 
 /**
@@ -633,6 +636,8 @@ typedef bool (*v4l2_ctrl_filter)(const struct v4l2_ctrl *ctrl);
  * @add:	The control handler whose controls you want to add to
  *		the @hdl control handler.
  * @filter:	This function will filter which controls should be added.
+ * @from_other_dev: If true, then the controls in @add were defined in another
+ *		device than @hdl.
  *
  * Does nothing if either of the two handlers is a NULL pointer.
  * If @filter is NULL, then all controls are added. Otherwise only those
@@ -642,7 +647,8 @@ typedef bool (*v4l2_ctrl_filter)(const struct v4l2_ctrl *ctrl);
  */
 int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 			  struct v4l2_ctrl_handler *add,
-			  v4l2_ctrl_filter filter);
+			  v4l2_ctrl_filter filter,
+			  bool from_other_dev);
 
 /**
  * v4l2_ctrl_radio_filter() - Standard filter for radio controls.
-- 
2.17.0

[PATCHv13 09/28] v4l2-ctrls: prepare internal structs for request API

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Embed and initialize a media_request_object in struct v4l2_ctrl_handler.

Add a p_req field to struct v4l2_ctrl_ref that will store the
request value.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
 include/media/v4l2-ctrls.h           | 7 +++++++
 2 files changed, 8 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index aa1dd2015e84..d09f49530d9e 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1880,6 +1880,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
 				      sizeof(hdl->buckets[0]),
 				      GFP_KERNEL | __GFP_ZERO);
 	hdl->error = hdl->buckets ? 0 : -ENOMEM;
+	media_request_object_init(&hdl->req_obj);
 	return hdl->error;
 }
 EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index d26b8ddebb56..76352eb59f14 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -20,6 +20,7 @@
 #include <linux/list.h>
 #include <linux/mutex.h>
 #include <linux/videodev2.h>
+#include <media/media-request.h>
 
 /* forward references */
 struct file;
@@ -249,6 +250,8 @@ struct v4l2_ctrl {
  *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
  * @from_other_dev: If true, then @ctrl was defined in another
  *		device than the &struct v4l2_ctrl_handler.
+ * @p_req:	The request value. Only used if the control handler
+ *		is bound to a media request.
  *
  * Each control handler has a list of these refs. The list_head is used to
  * keep a sorted-by-control-ID list of all controls, while the next pointer
@@ -260,6 +263,7 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
 	bool from_other_dev;
+	union v4l2_ctrl_ptr p_req;
 };
 
 /**
@@ -283,6 +287,8 @@ struct v4l2_ctrl_ref {
  * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
  * @nr_of_buckets: Total number of buckets in the array.
  * @error:	The error code of the first failed control addition.
+ * @req_obj:	The &struct media_request_object, used to link into a
+ *		&struct media_request.
  */
 struct v4l2_ctrl_handler {
 	struct mutex _lock;
@@ -295,6 +301,7 @@ struct v4l2_ctrl_handler {
 	void *notify_priv;
 	u16 nr_of_buckets;
 	int error;
+	struct media_request_object req_obj;
 };
 
 /**
-- 
2.17.0

[PATCHv13 10/28] v4l2-ctrls: alloc memory for p_req

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

To store request data the handler_new_ref() allocates memory
for it if needed.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 20 ++++++++++++++++----
 1 file changed, 16 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index d09f49530d9e..3c1b00baa8d0 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1997,13 +1997,18 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 			   struct v4l2_ctrl *ctrl,
-			   bool from_other_dev)
+			   struct v4l2_ctrl_ref **ctrl_ref,
+			   bool from_other_dev, bool allocate_req)
 {
 	struct v4l2_ctrl_ref *ref;
 	struct v4l2_ctrl_ref *new_ref;
 	u32 id = ctrl->id;
 	u32 class_ctrl = V4L2_CTRL_ID2WHICH(id) | 1;
 	int bucket = id % hdl->nr_of_buckets;	/* which bucket to use */
+	unsigned int sz_extra = 0;
+
+	if (ctrl_ref)
+		*ctrl_ref = NULL;
 
 	/*
 	 * Automatically add the control class if it is not yet present and
@@ -2017,11 +2022,16 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	if (hdl->error)
 		return hdl->error;
 
-	new_ref = kzalloc(sizeof(*new_ref), GFP_KERNEL);
+	if (allocate_req)
+		sz_extra = ctrl->elems * ctrl->elem_size;
+	new_ref = kzalloc(sizeof(*new_ref) + sz_extra, GFP_KERNEL);
 	if (!new_ref)
 		return handler_set_err(hdl, -ENOMEM);
 	new_ref->ctrl = ctrl;
 	new_ref->from_other_dev = from_other_dev;
+	if (sz_extra)
+		new_ref->p_req.p = &new_ref[1];
+
 	if (ctrl->handler == hdl) {
 		/* By default each control starts in a cluster of its own.
 		   new_ref->ctrl is basically a cluster array with one
@@ -2061,6 +2071,8 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	/* Insert the control node in the hash */
 	new_ref->next = hdl->buckets[bucket];
 	hdl->buckets[bucket] = new_ref;
+	if (ctrl_ref)
+		*ctrl_ref = new_ref;
 
 unlock:
 	mutex_unlock(hdl->lock);
@@ -2202,7 +2214,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
 		ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
 	}
 
-	if (handler_new_ref(hdl, ctrl, false)) {
+	if (handler_new_ref(hdl, ctrl, NULL, false, false)) {
 		kvfree(ctrl);
 		return NULL;
 	}
@@ -2395,7 +2407,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 		/* Filter any unwanted controls */
 		if (filter && !filter(ctrl))
 			continue;
-		ret = handler_new_ref(hdl, ctrl, from_other_dev);
+		ret = handler_new_ref(hdl, ctrl, NULL, from_other_dev, false);
 		if (ret)
 			break;
 	}
-- 
2.17.0

[PATCHv13 11/28] v4l2-ctrls: use ref in helper instead of ctrl

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

The next patch needs the reference to a control instead of the
control itself, so change struct v4l2_ctrl_helper accordingly.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 3c1b00baa8d0..da4cc1485dc4 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -37,8 +37,8 @@
 struct v4l2_ctrl_helper {
 	/* Pointer to the control reference of the master control */
 	struct v4l2_ctrl_ref *mref;
-	/* The control corresponding to the v4l2_ext_control ID field. */
-	struct v4l2_ctrl *ctrl;
+	/* The control ref corresponding to the v4l2_ext_control ID field. */
+	struct v4l2_ctrl_ref *ref;
 	/* v4l2_ext_control index of the next control belonging to the
 	   same cluster, or 0 if there isn't any. */
 	u32 next;
@@ -2887,6 +2887,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 		ref = find_ref_lock(hdl, id);
 		if (ref == NULL)
 			return -EINVAL;
+		h->ref = ref;
 		ctrl = ref->ctrl;
 		if (ctrl->flags & V4L2_CTRL_FLAG_DISABLED)
 			return -EINVAL;
@@ -2909,7 +2910,6 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 		}
 		/* Store the ref to the master control of the cluster */
 		h->mref = ref;
-		h->ctrl = ctrl;
 		/* Initially set next to 0, meaning that there is no other
 		   control in this helper array belonging to the same
 		   cluster */
@@ -2994,7 +2994,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 	cs->error_idx = cs->count;
 
 	for (i = 0; !ret && i < cs->count; i++)
-		if (helpers[i].ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
+		if (helpers[i].ref->ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
 			ret = -EACCES;
 
 	for (i = 0; !ret && i < cs->count; i++) {
@@ -3029,7 +3029,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 
 			do {
 				ret = ctrl_to_user(cs->controls + idx,
-						   helpers[idx].ctrl);
+						   helpers[idx].ref->ctrl);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
@@ -3168,7 +3168,7 @@ static int validate_ctrls(struct v4l2_ext_controls *cs,
 
 	cs->error_idx = cs->count;
 	for (i = 0; i < cs->count; i++) {
-		struct v4l2_ctrl *ctrl = helpers[i].ctrl;
+		struct v4l2_ctrl *ctrl = helpers[i].ref->ctrl;
 		union v4l2_ctrl_ptr p_new;
 
 		cs->error_idx = i;
@@ -3280,7 +3280,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 			do {
 				/* Check if the auto control is part of the
 				   list, and remember the new value. */
-				if (helpers[tmp_idx].ctrl == master)
+				if (helpers[tmp_idx].ref->ctrl == master)
 					new_auto_val = cs->controls[tmp_idx].value;
 				tmp_idx = helpers[tmp_idx].next;
 			} while (tmp_idx);
@@ -3293,7 +3293,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 		/* Copy the new caller-supplied control values.
 		   user_to_new() sets 'is_new' to 1. */
 		do {
-			struct v4l2_ctrl *ctrl = helpers[idx].ctrl;
+			struct v4l2_ctrl *ctrl = helpers[idx].ref->ctrl;
 
 			ret = user_to_new(cs->controls + idx, ctrl);
 			if (!ret && ctrl->is_ptr)
@@ -3309,7 +3309,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 			idx = i;
 			do {
 				ret = new_to_user(cs->controls + idx,
-						helpers[idx].ctrl);
+						helpers[idx].ref->ctrl);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
-- 
2.17.0

[PATCHv13 12/28] v4l2-ctrls: add core request support

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Integrate the request support. This adds the v4l2_ctrl_request_complete
and v4l2_ctrl_request_setup functions to complete a request and (as a
helper function) to apply a request to the hardware.

It takes care of queuing requests and correctly chaining control values
in the request queue.

Note that when a request is marked completed it will copy control values
to the internal request state. This can be optimized in the future since
this is sub-optimal when dealing with large compound and/or array controls.

For the initial 'stateless codec' use-case the current implementation is
sufficient.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
 include/media/v4l2-ctrls.h           |  23 ++
 2 files changed, 348 insertions(+), 6 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index da4cc1485dc4..56b986185463 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1647,6 +1647,13 @@ static int new_to_user(struct v4l2_ext_control *c,
 	return ptr_to_user(c, ctrl, ctrl->p_new);
 }
 
+/* Helper function: copy the request value back to the caller */
+static int req_to_user(struct v4l2_ext_control *c,
+		       struct v4l2_ctrl_ref *ref)
+{
+	return ptr_to_user(c, ref->ctrl, ref->p_req);
+}
+
 /* Helper function: copy the initial control value back to the caller */
 static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
 {
@@ -1766,6 +1773,26 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
 	ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
 }
 
+/* Copy the new value to the request value */
+static void new_to_req(struct v4l2_ctrl_ref *ref)
+{
+	if (!ref)
+		return;
+	ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
+	ref->req = ref;
+}
+
+/* Copy the request value to the new value */
+static void req_to_new(struct v4l2_ctrl_ref *ref)
+{
+	if (!ref)
+		return;
+	if (ref->req)
+		ptr_to_ptr(ref->ctrl, ref->req->p_req, ref->ctrl->p_new);
+	else
+		ptr_to_ptr(ref->ctrl, ref->ctrl->p_cur, ref->ctrl->p_new);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
    value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -1875,6 +1902,9 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
 	lockdep_set_class_and_name(hdl->lock, key, name);
 	INIT_LIST_HEAD(&hdl->ctrls);
 	INIT_LIST_HEAD(&hdl->ctrl_refs);
+	INIT_LIST_HEAD(&hdl->requests);
+	INIT_LIST_HEAD(&hdl->requests_queued);
+	hdl->request_is_queued = false;
 	hdl->nr_of_buckets = 1 + nr_of_controls_hint / 8;
 	hdl->buckets = kvmalloc_array(hdl->nr_of_buckets,
 				      sizeof(hdl->buckets[0]),
@@ -1895,6 +1925,14 @@ void v4l2_ctrl_handler_free(struct v4l2_ctrl_handler *hdl)
 	if (hdl == NULL || hdl->buckets == NULL)
 		return;
 
+	if (!hdl->req_obj.req && !list_empty(&hdl->requests)) {
+		struct v4l2_ctrl_handler *req, *next_req;
+
+		list_for_each_entry_safe(req, next_req, &hdl->requests, requests) {
+			media_request_object_unbind(&req->req_obj);
+			media_request_object_put(&req->req_obj);
+		}
+	}
 	mutex_lock(hdl->lock);
 	/* Free all nodes */
 	list_for_each_entry_safe(ref, next_ref, &hdl->ctrl_refs, node) {
@@ -2816,6 +2854,128 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct v4l2_querymenu *qm)
 }
 EXPORT_SYMBOL(v4l2_querymenu);
 
+static int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+				   const struct v4l2_ctrl_handler *from)
+{
+	struct v4l2_ctrl_ref *ref;
+	int err;
+
+	if (WARN_ON(!hdl || hdl == from))
+		return -EINVAL;
+
+	if (hdl->error)
+		return hdl->error;
+
+	WARN_ON(hdl->lock != &hdl->_lock);
+
+	mutex_lock(from->lock);
+	list_for_each_entry(ref, &from->ctrl_refs, node) {
+		struct v4l2_ctrl *ctrl = ref->ctrl;
+		struct v4l2_ctrl_ref *new_ref;
+
+		/* Skip refs inherited from other devices */
+		if (ref->from_other_dev)
+			continue;
+		/* And buttons */
+		if (ctrl->type == V4L2_CTRL_TYPE_BUTTON)
+			continue;
+		err = handler_new_ref(hdl, ctrl, &new_ref, false, true);
+		if (err) {
+			printk("%s: handler_new_ref on control %x (%s) returned %d\n", __func__, ctrl->id, ctrl->name, err);
+			err = 0;
+			continue;
+		}
+		if (err)
+			break;
+	}
+	mutex_unlock(from->lock);
+	return err;
+}
+
+static void v4l2_ctrl_request_queue(struct media_request_object *obj)
+{
+	struct v4l2_ctrl_handler *hdl =
+		container_of(obj, struct v4l2_ctrl_handler, req_obj);
+	struct v4l2_ctrl_handler *main_hdl = obj->priv;
+	struct v4l2_ctrl_handler *prev_hdl = NULL;
+	struct v4l2_ctrl_ref *ref_ctrl, *ref_ctrl_prev = NULL;
+
+	if (list_empty(&main_hdl->requests_queued))
+		goto queue;
+
+	prev_hdl = list_last_entry(&main_hdl->requests_queued,
+				   struct v4l2_ctrl_handler, requests_queued);
+	/*
+	 * Note: prev_hdl and hdl must contain the same list of control
+	 * references, so if any differences are detected then that is a
+	 * driver bug and the WARN_ON is triggered.
+	 */
+	mutex_lock(prev_hdl->lock);
+	ref_ctrl_prev = list_first_entry(&prev_hdl->ctrl_refs,
+					 struct v4l2_ctrl_ref, node);
+	list_for_each_entry(ref_ctrl, &hdl->ctrl_refs, node) {
+		if (ref_ctrl->req)
+			continue;
+		while (ref_ctrl_prev->ctrl->id < ref_ctrl->ctrl->id) {
+			/* Should never happen, but just in case... */
+			if (list_is_last(&ref_ctrl_prev->node,
+					 &prev_hdl->ctrl_refs))
+				break;
+			ref_ctrl_prev = list_next_entry(ref_ctrl_prev, node);
+		}
+		if (WARN_ON(ref_ctrl_prev->ctrl->id != ref_ctrl->ctrl->id))
+			break;
+		ref_ctrl->req = ref_ctrl_prev->req;
+	}
+	mutex_unlock(prev_hdl->lock);
+queue:
+	list_add_tail(&hdl->requests_queued, &main_hdl->requests_queued);
+	hdl->request_is_queued = true;
+}
+
+static void v4l2_ctrl_request_unbind(struct media_request_object *obj)
+{
+	struct v4l2_ctrl_handler *hdl =
+		container_of(obj, struct v4l2_ctrl_handler, req_obj);
+
+	list_del_init(&hdl->requests);
+	if (hdl->request_is_queued) {
+		list_del_init(&hdl->requests_queued);
+		hdl->request_is_queued = false;
+	}
+}
+
+static void v4l2_ctrl_request_release(struct media_request_object *obj)
+{
+	struct v4l2_ctrl_handler *hdl =
+		container_of(obj, struct v4l2_ctrl_handler, req_obj);
+
+	v4l2_ctrl_handler_free(hdl);
+	kfree(hdl);
+}
+
+static const struct media_request_object_ops req_ops = {
+	.queue = v4l2_ctrl_request_queue,
+	.unbind = v4l2_ctrl_request_unbind,
+	.release = v4l2_ctrl_request_release,
+};
+
+static int v4l2_ctrl_request_bind(struct media_request *req,
+			   struct v4l2_ctrl_handler *hdl,
+			   struct v4l2_ctrl_handler *from)
+{
+	int ret;
+
+	ret = v4l2_ctrl_request_clone(hdl, from);
+
+	if (!ret) {
+		ret = media_request_object_bind(req, &req_ops,
+						from, &hdl->req_obj);
+		if (!ret)
+			list_add_tail(&hdl->requests, &from->requests);
+	}
+	return ret;
+}
 
 /* Some general notes on the atomic requirements of VIDIOC_G/TRY/S_EXT_CTRLS:
 
@@ -2877,6 +3037,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 
 		if (cs->which &&
 		    cs->which != V4L2_CTRL_WHICH_DEF_VAL &&
+		    cs->which != V4L2_CTRL_WHICH_REQUEST_VAL &&
 		    V4L2_CTRL_ID2WHICH(id) != cs->which)
 			return -EINVAL;
 
@@ -2956,13 +3117,12 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
    whether there are any controls at all. */
 static int class_check(struct v4l2_ctrl_handler *hdl, u32 which)
 {
-	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL)
+	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL ||
+	    which == V4L2_CTRL_WHICH_REQUEST_VAL)
 		return 0;
 	return find_ref_lock(hdl, which | 1) ? 0 : -EINVAL;
 }
 
-
-
 /* Get extended controls. Allocates the helpers array if needed. */
 int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs)
 {
@@ -3028,8 +3188,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 			u32 idx = i;
 
 			do {
-				ret = ctrl_to_user(cs->controls + idx,
-						   helpers[idx].ref->ctrl);
+				if (helpers[idx].ref->req)
+					ret = req_to_user(cs->controls + idx,
+						helpers[idx].ref->req);
+				else
+					ret = ctrl_to_user(cs->controls + idx,
+						helpers[idx].ref->ctrl);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
@@ -3302,7 +3466,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 		} while (!ret && idx);
 
 		if (!ret)
-			ret = try_or_set_cluster(fh, master, set, 0);
+			ret = try_or_set_cluster(fh, master,
+						 !hdl->req_obj.req && set, 0);
+		if (!ret && hdl->req_obj.req && set) {
+			for (j = 0; j < master->ncontrols; j++) {
+				struct v4l2_ctrl_ref *ref =
+					find_ref(hdl, master->cluster[j]->id);
+
+				new_to_req(ref);
+			}
+		}
 
 		/* Copy the new values back to userspace. */
 		if (!ret) {
@@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
 }
 EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
 
+void v4l2_ctrl_request_complete(struct media_request *req,
+				struct v4l2_ctrl_handler *main_hdl)
+{
+	struct media_request_object *obj;
+	struct v4l2_ctrl_handler *hdl;
+	struct v4l2_ctrl_ref *ref;
+
+	if (!req || !main_hdl)
+		return;
+
+	obj = media_request_object_find(req, &req_ops, main_hdl);
+	if (!obj)
+		return;
+	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
+
+	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
+		struct v4l2_ctrl *ctrl = ref->ctrl;
+		struct v4l2_ctrl *master = ctrl->cluster[0];
+		unsigned int i;
+
+		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
+			ref->req = ref;
+
+			v4l2_ctrl_lock(master);
+			/* g_volatile_ctrl will update the current control values */
+			for (i = 0; i < master->ncontrols; i++)
+				cur_to_new(master->cluster[i]);
+			call_op(master, g_volatile_ctrl);
+			new_to_req(ref);
+			v4l2_ctrl_unlock(master);
+			continue;
+		}
+		if (ref->req == ref)
+			continue;
+
+		v4l2_ctrl_lock(ctrl);
+		if (ref->req)
+			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
+		else
+			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
+		v4l2_ctrl_unlock(ctrl);
+	}
+
+	WARN_ON(!hdl->request_is_queued);
+	list_del_init(&hdl->requests_queued);
+	hdl->request_is_queued = false;
+	media_request_object_complete(obj);
+	media_request_object_put(obj);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_complete);
+
+void v4l2_ctrl_request_setup(struct media_request *req,
+			     struct v4l2_ctrl_handler *main_hdl)
+{
+	struct media_request_object *obj;
+	struct v4l2_ctrl_handler *hdl;
+	struct v4l2_ctrl_ref *ref;
+
+	if (!req || !main_hdl)
+		return;
+
+	obj = media_request_object_find(req, &req_ops, main_hdl);
+	if (!obj)
+		return;
+	if (obj->completed) {
+		media_request_object_put(obj);
+		return;
+	}
+	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
+
+	mutex_lock(hdl->lock);
+
+	list_for_each_entry(ref, &hdl->ctrl_refs, node)
+		ref->done = false;
+
+	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
+		struct v4l2_ctrl *ctrl = ref->ctrl;
+		struct v4l2_ctrl *master = ctrl->cluster[0];
+		bool have_new_data = false;
+		int i;
+
+		/*
+		 * Skip if this control was already handled by a cluster.
+		 * Skip button controls and read-only controls.
+		 */
+		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
+			continue;
+
+		v4l2_ctrl_lock(master);
+		for (i = 0; i < master->ncontrols; i++) {
+			if (master->cluster[i]) {
+				struct v4l2_ctrl_ref *r =
+					find_ref(hdl, master->cluster[i]->id);
+
+				if (r->req && r == r->req) {
+					have_new_data = true;
+					break;
+				}
+			}
+		}
+		if (!have_new_data) {
+			v4l2_ctrl_unlock(master);
+			continue;
+		}
+
+		for (i = 0; i < master->ncontrols; i++) {
+			if (master->cluster[i]) {
+				struct v4l2_ctrl_ref *r =
+					find_ref(hdl, master->cluster[i]->id);
+
+				req_to_new(r);
+				master->cluster[i]->is_new = 1;
+				r->done = true;
+			}
+		}
+		/*
+		 * For volatile autoclusters that are currently in auto mode
+		 * we need to discover if it will be set to manual mode.
+		 * If so, then we have to copy the current volatile values
+		 * first since those will become the new manual values (which
+		 * may be overwritten by explicit new values from this set
+		 * of controls).
+		 */
+		if (master->is_auto && master->has_volatiles &&
+		    !is_cur_manual(master)) {
+			s32 new_auto_val = *master->p_new.p_s32;
+
+			/*
+			 * If the new value == the manual value, then copy
+			 * the current volatile values.
+			 */
+			if (new_auto_val == master->manual_mode_value)
+				update_from_auto_cluster(master);
+		}
+
+		try_or_set_cluster(NULL, master, true, 0);
+
+		v4l2_ctrl_unlock(master);
+	}
+
+	mutex_unlock(hdl->lock);
+	media_request_object_put(obj);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_setup);
+
 void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
 {
 	if (ctrl == NULL)
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 76352eb59f14..a0f7c38d1a90 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -250,6 +250,10 @@ struct v4l2_ctrl {
  *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
  * @from_other_dev: If true, then @ctrl was defined in another
  *		device than the &struct v4l2_ctrl_handler.
+ * @done:	If true, then this control reference is part of a
+ *		control cluster that was already set while applying
+ *		the controls in this media request object.
+ * @req:	If set, this refers to another request that sets this control.
  * @p_req:	The request value. Only used if the control handler
  *		is bound to a media request.
  *
@@ -263,6 +267,8 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
 	bool from_other_dev;
+	bool done;
+	struct v4l2_ctrl_ref *req;
 	union v4l2_ctrl_ptr p_req;
 };
 
@@ -287,6 +293,15 @@ struct v4l2_ctrl_ref {
  * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
  * @nr_of_buckets: Total number of buckets in the array.
  * @error:	The error code of the first failed control addition.
+ * @request_is_queued: True if the request was queued.
+ * @requests:	List to keep track of open control handler request objects.
+ *		For the parent control handler (@req_obj.req == NULL) this
+ *		is the list header. When the parent control handler is
+ *		removed, it has to unbind and put all these requests since
+ *		they refer to the parent.
+ * @requests_queued: List of the queued requests. This determines the order
+ *		in which these controls are applied. Once the request is
+ *		completed it is removed from this list.
  * @req_obj:	The &struct media_request_object, used to link into a
  *		&struct media_request.
  */
@@ -301,6 +316,9 @@ struct v4l2_ctrl_handler {
 	void *notify_priv;
 	u16 nr_of_buckets;
 	int error;
+	bool request_is_queued;
+	struct list_head requests;
+	struct list_head requests_queued;
 	struct media_request_object req_obj;
 };
 
@@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
  */
 __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
 
+void v4l2_ctrl_request_setup(struct media_request *req,
+			     struct v4l2_ctrl_handler *hdl);
+void v4l2_ctrl_request_complete(struct media_request *req,
+				struct v4l2_ctrl_handler *hdl);
+
 /* Helpers for ioctl_ops */
 
 /**
-- 
2.17.0

[PATCHv13 13/28] v4l2-ctrls: support g/s_ext_ctrls for requests

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

The v4l2_g/s_ext_ctrls functions now support control handlers that
represent requests.

The v4l2_ctrls_find_req_obj() function is responsible for finding the
request from the fd.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/omap3isp/ispvideo.c |   2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c       | 113 +++++++++++++++++++--
 drivers/media/v4l2-core/v4l2-ioctl.c       |  12 +--
 drivers/media/v4l2-core/v4l2-subdev.c      |   9 +-
 include/media/v4l2-ctrls.h                 |   7 +-
 5 files changed, 124 insertions(+), 19 deletions(-)

diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 9d228eac24ea..674e7fd3ad99 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1028,7 +1028,7 @@ static int isp_video_check_external_subdevs(struct isp_video *video,
 	ctrls.count = 1;
 	ctrls.controls = &ctrl;
 
-	ret = v4l2_g_ext_ctrls(pipe->external->ctrl_handler, &ctrls);
+	ret = v4l2_g_ext_ctrls(pipe->external->ctrl_handler, NULL, &ctrls);
 	if (ret < 0) {
 		dev_warn(isp->dev, "no pixel rate control in subdev %s\n",
 			 pipe->external->name);
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 56b986185463..df58a23eb731 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -3124,7 +3124,8 @@ static int class_check(struct v4l2_ctrl_handler *hdl, u32 which)
 }
 
 /* Get extended controls. Allocates the helpers array if needed. */
-int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs)
+int __v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
+		       struct v4l2_ext_controls *cs)
 {
 	struct v4l2_ctrl_helper helper[4];
 	struct v4l2_ctrl_helper *helpers = helper;
@@ -3204,6 +3205,73 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 		kvfree(helpers);
 	return ret;
 }
+
+static struct media_request_object *
+v4l2_ctrls_find_req_obj(struct v4l2_ctrl_handler *hdl,
+			struct media_device *mdev, s32 fd, bool set)
+{
+	struct media_request *req = media_request_get_by_fd(mdev, fd);
+	struct media_request_object *obj;
+	struct v4l2_ctrl_handler *new_hdl;
+	int ret;
+
+	if (IS_ERR(req))
+		return ERR_CAST(req);
+
+	if (set && atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE) {
+		media_request_put(req);
+		return ERR_PTR(-EBUSY);
+	}
+
+	obj = media_request_object_find(req, &req_ops, hdl);
+	if (obj) {
+		media_request_put(req);
+		return obj;
+	}
+
+	new_hdl = kzalloc(sizeof(*new_hdl), GFP_KERNEL);
+	if (!new_hdl) {
+		ret = -ENOMEM;
+		goto put;
+	}
+	obj = &new_hdl->req_obj;
+	ret = v4l2_ctrl_handler_init(new_hdl, (hdl->nr_of_buckets - 1) * 8);
+	if (!ret)
+		ret = v4l2_ctrl_request_bind(req, new_hdl, hdl);
+	if (!ret) {
+		media_request_object_get(obj);
+		media_request_put(req);
+		return obj;
+	}
+	kfree(new_hdl);
+
+put:
+	media_request_put(req);
+	return ERR_PTR(ret);
+}
+
+int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct media_device *mdev,
+		     struct v4l2_ext_controls *cs)
+{
+	struct media_request_object *obj = NULL;
+	int ret;
+
+	if (cs->which == V4L2_CTRL_WHICH_REQUEST_VAL) {
+		if (!mdev || cs->request_fd < 0)
+			return -EINVAL;
+		obj = v4l2_ctrls_find_req_obj(hdl, mdev, cs->request_fd, false);
+		if (IS_ERR(obj))
+			return PTR_ERR(obj);
+		hdl = container_of(obj, struct v4l2_ctrl_handler,
+				   req_obj);
+	}
+
+	ret = __v4l2_g_ext_ctrls(hdl, cs);
+
+	if (obj)
+		media_request_object_put(obj);
+	return ret;
+}
 EXPORT_SYMBOL(v4l2_g_ext_ctrls);
 
 /* Helper function to get a single control */
@@ -3379,9 +3447,9 @@ static void update_from_auto_cluster(struct v4l2_ctrl *master)
 }
 
 /* Try or try-and-set controls */
-static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
-			     struct v4l2_ext_controls *cs,
-			     bool set)
+static int __try_set_ext_ctrls(struct v4l2_fh *fh,
+			       struct v4l2_ctrl_handler *hdl,
+			       struct v4l2_ext_controls *cs, bool set)
 {
 	struct v4l2_ctrl_helper helper[4];
 	struct v4l2_ctrl_helper *helpers = helper;
@@ -3494,16 +3562,45 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 	return ret;
 }
 
-int v4l2_try_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs)
+static int try_set_ext_ctrls(struct v4l2_fh *fh,
+			     struct v4l2_ctrl_handler *hdl, struct media_device *mdev,
+			     struct v4l2_ext_controls *cs, bool set)
+{
+	struct media_request_object *obj = NULL;
+	int ret;
+
+	if (cs->which == V4L2_CTRL_WHICH_REQUEST_VAL) {
+		if (!mdev || cs->request_fd < 0)
+			return -EINVAL;
+		obj = v4l2_ctrls_find_req_obj(hdl, mdev, cs->request_fd, true);
+		if (IS_ERR(obj))
+			return PTR_ERR(obj);
+		if (atomic_read(&obj->req->state) != MEDIA_REQUEST_STATE_IDLE) {
+			media_request_object_put(obj);
+			return -EBUSY;
+		}
+		hdl = container_of(obj, struct v4l2_ctrl_handler,
+				   req_obj);
+	}
+
+	ret = __try_set_ext_ctrls(fh, hdl, cs, set);
+
+	if (obj)
+		media_request_object_put(obj);
+	return ret;
+}
+
+int v4l2_try_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct media_device *mdev,
+		       struct v4l2_ext_controls *cs)
 {
-	return try_set_ext_ctrls(NULL, hdl, cs, false);
+	return try_set_ext_ctrls(NULL, hdl, mdev, cs, false);
 }
 EXPORT_SYMBOL(v4l2_try_ext_ctrls);
 
 int v4l2_s_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
-					struct v4l2_ext_controls *cs)
+		     struct media_device *mdev, struct v4l2_ext_controls *cs)
 {
-	return try_set_ext_ctrls(fh, hdl, cs, true);
+	return try_set_ext_ctrls(fh, hdl, mdev, cs, true);
 }
 EXPORT_SYMBOL(v4l2_s_ext_ctrls);
 
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 9ce23e23c5bf..56741c4a48fc 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -2079,9 +2079,9 @@ static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops *ops,
 
 	p->error_idx = p->count;
 	if (vfh && vfh->ctrl_handler)
-		return v4l2_g_ext_ctrls(vfh->ctrl_handler, p);
+		return v4l2_g_ext_ctrls(vfh->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (vfd->ctrl_handler)
-		return v4l2_g_ext_ctrls(vfd->ctrl_handler, p);
+		return v4l2_g_ext_ctrls(vfd->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (ops->vidioc_g_ext_ctrls == NULL)
 		return -ENOTTY;
 	return check_ext_ctrls(p, 0) ? ops->vidioc_g_ext_ctrls(file, fh, p) :
@@ -2098,9 +2098,9 @@ static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops *ops,
 
 	p->error_idx = p->count;
 	if (vfh && vfh->ctrl_handler)
-		return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, p);
+		return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (vfd->ctrl_handler)
-		return v4l2_s_ext_ctrls(NULL, vfd->ctrl_handler, p);
+		return v4l2_s_ext_ctrls(NULL, vfd->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (ops->vidioc_s_ext_ctrls == NULL)
 		return -ENOTTY;
 	return check_ext_ctrls(p, 0) ? ops->vidioc_s_ext_ctrls(file, fh, p) :
@@ -2117,9 +2117,9 @@ static int v4l_try_ext_ctrls(const struct v4l2_ioctl_ops *ops,
 
 	p->error_idx = p->count;
 	if (vfh && vfh->ctrl_handler)
-		return v4l2_try_ext_ctrls(vfh->ctrl_handler, p);
+		return v4l2_try_ext_ctrls(vfh->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (vfd->ctrl_handler)
-		return v4l2_try_ext_ctrls(vfd->ctrl_handler, p);
+		return v4l2_try_ext_ctrls(vfd->ctrl_handler, vfd->v4l2_dev->mdev, p);
 	if (ops->vidioc_try_ext_ctrls == NULL)
 		return -ENOTTY;
 	return check_ext_ctrls(p, 0) ? ops->vidioc_try_ext_ctrls(file, fh, p) :
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index f9eed938d348..ce8c133e0ccd 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -222,17 +222,20 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_G_EXT_CTRLS:
 		if (!vfh->ctrl_handler)
 			return -ENOTTY;
-		return v4l2_g_ext_ctrls(vfh->ctrl_handler, arg);
+		return v4l2_g_ext_ctrls(vfh->ctrl_handler,
+					sd->v4l2_dev->mdev, arg);
 
 	case VIDIOC_S_EXT_CTRLS:
 		if (!vfh->ctrl_handler)
 			return -ENOTTY;
-		return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, arg);
+		return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler,
+					sd->v4l2_dev->mdev, arg);
 
 	case VIDIOC_TRY_EXT_CTRLS:
 		if (!vfh->ctrl_handler)
 			return -ENOTTY;
-		return v4l2_try_ext_ctrls(vfh->ctrl_handler, arg);
+		return v4l2_try_ext_ctrls(vfh->ctrl_handler,
+					  sd->v4l2_dev->mdev, arg);
 
 	case VIDIOC_DQEVENT:
 		if (!(sd->flags & V4L2_SUBDEV_FL_HAS_EVENTS))
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index a0f7c38d1a90..963c37b02363 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -1148,11 +1148,12 @@ int v4l2_s_ctrl(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
  *	:ref:`VIDIOC_G_EXT_CTRLS <vidioc_g_ext_ctrls>` ioctl
  *
  * @hdl: pointer to &struct v4l2_ctrl_handler
+ * @mdev: pointer to &struct media_device
  * @c: pointer to &struct v4l2_ext_controls
  *
  * If hdl == NULL then they will all return -EINVAL.
  */
-int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
+int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct media_device *mdev,
 		     struct v4l2_ext_controls *c);
 
 /**
@@ -1160,11 +1161,13 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl,
  *	:ref:`VIDIOC_TRY_EXT_CTRLS <vidioc_g_ext_ctrls>` ioctl
  *
  * @hdl: pointer to &struct v4l2_ctrl_handler
+ * @mdev: pointer to &struct media_device
  * @c: pointer to &struct v4l2_ext_controls
  *
  * If hdl == NULL then they will all return -EINVAL.
  */
 int v4l2_try_ext_ctrls(struct v4l2_ctrl_handler *hdl,
+		       struct media_device *mdev,
 		       struct v4l2_ext_controls *c);
 
 /**
@@ -1173,11 +1176,13 @@ int v4l2_try_ext_ctrls(struct v4l2_ctrl_handler *hdl,
  *
  * @fh: pointer to &struct v4l2_fh
  * @hdl: pointer to &struct v4l2_ctrl_handler
+ * @mdev: pointer to &struct media_device
  * @c: pointer to &struct v4l2_ext_controls
  *
  * If hdl == NULL then they will all return -EINVAL.
  */
 int v4l2_s_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
+		     struct media_device *mdev,
 		     struct v4l2_ext_controls *c);
 
 /**
-- 
2.17.0

[PATCHv13 14/28] videodev2.h: Add request_fd field to v4l2_buffer

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

When queuing buffers allow for passing the request that should
be associated with this buffer.

If V4L2_BUF_FLAG_REQUEST_FD is set, then request_fd is used as
the file descriptor.

If a buffer is stored in a request, but not yet queued to the
driver, then V4L2_BUF_FLAG_IN_REQUEST is set.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  2 +-
 drivers/media/usb/cpia2/cpia2_v4l.c             |  2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   |  9 ++++++---
 drivers/media/v4l2-core/v4l2-ioctl.c            |  4 ++--
 include/uapi/linux/videodev2.h                  | 10 +++++++++-
 5 files changed, 19 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 886a2d8d5c6c..4e9c77f21858 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -204,7 +204,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
 	b->reserved2 = 0;
-	b->reserved = 0;
+	b->request_fd = 0;
 
 	if (q->is_multiplanar) {
 		/*
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c b/drivers/media/usb/cpia2/cpia2_v4l.c
index 99f106b13280..13aee9f67d05 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -949,7 +949,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
 	buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
 	buf->length = cam->frame_size;
 	buf->reserved2 = 0;
-	buf->reserved = 0;
+	buf->request_fd = 0;
 	memset(&buf->timecode, 0, sizeof(buf->timecode));
 
 	DBG("DQBUF #%d status:%d seq:%d length:%d\n", buf->index,
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index dcce86c1fe40..633465d21d04 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -482,7 +482,7 @@ struct v4l2_buffer32 {
 	} m;
 	__u32			length;
 	__u32			reserved2;
-	__u32			reserved;
+	__s32			request_fd;
 };
 
 static int get_v4l2_plane32(struct v4l2_plane __user *p64,
@@ -581,6 +581,7 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *p64,
 {
 	u32 type;
 	u32 length;
+	s32 request_fd;
 	enum v4l2_memory memory;
 	struct v4l2_plane32 __user *uplane32;
 	struct v4l2_plane __user *uplane;
@@ -595,7 +596,9 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *p64,
 	    get_user(memory, &p32->memory) ||
 	    put_user(memory, &p64->memory) ||
 	    get_user(length, &p32->length) ||
-	    put_user(length, &p64->length))
+	    put_user(length, &p64->length) ||
+	    get_user(request_fd, &p32->request_fd) ||
+	    put_user(request_fd, &p64->request_fd))
 		return -EFAULT;
 
 	if (V4L2_TYPE_IS_OUTPUT(type))
@@ -699,7 +702,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *p64,
 	    copy_in_user(&p32->timecode, &p64->timecode, sizeof(p64->timecode)) ||
 	    assign_in_user(&p32->sequence, &p64->sequence) ||
 	    assign_in_user(&p32->reserved2, &p64->reserved2) ||
-	    assign_in_user(&p32->reserved, &p64->reserved) ||
+	    assign_in_user(&p32->request_fd, &p64->request_fd) ||
 	    get_user(length, &p64->length) ||
 	    put_user(length, &p32->length))
 		return -EFAULT;
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 56741c4a48fc..561a1fe3160b 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 	const struct v4l2_plane *plane;
 	int i;
 
-	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s",
+	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request_fd=%d, flags=0x%08x, field=%s, sequence=%d, memory=%s",
 			p->timestamp.tv_sec / 3600,
 			(int)(p->timestamp.tv_sec / 60) % 60,
 			(int)(p->timestamp.tv_sec % 60),
 			(long)p->timestamp.tv_usec,
 			p->index,
-			prt_names(p->type, v4l2_type_names),
+			prt_names(p->type, v4l2_type_names), p->request_fd,
 			p->flags, prt_names(p->field, v4l2_field_names),
 			p->sequence, prt_names(p->memory, v4l2_memory_names));
 
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 16b53b82496c..1f6c4b52baae 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -910,6 +910,7 @@ struct v4l2_plane {
  * @length:	size in bytes of the buffer (NOT its payload) for single-plane
  *		buffers (when type != *_MPLANE); number of elements in the
  *		planes array for multi-plane buffers
+ * @request_fd: fd of the request that this buffer should use
  *
  * Contains data exchanged by application and driver using one of the Streaming
  * I/O methods.
@@ -934,7 +935,10 @@ struct v4l2_buffer {
 	} m;
 	__u32			length;
 	__u32			reserved2;
-	__u32			reserved;
+	union {
+		__s32		request_fd;
+		__u32		reserved;
+	};
 };
 
 /*  Flags for 'flags' field */
@@ -952,6 +956,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_BFRAME			0x00000020
 /* Buffer is ready, but the data contained within is corrupted. */
 #define V4L2_BUF_FLAG_ERROR			0x00000040
+/* Buffer is added to an unqueued request */
+#define V4L2_BUF_FLAG_IN_REQUEST		0x00000080
 /* timecode field is valid */
 #define V4L2_BUF_FLAG_TIMECODE			0x00000100
 /* Buffer is prepared for queuing */
@@ -970,6 +976,8 @@ struct v4l2_buffer {
 #define V4L2_BUF_FLAG_TSTAMP_SRC_SOE		0x00010000
 /* mem2mem encoder/decoder */
 #define V4L2_BUF_FLAG_LAST			0x00100000
+/* request_fd is valid */
+#define V4L2_BUF_FLAG_REQUEST_FD		0x00800000
 
 /**
  * struct v4l2_exportbuffer - export of video buffer as DMABUF file descriptor
-- 
2.17.0

[PATCHv13 15/28] vb2: store userspace data in vb2_v4l2_buffer

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

The userspace-provided plane data needs to be stored in
vb2_v4l2_buffer. Currently this information is applied by
__fill_vb2_buffer() which is called by the core prepare_buf
and qbuf functions, but when using requests these functions
aren't called yet since the buffer won't be prepared until
the media request is actually queued.

In the meantime this information has to be stored somewhere
and vb2_v4l2_buffer is a good place for it.

The __fill_vb2_buffer callback now just copies the relevant
information from vb2_v4l2_buffer into the planes array.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 .../media/common/videobuf2/videobuf2-core.c   |  43 +--
 .../media/common/videobuf2/videobuf2-v4l2.c   | 324 ++++++++++--------
 drivers/media/dvb-core/dvb_vb2.c              |   3 +-
 include/media/videobuf2-core.h                |   3 +-
 include/media/videobuf2-v4l2.h                |   2 +
 5 files changed, 206 insertions(+), 169 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index d3f7bb33a54d..70ba43c5b3e2 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -964,20 +964,19 @@ EXPORT_SYMBOL_GPL(vb2_discard_done);
 /*
  * __prepare_mmap() - prepare an MMAP buffer
  */
-static int __prepare_mmap(struct vb2_buffer *vb, const void *pb)
+static int __prepare_mmap(struct vb2_buffer *vb)
 {
 	int ret = 0;
 
-	if (pb)
-		ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
-				 vb, pb, vb->planes);
+	ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+			 vb, vb->planes);
 	return ret ? ret : call_vb_qop(vb, buf_prepare, vb);
 }
 
 /*
  * __prepare_userptr() - prepare a USERPTR buffer
  */
-static int __prepare_userptr(struct vb2_buffer *vb, const void *pb)
+static int __prepare_userptr(struct vb2_buffer *vb)
 {
 	struct vb2_plane planes[VB2_MAX_PLANES];
 	struct vb2_queue *q = vb->vb2_queue;
@@ -988,12 +987,10 @@ static int __prepare_userptr(struct vb2_buffer *vb, const void *pb)
 
 	memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
 	/* Copy relevant information provided by the userspace */
-	if (pb) {
-		ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
-				 vb, pb, planes);
-		if (ret)
-			return ret;
-	}
+	ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+			 vb, planes);
+	if (ret)
+		return ret;
 
 	for (plane = 0; plane < vb->num_planes; ++plane) {
 		/* Skip the plane if already verified */
@@ -1093,7 +1090,7 @@ static int __prepare_userptr(struct vb2_buffer *vb, const void *pb)
 /*
  * __prepare_dmabuf() - prepare a DMABUF buffer
  */
-static int __prepare_dmabuf(struct vb2_buffer *vb, const void *pb)
+static int __prepare_dmabuf(struct vb2_buffer *vb)
 {
 	struct vb2_plane planes[VB2_MAX_PLANES];
 	struct vb2_queue *q = vb->vb2_queue;
@@ -1104,12 +1101,10 @@ static int __prepare_dmabuf(struct vb2_buffer *vb, const void *pb)
 
 	memset(planes, 0, sizeof(planes[0]) * vb->num_planes);
 	/* Copy relevant information provided by the userspace */
-	if (pb) {
-		ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
-				 vb, pb, planes);
-		if (ret)
-			return ret;
-	}
+	ret = call_bufop(vb->vb2_queue, fill_vb2_buffer,
+			 vb, planes);
+	if (ret)
+		return ret;
 
 	for (plane = 0; plane < vb->num_planes; ++plane) {
 		struct dma_buf *dbuf = dma_buf_get(planes[plane].m.fd);
@@ -1238,7 +1233,7 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
 	call_void_vb_qop(vb, buf_queue, vb);
 }
 
-static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
+static int __buf_prepare(struct vb2_buffer *vb)
 {
 	struct vb2_queue *q = vb->vb2_queue;
 	unsigned int plane;
@@ -1253,13 +1248,13 @@ static int __buf_prepare(struct vb2_buffer *vb, const void *pb)
 
 	switch (q->memory) {
 	case VB2_MEMORY_MMAP:
-		ret = __prepare_mmap(vb, pb);
+		ret = __prepare_mmap(vb);
 		break;
 	case VB2_MEMORY_USERPTR:
-		ret = __prepare_userptr(vb, pb);
+		ret = __prepare_userptr(vb);
 		break;
 	case VB2_MEMORY_DMABUF:
-		ret = __prepare_dmabuf(vb, pb);
+		ret = __prepare_dmabuf(vb);
 		break;
 	default:
 		WARN(1, "Invalid queue type\n");
@@ -1293,7 +1288,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 		return -EINVAL;
 	}
 
-	ret = __buf_prepare(vb, pb);
+	ret = __buf_prepare(vb);
 	if (ret)
 		return ret;
 
@@ -1378,7 +1373,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 
 	switch (vb->state) {
 	case VB2_BUF_STATE_DEQUEUED:
-		ret = __buf_prepare(vb, pb);
+		ret = __buf_prepare(vb);
 		if (ret)
 			return ret;
 		break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 4e9c77f21858..bf7a3ba9fed0 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -154,9 +154,177 @@ static void vb2_warn_zero_bytesused(struct vb2_buffer *vb)
 		pr_warn("use the actual size instead.\n");
 }
 
+static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b)
+{
+	struct vb2_queue *q = vb->vb2_queue;
+	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
+	struct vb2_plane *planes = vbuf->planes;
+	unsigned int plane;
+	int ret;
+
+	ret = __verify_length(vb, b);
+	if (ret < 0) {
+		dprintk(1, "plane parameters verification failed: %d\n", ret);
+		return ret;
+	}
+	if (b->field == V4L2_FIELD_ALTERNATE && q->is_output) {
+		/*
+		 * If the format's field is ALTERNATE, then the buffer's field
+		 * should be either TOP or BOTTOM, not ALTERNATE since that
+		 * makes no sense. The driver has to know whether the
+		 * buffer represents a top or a bottom field in order to
+		 * program any DMA correctly. Using ALTERNATE is wrong, since
+		 * that just says that it is either a top or a bottom field,
+		 * but not which of the two it is.
+		 */
+		dprintk(1, "the field is incorrectly set to ALTERNATE for an output buffer\n");
+		return -EINVAL;
+	}
+	vbuf->sequence = 0;
+
+	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
+		switch (b->memory) {
+		case VB2_MEMORY_USERPTR:
+			for (plane = 0; plane < vb->num_planes; ++plane) {
+				planes[plane].m.userptr =
+					b->m.planes[plane].m.userptr;
+				planes[plane].length =
+					b->m.planes[plane].length;
+			}
+			break;
+		case VB2_MEMORY_DMABUF:
+			for (plane = 0; plane < vb->num_planes; ++plane) {
+				planes[plane].m.fd =
+					b->m.planes[plane].m.fd;
+				planes[plane].length =
+					b->m.planes[plane].length;
+			}
+			break;
+		default:
+			for (plane = 0; plane < vb->num_planes; ++plane) {
+				planes[plane].m.offset =
+					vb->planes[plane].m.offset;
+				planes[plane].length =
+					vb->planes[plane].length;
+			}
+			break;
+		}
+
+		/* Fill in driver-provided information for OUTPUT types */
+		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
+			/*
+			 * Will have to go up to b->length when API starts
+			 * accepting variable number of planes.
+			 *
+			 * If bytesused == 0 for the output buffer, then fall
+			 * back to the full buffer size. In that case
+			 * userspace clearly never bothered to set it and
+			 * it's a safe assumption that they really meant to
+			 * use the full plane sizes.
+			 *
+			 * Some drivers, e.g. old codec drivers, use bytesused == 0
+			 * as a way to indicate that streaming is finished.
+			 * In that case, the driver should use the
+			 * allow_zero_bytesused flag to keep old userspace
+			 * applications working.
+			 */
+			for (plane = 0; plane < vb->num_planes; ++plane) {
+				struct vb2_plane *pdst = &planes[plane];
+				struct v4l2_plane *psrc = &b->m.planes[plane];
+
+				if (psrc->bytesused == 0)
+					vb2_warn_zero_bytesused(vb);
+
+				if (vb->vb2_queue->allow_zero_bytesused)
+					pdst->bytesused = psrc->bytesused;
+				else
+					pdst->bytesused = psrc->bytesused ?
+						psrc->bytesused : pdst->length;
+				pdst->data_offset = psrc->data_offset;
+			}
+		}
+	} else {
+		/*
+		 * Single-planar buffers do not use planes array,
+		 * so fill in relevant v4l2_buffer struct fields instead.
+		 * In videobuf we use our internal V4l2_planes struct for
+		 * single-planar buffers as well, for simplicity.
+		 *
+		 * If bytesused == 0 for the output buffer, then fall back
+		 * to the full buffer size as that's a sensible default.
+		 *
+		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
+		 * a way to indicate that streaming is finished. In that case,
+		 * the driver should use the allow_zero_bytesused flag to keep
+		 * old userspace applications working.
+		 */
+		switch (b->memory) {
+		case VB2_MEMORY_USERPTR:
+			planes[0].m.userptr = b->m.userptr;
+			planes[0].length = b->length;
+			break;
+		case VB2_MEMORY_DMABUF:
+			planes[0].m.fd = b->m.fd;
+			planes[0].length = b->length;
+			break;
+		default:
+			planes[0].m.offset = vb->planes[0].m.offset;
+			planes[0].length = vb->planes[0].length;
+			break;
+		}
+
+		planes[0].data_offset = 0;
+		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
+			if (b->bytesused == 0)
+				vb2_warn_zero_bytesused(vb);
+
+			if (vb->vb2_queue->allow_zero_bytesused)
+				planes[0].bytesused = b->bytesused;
+			else
+				planes[0].bytesused = b->bytesused ?
+					b->bytesused : planes[0].length;
+		} else
+			planes[0].bytesused = 0;
+
+	}
+
+	/* Zero flags that we handle */
+	vbuf->flags = b->flags & ~V4L2_BUFFER_MASK_FLAGS;
+	if (!vb->vb2_queue->copy_timestamp || !V4L2_TYPE_IS_OUTPUT(b->type)) {
+		/*
+		 * Non-COPY timestamps and non-OUTPUT queues will get
+		 * their timestamp and timestamp source flags from the
+		 * queue.
+		 */
+		vbuf->flags &= ~V4L2_BUF_FLAG_TSTAMP_SRC_MASK;
+	}
+
+	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
+		/*
+		 * For output buffers mask out the timecode flag:
+		 * this will be handled later in vb2_qbuf().
+		 * The 'field' is valid metadata for this output buffer
+		 * and so that needs to be copied here.
+		 */
+		vbuf->flags &= ~V4L2_BUF_FLAG_TIMECODE;
+		vbuf->field = b->field;
+	} else {
+		/* Zero any output buffer flags as this is a capture buffer */
+		vbuf->flags &= ~V4L2_BUFFER_OUT_FLAGS;
+		/* Zero last flag, this is a signal from driver to userspace */
+		vbuf->flags &= ~V4L2_BUF_FLAG_LAST;
+	}
+
+	return 0;
+}
+
 static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 				    const char *opname)
 {
+	struct vb2_v4l2_buffer *vbuf;
+	struct vb2_buffer *vb;
+	int ret;
+
 	if (b->type != q->type) {
 		dprintk(1, "%s: invalid buffer type\n", opname);
 		return -EINVAL;
@@ -178,7 +346,15 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 		return -EINVAL;
 	}
 
-	return __verify_planes_array(q->bufs[b->index], b);
+	vb = q->bufs[b->index];
+	vbuf = to_vb2_v4l2_buffer(vb);
+	ret = __verify_planes_array(vb, b);
+	if (ret)
+		return ret;
+
+	/* Copy relevant information provided by the userspace */
+	memset(vbuf->planes, 0, sizeof(vbuf->planes[0]) * vb->num_planes);
+	return vb2_fill_vb2_v4l2_buffer(vb, b);
 }
 
 /*
@@ -291,153 +467,19 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
  * v4l2_buffer by the userspace. It also verifies that struct
  * v4l2_buffer has a valid number of planes.
  */
-static int __fill_vb2_buffer(struct vb2_buffer *vb,
-		const void *pb, struct vb2_plane *planes)
+static int __fill_vb2_buffer(struct vb2_buffer *vb, struct vb2_plane *planes)
 {
-	struct vb2_queue *q = vb->vb2_queue;
-	const struct v4l2_buffer *b = pb;
 	struct vb2_v4l2_buffer *vbuf = to_vb2_v4l2_buffer(vb);
 	unsigned int plane;
-	int ret;
 
-	ret = __verify_length(vb, b);
-	if (ret < 0) {
-		dprintk(1, "plane parameters verification failed: %d\n", ret);
-		return ret;
-	}
-	if (b->field == V4L2_FIELD_ALTERNATE && q->is_output) {
-		/*
-		 * If the format's field is ALTERNATE, then the buffer's field
-		 * should be either TOP or BOTTOM, not ALTERNATE since that
-		 * makes no sense. The driver has to know whether the
-		 * buffer represents a top or a bottom field in order to
-		 * program any DMA correctly. Using ALTERNATE is wrong, since
-		 * that just says that it is either a top or a bottom field,
-		 * but not which of the two it is.
-		 */
-		dprintk(1, "the field is incorrectly set to ALTERNATE for an output buffer\n");
-		return -EINVAL;
-	}
 	vb->timestamp = 0;
-	vbuf->sequence = 0;
-
-	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
-		if (b->memory == VB2_MEMORY_USERPTR) {
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.userptr =
-					b->m.planes[plane].m.userptr;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-		}
-		if (b->memory == VB2_MEMORY_DMABUF) {
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				planes[plane].m.fd =
-					b->m.planes[plane].m.fd;
-				planes[plane].length =
-					b->m.planes[plane].length;
-			}
-		}
-
-		/* Fill in driver-provided information for OUTPUT types */
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			/*
-			 * Will have to go up to b->length when API starts
-			 * accepting variable number of planes.
-			 *
-			 * If bytesused == 0 for the output buffer, then fall
-			 * back to the full buffer size. In that case
-			 * userspace clearly never bothered to set it and
-			 * it's a safe assumption that they really meant to
-			 * use the full plane sizes.
-			 *
-			 * Some drivers, e.g. old codec drivers, use bytesused == 0
-			 * as a way to indicate that streaming is finished.
-			 * In that case, the driver should use the
-			 * allow_zero_bytesused flag to keep old userspace
-			 * applications working.
-			 */
-			for (plane = 0; plane < vb->num_planes; ++plane) {
-				struct vb2_plane *pdst = &planes[plane];
-				struct v4l2_plane *psrc = &b->m.planes[plane];
-
-				if (psrc->bytesused == 0)
-					vb2_warn_zero_bytesused(vb);
 
-				if (vb->vb2_queue->allow_zero_bytesused)
-					pdst->bytesused = psrc->bytesused;
-				else
-					pdst->bytesused = psrc->bytesused ?
-						psrc->bytesused : pdst->length;
-				pdst->data_offset = psrc->data_offset;
-			}
-		}
-	} else {
-		/*
-		 * Single-planar buffers do not use planes array,
-		 * so fill in relevant v4l2_buffer struct fields instead.
-		 * In videobuf we use our internal V4l2_planes struct for
-		 * single-planar buffers as well, for simplicity.
-		 *
-		 * If bytesused == 0 for the output buffer, then fall back
-		 * to the full buffer size as that's a sensible default.
-		 *
-		 * Some drivers, e.g. old codec drivers, use bytesused == 0 as
-		 * a way to indicate that streaming is finished. In that case,
-		 * the driver should use the allow_zero_bytesused flag to keep
-		 * old userspace applications working.
-		 */
-		if (b->memory == VB2_MEMORY_USERPTR) {
-			planes[0].m.userptr = b->m.userptr;
-			planes[0].length = b->length;
-		}
-
-		if (b->memory == VB2_MEMORY_DMABUF) {
-			planes[0].m.fd = b->m.fd;
-			planes[0].length = b->length;
-		}
-
-		if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-			if (b->bytesused == 0)
-				vb2_warn_zero_bytesused(vb);
-
-			if (vb->vb2_queue->allow_zero_bytesused)
-				planes[0].bytesused = b->bytesused;
-			else
-				planes[0].bytesused = b->bytesused ?
-					b->bytesused : planes[0].length;
-		} else
-			planes[0].bytesused = 0;
-
-	}
-
-	/* Zero flags that the vb2 core handles */
-	vbuf->flags = b->flags & ~V4L2_BUFFER_MASK_FLAGS;
-	if (!vb->vb2_queue->copy_timestamp || !V4L2_TYPE_IS_OUTPUT(b->type)) {
-		/*
-		 * Non-COPY timestamps and non-OUTPUT queues will get
-		 * their timestamp and timestamp source flags from the
-		 * queue.
-		 */
-		vbuf->flags &= ~V4L2_BUF_FLAG_TSTAMP_SRC_MASK;
+	for (plane = 0; plane < vb->num_planes; ++plane) {
+		planes[plane].m = vbuf->planes[plane].m;
+		planes[plane].length = vbuf->planes[plane].length;
+		planes[plane].bytesused = vbuf->planes[plane].bytesused;
+		planes[plane].data_offset = vbuf->planes[plane].data_offset;
 	}
-
-	if (V4L2_TYPE_IS_OUTPUT(b->type)) {
-		/*
-		 * For output buffers mask out the timecode flag:
-		 * this will be handled later in vb2_qbuf().
-		 * The 'field' is valid metadata for this output buffer
-		 * and so that needs to be copied here.
-		 */
-		vbuf->flags &= ~V4L2_BUF_FLAG_TIMECODE;
-		vbuf->field = b->field;
-	} else {
-		/* Zero any output buffer flags as this is a capture buffer */
-		vbuf->flags &= ~V4L2_BUFFER_OUT_FLAGS;
-		/* Zero last flag, this is a signal from driver to userspace */
-		vbuf->flags &= ~V4L2_BUF_FLAG_LAST;
-	}
-
 	return 0;
 }
 
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index b811adf88afa..da6a8cec7d42 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -146,8 +146,7 @@ static void _fill_dmx_buffer(struct vb2_buffer *vb, void *pb)
 	dprintk(3, "[%s]\n", ctx->name);
 }
 
-static int _fill_vb2_buffer(struct vb2_buffer *vb,
-			    const void *pb, struct vb2_plane *planes)
+static int _fill_vb2_buffer(struct vb2_buffer *vb, struct vb2_plane *planes)
 {
 	struct dvb_vb2_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
 
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f6818f732f34..224c4820a044 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -417,8 +417,7 @@ struct vb2_ops {
 struct vb2_buf_ops {
 	int (*verify_planes_array)(struct vb2_buffer *vb, const void *pb);
 	void (*fill_user_buffer)(struct vb2_buffer *vb, void *pb);
-	int (*fill_vb2_buffer)(struct vb2_buffer *vb, const void *pb,
-				struct vb2_plane *planes);
+	int (*fill_vb2_buffer)(struct vb2_buffer *vb, struct vb2_plane *planes);
 	void (*copy_timestamp)(struct vb2_buffer *vb, const void *pb);
 };
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 3d5e2d739f05..097bf3e6951d 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -32,6 +32,7 @@
  *		&enum v4l2_field.
  * @timecode:	frame timecode.
  * @sequence:	sequence count of this frame.
+ * @planes:	plane information (userptr/fd, length, bytesused, data_offset).
  *
  * Should contain enough information to be able to cover all the fields
  * of &struct v4l2_buffer at ``videodev2.h``.
@@ -43,6 +44,7 @@ struct vb2_v4l2_buffer {
 	__u32			field;
 	struct v4l2_timecode	timecode;
 	__u32			sequence;
+	struct vb2_plane	planes[VB2_MAX_PLANES];
 };
 
 /*
-- 
2.17.0

[PATCHv13 16/28] videobuf2-core: embed media_request_object

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Make vb2_buffer a request object.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 include/media/videobuf2-core.h | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 224c4820a044..3d54654c3cd4 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -17,6 +17,7 @@
 #include <linux/poll.h>
 #include <linux/dma-buf.h>
 #include <linux/bitops.h>
+#include <media/media-request.h>
 
 #define VB2_MAX_FRAME	(32)
 #define VB2_MAX_PLANES	(8)
@@ -238,6 +239,7 @@ struct vb2_queue;
  * @num_planes:		number of planes in the buffer
  *			on an internal driver queue.
  * @timestamp:		frame timestamp in ns.
+ * @req_obj:		used to bind this buffer to a request
  */
 struct vb2_buffer {
 	struct vb2_queue	*vb2_queue;
@@ -246,6 +248,7 @@ struct vb2_buffer {
 	unsigned int		memory;
 	unsigned int		num_planes;
 	u64			timestamp;
+	struct media_request_object	req_obj;
 
 	/* private: internal use only
 	 *
-- 
2.17.0

[PATCHv13 17/28] videobuf2-core: integrate with media requests

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Buffers can now be prepared or queued for a request.

A buffer is unbound from the request at vb2_buffer_done time or
when the queue is cancelled.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 .../media/common/videobuf2/videobuf2-core.c   | 124 ++++++++++++++++--
 .../media/common/videobuf2/videobuf2-v4l2.c   |   4 +-
 drivers/media/dvb-core/dvb_vb2.c              |   2 +-
 include/media/videobuf2-core.h                |  18 ++-
 4 files changed, 135 insertions(+), 13 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 70ba43c5b3e2..4064fa15bc40 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -497,8 +497,9 @@ static int __vb2_queue_free(struct vb2_queue *q, unsigned int buffers)
 			pr_info("     buf_init: %u buf_cleanup: %u buf_prepare: %u buf_finish: %u\n",
 				vb->cnt_buf_init, vb->cnt_buf_cleanup,
 				vb->cnt_buf_prepare, vb->cnt_buf_finish);
-			pr_info("     buf_queue: %u buf_done: %u\n",
-				vb->cnt_buf_queue, vb->cnt_buf_done);
+			pr_info("     buf_queue: %u buf_done: %u buf_request_complete: %u\n",
+				vb->cnt_buf_queue, vb->cnt_buf_done,
+				vb->cnt_buf_request_complete);
 			pr_info("     alloc: %u put: %u prepare: %u finish: %u mmap: %u\n",
 				vb->cnt_mem_alloc, vb->cnt_mem_put,
 				vb->cnt_mem_prepare, vb->cnt_mem_finish,
@@ -930,6 +931,14 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		vb->state = state;
 	}
 	atomic_dec(&q->owned_by_drv_count);
+
+	if (vb->req_obj.req) {
+		/* This is not supported at the moment */
+		WARN_ON(state == VB2_BUF_STATE_REQUEUEING);
+		media_request_object_unbind(&vb->req_obj);
+		media_request_object_put(&vb->req_obj);
+	}
+
 	spin_unlock_irqrestore(&q->done_lock, flags);
 
 	trace_vb2_buf_done(q, vb);
@@ -1236,6 +1245,7 @@ static void __enqueue_in_driver(struct vb2_buffer *vb)
 static int __buf_prepare(struct vb2_buffer *vb)
 {
 	struct vb2_queue *q = vb->vb2_queue;
+	enum vb2_buffer_state orig_state = vb->state;
 	unsigned int plane;
 	int ret;
 
@@ -1263,7 +1273,7 @@ static int __buf_prepare(struct vb2_buffer *vb)
 
 	if (ret) {
 		dprintk(1, "buffer preparation failed: %d\n", ret);
-		vb->state = VB2_BUF_STATE_DEQUEUED;
+		vb->state = orig_state;
 		return ret;
 	}
 
@@ -1276,6 +1286,60 @@ static int __buf_prepare(struct vb2_buffer *vb)
 	return 0;
 }
 
+static int vb2_req_prepare(struct media_request_object *obj)
+{
+	struct vb2_buffer *vb = container_of(obj, struct vb2_buffer, req_obj);
+	int ret;
+
+	if (WARN_ON(vb->state != VB2_BUF_STATE_IN_REQUEST))
+		return -EINVAL;
+
+	mutex_lock(vb->vb2_queue->lock);
+	ret = __buf_prepare(vb);
+	mutex_unlock(vb->vb2_queue->lock);
+	return ret;
+}
+
+static void __vb2_dqbuf(struct vb2_buffer *vb);
+
+static void vb2_req_unprepare(struct media_request_object *obj)
+{
+	struct vb2_buffer *vb = container_of(obj, struct vb2_buffer, req_obj);
+
+	mutex_lock(vb->vb2_queue->lock);
+	__vb2_dqbuf(vb);
+	vb->state = VB2_BUF_STATE_IN_REQUEST;
+	mutex_unlock(vb->vb2_queue->lock);
+	WARN_ON(!vb->req_obj.req);
+}
+
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct media_request *req);
+
+static void vb2_req_queue(struct media_request_object *obj)
+{
+	struct vb2_buffer *vb = container_of(obj, struct vb2_buffer, req_obj);
+
+	mutex_lock(vb->vb2_queue->lock);
+	vb2_core_qbuf(vb->vb2_queue, vb->index, NULL, NULL);
+	mutex_unlock(vb->vb2_queue->lock);
+}
+
+static void vb2_req_release(struct media_request_object *obj)
+{
+	struct vb2_buffer *vb = container_of(obj, struct vb2_buffer, req_obj);
+
+	if (vb->state == VB2_BUF_STATE_IN_REQUEST)
+		vb->state = VB2_BUF_STATE_DEQUEUED;
+}
+
+static const struct media_request_object_ops vb2_core_req_ops = {
+	.prepare = vb2_req_prepare,
+	.unprepare = vb2_req_unprepare,
+	.queue = vb2_req_queue,
+	.release = vb2_req_release,
+};
+
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 {
 	struct vb2_buffer *vb;
@@ -1297,7 +1361,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 
 	dprintk(2, "prepare of buffer %d succeeded\n", vb->index);
 
-	return ret;
+	return 0;
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
@@ -1364,13 +1428,39 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct media_request *req)
 {
 	struct vb2_buffer *vb;
 	int ret;
 
 	vb = q->bufs[index];
 
+	if (vb->state == VB2_BUF_STATE_DEQUEUED && req) {
+		int ret;
+
+		media_request_object_init(&vb->req_obj);
+		/*
+		 * Note: the bind function checks if the request is in
+		 * IDLE state and returns an error if it isn't. It uses
+		 * the req->lock spinlock and so we do not need to take
+		 * the media_device req_queue_mutex here in order to
+		 * prevent a race condition with the QUEUE ioctl.
+		 */
+		ret = media_request_object_bind(req, &vb2_core_req_ops,
+						q, &vb->req_obj);
+		if (ret)
+			return ret;
+
+		vb->state = VB2_BUF_STATE_IN_REQUEST;
+		/* Fill buffer information for the userspace */
+		if (pb)
+			call_void_bufop(q, fill_user_buffer, vb, pb);
+
+		dprintk(2, "qbuf of buffer %d succeeded\n", vb->index);
+		return 0;
+	}
+
 	switch (vb->state) {
 	case VB2_BUF_STATE_DEQUEUED:
 		ret = __buf_prepare(vb);
@@ -1578,6 +1668,10 @@ static void __vb2_dqbuf(struct vb2_buffer *vb)
 			call_void_memop(vb, unmap_dmabuf, vb->planes[i].mem_priv);
 			vb->planes[i].dbuf_mapped = 0;
 		}
+	if (vb->req_obj.req) {
+		media_request_object_unbind(&vb->req_obj);
+		media_request_object_put(&vb->req_obj);
+	}
 }
 
 int vb2_core_dqbuf(struct vb2_queue *q, unsigned int *pindex, void *pb,
@@ -1691,6 +1785,17 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 	for (i = 0; i < q->num_buffers; ++i) {
 		struct vb2_buffer *vb = q->bufs[i];
 
+		/*
+		 * If a request is associated with this buffer, then
+		 * call buf_request_cancel() to give the driver to complete()
+		 * related request objects. Otherwise those objects would
+		 * never complete.
+		 */
+		if (vb->req_obj.req &&
+		    atomic_read(&vb->req_obj.req->state) ==
+						MEDIA_REQUEST_STATE_QUEUED)
+			call_void_vb_qop(vb, buf_request_complete, vb);
+
 		if (vb->state == VB2_BUF_STATE_PREPARED ||
 		    vb->state == VB2_BUF_STATE_QUEUED) {
 			unsigned int plane;
@@ -1700,7 +1805,8 @@ static void __vb2_queue_cancel(struct vb2_queue *q)
 						vb->planes[plane].mem_priv);
 		}
 
-		if (vb->state != VB2_BUF_STATE_DEQUEUED) {
+		if (vb->state != VB2_BUF_STATE_DEQUEUED &&
+		    vb->state != VB2_BUF_STATE_IN_REQUEST) {
 			vb->state = VB2_BUF_STATE_PREPARED;
 			call_void_vb_qop(vb, buf_finish, vb);
 		}
@@ -2259,7 +2365,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2438,7 +2544,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2541,7 +2647,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index bf7a3ba9fed0..b8d370b97cca 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -544,7 +544,7 @@ int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b)
 
 	ret = vb2_queue_or_prepare_buf(q, b, "prepare_buf");
 
-	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
+	return ret ? ret : vb2_core_prepare_buf(q, b->index, b, NULL);
 }
 EXPORT_SYMBOL_GPL(vb2_prepare_buf);
 
@@ -612,7 +612,7 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 	}
 
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	return ret ? ret : vb2_core_qbuf(q, b->index, b, NULL);
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/drivers/media/dvb-core/dvb_vb2.c b/drivers/media/dvb-core/dvb_vb2.c
index da6a8cec7d42..f1e7f0536028 100644
--- a/drivers/media/dvb-core/dvb_vb2.c
+++ b/drivers/media/dvb-core/dvb_vb2.c
@@ -384,7 +384,7 @@ int dvb_vb2_qbuf(struct dvb_vb2_ctx *ctx, struct dmx_buffer *b)
 {
 	int ret;
 
-	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b);
+	ret = vb2_core_qbuf(&ctx->vb_q, b->index, b, NULL);
 	if (ret) {
 		dprintk(1, "[%s] index=%d errno=%d\n", ctx->name,
 			b->index, ret);
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 3d54654c3cd4..0b0e8b56aed1 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -204,6 +204,7 @@ enum vb2_io_modes {
 /**
  * enum vb2_buffer_state - current video buffer state.
  * @VB2_BUF_STATE_DEQUEUED:	buffer under userspace control.
+ * @VB2_BUF_STATE_IN_REQUEST:	buffer is queued in media request.
  * @VB2_BUF_STATE_PREPARING:	buffer is being prepared in videobuf.
  * @VB2_BUF_STATE_PREPARED:	buffer prepared in videobuf and by the driver.
  * @VB2_BUF_STATE_QUEUED:	buffer queued in videobuf, but not in driver.
@@ -218,6 +219,7 @@ enum vb2_io_modes {
  */
 enum vb2_buffer_state {
 	VB2_BUF_STATE_DEQUEUED,
+	VB2_BUF_STATE_IN_REQUEST,
 	VB2_BUF_STATE_PREPARING,
 	VB2_BUF_STATE_PREPARED,
 	VB2_BUF_STATE_QUEUED,
@@ -290,6 +292,7 @@ struct vb2_buffer {
 	u32		cnt_buf_finish;
 	u32		cnt_buf_cleanup;
 	u32		cnt_buf_queue;
+	u32		cnt_buf_request_complete;
 
 	/* This counts the number of calls to vb2_buffer_done() */
 	u32		cnt_buf_done;
@@ -383,6 +386,11 @@ struct vb2_buffer {
  *			ioctl; might be called before @start_streaming callback
  *			if user pre-queued buffers before calling
  *			VIDIOC_STREAMON().
+ * @buf_request_complete: a buffer that was never queued to the driver but is
+ *			associated with a queued request was canceled.
+ *			The driver will have to mark associated objects in the
+ *			request as completed; required if requests are
+ *			supported.
  */
 struct vb2_ops {
 	int (*queue_setup)(struct vb2_queue *q,
@@ -401,6 +409,8 @@ struct vb2_ops {
 	void (*stop_streaming)(struct vb2_queue *q);
 
 	void (*buf_queue)(struct vb2_buffer *vb);
+
+	void (*buf_request_complete)(struct vb2_buffer *vb);
 };
 
 /**
@@ -754,12 +764,17 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * @index:	id number of the buffer
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
+ * @req:	pointer to &struct media_request, may be NULL.
  *
  * Videobuf2 core helper to implement VIDIOC_QBUF() operation. It is called
  * internally by VB2 by an API-specific handler, like ``videobuf2-v4l2.h``.
  *
  * This function:
  *
+ * #) If @req is non-NULL, then the buffer will be bound to this
+ *    media request and it returns. The buffer will be prepared and
+ *    queued to the driver (i.e. the next two steps) when the request
+ *    itself is queued.
  * #) if necessary, calls &vb2_ops->buf_prepare callback in the driver
  *    (if provided), in which driver-specific buffer initialization can
  *    be performed;
@@ -768,7 +783,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * Return: returns zero on success; an error code otherwise.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb,
+		  struct media_request *req);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.17.0

[PATCHv13 18/28] videobuf2-v4l2: integrate with media requests

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

This implements the V4L2 part of the request support. The main
change is that vb2_qbuf and vb2_prepare_buf now have a new
media_device pointer. This required changes to several drivers
that did not use the vb2_ioctl_qbuf/prepare_buf helper functions.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 .../media/common/videobuf2/videobuf2-v4l2.c   | 97 ++++++++++++++++---
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 .../media/platform/s3c-camif/camif-capture.c  |  4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_dec.c  |  4 +-
 drivers/media/platform/s5p-mfc/s5p_mfc_enc.c  |  4 +-
 .../media/platform/soc_camera/soc_camera.c    |  4 +-
 drivers/media/usb/uvc/uvc_queue.c             |  5 +-
 drivers/media/usb/uvc/uvc_v4l2.c              |  3 +-
 drivers/media/usb/uvc/uvcvideo.h              |  1 +
 drivers/media/v4l2-core/v4l2-mem2mem.c        |  7 +-
 .../staging/media/davinci_vpfe/vpfe_video.c   |  3 +-
 drivers/staging/media/omap4iss/iss_video.c    |  3 +-
 drivers/usb/gadget/function/uvc_queue.c       |  2 +-
 include/media/videobuf2-v4l2.h                | 14 ++-
 14 files changed, 122 insertions(+), 31 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index b8d370b97cca..0a68b19b40da 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -25,6 +25,7 @@
 #include <linux/kthread.h>
 
 #include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-common.h>
@@ -40,10 +41,12 @@ module_param(debug, int, 0644);
 			pr_info("vb2-v4l2: %s: " fmt, __func__, ## arg); \
 	} while (0)
 
-/* Flags that are set by the vb2 core */
+/* Flags that are set by us */
 #define V4L2_BUFFER_MASK_FLAGS	(V4L2_BUF_FLAG_MAPPED | V4L2_BUF_FLAG_QUEUED | \
 				 V4L2_BUF_FLAG_DONE | V4L2_BUF_FLAG_ERROR | \
 				 V4L2_BUF_FLAG_PREPARED | \
+				 V4L2_BUF_FLAG_IN_REQUEST | \
+				 V4L2_BUF_FLAG_REQUEST_FD | \
 				 V4L2_BUF_FLAG_TIMESTAMP_MASK)
 /* Output buffer flags that should be passed on to the driver */
 #define V4L2_BUFFER_OUT_FLAGS	(V4L2_BUF_FLAG_PFRAME | V4L2_BUF_FLAG_BFRAME | \
@@ -181,6 +184,7 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 		return -EINVAL;
 	}
 	vbuf->sequence = 0;
+	vbuf->request_fd = -1;
 
 	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
 		switch (b->memory) {
@@ -318,9 +322,12 @@ static int vb2_fill_vb2_v4l2_buffer(struct vb2_buffer *vb, struct v4l2_buffer *b
 	return 0;
 }
 
-static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
-				    const char *opname)
+static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+				    struct v4l2_buffer *b,
+				    const char *opname,
+				    struct media_request **p_req)
 {
+	struct media_request *req;
 	struct vb2_v4l2_buffer *vbuf;
 	struct vb2_buffer *vb;
 	int ret;
@@ -354,7 +361,55 @@ static int vb2_queue_or_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b,
 
 	/* Copy relevant information provided by the userspace */
 	memset(vbuf->planes, 0, sizeof(vbuf->planes[0]) * vb->num_planes);
-	return vb2_fill_vb2_v4l2_buffer(vb, b);
+	ret = vb2_fill_vb2_v4l2_buffer(vb, b);
+	if (ret)
+		return ret;
+
+	if (!(b->flags & V4L2_BUF_FLAG_REQUEST_FD))
+		return 0;
+
+	/*
+	 * For proper locking when queueing a request you need to be able
+	 * to lock access to the vb2 queue, so check that there is a lock
+	 * that we can use. In addition p_req must be non-NULL.
+	 */
+	if (WARN_ON(!q->lock || !p_req))
+		return -EINVAL;
+
+	/*
+	 * Make sure this op is implemented by the driver. It's easy to forget
+	 * this callback, but is it important when canceling a buffer in a
+	 * queued request.
+	 */
+	if (WARN_ON(!q->ops->buf_request_complete))
+		return -EINVAL;
+
+	if (vb->state != VB2_BUF_STATE_DEQUEUED) {
+		dprintk(1, "%s: buffer is not in dequeued state\n", opname);
+		return -EINVAL;
+	}
+
+	if (b->request_fd < 0) {
+		dprintk(1, "%s: request_fd < 0\n", opname);
+		return -EINVAL;
+	}
+
+	req = media_request_get_by_fd(mdev, b->request_fd);
+	if (IS_ERR(req)) {
+		dprintk(1, "%s: invalid request_fd\n", opname);
+		return PTR_ERR(req);
+	}
+
+	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE) {
+		dprintk(1, "%s: request is not idle\n", opname);
+		media_request_put(req);
+		return -EBUSY;
+	}
+
+	*p_req = req;
+	vbuf->request_fd = b->request_fd;
+
+	return 0;
 }
 
 /*
@@ -437,6 +492,9 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	case VB2_BUF_STATE_ACTIVE:
 		b->flags |= V4L2_BUF_FLAG_QUEUED;
 		break;
+	case VB2_BUF_STATE_IN_REQUEST:
+		b->flags |= V4L2_BUF_FLAG_IN_REQUEST;
+		break;
 	case VB2_BUF_STATE_ERROR:
 		b->flags |= V4L2_BUF_FLAG_ERROR;
 		/* fall through */
@@ -455,6 +513,10 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 
 	if (vb2_buffer_in_use(q, vb))
 		b->flags |= V4L2_BUF_FLAG_MAPPED;
+	if (vbuf->request_fd >= 0) {
+		b->flags |= V4L2_BUF_FLAG_REQUEST_FD;
+		b->request_fd = vbuf->request_fd;
+	}
 
 	if (!q->is_output &&
 		b->flags & V4L2_BUF_FLAG_DONE &&
@@ -533,7 +595,8 @@ int vb2_reqbufs(struct vb2_queue *q, struct v4l2_requestbuffers *req)
 }
 EXPORT_SYMBOL_GPL(vb2_reqbufs);
 
-int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b)
+int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+		    struct v4l2_buffer *b)
 {
 	int ret;
 
@@ -542,9 +605,12 @@ int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b)
 		return -EBUSY;
 	}
 
-	ret = vb2_queue_or_prepare_buf(q, b, "prepare_buf");
+	if (b->flags & V4L2_BUF_FLAG_REQUEST_FD)
+		return -EINVAL;
+
+	ret = vb2_queue_or_prepare_buf(q, mdev, b, "prepare_buf", NULL);
 
-	return ret ? ret : vb2_core_prepare_buf(q, b->index, b, NULL);
+	return ret ? ret : vb2_core_prepare_buf(q, b->index, b);
 }
 EXPORT_SYMBOL_GPL(vb2_prepare_buf);
 
@@ -602,8 +668,10 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create)
 }
 EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
-int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
+int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
+	     struct v4l2_buffer *b)
 {
+	struct media_request *req = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -611,8 +679,13 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		return -EBUSY;
 	}
 
-	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b, NULL);
+	ret = vb2_queue_or_prepare_buf(q, mdev, b, "qbuf", &req);
+	if (ret)
+		return ret;
+	ret = vb2_core_qbuf(q, b->index, b, req);
+	if (req)
+		media_request_put(req);
+	return ret;
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
@@ -802,7 +875,7 @@ int vb2_ioctl_prepare_buf(struct file *file, void *priv,
 
 	if (vb2_queue_is_busy(vdev, file))
 		return -EBUSY;
-	return vb2_prepare_buf(vdev->queue, p);
+	return vb2_prepare_buf(vdev->queue, vdev->v4l2_dev->mdev, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_prepare_buf);
 
@@ -821,7 +894,7 @@ int vb2_ioctl_qbuf(struct file *file, void *priv, struct v4l2_buffer *p)
 
 	if (vb2_queue_is_busy(vdev, file))
 		return -EBUSY;
-	return vb2_qbuf(vdev->queue, p);
+	return vb2_qbuf(vdev->queue, vdev->v4l2_dev->mdev, p);
 }
 EXPORT_SYMBOL_GPL(vb2_ioctl_qbuf);
 
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 674e7fd3ad99..1de7b6b13f67 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -940,7 +940,7 @@ isp_video_qbuf(struct file *file, void *fh, struct v4l2_buffer *b)
 	int ret;
 
 	mutex_lock(&video->queue_lock);
-	ret = vb2_qbuf(&vfh->queue, b);
+	ret = vb2_qbuf(&vfh->queue, video->video.v4l2_dev->mdev, b);
 	mutex_unlock(&video->queue_lock);
 
 	return ret;
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 9ab8e7ee2e1e..fafb6da3e804 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -941,7 +941,7 @@ static int s3c_camif_qbuf(struct file *file, void *priv,
 	if (vp->owner && vp->owner != priv)
 		return -EBUSY;
 
-	return vb2_qbuf(&vp->vb_queue, buf);
+	return vb2_qbuf(&vp->vb_queue, vp->vdev.v4l2_dev->mdev, buf);
 }
 
 static int s3c_camif_dqbuf(struct file *file, void *priv,
@@ -979,7 +979,7 @@ static int s3c_camif_prepare_buf(struct file *file, void *priv,
 				 struct v4l2_buffer *b)
 {
 	struct camif_vp *vp = video_drvdata(file);
-	return vb2_prepare_buf(&vp->vb_queue, b);
+	return vb2_prepare_buf(&vp->vb_queue, vp->vdev.v4l2_dev->mdev, b);
 }
 
 static int s3c_camif_g_selection(struct file *file, void *priv,
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
index 5cf4d9921264..3d863e3f5798 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_dec.c
@@ -632,9 +632,9 @@ static int vidioc_qbuf(struct file *file, void *priv, struct v4l2_buffer *buf)
 		return -EIO;
 	}
 	if (buf->type == V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE)
-		return vb2_qbuf(&ctx->vq_src, buf);
+		return vb2_qbuf(&ctx->vq_src, NULL, buf);
 	else if (buf->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE)
-		return vb2_qbuf(&ctx->vq_dst, buf);
+		return vb2_qbuf(&ctx->vq_dst, NULL, buf);
 	return -EINVAL;
 }
 
diff --git a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
index 5c0462ca9993..ed12d5d062e8 100644
--- a/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
+++ b/drivers/media/platform/s5p-mfc/s5p_mfc_enc.c
@@ -1621,9 +1621,9 @@ static int vidioc_qbuf(struct file *file, void *priv, struct v4l2_buffer *buf)
 			mfc_err("Call on QBUF after EOS command\n");
 			return -EIO;
 		}
-		return vb2_qbuf(&ctx->vq_src, buf);
+		return vb2_qbuf(&ctx->vq_src, NULL, buf);
 	} else if (buf->type == V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE) {
-		return vb2_qbuf(&ctx->vq_dst, buf);
+		return vb2_qbuf(&ctx->vq_dst, NULL, buf);
 	}
 	return -EINVAL;
 }
diff --git a/drivers/media/platform/soc_camera/soc_camera.c b/drivers/media/platform/soc_camera/soc_camera.c
index e6787abc34ae..08adf9a79420 100644
--- a/drivers/media/platform/soc_camera/soc_camera.c
+++ b/drivers/media/platform/soc_camera/soc_camera.c
@@ -394,7 +394,7 @@ static int soc_camera_qbuf(struct file *file, void *priv,
 	if (icd->streamer != file)
 		return -EBUSY;
 
-	return vb2_qbuf(&icd->vb2_vidq, p);
+	return vb2_qbuf(&icd->vb2_vidq, NULL, p);
 }
 
 static int soc_camera_dqbuf(struct file *file, void *priv,
@@ -430,7 +430,7 @@ static int soc_camera_prepare_buf(struct file *file, void *priv,
 {
 	struct soc_camera_device *icd = file->private_data;
 
-	return vb2_prepare_buf(&icd->vb2_vidq, b);
+	return vb2_prepare_buf(&icd->vb2_vidq, NULL, b);
 }
 
 static int soc_camera_expbuf(struct file *file, void *priv,
diff --git a/drivers/media/usb/uvc/uvc_queue.c b/drivers/media/usb/uvc/uvc_queue.c
index fecccb5e7628..8964e16f2b22 100644
--- a/drivers/media/usb/uvc/uvc_queue.c
+++ b/drivers/media/usb/uvc/uvc_queue.c
@@ -300,12 +300,13 @@ int uvc_create_buffers(struct uvc_video_queue *queue,
 	return ret;
 }
 
-int uvc_queue_buffer(struct uvc_video_queue *queue, struct v4l2_buffer *buf)
+int uvc_queue_buffer(struct uvc_video_queue *queue,
+		     struct media_device *mdev, struct v4l2_buffer *buf)
 {
 	int ret;
 
 	mutex_lock(&queue->mutex);
-	ret = vb2_qbuf(&queue->queue, buf);
+	ret = vb2_qbuf(&queue->queue, mdev, buf);
 	mutex_unlock(&queue->mutex);
 
 	return ret;
diff --git a/drivers/media/usb/uvc/uvc_v4l2.c b/drivers/media/usb/uvc/uvc_v4l2.c
index bd32914259ae..3da5fdc002ac 100644
--- a/drivers/media/usb/uvc/uvc_v4l2.c
+++ b/drivers/media/usb/uvc/uvc_v4l2.c
@@ -751,7 +751,8 @@ static int uvc_ioctl_qbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
 	if (!uvc_has_privileges(handle))
 		return -EBUSY;
 
-	return uvc_queue_buffer(&stream->queue, buf);
+	return uvc_queue_buffer(&stream->queue,
+				stream->vdev.v4l2_dev->mdev, buf);
 }
 
 static int uvc_ioctl_expbuf(struct file *file, void *fh,
diff --git a/drivers/media/usb/uvc/uvcvideo.h b/drivers/media/usb/uvc/uvcvideo.h
index be5cf179228b..bc9ed18f043c 100644
--- a/drivers/media/usb/uvc/uvcvideo.h
+++ b/drivers/media/usb/uvc/uvcvideo.h
@@ -680,6 +680,7 @@ int uvc_query_buffer(struct uvc_video_queue *queue,
 int uvc_create_buffers(struct uvc_video_queue *queue,
 		       struct v4l2_create_buffers *v4l2_cb);
 int uvc_queue_buffer(struct uvc_video_queue *queue,
+		     struct media_device *mdev,
 		     struct v4l2_buffer *v4l2_buf);
 int uvc_export_buffer(struct uvc_video_queue *queue,
 		      struct v4l2_exportbuffer *exp);
diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index c4f963d96a79..438f1b869319 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -20,6 +20,7 @@
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mem2mem.h>
 #include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 
@@ -388,11 +389,12 @@ EXPORT_SYMBOL_GPL(v4l2_m2m_querybuf);
 int v4l2_m2m_qbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 		  struct v4l2_buffer *buf)
 {
+	struct video_device *vdev = video_devdata(file);
 	struct vb2_queue *vq;
 	int ret;
 
 	vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
-	ret = vb2_qbuf(vq, buf);
+	ret = vb2_qbuf(vq, vdev->v4l2_dev->mdev, buf);
 	if (!ret)
 		v4l2_m2m_try_schedule(m2m_ctx);
 
@@ -413,11 +415,12 @@ EXPORT_SYMBOL_GPL(v4l2_m2m_dqbuf);
 int v4l2_m2m_prepare_buf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 			 struct v4l2_buffer *buf)
 {
+	struct video_device *vdev = video_devdata(file);
 	struct vb2_queue *vq;
 	int ret;
 
 	vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
-	ret = vb2_prepare_buf(vq, buf);
+	ret = vb2_prepare_buf(vq, vdev->v4l2_dev->mdev, buf);
 	if (!ret)
 		v4l2_m2m_try_schedule(m2m_ctx);
 
diff --git a/drivers/staging/media/davinci_vpfe/vpfe_video.c b/drivers/staging/media/davinci_vpfe/vpfe_video.c
index 390fc98d07dd..0d243a71d82c 100644
--- a/drivers/staging/media/davinci_vpfe/vpfe_video.c
+++ b/drivers/staging/media/davinci_vpfe/vpfe_video.c
@@ -1426,7 +1426,8 @@ static int vpfe_qbuf(struct file *file, void *priv,
 		return -EACCES;
 	}
 
-	return vb2_qbuf(&video->buffer_queue, p);
+	return vb2_qbuf(&video->buffer_queue,
+			video->video_dev.v4l2_dev->mdev, p);
 }
 
 /*
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index a3a83424a926..a35d1004b522 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -805,9 +805,10 @@ iss_video_querybuf(struct file *file, void *fh, struct v4l2_buffer *b)
 static int
 iss_video_qbuf(struct file *file, void *fh, struct v4l2_buffer *b)
 {
+	struct iss_video *video = video_drvdata(file);
 	struct iss_video_fh *vfh = to_iss_video_fh(fh);
 
-	return vb2_qbuf(&vfh->queue, b);
+	return vb2_qbuf(&vfh->queue, video->video.v4l2_dev->mdev, b);
 }
 
 static int
diff --git a/drivers/usb/gadget/function/uvc_queue.c b/drivers/usb/gadget/function/uvc_queue.c
index 9e33d5206d54..f2497cb96abb 100644
--- a/drivers/usb/gadget/function/uvc_queue.c
+++ b/drivers/usb/gadget/function/uvc_queue.c
@@ -166,7 +166,7 @@ int uvcg_queue_buffer(struct uvc_video_queue *queue, struct v4l2_buffer *buf)
 	unsigned long flags;
 	int ret;
 
-	ret = vb2_qbuf(&queue->queue, buf);
+	ret = vb2_qbuf(&queue->queue, NULL, buf);
 	if (ret < 0)
 		return ret;
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 097bf3e6951d..91a2b3e1a642 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -32,6 +32,7 @@
  *		&enum v4l2_field.
  * @timecode:	frame timecode.
  * @sequence:	sequence count of this frame.
+ * @request_fd:	the request_fd associated with this buffer
  * @planes:	plane information (userptr/fd, length, bytesused, data_offset).
  *
  * Should contain enough information to be able to cover all the fields
@@ -44,6 +45,7 @@ struct vb2_v4l2_buffer {
 	__u32			field;
 	struct v4l2_timecode	timecode;
 	__u32			sequence;
+	__s32			request_fd;
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 };
 
@@ -79,6 +81,7 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
  * vb2_prepare_buf() - Pass ownership of a buffer from userspace to the kernel
  *
  * @q:		pointer to &struct vb2_queue with videobuf2 queue.
+ * @mdev:	pointer to &struct media_device, may be NULL.
  * @b:		buffer structure passed from userspace to
  *		&v4l2_ioctl_ops->vidioc_prepare_buf handler in driver
  *
@@ -90,15 +93,19 @@ int vb2_create_bufs(struct vb2_queue *q, struct v4l2_create_buffers *create);
  * #) verifies the passed buffer,
  * #) calls &vb2_ops->buf_prepare callback in the driver (if provided),
  *    in which driver-specific buffer initialization can be performed.
+ * #) if @b->request_fd is non-zero and @mdev->ops->req_queue is set,
+ *    then bind the prepared buffer to the request.
  *
  * The return values from this function are intended to be directly returned
  * from &v4l2_ioctl_ops->vidioc_prepare_buf handler in driver.
  */
-int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_prepare_buf(struct vb2_queue *q, struct media_device *mdev,
+		    struct v4l2_buffer *b);
 
 /**
  * vb2_qbuf() - Queue a buffer from userspace
  * @q:		pointer to &struct vb2_queue with videobuf2 queue.
+ * @mdev:	pointer to &struct media_device, may be NULL.
  * @b:		buffer structure passed from userspace to
  *		&v4l2_ioctl_ops->vidioc_qbuf handler in driver
  *
@@ -107,6 +114,8 @@ int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b);
  * This function:
  *
  * #) verifies the passed buffer;
+ * #) if @b->request_fd is non-zero and @mdev->ops->req_queue is set,
+ *    then bind the buffer to the request.
  * #) if necessary, calls &vb2_ops->buf_prepare callback in the driver
  *    (if provided), in which driver-specific buffer initialization can
  *    be performed;
@@ -116,7 +125,8 @@ int vb2_prepare_buf(struct vb2_queue *q, struct v4l2_buffer *b);
  * The return values from this function are intended to be directly returned
  * from &v4l2_ioctl_ops->vidioc_qbuf handler in driver.
  */
-int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b);
+int vb2_qbuf(struct vb2_queue *q, struct media_device *mdev,
+	     struct v4l2_buffer *b);
 
 /**
  * vb2_expbuf() - Export a buffer as a file descriptor
-- 
2.17.0

[PATCHv13 19/28] videobuf2-core: add request helper functions

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add a new helper function to tell if a request object is a buffer.

Add a new helper function that returns true if a media_request
contains at least one buffer.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 17 +++++++++++++++++
 include/media/videobuf2-core.h                  | 15 +++++++++++++++
 2 files changed, 32 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index 4064fa15bc40..e18b87ada892 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -1340,6 +1340,23 @@ static const struct media_request_object_ops vb2_core_req_ops = {
 	.release = vb2_req_release,
 };
 
+bool vb2_request_object_is_buffer(struct media_request_object *obj)
+{
+	return obj->ops == &vb2_core_req_ops;
+}
+EXPORT_SYMBOL_GPL(vb2_request_object_is_buffer);
+
+bool vb2_request_has_buffers(struct media_request *req)
+{
+	struct media_request_object *obj;
+
+	list_for_each_entry(obj, &req->objects, list)
+		if (vb2_request_object_is_buffer(obj))
+			return true;
+	return false;
+}
+EXPORT_SYMBOL_GPL(vb2_request_has_buffers);
+
 int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 {
 	struct vb2_buffer *vb;
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index 0b0e8b56aed1..c9045f113791 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -1161,4 +1161,19 @@ bool vb2_buffer_in_use(struct vb2_queue *q, struct vb2_buffer *vb);
  */
 int vb2_verify_memory_type(struct vb2_queue *q,
 		enum vb2_memory memory, unsigned int type);
+
+/**
+ * vb2_request_object_is_buffer() - return true if the object is a buffer
+ *
+ * @obj:	the request object.
+ */
+bool vb2_request_object_is_buffer(struct media_request_object *obj);
+
+/**
+ * vb2_request_has_buffers() - return true if the request contains buffers
+ *
+ * @req:	the request.
+ */
+bool vb2_request_has_buffers(struct media_request *req);
+
 #endif /* _MEDIA_VIDEOBUF2_CORE_H */
-- 
2.17.0

[PATCHv13 20/28] videobuf2-v4l2: add vb2_request_queue/validate helpers

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

The generic vb2_request_validate helper function checks if
there are buffers in the request and if so, prepares (validates)
all objects in the request.

The generic vb2_request_queue helper function queues all buffer
objects in the validated request.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 .../media/common/videobuf2/videobuf2-v4l2.c   | 38 +++++++++++++++++++
 include/media/videobuf2-v4l2.h                |  4 ++
 2 files changed, 42 insertions(+)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 0a68b19b40da..674bafe808ed 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -1076,6 +1076,44 @@ void vb2_ops_wait_finish(struct vb2_queue *vq)
 }
 EXPORT_SYMBOL_GPL(vb2_ops_wait_finish);
 
+int vb2_request_validate(struct media_request *req)
+{
+	struct media_request_object *obj;
+	int ret = 0;
+
+	if (!vb2_request_has_buffers(req))
+		return -ENOENT;
+
+	list_for_each_entry(obj, &req->objects, list) {
+		if (!obj->ops->prepare)
+			continue;
+
+		ret = obj->ops->prepare(obj);
+		if (ret)
+			break;
+	}
+
+	if (ret) {
+		list_for_each_entry_continue_reverse(obj, &req->objects, list)
+			if (obj->ops->unprepare)
+				obj->ops->unprepare(obj);
+		return ret;
+	}
+	return 0;
+}
+EXPORT_SYMBOL_GPL(vb2_request_validate);
+
+void vb2_request_queue(struct media_request *req)
+{
+	struct media_request_object *obj;
+
+	list_for_each_entry(obj, &req->objects, list) {
+		if (obj->ops->queue)
+			obj->ops->queue(obj);
+	}
+}
+EXPORT_SYMBOL_GPL(vb2_request_queue);
+
 MODULE_DESCRIPTION("Driver helper framework for Video for Linux 2");
 MODULE_AUTHOR("Pawel Osciak <pawel@osciak.com>, Marek Szyprowski");
 MODULE_LICENSE("GPL");
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 91a2b3e1a642..727855463838 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -303,4 +303,8 @@ void vb2_ops_wait_prepare(struct vb2_queue *vq);
  */
 void vb2_ops_wait_finish(struct vb2_queue *vq);
 
+struct media_request;
+int vb2_request_validate(struct media_request *req);
+void vb2_request_queue(struct media_request *req);
+
 #endif /* _MEDIA_VIDEOBUF2_V4L2_H */
-- 
2.17.0

[PATCHv13 21/28] v4l2-mem2mem: add vb2_m2m_request_queue

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

For mem2mem devices we have to make sure that v4l2_m2m_try_schedule()
is called whenever a request is queued.

We do that by creating a vb2_m2m_request_queue() helper that should
be used instead of the 'normal' vb2_request_queue() helper. The m2m
helper function will call v4l2_m2m_try_schedule() as needed.

In addition we also avoid calling v4l2_m2m_try_schedule() when preparing
or queueing a buffer for a request since that is no longer needed.
Instead this helper function will do that when the request is actually
queued.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 33 ++++++++++++++++++++++++--
 include/media/v4l2-mem2mem.h           |  4 ++++
 2 files changed, 35 insertions(+), 2 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index 438f1b869319..80233bf6523a 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -395,7 +395,7 @@ int v4l2_m2m_qbuf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 
 	vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
 	ret = vb2_qbuf(vq, vdev->v4l2_dev->mdev, buf);
-	if (!ret)
+	if (!ret && !(buf->flags & V4L2_BUF_FLAG_IN_REQUEST))
 		v4l2_m2m_try_schedule(m2m_ctx);
 
 	return ret;
@@ -421,7 +421,7 @@ int v4l2_m2m_prepare_buf(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 
 	vq = v4l2_m2m_get_vq(m2m_ctx, buf->type);
 	ret = vb2_prepare_buf(vq, vdev->v4l2_dev->mdev, buf);
-	if (!ret)
+	if (!ret && !(buf->flags & V4L2_BUF_FLAG_IN_REQUEST))
 		v4l2_m2m_try_schedule(m2m_ctx);
 
 	return ret;
@@ -701,6 +701,35 @@ void v4l2_m2m_buf_queue(struct v4l2_m2m_ctx *m2m_ctx,
 }
 EXPORT_SYMBOL_GPL(v4l2_m2m_buf_queue);
 
+void vb2_m2m_request_queue(struct media_request *req)
+{
+	struct media_request_object *obj;
+
+	list_for_each_entry(obj, &req->objects, list) {
+		struct v4l2_m2m_ctx *m2m_ctx;
+		struct vb2_buffer *vb;
+
+		if (!obj->ops->queue)
+			continue;
+
+		obj->ops->queue(obj);
+		if (!vb2_request_object_is_buffer(obj))
+			continue;
+
+		vb = container_of(obj, struct vb2_buffer, req_obj);
+		if (V4L2_TYPE_IS_OUTPUT(vb->vb2_queue->type))
+			m2m_ctx = container_of(vb->vb2_queue,
+					       struct v4l2_m2m_ctx,
+					       out_q_ctx.q);
+		else
+			m2m_ctx = container_of(vb->vb2_queue,
+					       struct v4l2_m2m_ctx,
+					       cap_q_ctx.q);
+		v4l2_m2m_try_schedule(m2m_ctx);
+	}
+}
+EXPORT_SYMBOL_GPL(vb2_m2m_request_queue);
+
 /* Videobuf2 ioctl helpers */
 
 int v4l2_m2m_ioctl_reqbufs(struct file *file, void *priv,
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index 3d07ba3a8262..f01b3fc13b0e 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -580,6 +580,10 @@ v4l2_m2m_dst_buf_remove_by_idx(struct v4l2_m2m_ctx *m2m_ctx, unsigned int idx)
 	return v4l2_m2m_buf_remove_by_idx(&m2m_ctx->cap_q_ctx, idx);
 }
 
+/* v4l2 request helper */
+
+void vb2_m2m_request_queue(struct media_request *req);
+
 /* v4l2 ioctl helpers */
 
 int v4l2_m2m_ioctl_reqbufs(struct file *file, void *priv,
-- 
2.17.0

[PATCHv13 22/28] Documentation: v4l: document request API

[Hans Verkuil]

From: Alexandre Courbot <acourbot@chromium.org>

Document the request API for V4L2 devices, and amend the documentation
of system calls influenced by it.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 .../media/uapi/mediactl/media-funcs.rst       |   3 +
 .../uapi/mediactl/media-ioc-request-alloc.rst |  71 ++++++
 .../uapi/mediactl/media-request-ioc-queue.rst |  46 ++++
 .../mediactl/media-request-ioc-reinit.rst     |  51 +++++
 Documentation/media/uapi/v4l/buffer.rst       |  18 +-
 Documentation/media/uapi/v4l/common.rst       |   1 +
 Documentation/media/uapi/v4l/request-api.rst  | 211 ++++++++++++++++++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst     |  25 ++-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
 .../media/videodev2.h.rst.exceptions          |   1 +
 10 files changed, 428 insertions(+), 6 deletions(-)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
 create mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst

diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst b/Documentation/media/uapi/mediactl/media-funcs.rst
index 076856501cdb..f4da9b3e17ec 100644
--- a/Documentation/media/uapi/mediactl/media-funcs.rst
+++ b/Documentation/media/uapi/mediactl/media-funcs.rst
@@ -16,3 +16,6 @@ Function Reference
     media-ioc-enum-entities
     media-ioc-enum-links
     media-ioc-setup-link
+    media-ioc-request-alloc
+    media-request-ioc-queue
+    media-request-ioc-reinit
diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
new file mode 100644
index 000000000000..d45a94c9e23c
--- /dev/null
+++ b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
@@ -0,0 +1,71 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+.. _media_ioc_request_alloc:
+
+*****************************
+ioctl MEDIA_IOC_REQUEST_ALLOC
+*****************************
+
+Name
+====
+
+MEDIA_IOC_REQUEST_ALLOC - Allocate a request
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_ALLOC, struct media_request_alloc *argp )
+    :name: MEDIA_IOC_REQUEST_ALLOC
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <media-func-open>`.
+
+``argp``
+
+
+Description
+===========
+
+If the media device supports :ref:`requests <media-request-api>`, then
+this ioctl can be used to allocate a request. A request is accessed through
+a file descriptor that is returned in struct :c:type:`media_request_alloc`.
+
+If the request was successfully allocated, then the request file descriptor
+can be passed to :ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>`,
+:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`,
+:ref:`ioctl VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` and
+:ref:`ioctl VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
+
+In addition, the request can be queued by calling
+:ref:`MEDIA_REQUEST_IOC_QUEUE` and re-initialized by calling
+:ref:`MEDIA_REQUEST_IOC_REINIT`.
+
+Finally, the file descriptor can be polled to wait for the request to
+complete.
+
+To free the request the file descriptor has to be closed.
+
+.. c:type:: media_request_alloc
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. flat-table:: struct media_request_alloc
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    *  -  __s32
+       -  ``fd``
+       -  The file descriptor of the request.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
new file mode 100644
index 000000000000..79f745a3d663
--- /dev/null
+++ b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
@@ -0,0 +1,46 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+.. _media_request_ioc_queue:
+
+*****************************
+ioctl MEDIA_REQUEST_IOC_QUEUE
+*****************************
+
+Name
+====
+
+MEDIA_REQUEST_IOC_QUEUE - Queue a request
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_QUEUE )
+    :name: MEDIA_REQUEST_IOC_QUEUE
+
+
+Arguments
+=========
+
+``request_fd``
+    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
+
+
+Description
+===========
+
+If the media device supports :ref:`requests <media-request-api>`, then
+this request ioctl can be used to queue a previously allocated request.
+
+If the request was successfully queued, then the file descriptor can be
+polled to wait for the request to complete.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EBUSY
+    The request was already queued.
diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
new file mode 100644
index 000000000000..7cbb7eb5d73e
--- /dev/null
+++ b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
@@ -0,0 +1,51 @@
+.. SPDX-License-Identifier: GPL-2.0-only
+
+.. _media_request_ioc_reinit:
+
+******************************
+ioctl MEDIA_REQUEST_IOC_REINIT
+******************************
+
+Name
+====
+
+MEDIA_REQUEST_IOC_REINIT - Re-initialize a request
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_REINIT )
+    :name: MEDIA_REQUEST_IOC_REINIT
+
+
+Arguments
+=========
+
+``request_fd``
+    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
+
+Description
+===========
+
+If the media device supports :ref:`requests <media-request-api>`, then
+this request ioctl can be used to re-initialize a previously allocated
+request.
+
+Re-initializing a request will clear any existing data from the request.
+This avoids having to close() a completed request and allocate a new
+request. Instead the completed request can just be re-initialized and
+it is ready to be used again.
+
+A request can only be re-initialized if it either has not been queued
+yet, or if it was queued and completed.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EBUSY
+    The request is queued but not yet completed.
diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
index e2c85ddc990b..8059fc0ed520 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -306,10 +306,12 @@ struct v4l2_buffer
       - A place holder for future extensions. Drivers and applications
 	must set this to 0.
     * - __u32
-      - ``reserved``
+      - ``request_fd``
       -
-      - A place holder for future extensions. Drivers and applications
-	must set this to 0.
+      - The file descriptor of the request to queue the buffer to. If specified
+        and flag ``V4L2_BUF_FLAG_REQUEST_FD`` is set, then the buffer will be
+	queued to that request. This is set by the user when calling
+	:ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>` and ignored by other ioctls.
 
 
 
@@ -514,6 +516,11 @@ Buffer Flags
 	streaming may continue as normal and the buffer may be reused
 	normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
 	called.
+    * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
+
+      - ``V4L2_BUF_FLAG_IN_REQUEST``
+      - 0x00000080
+      - This buffer is part of a request that hasn't been queued yet.
     * .. _`V4L2-BUF-FLAG-KEYFRAME`:
 
       - ``V4L2_BUF_FLAG_KEYFRAME``
@@ -589,6 +596,11 @@ Buffer Flags
 	the format. Any Any subsequent call to the
 	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
 	but return an ``EPIPE`` error code.
+    * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
+
+      - ``V4L2_BUF_FLAG_REQUEST_FD``
+      - 0x00800000
+      - The ``request_fd`` field contains a valid file descriptor.
     * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
 
       - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
diff --git a/Documentation/media/uapi/v4l/common.rst b/Documentation/media/uapi/v4l/common.rst
index 13f2ed3fc5a6..a4aa0059d45a 100644
--- a/Documentation/media/uapi/v4l/common.rst
+++ b/Documentation/media/uapi/v4l/common.rst
@@ -44,3 +44,4 @@ applicable to all devices.
     crop
     selection-api
     streaming-par
+    request-api
diff --git a/Documentation/media/uapi/v4l/request-api.rst b/Documentation/media/uapi/v4l/request-api.rst
new file mode 100644
index 000000000000..0ce75b254956
--- /dev/null
+++ b/Documentation/media/uapi/v4l/request-api.rst
@@ -0,0 +1,211 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media-request-api:
+
+Request API
+===========
+
+The Request API has been designed to allow V4L2 to deal with requirements of
+modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
+v2). One such requirement is the ability for devices belonging to the same
+pipeline to reconfigure and collaborate closely on a per-frame basis. Another is
+efficient support of stateless codecs, which need per-frame controls to be set
+asynchronously in order to be used efficiently.
+
+Supporting these features without the Request API is possible but terribly
+inefficient: user-space would have to flush all activity on the media pipeline,
+reconfigure it for the next frame, queue the buffers to be processed with that
+configuration, and wait until they are all available for dequeuing before
+considering the next frame. This defeats the purpose of having buffer queues
+since in practice only one buffer would be queued at a time.
+
+The Request API allows a specific configuration of the pipeline (media
+controller topology + controls for each device) to be associated with specific
+buffers. The parameters are applied by each participating device as buffers
+associated to a request flow in. This allows user-space to schedule several
+tasks ("requests") with different parameters in advance, knowing that the
+parameters will be applied when needed to get the expected result. Control
+values at the time of request completion are also available for reading.
+
+Usage
+=====
+
+The Request API is used on top of standard media controller and V4L2 calls,
+which are augmented with an extra ``request_fd`` parameter. Requests themselves
+are allocated from the supporting media controller node.
+
+Request Allocation
+------------------
+
+User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
+for the media device node. This returns a file descriptor representing the
+request. Typically, several such requests will be allocated.
+
+Request Preparation
+-------------------
+
+Standard V4L2 ioctls can then receive a request file descriptor to express the
+fact that the ioctl is part of said request, and is not to be applied
+immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
+support this. Controls set with a ``request_fd`` parameter are stored instead
+of being immediately applied, and buffers queued to a request do not enter the
+regular buffer queue until the request itself is queued.
+
+Request Submission
+------------------
+
+Once the parameters and buffers of the request are specified, it can be
+queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request FD.
+This will make the buffers associated to the request available to their driver,
+which can then apply the associated controls as buffers are processed. A queued
+request cannot be modified anymore.
+
+If several devices are part of the request, individual drivers may synchronize
+so the requested pipeline's topology is applied before the buffers are
+processed. This is at the discretion of media controller drivers and is not a
+requirement.
+
+Buffers queued without an associated request after a request-bound buffer will
+be processed using the state of the hardware at the time of the request
+completion. All the same, controls set without a request are applied
+immediately, regardless of whether a request is in use or not.
+
+User-space can ``poll()`` a request FD in order to wait until the request
+completes. A request is considered complete once all its associated buffers are
+available for dequeuing and all the associated controls have been updated with
+the values at the time of completion. Note that user-space does not need to wait
+for the request to complete to dequeue its buffers: buffers that are available
+halfway through a request can be dequeued independently of the request's state.
+
+A completed request contains the state of the request at the time of the
+request completion. User-space can query that state by calling
+:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request FD.
+
+Recycling and Destruction
+-------------------------
+
+Finally, completed request can either be discarded or be reused. Calling
+``close()`` on a request FD will make that FD unusable, freeing the request if
+it is not referenced elsewhere. The :ref:`MEDIA_REQUEST_IOC_REINIT` will
+clear a request's state and make it available again. No state is retained by
+this operation: the request is as if it had just been allocated.
+
+Example for a Memory-to-Memory Device
+-------------------------------------
+
+M2M devices are single-node V4L2 devices providing one OUTPUT queue (for
+user-space to provide input buffers) and one CAPTURE queue (to retrieve
+processed data). These devices are commonly used for frame processors or
+stateless codecs.
+
+In this use-case, the request API can be used to associate specific controls to
+be applied by the driver before processing an OUTPUT buffer, allowing user-space
+to queue many such buffers in advance. It can also take advantage of requests'
+ability to capture the state of controls when the request completes to read back
+information that may be subject to change.
+
+Put into code, after obtaining a request, user-space can assign controls and one
+OUTPUT buffer to it:
+
+.. code-block:: c
+
+	struct v4l2_buffer buf;
+	struct v4l2_ext_controls ctrls;
+	struct media_request_alloc alloc = { 0 };
+	int req_fd;
+	...
+	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
+	req_fd = alloc.fd;
+	...
+	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
+	ctrls.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
+	...
+	buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
+	buf.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_QBUF, &buf);
+
+Note that there is typically no need to use the Request API for CAPTURE buffers
+since there are per-frame settings to report there.
+
+Once the request is fully prepared, it can be queued to the driver:
+
+.. code-block:: c
+
+	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
+
+User-space can then either wait for the request to complete by calling poll() on
+its file descriptor, or start dequeuing CAPTURE buffers. Most likely, it will
+want to get CAPTURE buffers as soon as possible and this can be done using a
+regular DQBUF:
+
+.. code-block:: c
+
+	struct v4l2_buffer buf;
+
+	memset(&buf, 0, sizeof(buf));
+	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	ioctl(codec_fd, VIDIOC_DQBUF, &buf);
+
+We can then, after ensuring that the request is completed via polling the
+request FD, query control values at the time of its completion via an
+annotated call to G_EXT_CTRLS. This is particularly useful for volatile controls
+for which we want to query values as soon as the capture buffer is produced.
+
+.. code-block:: c
+
+	struct pollfd pfd = { .events = POLLPRI, .fd = request_fd };
+	poll(&pfd, 1, -1);
+	...
+	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
+	ctrls.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls);
+
+Once we don't need the request anymore, we can either recycle it for reuse with
+:ref:`MEDIA_REQUEST_IOC_REINIT`...
+
+.. code-block:: c
+
+	ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT);
+
+... or close its file descriptor to completely dispose of it.
+
+.. code-block:: c
+
+	close(req_fd);
+
+Example for a Simple Capture Device
+-----------------------------------
+
+With a simple capture device, requests can be used to specify controls to apply
+to a given CAPTURE buffer. The driver will apply these controls before producing
+the marked CAPTURE buffer.
+
+.. code-block:: c
+
+	struct v4l2_buffer buf;
+	struct v4l2_ext_controls ctrls;
+	struct media_request_alloc alloc = { 0 };
+	int req_fd;
+	...
+	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
+	req_fd = alloc.fd;
+	...
+	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
+	ctrls.request_fd = req_fd;
+	ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
+	...
+	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
+	buf.request_fd = req_fd;
+	ioctl(camera_fd, VIDIOC_QBUF, &buf);
+
+Once the request is fully prepared, it can be queued to the driver:
+
+.. code-block:: c
+
+	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
+
+User-space can then dequeue buffers, wait for the request completion, query
+controls and recycle the request as in the M2M example above.
diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
index 2011c2b2ee67..8f788c444347 100644
--- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
+++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
@@ -95,6 +95,19 @@ appropriate. In the first case the new value is set in struct
 is inappropriate (e.g. the given menu index is not supported by the menu
 control), then this will also result in an ``EINVAL`` error code error.
 
+If ``request_fd`` is set to a not-submitted request file descriptor and
+``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``, then the
+controls are not applied immediately when calling
+:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`, but instead are applied right
+before the driver starts processing a buffer associated to the same request.
+
+If ``request_fd`` is specified and ``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``
+during a call to :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`, then the
+returned values will be the values currently set for the request (or the
+hardware value if none is set) if the request has not yet completed, or the
+values of the controls at the time of request completion if it has already
+completed.
+
 The driver will only set/get these controls if all control values are
 correct. This prevents the situation where only some of the controls
 were set/get. Only low-level errors (e. g. a failed i2c command) can
@@ -209,8 +222,10 @@ still cause this situation.
       - ``which``
       - Which value of the control to get/set/try.
 	``V4L2_CTRL_WHICH_CUR_VAL`` will return the current value of the
-	control and ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
-	value of the control.
+	control, ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
+	value of the control and ``V4L2_CTRL_WHICH_REQUEST_VAL`` indicates that
+	these controls have to be retrieved from a request or tried/set for
+	a request.
 
 	.. note::
 
@@ -272,8 +287,12 @@ still cause this situation.
 	then you can call :ref:`VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` to try to discover the
 	actual control that failed the validation step. Unfortunately,
 	there is no ``TRY`` equivalent for :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
+    * - __s32
+      - ``request_fd``
+	File descriptor of the request to be used by this operation. Only
+	valid if ``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``.
     * - __u32
-      - ``reserved``\ [2]
+      - ``reserved``\ [1]
       - Reserved for future extensions.
 
 	Drivers and applications must set the array to zero.
diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..c9f55d8340d1 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -98,6 +98,13 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
 :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
 device is closed.
 
+The ``request_fd`` field can be used when queuing to specify the file
+descriptor of a request, if requests are in use. Setting it means that the
+buffer will not be passed to the driver until the request itself is queued.
+Also, the driver will apply any setting associated with the request before
+processing the buffer. This field will be ignored unless the
+``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
+
 Applications call the ``VIDIOC_DQBUF`` ioctl to dequeue a filled
 (capturing) or displayed (output) buffer from the driver's outgoing
 queue. They just set the ``type``, ``memory`` and ``reserved`` fields of
diff --git a/Documentation/media/videodev2.h.rst.exceptions b/Documentation/media/videodev2.h.rst.exceptions
index a5cb0a8686ac..5a5a1c772053 100644
--- a/Documentation/media/videodev2.h.rst.exceptions
+++ b/Documentation/media/videodev2.h.rst.exceptions
@@ -514,6 +514,7 @@ ignore define V4L2_CTRL_DRIVER_PRIV
 ignore define V4L2_CTRL_MAX_DIMS
 ignore define V4L2_CTRL_WHICH_CUR_VAL
 ignore define V4L2_CTRL_WHICH_DEF_VAL
+ignore define V4L2_CTRL_WHICH_REQUEST_VAL
 ignore define V4L2_OUT_CAP_CUSTOM_TIMINGS
 ignore define V4L2_CID_MAX_CTRLS
 
-- 
2.17.0

[PATCHv13 23/28] media: vim2m: add media device

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

This probably needs a bit more work to correctly represent m2m
hardware in the media topology.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/vim2m.c | 42 ++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 5 deletions(-)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..9be4da3b8577 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,10 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
 	struct v4l2_device	v4l2_dev;
 	struct video_device	vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_device	mdev;
+	struct media_pad	pad[2];
+#endif
 
 	atomic_t		num_inst;
 	struct mutex		dev_mutex;
@@ -1000,11 +1004,6 @@ static int vim2m_probe(struct platform_device *pdev)
 		return -ENOMEM;
 
 	spin_lock_init(&dev->irqlock);
-
-	ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
-	if (ret)
-		return ret;
-
 	atomic_set(&dev->num_inst, 0);
 	mutex_init(&dev->dev_mutex);
 
@@ -1013,6 +1012,22 @@ static int vim2m_probe(struct platform_device *pdev)
 	vfd->lock = &dev->dev_mutex;
 	vfd->v4l2_dev = &dev->v4l2_dev;
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->mdev.dev = &pdev->dev;
+	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+	media_device_init(&dev->mdev);
+	dev->v4l2_dev.mdev = &dev->mdev;
+	dev->pad[0].flags = MEDIA_PAD_FL_SINK;
+	dev->pad[1].flags = MEDIA_PAD_FL_SOURCE;
+	ret = media_entity_pads_init(&vfd->entity, 2, dev->pad);
+	if (ret)
+		return ret;
+#endif
+
+	ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
+	if (ret)
+		goto unreg_media;
+
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
 	if (ret) {
 		v4l2_err(&dev->v4l2_dev, "Failed to register video device\n");
@@ -1034,6 +1049,13 @@ static int vim2m_probe(struct platform_device *pdev)
 		goto err_m2m;
 	}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	/* Register the media device node */
+	ret = media_device_register(&dev->mdev);
+	if (ret)
+		goto err_m2m;
+#endif
+
 	return 0;
 
 err_m2m:
@@ -1041,6 +1063,10 @@ static int vim2m_probe(struct platform_device *pdev)
 	video_unregister_device(&dev->vfd);
 unreg_dev:
 	v4l2_device_unregister(&dev->v4l2_dev);
+unreg_media:
+#ifdef CONFIG_MEDIA_CONTROLLER
+	media_device_unregister(&dev->mdev);
+#endif
 
 	return ret;
 }
@@ -1050,6 +1076,12 @@ static int vim2m_remove(struct platform_device *pdev)
 	struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
 	v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	media_device_unregister(&dev->mdev);
+	media_device_cleanup(&dev->mdev);
+#endif
+
 	v4l2_m2m_release(dev->m2m_dev);
 	del_timer_sync(&dev->timer);
 	video_unregister_device(&dev->vfd);
-- 
2.17.0

[PATCHv13 24/28] vim2m: use workqueue

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

v4l2_ctrl uses mutexes, so we can't setup a ctrl_handler in
interrupt context. Switch to a workqueue instead.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/vim2m.c | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 9be4da3b8577..a1b0bb08668d 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -150,6 +150,7 @@ struct vim2m_dev {
 	spinlock_t		irqlock;
 
 	struct timer_list	timer;
+	struct work_struct	work_run;
 
 	struct v4l2_m2m_dev	*m2m_dev;
 };
@@ -392,9 +393,10 @@ static void device_run(void *priv)
 	schedule_irq(dev, ctx->transtime);
 }
 
-static void device_isr(struct timer_list *t)
+static void device_work(struct work_struct *w)
 {
-	struct vim2m_dev *vim2m_dev = from_timer(vim2m_dev, t, timer);
+	struct vim2m_dev *vim2m_dev =
+		container_of(w, struct vim2m_dev, work_run);
 	struct vim2m_ctx *curr_ctx;
 	struct vb2_v4l2_buffer *src_vb, *dst_vb;
 	unsigned long flags;
@@ -426,6 +428,13 @@ static void device_isr(struct timer_list *t)
 	}
 }
 
+static void device_isr(struct timer_list *t)
+{
+	struct vim2m_dev *vim2m_dev = from_timer(vim2m_dev, t, timer);
+
+	schedule_work(&vim2m_dev->work_run);
+}
+
 /*
  * video ioctls
  */
@@ -806,6 +815,7 @@ static void vim2m_stop_streaming(struct vb2_queue *q)
 	struct vb2_v4l2_buffer *vbuf;
 	unsigned long flags;
 
+	flush_scheduled_work();
 	for (;;) {
 		if (V4L2_TYPE_IS_OUTPUT(q->type))
 			vbuf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx);
@@ -1011,6 +1021,7 @@ static int vim2m_probe(struct platform_device *pdev)
 	vfd = &dev->vfd;
 	vfd->lock = &dev->dev_mutex;
 	vfd->v4l2_dev = &dev->v4l2_dev;
+	INIT_WORK(&dev->work_run, device_work);
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 	dev->mdev.dev = &pdev->dev;
-- 
2.17.0

[PATCHv13 25/28] vim2m: support requests

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add support for requests to vim2m.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/vim2m.c | 34 ++++++++++++++++++++++++++++++++++
 1 file changed, 34 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index a1b0bb08668d..7c86c711a4cd 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -387,8 +387,26 @@ static void device_run(void *priv)
 	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
 	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
 
+	/* Apply request controls if needed */
+	if (src_buf->vb2_buf.req_obj.req)
+		v4l2_ctrl_request_setup(src_buf->vb2_buf.req_obj.req,
+					&ctx->hdl);
+	if (dst_buf->vb2_buf.req_obj.req &&
+	    dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
+		v4l2_ctrl_request_setup(dst_buf->vb2_buf.req_obj.req,
+					&ctx->hdl);
+
 	device_process(ctx, src_buf, dst_buf);
 
+	/* Complete request controls if needed */
+	if (src_buf->vb2_buf.req_obj.req)
+		v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req,
+					&ctx->hdl);
+	if (dst_buf->vb2_buf.req_obj.req &&
+	    dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
+		v4l2_ctrl_request_complete(dst_buf->vb2_buf.req_obj.req,
+					&ctx->hdl);
+
 	/* Run a timer, which simulates a hardware irq  */
 	schedule_irq(dev, ctx->transtime);
 }
@@ -823,12 +841,21 @@ static void vim2m_stop_streaming(struct vb2_queue *q)
 			vbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx);
 		if (vbuf == NULL)
 			return;
+		v4l2_ctrl_request_complete(vbuf->vb2_buf.req_obj.req,
+					   &ctx->hdl);
 		spin_lock_irqsave(&ctx->dev->irqlock, flags);
 		v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR);
 		spin_unlock_irqrestore(&ctx->dev->irqlock, flags);
 	}
 }
 
+static void vim2m_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vim2m_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &ctx->hdl);
+}
+
 static const struct vb2_ops vim2m_qops = {
 	.queue_setup	 = vim2m_queue_setup,
 	.buf_prepare	 = vim2m_buf_prepare,
@@ -837,6 +864,7 @@ static const struct vb2_ops vim2m_qops = {
 	.stop_streaming  = vim2m_stop_streaming,
 	.wait_prepare	 = vb2_ops_wait_prepare,
 	.wait_finish	 = vb2_ops_wait_finish,
+	.buf_request_complete = vim2m_buf_request_complete,
 };
 
 static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
@@ -1003,6 +1031,11 @@ static const struct v4l2_m2m_ops m2m_ops = {
 	.job_abort	= job_abort,
 };
 
+static const struct media_device_ops m2m_media_ops = {
+	.req_validate = vb2_request_validate,
+	.req_queue = vb2_m2m_request_queue,
+};
+
 static int vim2m_probe(struct platform_device *pdev)
 {
 	struct vim2m_dev *dev;
@@ -1027,6 +1060,7 @@ static int vim2m_probe(struct platform_device *pdev)
 	dev->mdev.dev = &pdev->dev;
 	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
 	media_device_init(&dev->mdev);
+	dev->mdev.ops = &m2m_media_ops;
 	dev->v4l2_dev.mdev = &dev->mdev;
 	dev->pad[0].flags = MEDIA_PAD_FL_SINK;
 	dev->pad[1].flags = MEDIA_PAD_FL_SOURCE;
-- 
2.17.0

[PATCHv13 26/28] vivid: add mc

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add support for the media_device to vivid. This is a prerequisite
for request support.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/vivid/vivid-core.c | 61 +++++++++++++++++++++++
 drivers/media/platform/vivid/vivid-core.h |  8 +++
 2 files changed, 69 insertions(+)

diff --git a/drivers/media/platform/vivid/vivid-core.c b/drivers/media/platform/vivid/vivid-core.c
index 82ec216f2ad8..69386b26d5dd 100644
--- a/drivers/media/platform/vivid/vivid-core.c
+++ b/drivers/media/platform/vivid/vivid-core.c
@@ -657,6 +657,15 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 
 	dev->inst = inst;
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->v4l2_dev.mdev = &dev->mdev;
+
+	/* Initialize media device */
+	strlcpy(dev->mdev.model, VIVID_MODULE_NAME, sizeof(dev->mdev.model));
+	dev->mdev.dev = &pdev->dev;
+	media_device_init(&dev->mdev);
+#endif
+
 	/* register v4l2_device */
 	snprintf(dev->v4l2_dev.name, sizeof(dev->v4l2_dev.name),
 			"%s-%03d", VIVID_MODULE_NAME, inst);
@@ -1173,6 +1182,13 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 		vfd->lock = &dev->mutex;
 		video_set_drvdata(vfd, dev);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		dev->vid_cap_pad.flags = MEDIA_PAD_FL_SINK;
+		ret = media_entity_pads_init(&vfd->entity, 1, &dev->vid_cap_pad);
+		if (ret)
+			goto unreg_dev;
+#endif
+
 #ifdef CONFIG_VIDEO_VIVID_CEC
 		if (in_type_counter[HDMI]) {
 			struct cec_adapter *adap;
@@ -1225,6 +1241,13 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 		vfd->lock = &dev->mutex;
 		video_set_drvdata(vfd, dev);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		dev->vid_out_pad.flags = MEDIA_PAD_FL_SOURCE;
+		ret = media_entity_pads_init(&vfd->entity, 1, &dev->vid_out_pad);
+		if (ret)
+			goto unreg_dev;
+#endif
+
 #ifdef CONFIG_VIDEO_VIVID_CEC
 		for (i = 0; i < dev->num_outputs; i++) {
 			struct cec_adapter *adap;
@@ -1274,6 +1297,13 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 		vfd->tvnorms = tvnorms_cap;
 		video_set_drvdata(vfd, dev);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		dev->vbi_cap_pad.flags = MEDIA_PAD_FL_SINK;
+		ret = media_entity_pads_init(&vfd->entity, 1, &dev->vbi_cap_pad);
+		if (ret)
+			goto unreg_dev;
+#endif
+
 		ret = video_register_device(vfd, VFL_TYPE_VBI, vbi_cap_nr[inst]);
 		if (ret < 0)
 			goto unreg_dev;
@@ -1299,6 +1329,13 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 		vfd->tvnorms = tvnorms_out;
 		video_set_drvdata(vfd, dev);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		dev->vbi_out_pad.flags = MEDIA_PAD_FL_SOURCE;
+		ret = media_entity_pads_init(&vfd->entity, 1, &dev->vbi_out_pad);
+		if (ret)
+			goto unreg_dev;
+#endif
+
 		ret = video_register_device(vfd, VFL_TYPE_VBI, vbi_out_nr[inst]);
 		if (ret < 0)
 			goto unreg_dev;
@@ -1322,6 +1359,13 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 		vfd->lock = &dev->mutex;
 		video_set_drvdata(vfd, dev);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		dev->sdr_cap_pad.flags = MEDIA_PAD_FL_SINK;
+		ret = media_entity_pads_init(&vfd->entity, 1, &dev->sdr_cap_pad);
+		if (ret)
+			goto unreg_dev;
+#endif
+
 		ret = video_register_device(vfd, VFL_TYPE_SDR, sdr_cap_nr[inst]);
 		if (ret < 0)
 			goto unreg_dev;
@@ -1368,12 +1412,25 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 					  video_device_node_name(vfd));
 	}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	/* Register the media device */
+	ret = media_device_register(&dev->mdev);
+	if (ret) {
+		dev_err(dev->mdev.dev,
+			"media device register failed (err=%d)\n", ret);
+		goto unreg_dev;
+	}
+#endif
+
 	/* Now that everything is fine, let's add it to device list */
 	vivid_devs[inst] = dev;
 
 	return 0;
 
 unreg_dev:
+#ifdef CONFIG_MEDIA_CONTROLLER
+	media_device_unregister(&dev->mdev);
+#endif
 	video_unregister_device(&dev->radio_tx_dev);
 	video_unregister_device(&dev->radio_rx_dev);
 	video_unregister_device(&dev->sdr_cap_dev);
@@ -1444,6 +1501,10 @@ static int vivid_remove(struct platform_device *pdev)
 		if (!dev)
 			continue;
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+		media_device_unregister(&dev->mdev);
+#endif
+
 		if (dev->has_vid_cap) {
 			v4l2_info(&dev->v4l2_dev, "unregistering %s\n",
 				video_device_node_name(&dev->vid_cap_dev));
diff --git a/drivers/media/platform/vivid/vivid-core.h b/drivers/media/platform/vivid/vivid-core.h
index 477c80a4d44c..6ccd1f5c1d91 100644
--- a/drivers/media/platform/vivid/vivid-core.h
+++ b/drivers/media/platform/vivid/vivid-core.h
@@ -136,6 +136,14 @@ struct vivid_cec_work {
 struct vivid_dev {
 	unsigned			inst;
 	struct v4l2_device		v4l2_dev;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_device		mdev;
+	struct media_pad		vid_cap_pad;
+	struct media_pad		vid_out_pad;
+	struct media_pad		vbi_cap_pad;
+	struct media_pad		vbi_out_pad;
+	struct media_pad		sdr_cap_pad;
+#endif
 	struct v4l2_ctrl_handler	ctrl_hdl_user_gen;
 	struct v4l2_ctrl_handler	ctrl_hdl_user_vid;
 	struct v4l2_ctrl_handler	ctrl_hdl_user_aud;
-- 
2.17.0

[PATCHv13 27/28] vivid: add request support

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add support for requests to vivid.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/platform/vivid/vivid-core.c        |  8 ++++++++
 drivers/media/platform/vivid/vivid-kthread-cap.c | 12 ++++++++++++
 drivers/media/platform/vivid/vivid-kthread-out.c | 12 ++++++++++++
 drivers/media/platform/vivid/vivid-sdr-cap.c     | 16 ++++++++++++++++
 drivers/media/platform/vivid/vivid-vbi-cap.c     | 10 ++++++++++
 drivers/media/platform/vivid/vivid-vbi-out.c     | 10 ++++++++++
 drivers/media/platform/vivid/vivid-vid-cap.c     | 10 ++++++++++
 drivers/media/platform/vivid/vivid-vid-out.c     | 10 ++++++++++
 8 files changed, 88 insertions(+)

diff --git a/drivers/media/platform/vivid/vivid-core.c b/drivers/media/platform/vivid/vivid-core.c
index 69386b26d5dd..a84c7e37c22b 100644
--- a/drivers/media/platform/vivid/vivid-core.c
+++ b/drivers/media/platform/vivid/vivid-core.c
@@ -627,6 +627,13 @@ static void vivid_dev_release(struct v4l2_device *v4l2_dev)
 	kfree(dev);
 }
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+static const struct media_device_ops vivid_media_ops = {
+	.req_validate = vb2_request_validate,
+	.req_queue = vb2_request_queue,
+};
+#endif
+
 static int vivid_create_instance(struct platform_device *pdev, int inst)
 {
 	static const struct v4l2_dv_timings def_dv_timings =
@@ -664,6 +671,7 @@ static int vivid_create_instance(struct platform_device *pdev, int inst)
 	strlcpy(dev->mdev.model, VIVID_MODULE_NAME, sizeof(dev->mdev.model));
 	dev->mdev.dev = &pdev->dev;
 	media_device_init(&dev->mdev);
+	dev->mdev.ops = &vivid_media_ops;
 #endif
 
 	/* register v4l2_device */
diff --git a/drivers/media/platform/vivid/vivid-kthread-cap.c b/drivers/media/platform/vivid/vivid-kthread-cap.c
index 3fdb280c36ca..c192b4b1b9de 100644
--- a/drivers/media/platform/vivid/vivid-kthread-cap.c
+++ b/drivers/media/platform/vivid/vivid-kthread-cap.c
@@ -703,6 +703,8 @@ static void vivid_thread_vid_cap_tick(struct vivid_dev *dev, int dropped_bufs)
 		goto update_mv;
 
 	if (vid_cap_buf) {
+		v4l2_ctrl_request_setup(vid_cap_buf->vb.vb2_buf.req_obj.req,
+					&dev->ctrl_hdl_vid_cap);
 		/* Fill buffer */
 		vivid_fillbuff(dev, vid_cap_buf);
 		dprintk(dev, 1, "filled buffer %d\n",
@@ -713,6 +715,8 @@ static void vivid_thread_vid_cap_tick(struct vivid_dev *dev, int dropped_bufs)
 			dev->fb_cap.fmt.pixelformat == dev->fmt_cap->fourcc)
 			vivid_overlay(dev, vid_cap_buf);
 
+		v4l2_ctrl_request_complete(vid_cap_buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_vid_cap);
 		vb2_buffer_done(&vid_cap_buf->vb.vb2_buf, dev->dqbuf_error ?
 				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
 		dprintk(dev, 2, "vid_cap buffer %d done\n",
@@ -720,10 +724,14 @@ static void vivid_thread_vid_cap_tick(struct vivid_dev *dev, int dropped_bufs)
 	}
 
 	if (vbi_cap_buf) {
+		v4l2_ctrl_request_setup(vbi_cap_buf->vb.vb2_buf.req_obj.req,
+					&dev->ctrl_hdl_vbi_cap);
 		if (dev->stream_sliced_vbi_cap)
 			vivid_sliced_vbi_cap_process(dev, vbi_cap_buf);
 		else
 			vivid_raw_vbi_cap_process(dev, vbi_cap_buf);
+		v4l2_ctrl_request_complete(vbi_cap_buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_vbi_cap);
 		vb2_buffer_done(&vbi_cap_buf->vb.vb2_buf, dev->dqbuf_error ?
 				VB2_BUF_STATE_ERROR : VB2_BUF_STATE_DONE);
 		dprintk(dev, 2, "vbi_cap %d done\n",
@@ -891,6 +899,8 @@ void vivid_stop_generating_vid_cap(struct vivid_dev *dev, bool *pstreaming)
 			buf = list_entry(dev->vid_cap_active.next,
 					 struct vivid_buffer, list);
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vid_cap);
 			vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
 			dprintk(dev, 2, "vid_cap buffer %d done\n",
 				buf->vb.vb2_buf.index);
@@ -904,6 +914,8 @@ void vivid_stop_generating_vid_cap(struct vivid_dev *dev, bool *pstreaming)
 			buf = list_entry(dev->vbi_cap_active.next,
 					 struct vivid_buffer, list);
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vbi_cap);
 			vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
 			dprintk(dev, 2, "vbi_cap buffer %d done\n",
 				buf->vb.vb2_buf.index);
diff --git a/drivers/media/platform/vivid/vivid-kthread-out.c b/drivers/media/platform/vivid/vivid-kthread-out.c
index 9981e7548019..5a14810eeb69 100644
--- a/drivers/media/platform/vivid/vivid-kthread-out.c
+++ b/drivers/media/platform/vivid/vivid-kthread-out.c
@@ -75,6 +75,10 @@ static void vivid_thread_vid_out_tick(struct vivid_dev *dev)
 		return;
 
 	if (vid_out_buf) {
+		v4l2_ctrl_request_setup(vid_out_buf->vb.vb2_buf.req_obj.req,
+					&dev->ctrl_hdl_vid_out);
+		v4l2_ctrl_request_complete(vid_out_buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_vid_out);
 		vid_out_buf->vb.sequence = dev->vid_out_seq_count;
 		if (dev->field_out == V4L2_FIELD_ALTERNATE) {
 			/*
@@ -92,6 +96,10 @@ static void vivid_thread_vid_out_tick(struct vivid_dev *dev)
 	}
 
 	if (vbi_out_buf) {
+		v4l2_ctrl_request_setup(vbi_out_buf->vb.vb2_buf.req_obj.req,
+					&dev->ctrl_hdl_vbi_out);
+		v4l2_ctrl_request_complete(vbi_out_buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_vbi_out);
 		if (dev->stream_sliced_vbi_out)
 			vivid_sliced_vbi_out_process(dev, vbi_out_buf);
 
@@ -262,6 +270,8 @@ void vivid_stop_generating_vid_out(struct vivid_dev *dev, bool *pstreaming)
 			buf = list_entry(dev->vid_out_active.next,
 					 struct vivid_buffer, list);
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vid_out);
 			vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
 			dprintk(dev, 2, "vid_out buffer %d done\n",
 				buf->vb.vb2_buf.index);
@@ -275,6 +285,8 @@ void vivid_stop_generating_vid_out(struct vivid_dev *dev, bool *pstreaming)
 			buf = list_entry(dev->vbi_out_active.next,
 					 struct vivid_buffer, list);
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vbi_out);
 			vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
 			dprintk(dev, 2, "vbi_out buffer %d done\n",
 				buf->vb.vb2_buf.index);
diff --git a/drivers/media/platform/vivid/vivid-sdr-cap.c b/drivers/media/platform/vivid/vivid-sdr-cap.c
index cfb7cb4d37a8..76cf8810a974 100644
--- a/drivers/media/platform/vivid/vivid-sdr-cap.c
+++ b/drivers/media/platform/vivid/vivid-sdr-cap.c
@@ -102,6 +102,10 @@ static void vivid_thread_sdr_cap_tick(struct vivid_dev *dev)
 
 	if (sdr_cap_buf) {
 		sdr_cap_buf->vb.sequence = dev->sdr_cap_seq_count;
+		v4l2_ctrl_request_setup(sdr_cap_buf->vb.vb2_buf.req_obj.req,
+					&dev->ctrl_hdl_sdr_cap);
+		v4l2_ctrl_request_complete(sdr_cap_buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_sdr_cap);
 		vivid_sdr_cap_process(dev, sdr_cap_buf);
 		sdr_cap_buf->vb.vb2_buf.timestamp =
 			ktime_get_ns() + dev->time_wrap_offset;
@@ -272,6 +276,8 @@ static int sdr_cap_start_streaming(struct vb2_queue *vq, unsigned count)
 
 		list_for_each_entry_safe(buf, tmp, &dev->sdr_cap_active, list) {
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_sdr_cap);
 			vb2_buffer_done(&buf->vb.vb2_buf,
 					VB2_BUF_STATE_QUEUED);
 		}
@@ -293,6 +299,8 @@ static void sdr_cap_stop_streaming(struct vb2_queue *vq)
 		buf = list_entry(dev->sdr_cap_active.next,
 				struct vivid_buffer, list);
 		list_del(&buf->list);
+		v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+					   &dev->ctrl_hdl_sdr_cap);
 		vb2_buffer_done(&buf->vb.vb2_buf, VB2_BUF_STATE_ERROR);
 	}
 
@@ -303,12 +311,20 @@ static void sdr_cap_stop_streaming(struct vb2_queue *vq)
 	mutex_lock(&dev->mutex);
 }
 
+static void sdr_cap_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vivid_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &dev->ctrl_hdl_sdr_cap);
+}
+
 const struct vb2_ops vivid_sdr_cap_qops = {
 	.queue_setup		= sdr_cap_queue_setup,
 	.buf_prepare		= sdr_cap_buf_prepare,
 	.buf_queue		= sdr_cap_buf_queue,
 	.start_streaming	= sdr_cap_start_streaming,
 	.stop_streaming		= sdr_cap_stop_streaming,
+	.buf_request_complete	= sdr_cap_buf_request_complete,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
 };
diff --git a/drivers/media/platform/vivid/vivid-vbi-cap.c b/drivers/media/platform/vivid/vivid-vbi-cap.c
index 92a852955173..903cebeb5ce5 100644
--- a/drivers/media/platform/vivid/vivid-vbi-cap.c
+++ b/drivers/media/platform/vivid/vivid-vbi-cap.c
@@ -204,6 +204,8 @@ static int vbi_cap_start_streaming(struct vb2_queue *vq, unsigned count)
 
 		list_for_each_entry_safe(buf, tmp, &dev->vbi_cap_active, list) {
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vbi_cap);
 			vb2_buffer_done(&buf->vb.vb2_buf,
 					VB2_BUF_STATE_QUEUED);
 		}
@@ -220,12 +222,20 @@ static void vbi_cap_stop_streaming(struct vb2_queue *vq)
 	vivid_stop_generating_vid_cap(dev, &dev->vbi_cap_streaming);
 }
 
+static void vbi_cap_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vivid_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &dev->ctrl_hdl_vbi_cap);
+}
+
 const struct vb2_ops vivid_vbi_cap_qops = {
 	.queue_setup		= vbi_cap_queue_setup,
 	.buf_prepare		= vbi_cap_buf_prepare,
 	.buf_queue		= vbi_cap_buf_queue,
 	.start_streaming	= vbi_cap_start_streaming,
 	.stop_streaming		= vbi_cap_stop_streaming,
+	.buf_request_complete	= vbi_cap_buf_request_complete,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
 };
diff --git a/drivers/media/platform/vivid/vivid-vbi-out.c b/drivers/media/platform/vivid/vivid-vbi-out.c
index 69486c130a7e..9357c07e30d6 100644
--- a/drivers/media/platform/vivid/vivid-vbi-out.c
+++ b/drivers/media/platform/vivid/vivid-vbi-out.c
@@ -96,6 +96,8 @@ static int vbi_out_start_streaming(struct vb2_queue *vq, unsigned count)
 
 		list_for_each_entry_safe(buf, tmp, &dev->vbi_out_active, list) {
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vbi_out);
 			vb2_buffer_done(&buf->vb.vb2_buf,
 					VB2_BUF_STATE_QUEUED);
 		}
@@ -115,12 +117,20 @@ static void vbi_out_stop_streaming(struct vb2_queue *vq)
 	dev->vbi_out_have_cc[1] = false;
 }
 
+static void vbi_out_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vivid_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &dev->ctrl_hdl_vbi_out);
+}
+
 const struct vb2_ops vivid_vbi_out_qops = {
 	.queue_setup		= vbi_out_queue_setup,
 	.buf_prepare		= vbi_out_buf_prepare,
 	.buf_queue		= vbi_out_buf_queue,
 	.start_streaming	= vbi_out_start_streaming,
 	.stop_streaming		= vbi_out_stop_streaming,
+	.buf_request_complete	= vbi_out_buf_request_complete,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
 };
diff --git a/drivers/media/platform/vivid/vivid-vid-cap.c b/drivers/media/platform/vivid/vivid-vid-cap.c
index 1599159f2574..b2aad441a071 100644
--- a/drivers/media/platform/vivid/vivid-vid-cap.c
+++ b/drivers/media/platform/vivid/vivid-vid-cap.c
@@ -240,6 +240,8 @@ static int vid_cap_start_streaming(struct vb2_queue *vq, unsigned count)
 
 		list_for_each_entry_safe(buf, tmp, &dev->vid_cap_active, list) {
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vid_cap);
 			vb2_buffer_done(&buf->vb.vb2_buf,
 					VB2_BUF_STATE_QUEUED);
 		}
@@ -257,6 +259,13 @@ static void vid_cap_stop_streaming(struct vb2_queue *vq)
 	dev->can_loop_video = false;
 }
 
+static void vid_cap_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vivid_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &dev->ctrl_hdl_vid_cap);
+}
+
 const struct vb2_ops vivid_vid_cap_qops = {
 	.queue_setup		= vid_cap_queue_setup,
 	.buf_prepare		= vid_cap_buf_prepare,
@@ -264,6 +273,7 @@ const struct vb2_ops vivid_vid_cap_qops = {
 	.buf_queue		= vid_cap_buf_queue,
 	.start_streaming	= vid_cap_start_streaming,
 	.stop_streaming		= vid_cap_stop_streaming,
+	.buf_request_complete	= vid_cap_buf_request_complete,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
 };
diff --git a/drivers/media/platform/vivid/vivid-vid-out.c b/drivers/media/platform/vivid/vivid-vid-out.c
index 51fec66d8d45..423a67133f28 100644
--- a/drivers/media/platform/vivid/vivid-vid-out.c
+++ b/drivers/media/platform/vivid/vivid-vid-out.c
@@ -162,6 +162,8 @@ static int vid_out_start_streaming(struct vb2_queue *vq, unsigned count)
 
 		list_for_each_entry_safe(buf, tmp, &dev->vid_out_active, list) {
 			list_del(&buf->list);
+			v4l2_ctrl_request_complete(buf->vb.vb2_buf.req_obj.req,
+						   &dev->ctrl_hdl_vid_out);
 			vb2_buffer_done(&buf->vb.vb2_buf,
 					VB2_BUF_STATE_QUEUED);
 		}
@@ -179,12 +181,20 @@ static void vid_out_stop_streaming(struct vb2_queue *vq)
 	dev->can_loop_video = false;
 }
 
+static void vid_out_buf_request_complete(struct vb2_buffer *vb)
+{
+	struct vivid_dev *dev = vb2_get_drv_priv(vb->vb2_queue);
+
+	v4l2_ctrl_request_complete(vb->req_obj.req, &dev->ctrl_hdl_vid_out);
+}
+
 const struct vb2_ops vivid_vid_out_qops = {
 	.queue_setup		= vid_out_queue_setup,
 	.buf_prepare		= vid_out_buf_prepare,
 	.buf_queue		= vid_out_buf_queue,
 	.start_streaming	= vid_out_start_streaming,
 	.stop_streaming		= vid_out_stop_streaming,
+	.buf_request_complete	= vid_out_buf_request_complete,
 	.wait_prepare		= vb2_ops_wait_prepare,
 	.wait_finish		= vb2_ops_wait_finish,
 };
-- 
2.17.0

[PATCHv13 28/28] RFC: media-requests: add debugfs node

[Hans Verkuil]

From: Hans Verkuil <hans.verkuil@cisco.com>

Keep track of the number of requests and request objects of a media
device. Helps to verify that all request-related memory is freed.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
---
 drivers/media/media-device.c  | 41 +++++++++++++++++++++++++++++++++++
 drivers/media/media-devnode.c | 17 +++++++++++++++
 drivers/media/media-request.c |  5 +++++
 include/media/media-device.h  | 11 ++++++++++
 include/media/media-devnode.h |  4 ++++
 include/media/media-request.h |  1 +
 6 files changed, 79 insertions(+)

diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index bb6a64acd3f0..e39731cc1add 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -680,6 +680,23 @@ void media_device_unregister_entity(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(media_device_unregister_entity);
 
+#ifdef CONFIG_DEBUG_FS
+/*
+ * Log the state of media requests.
+ * Very useful for debugging.
+ */
+static int media_device_requests(struct seq_file *file, void *priv)
+{
+	struct media_device *dev = dev_get_drvdata(file->private);
+
+	seq_printf(file, "number of requests: %d\n",
+		   atomic_read(&dev->num_requests));
+	seq_printf(file, "number of request objects: %d\n",
+		   atomic_read(&dev->num_request_objects));
+	return 0;
+}
+#endif
+
 /**
  * media_device_init() - initialize a media device
  * @mdev:	The media device
@@ -702,6 +719,9 @@ void media_device_init(struct media_device *mdev)
 	mutex_init(&mdev->graph_mutex);
 	ida_init(&mdev->entity_internal_idx);
 
+	atomic_set(&mdev->num_requests, 0);
+	atomic_set(&mdev->num_request_objects, 0);
+
 	dev_dbg(mdev->dev, "Media device initialized\n");
 }
 EXPORT_SYMBOL_GPL(media_device_init);
@@ -753,6 +773,26 @@ int __must_check __media_device_register(struct media_device *mdev,
 
 	dev_dbg(mdev->dev, "Media device registered\n");
 
+#ifdef CONFIG_DEBUG_FS
+	if (!media_top_dir)
+		return 0;
+
+	mdev->media_dir = debugfs_create_dir(dev_name(&devnode->dev),
+					     media_top_dir);
+	if (IS_ERR_OR_NULL(mdev->media_dir)) {
+		dev_warn(mdev->dev, "Failed to create debugfs dir\n");
+		return 0;
+	}
+	mdev->requests_file = debugfs_create_devm_seqfile(&devnode->dev,
+		"requests", mdev->media_dir, media_device_requests);
+	if (IS_ERR_OR_NULL(mdev->requests_file)) {
+		dev_warn(mdev->dev, "Failed to create requests file\n");
+		debugfs_remove_recursive(mdev->media_dir);
+		mdev->media_dir = NULL;
+		return 0;
+	}
+#endif
+
 	return 0;
 }
 EXPORT_SYMBOL_GPL(__media_device_register);
@@ -830,6 +870,7 @@ void media_device_unregister(struct media_device *mdev)
 
 	dev_dbg(mdev->dev, "Media device unregistered\n");
 
+	debugfs_remove_recursive(mdev->media_dir);
 	device_remove_file(&mdev->devnode->dev, &dev_attr_model);
 	media_devnode_unregister(mdev->devnode);
 	/* devnode free is handled in media_devnode_*() */
diff --git a/drivers/media/media-devnode.c b/drivers/media/media-devnode.c
index 67ac51eff15c..2656c2d7a312 100644
--- a/drivers/media/media-devnode.c
+++ b/drivers/media/media-devnode.c
@@ -53,6 +53,12 @@ static dev_t media_dev_t;
 static DEFINE_MUTEX(media_devnode_lock);
 static DECLARE_BITMAP(media_devnode_nums, MEDIA_NUM_DEVICES);
 
+/*
+ * debugfs
+ */
+struct dentry *media_top_dir;
+
+
 /* Called when the last user of the media device exits. */
 static void media_devnode_release(struct device *cd)
 {
@@ -259,6 +265,8 @@ int __must_check media_devnode_register(struct media_device *mdev,
 		goto cdev_add_error;
 	}
 
+	dev_set_drvdata(&devnode->dev, mdev);
+
 	/* Part 4: Activate this minor. The char device can now be used. */
 	set_bit(MEDIA_FLAG_REGISTERED, &devnode->flags);
 
@@ -310,6 +318,14 @@ static int __init media_devnode_init(void)
 		return ret;
 	}
 
+#ifdef CONFIG_DEBUG_FS
+	media_top_dir = debugfs_create_dir("media", NULL);
+	if (IS_ERR_OR_NULL(media_top_dir)) {
+		pr_warn("Failed to create debugfs media dir\n");
+		media_top_dir = NULL;
+	}
+#endif
+
 	ret = bus_register(&media_bus_type);
 	if (ret < 0) {
 		unregister_chrdev_region(media_dev_t, MEDIA_NUM_DEVICES);
@@ -322,6 +338,7 @@ static int __init media_devnode_init(void)
 
 static void __exit media_devnode_exit(void)
 {
+	debugfs_remove_recursive(media_top_dir);
 	bus_unregister(&media_bus_type);
 	unregister_chrdev_region(media_dev_t, MEDIA_NUM_DEVICES);
 }
diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
index c7e11e816e27..a24706d9e33e 100644
--- a/drivers/media/media-request.c
+++ b/drivers/media/media-request.c
@@ -63,6 +63,7 @@ static void media_request_release(struct kref *kref)
 		mdev->ops->req_free(req);
 	else
 		kfree(req);
+	atomic_dec(&mdev->num_requests);
 }
 
 void media_request_put(struct media_request *req)
@@ -296,6 +297,7 @@ int media_request_alloc(struct media_device *mdev,
 	get_task_comm(comm, current);
 	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
 		 comm, fd);
+	atomic_inc(&mdev->num_requests);
 	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
 
 	fd_install(fd, filp);
@@ -320,6 +322,7 @@ static void media_request_object_release(struct kref *kref)
 	if (req)
 		media_request_object_unbind(obj);
 	obj->ops->release(obj);
+	atomic_dec(&obj->mdev->num_request_objects);
 }
 
 struct media_request_object *
@@ -378,6 +381,7 @@ int media_request_object_bind(struct media_request *req,
 	obj->req = req;
 	obj->ops = ops;
 	obj->priv = priv;
+	obj->mdev = req->mdev;
 
 	spin_lock_irqsave(&req->lock, flags);
 
@@ -387,6 +391,7 @@ int media_request_object_bind(struct media_request *req,
 	list_add_tail(&obj->list, &req->objects);
 	req->num_incomplete_objects++;
 	ret = 0;
+	atomic_inc(&obj->mdev->num_request_objects);
 
 unlock:
 	spin_unlock_irqrestore(&req->lock, flags);
diff --git a/include/media/media-device.h b/include/media/media-device.h
index 7d855823341c..2e8e2d69f27f 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -21,6 +21,7 @@
 
 #include <linux/list.h>
 #include <linux/mutex.h>
+#include <linux/atomic.h>
 
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
@@ -102,6 +103,10 @@ struct media_device_ops {
  * @ops:	Operation handler callbacks
  * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
  *		     media device.
+ * @num_requests: number of associated requests
+ * @num_request_objects: number of associated request objects
+ * @media_dir:	DebugFS media directory
+ * @requests_file: DebugFS requests file
  *
  * This structure represents an abstract high-level media device. It allows easy
  * access to entities and provides basic media device-level support. The
@@ -174,6 +179,12 @@ struct media_device {
 	const struct media_device_ops *ops;
 
 	struct mutex req_queue_mutex;
+	atomic_t num_requests;
+	atomic_t num_request_objects;
+
+	/* debugfs */
+	struct dentry *media_dir;
+	struct dentry *requests_file;
 };
 
 /* We don't need to include pci.h or usb.h here */
diff --git a/include/media/media-devnode.h b/include/media/media-devnode.h
index dc2f64e1b08f..984b62b634d3 100644
--- a/include/media/media-devnode.h
+++ b/include/media/media-devnode.h
@@ -28,9 +28,13 @@
 #include <linux/fs.h>
 #include <linux/device.h>
 #include <linux/cdev.h>
+#include <linux/debugfs.h>
 
 struct media_device;
 
+/* DebugFS top-level media directory */
+extern struct dentry *media_top_dir;
+
 /*
  * Flag to mark the media_devnode struct as registered. Drivers must not touch
  * this flag directly, it will be set and cleared by media_devnode_register and
diff --git a/include/media/media-request.h b/include/media/media-request.h
index 5367b4a2f91c..2671b570c120 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -164,6 +164,7 @@ struct media_request_object_ops {
  * larger object data.
  */
 struct media_request_object {
+	struct media_device *mdev;
 	const struct media_request_object_ops *ops;
 	void *priv;
 	struct media_request *req;
-- 
2.17.0

Re: [PATCHv13 01/28] v4l2-device.h: always expose mdev

[Sakari Ailus]

On Thu, May 03, 2018 at 04:52:51PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
> But since we will need to pass the media_device to vb2 and the
> control framework it is very convenient to just make this field
> available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
> then it will just be NULL.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>

Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 02/28] uapi/linux/media.h: add request API

[Sakari Ailus]

On Thu, May 03, 2018 at 04:52:52PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Define the public request API.
> 
> This adds the new MEDIA_IOC_REQUEST_ALLOC ioctl to allocate a request
> and two ioctls that operate on a request in order to queue the
> contents of the request to the driver and to re-initialize the
> request.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>

Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 24/28] vim2m: use workqueue

[Sakari Ailus]

Hi Hans,

On Thu, May 03, 2018 at 04:53:14PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> v4l2_ctrl uses mutexes, so we can't setup a ctrl_handler in
> interrupt context. Switch to a workqueue instead.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/platform/vim2m.c | 15 +++++++++++++--
>  1 file changed, 13 insertions(+), 2 deletions(-)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index 9be4da3b8577..a1b0bb08668d 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -150,6 +150,7 @@ struct vim2m_dev {
>  	spinlock_t		irqlock;
>  
>  	struct timer_list	timer;
> +	struct work_struct	work_run;
>  
>  	struct v4l2_m2m_dev	*m2m_dev;
>  };
> @@ -392,9 +393,10 @@ static void device_run(void *priv)
>  	schedule_irq(dev, ctx->transtime);
>  }
>  
> -static void device_isr(struct timer_list *t)
> +static void device_work(struct work_struct *w)
>  {
> -	struct vim2m_dev *vim2m_dev = from_timer(vim2m_dev, t, timer);
> +	struct vim2m_dev *vim2m_dev =
> +		container_of(w, struct vim2m_dev, work_run);
>  	struct vim2m_ctx *curr_ctx;
>  	struct vb2_v4l2_buffer *src_vb, *dst_vb;
>  	unsigned long flags;
> @@ -426,6 +428,13 @@ static void device_isr(struct timer_list *t)
>  	}
>  }
>  
> +static void device_isr(struct timer_list *t)
> +{
> +	struct vim2m_dev *vim2m_dev = from_timer(vim2m_dev, t, timer);
> +
> +	schedule_work(&vim2m_dev->work_run);
> +}
> +
>  /*
>   * video ioctls
>   */
> @@ -806,6 +815,7 @@ static void vim2m_stop_streaming(struct vb2_queue *q)
>  	struct vb2_v4l2_buffer *vbuf;
>  	unsigned long flags;
>  
> +	flush_scheduled_work();
>  	for (;;) {
>  		if (V4L2_TYPE_IS_OUTPUT(q->type))
>  			vbuf = v4l2_m2m_src_buf_remove(ctx->fh.m2m_ctx);
> @@ -1011,6 +1021,7 @@ static int vim2m_probe(struct platform_device *pdev)
>  	vfd = &dev->vfd;
>  	vfd->lock = &dev->dev_mutex;
>  	vfd->v4l2_dev = &dev->v4l2_dev;
> +	INIT_WORK(&dev->work_run, device_work);
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  	dev->mdev.dev = &pdev->dev;

How about just removing the time and using schedule_delayed_work()
instead? That'd be quite a bit more simple.

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 04/28] media-request: add media_request_get_by_fd

[Sakari Ailus]

On Thu, May 03, 2018 at 04:52:54PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Add media_request_get_by_fd() to find a request based on the file
> descriptor.
> 
> The caller has to call media_request_put() for the returned
> request since this function increments the refcount.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>

Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 03/28] media-request: implement media requests

[Sakari Ailus]

Hi Hans,

I've read this patch a large number of times and I think also the details
begin to seem sound. A few comments below.

On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Add initial media request support:
> 
> 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> 2) Add struct media_request to store request objects.
> 3) Add struct media_request_object to represent a request object.
> 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> 
> Basic lifecycle: the application allocates a request, adds
> objects to it, queues the request, polls until it is completed
> and can then read the final values of the objects at the time
> of completion. When it closes the file descriptor the request
> memory will be freed (actually, when the last user of that request
> releases the request).
> 
> Drivers will bind an object to a request (the 'adds objects to it'
> phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> validated (req_validate op), then queued (the req_queue op).
> 
> When done with an object it can either be unbound from the request
> (e.g. when the driver has finished with a vb2 buffer) or marked as
> completed (e.g. for controls associated with a buffer). When all
> objects in the request are completed (or unbound), then the request
> fd will signal an exception (poll).
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/Makefile        |   3 +-
>  drivers/media/media-device.c  |  14 ++
>  drivers/media/media-request.c | 407 ++++++++++++++++++++++++++++++++++
>  include/media/media-device.h  |  16 ++
>  include/media/media-request.h | 244 ++++++++++++++++++++
>  5 files changed, 683 insertions(+), 1 deletion(-)
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 
> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> index 594b462ddf0e..985d35ec6b29 100644
> --- a/drivers/media/Makefile
> +++ b/drivers/media/Makefile
> @@ -3,7 +3,8 @@
>  # Makefile for the kernel multimedia device drivers.
>  #
>  
> -media-objs	:= media-device.o media-devnode.o media-entity.o
> +media-objs	:= media-device.o media-devnode.o media-entity.o \
> +		   media-request.o
>  
>  #
>  # I2C drivers should come before other drivers, otherwise they'll fail
> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> index 35e81f7c0d2f..bb6a64acd3f0 100644
> --- a/drivers/media/media-device.c
> +++ b/drivers/media/media-device.c
> @@ -32,6 +32,7 @@
>  #include <media/media-device.h>
>  #include <media/media-devnode.h>
>  #include <media/media-entity.h>
> +#include <media/media-request.h>
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  
> @@ -366,6 +367,15 @@ static long media_device_get_topology(struct media_device *mdev,
>  	return ret;
>  }
>  
> +static long media_device_request_alloc(struct media_device *mdev,
> +				       struct media_request_alloc *alloc)
> +{
> +	if (!mdev->ops || !mdev->ops->req_validate || !mdev->ops->req_queue)
> +		return -ENOTTY;
> +
> +	return media_request_alloc(mdev, alloc);
> +}
> +
>  static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
>  {
>  	/* All media IOCTLs are _IOWR() */
> @@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
> +	MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
>  };
>  
>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
> @@ -686,6 +697,8 @@ void media_device_init(struct media_device *mdev)
>  	INIT_LIST_HEAD(&mdev->pads);
>  	INIT_LIST_HEAD(&mdev->links);
>  	INIT_LIST_HEAD(&mdev->entity_notify);
> +
> +	mutex_init(&mdev->req_queue_mutex);
>  	mutex_init(&mdev->graph_mutex);
>  	ida_init(&mdev->entity_internal_idx);
>  
> @@ -699,6 +712,7 @@ void media_device_cleanup(struct media_device *mdev)
>  	mdev->entity_internal_idx_max = 0;
>  	media_graph_walk_cleanup(&mdev->pm_count_walk);
>  	mutex_destroy(&mdev->graph_mutex);
> +	mutex_destroy(&mdev->req_queue_mutex);
>  }
>  EXPORT_SYMBOL_GPL(media_device_cleanup);
>  
> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> new file mode 100644
> index 000000000000..c216c4ab628b
> --- /dev/null
> +++ b/drivers/media/media-request.c
> @@ -0,0 +1,407 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media device request objects
> + *
> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
> + * Copyright (C) 2018 Intel Corporation
> + * Copyright (C) 2018 Google, Inc.
> + *
> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
> + */
> +
> +#include <linux/anon_inodes.h>
> +#include <linux/file.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-request.h>
> +
> +static const char * const request_state[] = {
> +	[MEDIA_REQUEST_STATE_IDLE]	 = "idle",
> +	[MEDIA_REQUEST_STATE_VALIDATING] = "validating",
> +	[MEDIA_REQUEST_STATE_QUEUED]	 = "queued",
> +	[MEDIA_REQUEST_STATE_COMPLETE]	 = "complete",
> +	[MEDIA_REQUEST_STATE_CLEANING]	 = "cleaning",
> +};
> +
> +static const char *
> +media_request_state_str(enum media_request_state state)
> +{
> +	if (WARN_ON(state >= ARRAY_SIZE(request_state)))
> +		return "invalid";
> +	return request_state[state];
> +}
> +
> +static void media_request_clean(struct media_request *req)
> +{
> +	struct media_request_object *obj, *obj_safe;
> +
> +	WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_CLEANING);
> +
> +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
> +		media_request_object_unbind(obj);
> +		media_request_object_put(obj);
> +	}
> +
> +	req->num_incomplete_objects = 0;

The number of incomplete objects should be already zero here. I'd think
that a different number would suggest that something has gone very wrong
and should be complained about. How about adding
WARN_ON(req->num_incomplete_objects) above this line? 

> +	wake_up_interruptible_all(&req->poll_wait);
> +}
> +
> +static void media_request_release(struct kref *kref)
> +{
> +	struct media_request *req =
> +		container_of(kref, struct media_request, kref);
> +	struct media_device *mdev = req->mdev;
> +
> +	dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
> +
> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
> +
> +	media_request_clean(req);
> +
> +	if (mdev->ops->req_free)
> +		mdev->ops->req_free(req);
> +	else
> +		kfree(req);
> +}
> +
> +void media_request_put(struct media_request *req)
> +{
> +	kref_put(&req->kref, media_request_release);
> +}
> +EXPORT_SYMBOL_GPL(media_request_put);
> +
> +static int media_request_close(struct inode *inode, struct file *filp)
> +{
> +	struct media_request *req = filp->private_data;
> +
> +	media_request_put(req);
> +	return 0;
> +}
> +
> +static unsigned int media_request_poll(struct file *filp,
> +				       struct poll_table_struct *wait)
> +{
> +	struct media_request *req = filp->private_data;
> +	unsigned long flags;
> +	unsigned int ret = 0;
> +	enum media_request_state state;
> +
> +	if (!(poll_requested_events(wait) & POLLPRI))
> +		return 0;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	state = atomic_read(&req->state);
> +
> +	if (state == MEDIA_REQUEST_STATE_COMPLETE) {
> +		ret = POLLPRI;
> +		goto unlock;
> +	}
> +	if (state != MEDIA_REQUEST_STATE_QUEUED) {
> +		ret = POLLERR;
> +		goto unlock;
> +	}
> +
> +	poll_wait(filp, &req->poll_wait, wait);
> +
> +unlock:
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	return ret;
> +}
> +
> +static long media_request_ioctl_queue(struct media_request *req)
> +{
> +	struct media_device *mdev = req->mdev;
> +	enum media_request_state state;
> +	unsigned long flags;
> +	int ret = 0;

ret is unconditionally assigned below, no need to initialise here.

> +
> +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
> +
> +	/*
> +	 * Ensure the request that is validated will be the one that gets queued
> +	 * next by serialising the queueing process. This mutex is also used
> +	 * to serialize with canceling a vb2 queue and with setting values such
> +	 * as controls in a request.
> +	 */
> +	mutex_lock(&mdev->req_queue_mutex);
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
> +			       MEDIA_REQUEST_STATE_VALIDATING);
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	if (state != MEDIA_REQUEST_STATE_IDLE) {
> +		dev_dbg(mdev->dev,
> +			"request: unable to queue %s, request in state %s\n",
> +			req->debug_str, media_request_state_str(state));
> +		mutex_unlock(&mdev->req_queue_mutex);
> +		return -EBUSY;
> +	}
> +
> +	ret = mdev->ops->req_validate(req);
> +
> +	/*
> +	 * If the req_validate was successful, then we mark the state as QUEUED
> +	 * and call req_queue. The reason we set the state first is that this
> +	 * allows req_queue to unbind or complete the queued objects in case
> +	 * they are immediately 'consumed'. State changes from QUEUED to another
> +	 * state can only happen if either the driver changes the state or if
> +	 * the user cancels the vb2 queue. The driver can only change the state
> +	 * after each object is queued through the req_queue op (and note that
> +	 * that op cannot fail), so setting the state to QUEUED up front is
> +	 * safe.
> +	 *
> +	 * The other reason for changing the state is if the vb2 queue is
> +	 * canceled, and that uses the req_queue_mutex which is still locked
> +	 * while req_queue is called, so that's safe as well.
> +	 */
> +	atomic_set(&req->state,
> +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);
> +
> +	if (!ret)
> +		mdev->ops->req_queue(req);
> +
> +	mutex_unlock(&mdev->req_queue_mutex);
> +
> +	if (ret)
> +		dev_dbg(mdev->dev, "request: can't queue %s (%d)\n",
> +			req->debug_str, ret);
> +	else
> +		media_request_get(req);
> +
> +	return ret;
> +}
> +
> +static long media_request_ioctl_reinit(struct media_request *req)
> +{
> +	struct media_device *mdev = req->mdev;
> +	unsigned long flags;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE &&
> +	    atomic_read(&req->state) != MEDIA_REQUEST_STATE_COMPLETE) {
> +		dev_dbg(mdev->dev,
> +			"request: %s not in idle or complete state, cannot reinit\n",
> +			req->debug_str);
> +		spin_unlock_irqrestore(&req->lock, flags);
> +		return -EBUSY;
> +	}
> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
> +	spin_unlock_irqrestore(&req->lock, flags);
> +
> +	media_request_clean(req);
> +
> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
> +
> +	return 0;
> +}
> +
> +static long media_request_ioctl(struct file *filp, unsigned int cmd,
> +				unsigned long arg)
> +{
> +	struct media_request *req = filp->private_data;
> +
> +	switch (cmd) {
> +	case MEDIA_REQUEST_IOC_QUEUE:
> +		return media_request_ioctl_queue(req);
> +	case MEDIA_REQUEST_IOC_REINIT:
> +		return media_request_ioctl_reinit(req);
> +	default:
> +		return -ENOIOCTLCMD;
> +	}
> +}
> +
> +static const struct file_operations request_fops = {
> +	.owner = THIS_MODULE,
> +	.poll = media_request_poll,
> +	.unlocked_ioctl = media_request_ioctl,
> +	.release = media_request_close,
> +};
> +
> +int media_request_alloc(struct media_device *mdev,
> +			struct media_request_alloc *alloc)
> +{
> +	struct media_request *req;
> +	struct file *filp;
> +	char comm[TASK_COMM_LEN];
> +	int fd;
> +	int ret;
> +
> +	/* Either both are NULL or both are non-NULL */
> +	if (WARN_ON(!mdev->ops->req_alloc ^ !mdev->ops->req_free))
> +		return -ENOMEM;
> +
> +	fd = get_unused_fd_flags(O_CLOEXEC);
> +	if (fd < 0)
> +		return fd;
> +
> +	filp = anon_inode_getfile("request", &request_fops, NULL, O_CLOEXEC);
> +	if (IS_ERR(filp)) {
> +		ret = PTR_ERR(filp);
> +		goto err_put_fd;
> +	}
> +
> +	if (mdev->ops->req_alloc)
> +		req = mdev->ops->req_alloc(mdev);
> +	else
> +		req = kzalloc(sizeof(*req), GFP_KERNEL);
> +	if (!req) {
> +		ret = -ENOMEM;
> +		goto err_fput;
> +	}
> +
> +	filp->private_data = req;
> +	req->mdev = mdev;
> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
> +	req->num_incomplete_objects = 0;
> +	kref_init(&req->kref);
> +	INIT_LIST_HEAD(&req->objects);
> +	spin_lock_init(&req->lock);
> +	init_waitqueue_head(&req->poll_wait);
> +
> +	alloc->fd = fd;
> +
> +	get_task_comm(comm, current);
> +	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
> +		 comm, fd);
> +	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
> +
> +	fd_install(fd, filp);
> +
> +	return 0;
> +
> +err_fput:
> +	fput(filp);
> +
> +err_put_fd:
> +	put_unused_fd(fd);
> +
> +	return ret;
> +}
> +
> +static void media_request_object_release(struct kref *kref)
> +{
> +	struct media_request_object *obj =
> +		container_of(kref, struct media_request_object, kref);
> +	struct media_request *req = obj->req;
> +
> +	if (req)
> +		media_request_object_unbind(obj);
> +	obj->ops->release(obj);
> +}
> +
> +void media_request_object_put(struct media_request_object *obj)
> +{
> +	kref_put(&obj->kref, media_request_object_release);
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_put);
> +
> +void media_request_object_init(struct media_request_object *obj)
> +{
> +	obj->ops = NULL;
> +	obj->req = NULL;
> +	obj->priv = NULL;
> +	obj->completed = false;
> +	INIT_LIST_HEAD(&obj->list);
> +	kref_init(&obj->kref);
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_init);
> +
> +int media_request_object_bind(struct media_request *req,
> +			      const struct media_request_object_ops *ops,
> +			      void *priv,
> +			      struct media_request_object *obj)
> +{
> +	unsigned long flags;
> +	int ret = -EBUSY;
> +
> +	if (WARN_ON(!ops->release))
> +		return -EPERM;
> +
> +	obj->req = req;
> +	obj->ops = ops;
> +	obj->priv = priv;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +
> +	if (WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE))
> +		goto unlock;

Is this worth a kernel warning, or rather how the drivers / other framework
bits (e.g. VB2) prevent user from binding objects to non-idle requests?
Even if you added a similar check to the caller, the request state could
well change in the meantime.

Perhaps add __must_check to the return value?

> +
> +	list_add_tail(&obj->list, &req->objects);
> +	req->num_incomplete_objects++;
> +	ret = 0;
> +
> +unlock:
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_bind);
> +
> +void media_request_object_unbind(struct media_request_object *obj)
> +{
> +	struct media_request *req = obj->req;
> +	enum media_request_state state;
> +	unsigned long flags;
> +	bool completed = false;
> +
> +	if (WARN_ON(!req))
> +		return;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	list_del(&obj->list);
> +	obj->req = NULL;
> +
> +	state = atomic_read(&req->state);
> +
> +	if (state == MEDIA_REQUEST_STATE_COMPLETE ||
> +	    state == MEDIA_REQUEST_STATE_CLEANING)
> +		goto unlock;
> +
> +	if (WARN_ON(state == MEDIA_REQUEST_STATE_VALIDATING))
> +		goto unlock;
> +
> +	if (WARN_ON(!req->num_incomplete_objects))
> +		goto unlock;
> +
> +	req->num_incomplete_objects--;
> +	if (state == MEDIA_REQUEST_STATE_QUEUED &&
> +	    !req->num_incomplete_objects) {
> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
> +		completed = true;
> +		wake_up_interruptible_all(&req->poll_wait);
> +	}
> +
> +unlock:
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	if (obj->ops->unbind)
> +		obj->ops->unbind(obj);
> +	if (completed)
> +		media_request_put(req);
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_unbind);
> +
> +void media_request_object_complete(struct media_request_object *obj)
> +{
> +	struct media_request *req = obj->req;
> +	unsigned long flags;
> +	bool completed = false;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	if (obj->completed)
> +		goto unlock;
> +	obj->completed = true;
> +	if (WARN_ON(!req->num_incomplete_objects) ||
> +	    WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_QUEUED))
> +		goto unlock;
> +
> +	if (!--req->num_incomplete_objects) {
> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
> +		wake_up_interruptible_all(&req->poll_wait);
> +		completed = true;
> +	}
> +unlock:
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	if (completed)
> +		media_request_put(req);
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_complete);
> diff --git a/include/media/media-device.h b/include/media/media-device.h
> index bcc6ec434f1f..7d855823341c 100644
> --- a/include/media/media-device.h
> +++ b/include/media/media-device.h
> @@ -27,6 +27,7 @@
>  
>  struct ida;
>  struct device;
> +struct media_device;
>  
>  /**
>   * struct media_entity_notify - Media Entity Notify
> @@ -50,10 +51,21 @@ struct media_entity_notify {
>   * struct media_device_ops - Media device operations
>   * @link_notify: Link state change notification callback. This callback is
>   *		 called with the graph_mutex held.
> + * @req_alloc: Allocate a request
> + * @req_free: Free a request
> + * @req_validate: Validate a request, but do not queue yet
> + * @req_queue: Queue a validated request, cannot fail. If something goes
> + *	       wrong when queueing this request then it should be marked
> + *	       as such internally in the driver and any related buffers
> + *	       must eventually return to vb2 with state VB2_BUF_STATE_ERROR.
>   */
>  struct media_device_ops {
>  	int (*link_notify)(struct media_link *link, u32 flags,
>  			   unsigned int notification);
> +	struct media_request *(*req_alloc)(struct media_device *mdev);
> +	void (*req_free)(struct media_request *req);
> +	int (*req_validate)(struct media_request *req);
> +	void (*req_queue)(struct media_request *req);
>  };
>  
>  /**
> @@ -88,6 +100,8 @@ struct media_device_ops {
>   * @disable_source: Disable Source Handler function pointer
>   *
>   * @ops:	Operation handler callbacks
> + * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
> + *		     media device.
>   *
>   * This structure represents an abstract high-level media device. It allows easy
>   * access to entities and provides basic media device-level support. The
> @@ -158,6 +172,8 @@ struct media_device {
>  	void (*disable_source)(struct media_entity *entity);
>  
>  	const struct media_device_ops *ops;
> +
> +	struct mutex req_queue_mutex;
>  };
>  
>  /* We don't need to include pci.h or usb.h here */
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> new file mode 100644
> index 000000000000..e39122dfd717
> --- /dev/null
> +++ b/include/media/media-request.h
> @@ -0,0 +1,244 @@
> +// SPDX-License-Identifier: GPL-2.0-only
> +/*
> + * Media device request objects
> + *
> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
> + * Copyright (C) 2018 Intel Corporation
> + *
> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
> + */
> +
> +#ifndef MEDIA_REQUEST_H
> +#define MEDIA_REQUEST_H
> +
> +#include <linux/list.h>
> +#include <linux/slab.h>
> +#include <linux/spinlock.h>
> +#include <linux/atomic.h>
> +
> +#include <media/media-device.h>
> +
> +/**
> + * enum media_request_state - media request state
> + *
> + * @MEDIA_REQUEST_STATE_IDLE:		Idle
> + * @MEDIA_REQUEST_STATE_VALIDATING:	Validating the request, no state changes
> + *					allowed
> + * @MEDIA_REQUEST_STATE_QUEUED:		Queued
> + * @MEDIA_REQUEST_STATE_COMPLETE:	Completed, the request is done
> + * @MEDIA_REQUEST_STATE_CLEANING:	Cleaning, the request is being re-inited
> + */
> +enum media_request_state {
> +	MEDIA_REQUEST_STATE_IDLE,
> +	MEDIA_REQUEST_STATE_VALIDATING,
> +	MEDIA_REQUEST_STATE_QUEUED,
> +	MEDIA_REQUEST_STATE_COMPLETE,
> +	MEDIA_REQUEST_STATE_CLEANING,
> +};
> +
> +struct media_request_object;
> +
> +/**
> + * struct media_request - Media device request
> + * @mdev: Media device this request belongs to
> + * @kref: Reference count
> + * @debug_str: Prefix for debug messages (process name:fd)
> + * @state: The state of the request
> + * @objects: List of @struct media_request_object request objects
> + * @num_objects: The number of objects in the request
> + * @num_incompleted_objects: The number of incomplete objects in the request
> + * @poll_wait: Wait queue for poll
> + * @lock: Serializes access to this struct
> + */
> +struct media_request {
> +	struct media_device *mdev;
> +	struct kref kref;
> +	char debug_str[TASK_COMM_LEN + 11];
> +	atomic_t state;
> +	struct list_head objects;
> +	unsigned int num_incomplete_objects;
> +	struct wait_queue_head poll_wait;
> +	spinlock_t lock;
> +};
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +
> +/**
> + * media_request_get - Get the media request
> + *
> + * @req: The request
> + *
> + * Get the media request.
> + */
> +static inline void media_request_get(struct media_request *req)
> +{
> +	kref_get(&req->kref);
> +}
> +
> +/**
> + * media_request_put - Put the media request
> + *
> + * @req: The request
> + *
> + * Put the media request. The media request will be released
> + * when the refcount reaches 0.
> + */
> +void media_request_put(struct media_request *req);
> +
> +/**
> + * media_request_alloc - Allocate the media request
> + *
> + * @mdev: Media device this request belongs to
> + * @alloc: Store the request's file descriptor in this struct
> + *
> + * Allocated the media request and put the fd in @alloc->fd.
> + */
> +int media_request_alloc(struct media_device *mdev,
> +			struct media_request_alloc *alloc);
> +
> +#else
> +
> +static inline void media_request_get(struct media_request *req)
> +{
> +}
> +
> +static inline void media_request_put(struct media_request *req)
> +{
> +}
> +
> +#endif
> +
> +/**
> + * struct media_request_object_ops - Media request object operations
> + * @prepare: Validate and prepare the request object, optional.
> + * @unprepare: Unprepare the request object, optional.
> + * @queue: Queue the request object, optional.
> + * @unbind: Unbind the request object, optional.
> + * @release: Release the request object, required.
> + */
> +struct media_request_object_ops {
> +	int (*prepare)(struct media_request_object *object);
> +	void (*unprepare)(struct media_request_object *object);
> +	void (*queue)(struct media_request_object *object);
> +	void (*unbind)(struct media_request_object *object);
> +	void (*release)(struct media_request_object *object);
> +};
> +
> +/**
> + * struct media_request_object - An opaque object that belongs to a media
> + *				 request
> + *
> + * @ops: object's operations
> + * @priv: object's priv pointer
> + * @req: the request this object belongs to (can be NULL)
> + * @list: List entry of the object for @struct media_request
> + * @kref: Reference count of the object, acquire before releasing req->lock
> + * @completed: If true, then this object was completed.
> + *
> + * An object related to the request. This struct is embedded in the
> + * larger object data.
> + */
> +struct media_request_object {
> +	const struct media_request_object_ops *ops;
> +	void *priv;
> +	struct media_request *req;
> +	struct list_head list;
> +	struct kref kref;
> +	bool completed;
> +};
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +
> +/**
> + * media_request_object_get - Get a media request object
> + *
> + * @obj: The object
> + *
> + * Get a media request object.
> + */
> +static inline void media_request_object_get(struct media_request_object *obj)
> +{
> +	kref_get(&obj->kref);
> +}
> +
> +/**
> + * media_request_object_put - Put a media request object
> + *
> + * @obj: The object
> + *
> + * Put a media request object. Once all references are gone, the
> + * object's memory is released.
> + */
> +void media_request_object_put(struct media_request_object *obj);
> +
> +/**
> + * media_request_object_init - Initialise a media request object
> + *
> + * Initialise a media request object. The object will be released using the
> + * release callback of the ops once it has no references (this function
> + * initialises references to one).
> + */
> +void media_request_object_init(struct media_request_object *obj);
> +
> +/**
> + * media_request_object_bind - Bind a media request object to a request

Argument documentation is missing.

I think you should also say that "every bound object must be unbound later
on".

> + */
> +int media_request_object_bind(struct media_request *req,
> +			      const struct media_request_object_ops *ops,
> +			      void *priv,
> +			      struct media_request_object *obj);
> +
> +/**
> + * media_request_object_unbind - Unbind a media request object
> + *
> + * @obj: The object
> + *
> + * Unbind the media request object from the request.
> + */
> +void media_request_object_unbind(struct media_request_object *obj);
> +
> +/**
> + * media_request_object_complete - Mark the media request object as complete
> + *
> + * @obj: The object
> + *
> + * Mark the media request object as complete.

Add:

Only bound request objects may be completed.

> + */
> +void media_request_object_complete(struct media_request_object *obj);
> +
> +#else
> +
> +static inline void media_request_object_get(struct media_request_object *obj)
> +{
> +}
> +
> +static inline void media_request_object_put(struct media_request_object *obj)
> +{
> +}
> +
> +static inline void media_request_object_init(struct media_request_object *obj)
> +{
> +	obj->ops = NULL;
> +	obj->req = NULL;
> +}
> +
> +static inline int media_request_object_bind(struct media_request *req,
> +			       const struct media_request_object_ops *ops,
> +			       void *priv,
> +			       struct media_request_object *obj)
> +{
> +	return 0;
> +}
> +
> +static inline void media_request_object_unbind(struct media_request_object *obj)
> +{
> +}
> +
> +static inline void media_request_object_complete(struct media_request_object *obj)
> +{
> +}
> +
> +#endif
> +
> +#endif

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 05/28] media-request: add media_request_object_find

[Sakari Ailus]

On Thu, May 03, 2018 at 04:52:55PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Add media_request_object_find to find a request object inside a
> request based on ops and/or priv values.
> 
> Objects of the same type (vb2 buffer, control handler) will have
> the same ops value. And objects that refer to the same 'parent'
> object (e.g. the v4l2_ctrl_handler that has the current driver
> state) will have the same priv value.
> 
> The caller has to call media_request_object_put() for the returned
> object since this function increments the refcount.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/media-request.c | 25 +++++++++++++++++++++++++
>  include/media/media-request.h | 24 ++++++++++++++++++++++++
>  2 files changed, 49 insertions(+)
> 
> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> index edc1c3af1959..c7e11e816e27 100644
> --- a/drivers/media/media-request.c
> +++ b/drivers/media/media-request.c
> @@ -322,6 +322,31 @@ static void media_request_object_release(struct kref *kref)
>  	obj->ops->release(obj);
>  }
>  
> +struct media_request_object *
> +media_request_object_find(struct media_request *req,
> +			  const struct media_request_object_ops *ops,
> +			  void *priv)
> +{
> +	struct media_request_object *obj;
> +	struct media_request_object *found = NULL;
> +	unsigned long flags;
> +
> +	if (WARN_ON(!ops || !priv))
> +		return NULL;
> +
> +	spin_lock_irqsave(&req->lock, flags);
> +	list_for_each_entry(obj, &req->objects, list) {
> +		if (obj->ops == ops && obj->priv == priv) {
> +			media_request_object_get(obj);
> +			found = obj;
> +			break;
> +		}
> +	}
> +	spin_unlock_irqrestore(&req->lock, flags);
> +	return found;
> +}
> +EXPORT_SYMBOL_GPL(media_request_object_find);
> +
>  void media_request_object_put(struct media_request_object *obj)
>  {
>  	kref_put(&obj->kref, media_request_object_release);
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> index 997e096d7128..5367b4a2f91c 100644
> --- a/include/media/media-request.h
> +++ b/include/media/media-request.h
> @@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
>   */
>  void media_request_object_put(struct media_request_object *obj);
>  
> +/**
> + * media_request_object_find - Find an object in a request
> + *
> + * @ops: Find an object with this ops value
> + * @priv: Find an object with this priv value
> + *
> + * Both @ops and @priv must be non-NULL.
> + *
> + * Returns NULL if not found or the object pointer. The caller must

I'd describe the successful case first. I.e. "Returns the object pointer or
NULL it not found".

Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>

> + * call media_request_object_put() once it finished using the object.
> + */
> +struct media_request_object *
> +media_request_object_find(struct media_request *req,
> +			  const struct media_request_object_ops *ops,
> +			  void *priv);
> +
>  /**
>   * media_request_object_init - Initialise a media request object
>   *
> @@ -241,6 +257,14 @@ static inline void media_request_object_put(struct media_request_object *obj)
>  {
>  }
>  
> +static inline struct media_request_object *
> +media_request_object_find(struct media_request *req,
> +			  const struct media_request_object_ops *ops,
> +			  void *priv)
> +{
> +	return NULL;
> +}
> +
>  static inline void media_request_object_init(struct media_request_object *obj)
>  {
>  	obj->ops = NULL;

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 01/28] v4l2-device.h: always expose mdev

[Mauro Carvalho Chehab]

Em Fri, 4 May 2018 13:51:28 +0300
Sakari Ailus <sakari.ailus@iki.fi> escreveu:

> On Thu, May 03, 2018 at 04:52:51PM +0200, Hans Verkuil wrote:
> > From: Hans Verkuil <hans.verkuil@cisco.com>
> > 
> > The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
> > But since we will need to pass the media_device to vb2 and the
> > control framework it is very convenient to just make this field
> > available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
> > then it will just be NULL.
> > 
> > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>  
> 
> Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 

This patch is no-brainer. It could be sent no matter what. However,
the patch is too simple :-)

There are a number of places where if CONFIG_MEDIA_CONTROLLER
(and for CONFIG_MEDIA_CONTROLLER_DVB - with is also optionally 
added at DVB core) is tested just because the field may or may
not be there.

If we're willing to always have it at the struct, then we should look
on all #ifs for CONFIG_MEDIA_CONTROLLER and get rid of most (or all)
of them, ensuring that function stubs will be enough for the code
itself to do the right thing if !CONFIG_MEDIA_CONTROLLER.

Thanks,
Mauro

Re: [PATCHv13 04/28] media-request: add media_request_get_by_fd

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:52:54 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Add media_request_get_by_fd() to find a request based on the file
> descriptor.
> 
> The caller has to call media_request_put() for the returned
> request since this function increments the refcount.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/media-request.c | 32 ++++++++++++++++++++++++++++++++
>  include/media/media-request.h | 24 ++++++++++++++++++++++++
>  2 files changed, 56 insertions(+)
> 
> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> index c216c4ab628b..edc1c3af1959 100644
> --- a/drivers/media/media-request.c
> +++ b/drivers/media/media-request.c
> @@ -218,6 +218,38 @@ static const struct file_operations request_fops = {
>  	.release = media_request_close,
>  };
>  
> +struct media_request *
> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
> +{
> +	struct file *filp;
> +	struct media_request *req;
> +
> +	if (!mdev || !mdev->ops ||
> +	    !mdev->ops->req_validate || !mdev->ops->req_queue)
> +		return ERR_PTR(-EPERM);
> +
> +	filp = fget(request_fd);
> +	if (!filp)
> +		return ERR_PTR(-ENOENT);
> +
> +	if (filp->f_op != &request_fops)
> +		goto err_fput;
> +	req = filp->private_data;
> +	if (req->mdev != mdev)
> +		goto err_fput;
> +
> +	media_request_get(req);

Hmm... this function changes the req struct (by calling kref_get) without
holding neither the mutex or the spin lock...

> +	fput(filp);
> +
> +	return req;
> +
> +err_fput:
> +	fput(filp);
> +
> +	return ERR_PTR(-ENOENT);
> +}
> +EXPORT_SYMBOL_GPL(media_request_get_by_fd);
> +
>  int media_request_alloc(struct media_device *mdev,
>  			struct media_request_alloc *alloc)
>  {
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> index e39122dfd717..997e096d7128 100644
> --- a/include/media/media-request.h
> +++ b/include/media/media-request.h
> @@ -86,6 +86,24 @@ static inline void media_request_get(struct media_request *req)
>   */
>  void media_request_put(struct media_request *req);
>  
> +/**
> + * media_request_get_by_fd - Get a media request by fd
> + *
> + * @mdev: Media device this request belongs to
> + * @request_fd: The file descriptor of the request
> + *
> + * Get the request represented by @request_fd that is owned
> + * by the media device.
> + *
> + * Return a -EPERM error pointer if requests are not supported
> + * by this driver. Return -ENOENT if the request was not found.
> + * Return the pointer to the request if found: the caller will
> + * have to call @media_request_put when it finished using the
> + * request.

... so, it should be said here how this should be serialized, in order
to avoid it to be destroyed by a task while some other task might be
trying to instantiate it.

> + */
> +struct media_request *
> +media_request_get_by_fd(struct media_device *mdev, int request_fd);
> +
>  /**
>   * media_request_alloc - Allocate the media request
>   *
> @@ -107,6 +125,12 @@ static inline void media_request_put(struct media_request *req)
>  {
>  }
>  
> +static inline struct media_request *
> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
> +{
> +	return ERR_PTR(-EPERM);
> +}
> +
>  #endif
>  
>  /**



Thanks,
Mauro

Re: [PATCHv13 05/28] media-request: add media_request_object_find

[Mauro Carvalho Chehab]

Em Fri, 4 May 2018 15:43:07 +0300
Sakari Ailus <sakari.ailus@iki.fi> escreveu:

> On Thu, May 03, 2018 at 04:52:55PM +0200, Hans Verkuil wrote:
> > From: Hans Verkuil <hans.verkuil@cisco.com>
> > 
> > Add media_request_object_find to find a request object inside a
> > request based on ops and/or priv values.
> > 
> > Objects of the same type (vb2 buffer, control handler) will have
> > the same ops value. And objects that refer to the same 'parent'
> > object (e.g. the v4l2_ctrl_handler that has the current driver
> > state) will have the same priv value.
> > 
> > The caller has to call media_request_object_put() for the returned
> > object since this function increments the refcount.
> > 
> > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> > ---
> >  drivers/media/media-request.c | 25 +++++++++++++++++++++++++
> >  include/media/media-request.h | 24 ++++++++++++++++++++++++
> >  2 files changed, 49 insertions(+)
> > 
> > diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> > index edc1c3af1959..c7e11e816e27 100644
> > --- a/drivers/media/media-request.c
> > +++ b/drivers/media/media-request.c
> > @@ -322,6 +322,31 @@ static void media_request_object_release(struct kref *kref)
> >  	obj->ops->release(obj);
> >  }
> >  
> > +struct media_request_object *
> > +media_request_object_find(struct media_request *req,
> > +			  const struct media_request_object_ops *ops,
> > +			  void *priv)
> > +{
> > +	struct media_request_object *obj;
> > +	struct media_request_object *found = NULL;
> > +	unsigned long flags;
> > +
> > +	if (WARN_ON(!ops || !priv))
> > +		return NULL;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	list_for_each_entry(obj, &req->objects, list) {
> > +		if (obj->ops == ops && obj->priv == priv) {
> > +			media_request_object_get(obj);
> > +			found = obj;
> > +			break;
> > +		}
> > +	}
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +	return found;
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_find);
> > +
> >  void media_request_object_put(struct media_request_object *obj)
> >  {
> >  	kref_put(&obj->kref, media_request_object_release);
> > diff --git a/include/media/media-request.h b/include/media/media-request.h
> > index 997e096d7128..5367b4a2f91c 100644
> > --- a/include/media/media-request.h
> > +++ b/include/media/media-request.h
> > @@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
> >   */
> >  void media_request_object_put(struct media_request_object *obj);
> >  
> > +/**
> > + * media_request_object_find - Find an object in a request
> > + *
> > + * @ops: Find an object with this ops value
> > + * @priv: Find an object with this priv value
> > + *
> > + * Both @ops and @priv must be non-NULL.
> > + *
> > + * Returns NULL if not found or the object pointer. The caller must  
> 
> I'd describe the successful case first. I.e. "Returns the object pointer or
> NULL it not found".

It would be good to also tell that this routine internally uses the
spin lock.

> 
> Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> > + * call media_request_object_put() once it finished using the object.
> > + */
> > +struct media_request_object *
> > +media_request_object_find(struct media_request *req,
> > +			  const struct media_request_object_ops *ops,
> > +			  void *priv);
> > +
> >  /**
> >   * media_request_object_init - Initialise a media request object
> >   *
> > @@ -241,6 +257,14 @@ static inline void media_request_object_put(struct media_request_object *obj)
> >  {
> >  }
> >  
> > +static inline struct media_request_object *
> > +media_request_object_find(struct media_request *req,
> > +			  const struct media_request_object_ops *ops,
> > +			  void *priv)
> > +{
> > +	return NULL;
> > +}
> > +
> >  static inline void media_request_object_init(struct media_request_object *obj)
> >  {
> >  	obj->ops = NULL;  
> 



Thanks,
Mauro

Re: [PATCHv13 06/28] v4l2-dev: lock req_queue_mutex

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:52:56 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> We need to serialize streamon/off with queueing new requests.
> These ioctls may trigger the cancellation of a streaming
> operation, and that should not be mixed with queuing a new
> request at the same time.
> 
> Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
> with MEDIA_REQUEST_IOC_QUEUE.
> 
> Finally close() needs this lock since that too can trigger the
> cancellation of a streaming operation.
> 
> We take the req_queue_mutex here before any other locks since
> it is a very high-level lock.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/v4l2-core/v4l2-dev.c | 37 +++++++++++++++++++++++++++++-
>  1 file changed, 36 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index 1d0b2208e8fb..b1c9efc0ecc4 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -353,13 +353,36 @@ static long v4l2_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>  
>  	if (vdev->fops->unlocked_ioctl) {
>  		struct mutex *lock = v4l2_ioctl_get_lock(vdev, cmd);
> +		struct mutex *queue_lock = NULL;
>  
> -		if (lock && mutex_lock_interruptible(lock))
> +		/*
> +		 * We need to serialize streamon/off with queueing new requests.
> +		 * These ioctls may trigger the cancellation of a streaming
> +		 * operation, and that should not be mixed with queueing a new
> +		 * request at the same time.
> +		 *
> +		 * Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
> +		 * with MEDIA_REQUEST_IOC_QUEUE.
> +		 */
> +		if (vdev->v4l2_dev->mdev &&
> +		    (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF ||
> +		     cmd == VIDIOC_S_EXT_CTRLS || cmd == VIDIOC_TRY_EXT_CTRLS))
> +			queue_lock = &vdev->v4l2_dev->mdev->req_queue_mutex;
> +
> +		if (queue_lock && mutex_lock_interruptible(queue_lock))
> +			return -ERESTARTSYS;

Taking both locks seems risky. Here you're taking first v4l2 lock, returned
by v4l2_ioctl_get_lock(vdev, cmd), and then you're taking the req_queue lock.

It is possible to call parts of the code that only handles req_queue
or v4l2 lock (for example, by mixing request API calls with non-requests
one). Worse than that, there are parts of the code where the request API
patches get both a mutex and a spin lock.

I didn't look too closely (nor ran any test), but I'm almost sure that
there are paths where it will end by leading into dead locks.

> +
> +		if (lock && mutex_lock_interruptible(lock)) {
> +			if (queue_lock)
> +				mutex_unlock(queue_lock);
>  			return -ERESTARTSYS;
> +		}
>  		if (video_is_registered(vdev))
>  			ret = vdev->fops->unlocked_ioctl(filp, cmd, arg);
>  		if (lock)
>  			mutex_unlock(lock);
> +		if (queue_lock)
> +			mutex_unlock(queue_lock);
>  	} else
>  		ret = -ENOTTY;
>  
> @@ -442,8 +465,20 @@ static int v4l2_release(struct inode *inode, struct file *filp)
>  	struct video_device *vdev = video_devdata(filp);
>  	int ret = 0;
>  
> +	/*
> +	 * We need to serialize the release() with queueing new requests.
> +	 * The release() may trigger the cancellation of a streaming
> +	 * operation, and that should not be mixed with queueing a new
> +	 * request at the same time.
> +	 */
> +	if (vdev->v4l2_dev->mdev)
> +		mutex_lock(&vdev->v4l2_dev->mdev->req_queue_mutex);
> +
>  	if (vdev->fops->release)
>  		ret = vdev->fops->release(filp);
> +
> +	if (vdev->v4l2_dev->mdev)
> +		mutex_unlock(&vdev->v4l2_dev->mdev->req_queue_mutex);
>  	if (vdev->dev_debug & V4L2_DEV_DEBUG_FOP)
>  		printk(KERN_DEBUG "%s: release\n",
>  			video_device_node_name(vdev));



Thanks,
Mauro

Re: [PATCHv13 09/28] v4l2-ctrls: prepare internal structs for request API

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:52:59 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Embed and initialize a media_request_object in struct v4l2_ctrl_handler.
> 
> Add a p_req field to struct v4l2_ctrl_ref that will store the
> request value.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
>  include/media/v4l2-ctrls.h           | 7 +++++++
>  2 files changed, 8 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> index aa1dd2015e84..d09f49530d9e 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -1880,6 +1880,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
>  				      sizeof(hdl->buckets[0]),
>  				      GFP_KERNEL | __GFP_ZERO);
>  	hdl->error = hdl->buckets ? 0 : -ENOMEM;
> +	media_request_object_init(&hdl->req_obj);
>  	return hdl->error;
>  }
>  EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> index d26b8ddebb56..76352eb59f14 100644
> --- a/include/media/v4l2-ctrls.h
> +++ b/include/media/v4l2-ctrls.h
> @@ -20,6 +20,7 @@
>  #include <linux/list.h>
>  #include <linux/mutex.h>
>  #include <linux/videodev2.h>
> +#include <media/media-request.h>
>  
>  /* forward references */
>  struct file;
> @@ -249,6 +250,8 @@ struct v4l2_ctrl {
>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>   * @from_other_dev: If true, then @ctrl was defined in another
>   *		device than the &struct v4l2_ctrl_handler.
> + * @p_req:	The request value. Only used if the control handler
> + *		is bound to a media request.

Could you please better elaborate the description of this field?

I read this patch dozen times to understand what you meant by
"request value", as I would be expecting here a "media_request_"
object or something similar. Instead, what you meant to say is that
@p_req will be used to cache the data passed via a request API call,
while the request is not handled yet, right?


>   *
>   * Each control handler has a list of these refs. The list_head is used to
>   * keep a sorted-by-control-ID list of all controls, while the next pointer
> @@ -260,6 +263,7 @@ struct v4l2_ctrl_ref {
>  	struct v4l2_ctrl *ctrl;
>  	struct v4l2_ctrl_helper *helper;
>  	bool from_other_dev;
> +	union v4l2_ctrl_ptr p_req;
>  };
>  
>  /**
> @@ -283,6 +287,8 @@ struct v4l2_ctrl_ref {
>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>   * @nr_of_buckets: Total number of buckets in the array.
>   * @error:	The error code of the first failed control addition.
> + * @req_obj:	The &struct media_request_object, used to link into a
> + *		&struct media_request.

I would document that there is kref is inside @req_obj, as we can't
add another kref here later.

>   */
>  struct v4l2_ctrl_handler {
>  	struct mutex _lock;
> @@ -295,6 +301,7 @@ struct v4l2_ctrl_handler {
>  	void *notify_priv;
>  	u16 nr_of_buckets;
>  	int error;
> +	struct media_request_object req_obj;
>  };
>  
>  /**



Thanks,
Mauro

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:53:02 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Integrate the request support. This adds the v4l2_ctrl_request_complete
> and v4l2_ctrl_request_setup functions to complete a request and (as a
> helper function) to apply a request to the hardware.
> 
> It takes care of queuing requests and correctly chaining control values
> in the request queue.
> 
> Note that when a request is marked completed it will copy control values
> to the internal request state. This can be optimized in the future since
> this is sub-optimal when dealing with large compound and/or array controls.
> 
> For the initial 'stateless codec' use-case the current implementation is
> sufficient.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
>  include/media/v4l2-ctrls.h           |  23 ++
>  2 files changed, 348 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> index da4cc1485dc4..56b986185463 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -1647,6 +1647,13 @@ static int new_to_user(struct v4l2_ext_control *c,
>  	return ptr_to_user(c, ctrl, ctrl->p_new);
>  }
>  
> +/* Helper function: copy the request value back to the caller */
> +static int req_to_user(struct v4l2_ext_control *c,
> +		       struct v4l2_ctrl_ref *ref)
> +{
> +	return ptr_to_user(c, ref->ctrl, ref->p_req);
> +}
> +
>  /* Helper function: copy the initial control value back to the caller */
>  static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
>  {
> @@ -1766,6 +1773,26 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
>  	ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
>  }
>  
> +/* Copy the new value to the request value */
> +static void new_to_req(struct v4l2_ctrl_ref *ref)
> +{
> +	if (!ref)
> +		return;
> +	ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
> +	ref->req = ref;
> +}
> +
> +/* Copy the request value to the new value */
> +static void req_to_new(struct v4l2_ctrl_ref *ref)
> +{
> +	if (!ref)
> +		return;
> +	if (ref->req)
> +		ptr_to_ptr(ref->ctrl, ref->req->p_req, ref->ctrl->p_new);
> +	else
> +		ptr_to_ptr(ref->ctrl, ref->ctrl->p_cur, ref->ctrl->p_new);
> +}
> +
>  /* Return non-zero if one or more of the controls in the cluster has a new
>     value that differs from the current value. */
>  static int cluster_changed(struct v4l2_ctrl *master)
> @@ -1875,6 +1902,9 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
>  	lockdep_set_class_and_name(hdl->lock, key, name);
>  	INIT_LIST_HEAD(&hdl->ctrls);
>  	INIT_LIST_HEAD(&hdl->ctrl_refs);
> +	INIT_LIST_HEAD(&hdl->requests);
> +	INIT_LIST_HEAD(&hdl->requests_queued);
> +	hdl->request_is_queued = false;
>  	hdl->nr_of_buckets = 1 + nr_of_controls_hint / 8;
>  	hdl->buckets = kvmalloc_array(hdl->nr_of_buckets,
>  				      sizeof(hdl->buckets[0]),
> @@ -1895,6 +1925,14 @@ void v4l2_ctrl_handler_free(struct v4l2_ctrl_handler *hdl)
>  	if (hdl == NULL || hdl->buckets == NULL)
>  		return;
>  
> +	if (!hdl->req_obj.req && !list_empty(&hdl->requests)) {
> +		struct v4l2_ctrl_handler *req, *next_req;
> +
> +		list_for_each_entry_safe(req, next_req, &hdl->requests, requests) {
> +			media_request_object_unbind(&req->req_obj);
> +			media_request_object_put(&req->req_obj);
> +		}
> +	}
>  	mutex_lock(hdl->lock);
>  	/* Free all nodes */
>  	list_for_each_entry_safe(ref, next_ref, &hdl->ctrl_refs, node) {
> @@ -2816,6 +2854,128 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct v4l2_querymenu *qm)
>  }
>  EXPORT_SYMBOL(v4l2_querymenu);
>  
> +static int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
> +				   const struct v4l2_ctrl_handler *from)
> +{
> +	struct v4l2_ctrl_ref *ref;
> +	int err;
> +
> +	if (WARN_ON(!hdl || hdl == from))
> +		return -EINVAL;
> +
> +	if (hdl->error)
> +		return hdl->error;
> +
> +	WARN_ON(hdl->lock != &hdl->_lock);
> +
> +	mutex_lock(from->lock);
> +	list_for_each_entry(ref, &from->ctrl_refs, node) {
> +		struct v4l2_ctrl *ctrl = ref->ctrl;
> +		struct v4l2_ctrl_ref *new_ref;
> +
> +		/* Skip refs inherited from other devices */
> +		if (ref->from_other_dev)
> +			continue;
> +		/* And buttons */
> +		if (ctrl->type == V4L2_CTRL_TYPE_BUTTON)
> +			continue;
> +		err = handler_new_ref(hdl, ctrl, &new_ref, false, true);
> +		if (err) {
> +			printk("%s: handler_new_ref on control %x (%s) returned %d\n", __func__, ctrl->id, ctrl->name, err);
> +			err = 0;
> +			continue;
> +		}
> +		if (err)
> +			break;
> +	}
> +	mutex_unlock(from->lock);
> +	return err;
> +}
> +
> +static void v4l2_ctrl_request_queue(struct media_request_object *obj)
> +{
> +	struct v4l2_ctrl_handler *hdl =
> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +	struct v4l2_ctrl_handler *main_hdl = obj->priv;
> +	struct v4l2_ctrl_handler *prev_hdl = NULL;
> +	struct v4l2_ctrl_ref *ref_ctrl, *ref_ctrl_prev = NULL;
> +
> +	if (list_empty(&main_hdl->requests_queued))
> +		goto queue;
> +
> +	prev_hdl = list_last_entry(&main_hdl->requests_queued,
> +				   struct v4l2_ctrl_handler, requests_queued);
> +	/*
> +	 * Note: prev_hdl and hdl must contain the same list of control
> +	 * references, so if any differences are detected then that is a
> +	 * driver bug and the WARN_ON is triggered.
> +	 */
> +	mutex_lock(prev_hdl->lock);
> +	ref_ctrl_prev = list_first_entry(&prev_hdl->ctrl_refs,
> +					 struct v4l2_ctrl_ref, node);
> +	list_for_each_entry(ref_ctrl, &hdl->ctrl_refs, node) {
> +		if (ref_ctrl->req)
> +			continue;
> +		while (ref_ctrl_prev->ctrl->id < ref_ctrl->ctrl->id) {
> +			/* Should never happen, but just in case... */
> +			if (list_is_last(&ref_ctrl_prev->node,
> +					 &prev_hdl->ctrl_refs))
> +				break;
> +			ref_ctrl_prev = list_next_entry(ref_ctrl_prev, node);
> +		}
> +		if (WARN_ON(ref_ctrl_prev->ctrl->id != ref_ctrl->ctrl->id))
> +			break;

Hmm... if the WARN_ON() is issued, should it still be queuing it? If
there's a driver's bug, I suspect it would be better to do, instead:

		if (WARN_ON(ref_ctrl_prev->ctrl->id != ref_ctrl->ctrl->id)) {
			mutex_unlock(prev_hdl->lock);
			return;
		}

> +		ref_ctrl->req = ref_ctrl_prev->req;
> +	}
> +	mutex_unlock(prev_hdl->lock);
> +queue:
> +	list_add_tail(&hdl->requests_queued, &main_hdl->requests_queued);
> +	hdl->request_is_queued = true;
> +}
> +
> +static void v4l2_ctrl_request_unbind(struct media_request_object *obj)
> +{
> +	struct v4l2_ctrl_handler *hdl =
> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	list_del_init(&hdl->requests);
> +	if (hdl->request_is_queued) {
> +		list_del_init(&hdl->requests_queued);
> +		hdl->request_is_queued = false;
> +	}
> +}
> +
> +static void v4l2_ctrl_request_release(struct media_request_object *obj)
> +{
> +	struct v4l2_ctrl_handler *hdl =
> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	v4l2_ctrl_handler_free(hdl);
> +	kfree(hdl);
> +}
> +
> +static const struct media_request_object_ops req_ops = {
> +	.queue = v4l2_ctrl_request_queue,
> +	.unbind = v4l2_ctrl_request_unbind,
> +	.release = v4l2_ctrl_request_release,
> +};
> +
> +static int v4l2_ctrl_request_bind(struct media_request *req,
> +			   struct v4l2_ctrl_handler *hdl,
> +			   struct v4l2_ctrl_handler *from)
> +{
> +	int ret;
> +
> +	ret = v4l2_ctrl_request_clone(hdl, from);
> +
> +	if (!ret) {
> +		ret = media_request_object_bind(req, &req_ops,
> +						from, &hdl->req_obj);
> +		if (!ret)
> +			list_add_tail(&hdl->requests, &from->requests);
> +	}
> +	return ret;
> +}
>  
>  /* Some general notes on the atomic requirements of VIDIOC_G/TRY/S_EXT_CTRLS:
>  
> @@ -2877,6 +3037,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
>  
>  		if (cs->which &&
>  		    cs->which != V4L2_CTRL_WHICH_DEF_VAL &&
> +		    cs->which != V4L2_CTRL_WHICH_REQUEST_VAL &&
>  		    V4L2_CTRL_ID2WHICH(id) != cs->which)
>  			return -EINVAL;
>  
> @@ -2956,13 +3117,12 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
>     whether there are any controls at all. */
>  static int class_check(struct v4l2_ctrl_handler *hdl, u32 which)
>  {
> -	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL)
> +	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL ||
> +	    which == V4L2_CTRL_WHICH_REQUEST_VAL)
>  		return 0;
>  	return find_ref_lock(hdl, which | 1) ? 0 : -EINVAL;
>  }
>  
> -
> -
>  /* Get extended controls. Allocates the helpers array if needed. */
>  int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs)
>  {
> @@ -3028,8 +3188,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
>  			u32 idx = i;
>  
>  			do {
> -				ret = ctrl_to_user(cs->controls + idx,
> -						   helpers[idx].ref->ctrl);
> +				if (helpers[idx].ref->req)
> +					ret = req_to_user(cs->controls + idx,
> +						helpers[idx].ref->req);
> +				else
> +					ret = ctrl_to_user(cs->controls + idx,
> +						helpers[idx].ref->ctrl);
>  				idx = helpers[idx].next;
>  			} while (!ret && idx);
>  		}
> @@ -3302,7 +3466,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
>  		} while (!ret && idx);
>  
>  		if (!ret)
> -			ret = try_or_set_cluster(fh, master, set, 0);
> +			ret = try_or_set_cluster(fh, master,
> +						 !hdl->req_obj.req && set, 0);
> +		if (!ret && hdl->req_obj.req && set) {
> +			for (j = 0; j < master->ncontrols; j++) {
> +				struct v4l2_ctrl_ref *ref =
> +					find_ref(hdl, master->cluster[j]->id);
> +
> +				new_to_req(ref);
> +			}
> +		}
>  
>  		/* Copy the new values back to userspace. */
>  		if (!ret) {
> @@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
>  }
>  EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
>  
> +void v4l2_ctrl_request_complete(struct media_request *req,
> +				struct v4l2_ctrl_handler *main_hdl)
> +{
> +	struct media_request_object *obj;
> +	struct v4l2_ctrl_handler *hdl;
> +	struct v4l2_ctrl_ref *ref;
> +
> +	if (!req || !main_hdl)
> +		return;
> +
> +	obj = media_request_object_find(req, &req_ops, main_hdl);
> +	if (!obj)
> +		return;
> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
> +		struct v4l2_ctrl *ctrl = ref->ctrl;
> +		struct v4l2_ctrl *master = ctrl->cluster[0];
> +		unsigned int i;
> +
> +		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
> +			ref->req = ref;
> +
> +			v4l2_ctrl_lock(master);
> +			/* g_volatile_ctrl will update the current control values */
> +			for (i = 0; i < master->ncontrols; i++)
> +				cur_to_new(master->cluster[i]);
> +			call_op(master, g_volatile_ctrl);
> +			new_to_req(ref);
> +			v4l2_ctrl_unlock(master);
> +			continue;
> +		}
> +		if (ref->req == ref)
> +			continue;
> +
> +		v4l2_ctrl_lock(ctrl);
> +		if (ref->req)
> +			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
> +		else
> +			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
> +		v4l2_ctrl_unlock(ctrl);
> +	}
> +
> +	WARN_ON(!hdl->request_is_queued);
> +	list_del_init(&hdl->requests_queued);
> +	hdl->request_is_queued = false;
> +	media_request_object_complete(obj);
> +	media_request_object_put(obj);
> +}
> +EXPORT_SYMBOL(v4l2_ctrl_request_complete);
> +
> +void v4l2_ctrl_request_setup(struct media_request *req,
> +			     struct v4l2_ctrl_handler *main_hdl)
> +{
> +	struct media_request_object *obj;
> +	struct v4l2_ctrl_handler *hdl;
> +	struct v4l2_ctrl_ref *ref;
> +
> +	if (!req || !main_hdl)
> +		return;
> +
> +	obj = media_request_object_find(req, &req_ops, main_hdl);
> +	if (!obj)
> +		return;
> +	if (obj->completed) {
> +		media_request_object_put(obj);
> +		return;
> +	}
> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	mutex_lock(hdl->lock);
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node)
> +		ref->done = false;
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
> +		struct v4l2_ctrl *ctrl = ref->ctrl;
> +		struct v4l2_ctrl *master = ctrl->cluster[0];
> +		bool have_new_data = false;
> +		int i;
> +
> +		/*
> +		 * Skip if this control was already handled by a cluster.
> +		 * Skip button controls and read-only controls.
> +		 */
> +		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
> +		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
> +			continue;
> +
> +		v4l2_ctrl_lock(master);
> +		for (i = 0; i < master->ncontrols; i++) {
> +			if (master->cluster[i]) {
> +				struct v4l2_ctrl_ref *r =
> +					find_ref(hdl, master->cluster[i]->id);
> +
> +				if (r->req && r == r->req) {
> +					have_new_data = true;
> +					break;
> +				}
> +			}
> +		}
> +		if (!have_new_data) {
> +			v4l2_ctrl_unlock(master);
> +			continue;
> +		}
> +
> +		for (i = 0; i < master->ncontrols; i++) {
> +			if (master->cluster[i]) {
> +				struct v4l2_ctrl_ref *r =
> +					find_ref(hdl, master->cluster[i]->id);
> +
> +				req_to_new(r);
> +				master->cluster[i]->is_new = 1;
> +				r->done = true;
> +			}
> +		}
> +		/*
> +		 * For volatile autoclusters that are currently in auto mode
> +		 * we need to discover if it will be set to manual mode.
> +		 * If so, then we have to copy the current volatile values
> +		 * first since those will become the new manual values (which
> +		 * may be overwritten by explicit new values from this set
> +		 * of controls).
> +		 */
> +		if (master->is_auto && master->has_volatiles &&
> +		    !is_cur_manual(master)) {
> +			s32 new_auto_val = *master->p_new.p_s32;
> +
> +			/*
> +			 * If the new value == the manual value, then copy
> +			 * the current volatile values.
> +			 */
> +			if (new_auto_val == master->manual_mode_value)
> +				update_from_auto_cluster(master);
> +		}
> +
> +		try_or_set_cluster(NULL, master, true, 0);
> +
> +		v4l2_ctrl_unlock(master);
> +	}
> +
> +	mutex_unlock(hdl->lock);
> +	media_request_object_put(obj);
> +}
> +EXPORT_SYMBOL(v4l2_ctrl_request_setup);
> +
>  void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
>  {
>  	if (ctrl == NULL)
> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> index 76352eb59f14..a0f7c38d1a90 100644
> --- a/include/media/v4l2-ctrls.h
> +++ b/include/media/v4l2-ctrls.h
> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>   * @from_other_dev: If true, then @ctrl was defined in another
>   *		device than the &struct v4l2_ctrl_handler.
> + * @done:	If true, then this control reference is part of a
> + *		control cluster that was already set while applying
> + *		the controls in this media request object.

Hmm... I would rename it for request_done or something similar, as it
seems that this applies only for requests.

> + * @req:	If set, this refers to another request that sets this control.
>   * @p_req:	The request value. Only used if the control handler
>   *		is bound to a media request.
>   *
> @@ -263,6 +267,8 @@ struct v4l2_ctrl_ref {
>  	struct v4l2_ctrl *ctrl;
>  	struct v4l2_ctrl_helper *helper;
>  	bool from_other_dev;
> +	bool done;
> +	struct v4l2_ctrl_ref *req;
>  	union v4l2_ctrl_ptr p_req;
>  };
>  
> @@ -287,6 +293,15 @@ struct v4l2_ctrl_ref {
>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>   * @nr_of_buckets: Total number of buckets in the array.
>   * @error:	The error code of the first failed control addition.
> + * @request_is_queued: True if the request was queued.
> + * @requests:	List to keep track of open control handler request objects.
> + *		For the parent control handler (@req_obj.req == NULL) this
> + *		is the list header. When the parent control handler is
> + *		removed, it has to unbind and put all these requests since
> + *		they refer to the parent.
> + * @requests_queued: List of the queued requests. This determines the order
> + *		in which these controls are applied. Once the request is
> + *		completed it is removed from this list.
>   * @req_obj:	The &struct media_request_object, used to link into a
>   *		&struct media_request.
>   */
> @@ -301,6 +316,9 @@ struct v4l2_ctrl_handler {
>  	void *notify_priv;
>  	u16 nr_of_buckets;
>  	int error;
> +	bool request_is_queued;
> +	struct list_head requests;
> +	struct list_head requests_queued;
>  	struct media_request_object req_obj;
>  };
>  
> @@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
>   */
>  __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
>  
> +void v4l2_ctrl_request_setup(struct media_request *req,
> +			     struct v4l2_ctrl_handler *hdl);
> +void v4l2_ctrl_request_complete(struct media_request *req,
> +				struct v4l2_ctrl_handler *hdl);
> +
>  /* Helpers for ioctl_ops */
>  
>  /**



Thanks,
Mauro

Re: [PATCHv13 04/28] media-request: add media_request_get_by_fd

[Hans Verkuil]

On 05/07/2018 07:01 PM, Mauro Carvalho Chehab wrote:
> Em Thu,  3 May 2018 16:52:54 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Add media_request_get_by_fd() to find a request based on the file
>> descriptor.
>>
>> The caller has to call media_request_put() for the returned
>> request since this function increments the refcount.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/media-request.c | 32 ++++++++++++++++++++++++++++++++
>>  include/media/media-request.h | 24 ++++++++++++++++++++++++
>>  2 files changed, 56 insertions(+)
>>
>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>> index c216c4ab628b..edc1c3af1959 100644
>> --- a/drivers/media/media-request.c
>> +++ b/drivers/media/media-request.c
>> @@ -218,6 +218,38 @@ static const struct file_operations request_fops = {
>>  	.release = media_request_close,
>>  };
>>  
>> +struct media_request *
>> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
>> +{
>> +	struct file *filp;
>> +	struct media_request *req;
>> +
>> +	if (!mdev || !mdev->ops ||
>> +	    !mdev->ops->req_validate || !mdev->ops->req_queue)
>> +		return ERR_PTR(-EPERM);
>> +
>> +	filp = fget(request_fd);
>> +	if (!filp)
>> +		return ERR_PTR(-ENOENT);
>> +
>> +	if (filp->f_op != &request_fops)
>> +		goto err_fput;
>> +	req = filp->private_data;
>> +	if (req->mdev != mdev)
>> +		goto err_fput;
>> +
>> +	media_request_get(req);
> 
> Hmm... this function changes the req struct (by calling kref_get) without
> holding neither the mutex or the spin lock...

kref_get is atomic, so why would you need to add additional locking?

Neither can the request be 'put' by another process before media_request_get()
is called due to the fget() above: while the fd has a refcount > 0 the
request refcount is also > 0. I'll add some comments to clarify this.

> 
>> +	fput(filp);
>> +
>> +	return req;
>> +
>> +err_fput:
>> +	fput(filp);
>> +
>> +	return ERR_PTR(-ENOENT);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get_by_fd);
>> +
>>  int media_request_alloc(struct media_device *mdev,
>>  			struct media_request_alloc *alloc)
>>  {
>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>> index e39122dfd717..997e096d7128 100644
>> --- a/include/media/media-request.h
>> +++ b/include/media/media-request.h
>> @@ -86,6 +86,24 @@ static inline void media_request_get(struct media_request *req)
>>   */
>>  void media_request_put(struct media_request *req);
>>  
>> +/**
>> + * media_request_get_by_fd - Get a media request by fd
>> + *
>> + * @mdev: Media device this request belongs to
>> + * @request_fd: The file descriptor of the request
>> + *
>> + * Get the request represented by @request_fd that is owned
>> + * by the media device.
>> + *
>> + * Return a -EPERM error pointer if requests are not supported
>> + * by this driver. Return -ENOENT if the request was not found.
>> + * Return the pointer to the request if found: the caller will
>> + * have to call @media_request_put when it finished using the
>> + * request.
> 
> ... so, it should be said here how this should be serialized, in order
> to avoid it to be destroyed by a task while some other task might be
> trying to instantiate it.

This doesn't need any serialization. If the request_fd is found, then
it will return the request with the request refcount increased.

I also do not understand what you mean with "some other task might be
trying to instantiate it".

I think there is some misunderstanding here.

Regards,

	Hans


> 
>> + */
>> +struct media_request *
>> +media_request_get_by_fd(struct media_device *mdev, int request_fd);
>> +
>>  /**
>>   * media_request_alloc - Allocate the media request
>>   *
>> @@ -107,6 +125,12 @@ static inline void media_request_put(struct media_request *req)
>>  {
>>  }
>>  
>> +static inline struct media_request *
>> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
>> +{
>> +	return ERR_PTR(-EPERM);
>> +}
>> +
>>  #endif
>>  
>>  /**
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 06/28] v4l2-dev: lock req_queue_mutex

[Hans Verkuil]

On 05/07/2018 07:20 PM, Mauro Carvalho Chehab wrote:
> Em Thu,  3 May 2018 16:52:56 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> We need to serialize streamon/off with queueing new requests.
>> These ioctls may trigger the cancellation of a streaming
>> operation, and that should not be mixed with queuing a new
>> request at the same time.
>>
>> Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
>> with MEDIA_REQUEST_IOC_QUEUE.
>>
>> Finally close() needs this lock since that too can trigger the
>> cancellation of a streaming operation.
>>
>> We take the req_queue_mutex here before any other locks since
>> it is a very high-level lock.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/v4l2-core/v4l2-dev.c | 37 +++++++++++++++++++++++++++++-
>>  1 file changed, 36 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index 1d0b2208e8fb..b1c9efc0ecc4 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -353,13 +353,36 @@ static long v4l2_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>>  
>>  	if (vdev->fops->unlocked_ioctl) {
>>  		struct mutex *lock = v4l2_ioctl_get_lock(vdev, cmd);
>> +		struct mutex *queue_lock = NULL;
>>  
>> -		if (lock && mutex_lock_interruptible(lock))
>> +		/*
>> +		 * We need to serialize streamon/off with queueing new requests.
>> +		 * These ioctls may trigger the cancellation of a streaming
>> +		 * operation, and that should not be mixed with queueing a new
>> +		 * request at the same time.
>> +		 *
>> +		 * Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
>> +		 * with MEDIA_REQUEST_IOC_QUEUE.
>> +		 */
>> +		if (vdev->v4l2_dev->mdev &&
>> +		    (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF ||
>> +		     cmd == VIDIOC_S_EXT_CTRLS || cmd == VIDIOC_TRY_EXT_CTRLS))
>> +			queue_lock = &vdev->v4l2_dev->mdev->req_queue_mutex;
>> +
>> +		if (queue_lock && mutex_lock_interruptible(queue_lock))
>> +			return -ERESTARTSYS;
> 
> Taking both locks seems risky. Here you're taking first v4l2 lock, returned
> by v4l2_ioctl_get_lock(vdev, cmd), and then you're taking the req_queue lock.

No,  v4l2_ioctl_get_lock() only returns a pointer to a mutex, it doesn't lock
anything. I think you got confused there. I'll reorganize the code a bit so
the call to  v4l2_ioctl_get_lock() happens after the queue_lock has been taken.

I'll also rename queue_lock to req_queue_lock (it's a bit more descriptive).

So we first take the high-level media_device req_queue_mutex if needed, and
then the ioctl serialization lock. Doing it the other way around will indeed
promptly deadlock (as I very quickly discovered after my initial implementation!).

So the order is:

	req_queue_mutex (serialize request state changes from/to IDLE)
	ioctl lock (serialize ioctls)
	request->lock (spinlock)

The last is only held for short periods when updating the media_request struct.

> 
> It is possible to call parts of the code that only handles req_queue
> or v4l2 lock (for example, by mixing request API calls with non-requests
> one). Worse than that, there are parts of the code where the request API
> patches get both a mutex and a spin lock.
> 
> I didn't look too closely (nor ran any test), but I'm almost sure that
> there are paths where it will end by leading into dead locks.

I've done extensive testing with this and actually been very careful about
the lock handling. It's also been tested with the cedrus driver.

Regards,

	Hans

> 
>> +
>> +		if (lock && mutex_lock_interruptible(lock)) {
>> +			if (queue_lock)
>> +				mutex_unlock(queue_lock);
>>  			return -ERESTARTSYS;
>> +		}
>>  		if (video_is_registered(vdev))
>>  			ret = vdev->fops->unlocked_ioctl(filp, cmd, arg);
>>  		if (lock)
>>  			mutex_unlock(lock);
>> +		if (queue_lock)
>> +			mutex_unlock(queue_lock);
>>  	} else
>>  		ret = -ENOTTY;
>>  
>> @@ -442,8 +465,20 @@ static int v4l2_release(struct inode *inode, struct file *filp)
>>  	struct video_device *vdev = video_devdata(filp);
>>  	int ret = 0;
>>  
>> +	/*
>> +	 * We need to serialize the release() with queueing new requests.
>> +	 * The release() may trigger the cancellation of a streaming
>> +	 * operation, and that should not be mixed with queueing a new
>> +	 * request at the same time.
>> +	 */
>> +	if (vdev->v4l2_dev->mdev)
>> +		mutex_lock(&vdev->v4l2_dev->mdev->req_queue_mutex);
>> +
>>  	if (vdev->fops->release)
>>  		ret = vdev->fops->release(filp);
>> +
>> +	if (vdev->v4l2_dev->mdev)
>> +		mutex_unlock(&vdev->v4l2_dev->mdev->req_queue_mutex);
>>  	if (vdev->dev_debug & V4L2_DEV_DEBUG_FOP)
>>  		printk(KERN_DEBUG "%s: release\n",
>>  			video_device_node_name(vdev));
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 09/28] v4l2-ctrls: prepare internal structs for request API

[Hans Verkuil]

On 05/07/2018 07:35 PM, Mauro Carvalho Chehab wrote:
> Em Thu,  3 May 2018 16:52:59 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Embed and initialize a media_request_object in struct v4l2_ctrl_handler.
>>
>> Add a p_req field to struct v4l2_ctrl_ref that will store the
>> request value.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
>>  include/media/v4l2-ctrls.h           | 7 +++++++
>>  2 files changed, 8 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
>> index aa1dd2015e84..d09f49530d9e 100644
>> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
>> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
>> @@ -1880,6 +1880,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
>>  				      sizeof(hdl->buckets[0]),
>>  				      GFP_KERNEL | __GFP_ZERO);
>>  	hdl->error = hdl->buckets ? 0 : -ENOMEM;
>> +	media_request_object_init(&hdl->req_obj);
>>  	return hdl->error;
>>  }
>>  EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
>> index d26b8ddebb56..76352eb59f14 100644
>> --- a/include/media/v4l2-ctrls.h
>> +++ b/include/media/v4l2-ctrls.h
>> @@ -20,6 +20,7 @@
>>  #include <linux/list.h>
>>  #include <linux/mutex.h>
>>  #include <linux/videodev2.h>
>> +#include <media/media-request.h>
>>  
>>  /* forward references */
>>  struct file;
>> @@ -249,6 +250,8 @@ struct v4l2_ctrl {
>>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>>   * @from_other_dev: If true, then @ctrl was defined in another
>>   *		device than the &struct v4l2_ctrl_handler.
>> + * @p_req:	The request value. Only used if the control handler
>> + *		is bound to a media request.
> 
> Could you please better elaborate the description of this field?
> 
> I read this patch dozen times to understand what you meant by
> "request value", as I would be expecting here a "media_request_"
> object or something similar. Instead, what you meant to say is that
> @p_req will be used to cache the data passed via a request API call,
> while the request is not handled yet, right?

The control handler has the request object. If the control handler is
part of a request, then the values of the controls that are owned by that
control handler are stored in p_req. When the request has been completed
p_req will contain the value at completion time.

I'll improve the documentation.

> 
> 
>>   *
>>   * Each control handler has a list of these refs. The list_head is used to
>>   * keep a sorted-by-control-ID list of all controls, while the next pointer
>> @@ -260,6 +263,7 @@ struct v4l2_ctrl_ref {
>>  	struct v4l2_ctrl *ctrl;
>>  	struct v4l2_ctrl_helper *helper;
>>  	bool from_other_dev;
>> +	union v4l2_ctrl_ptr p_req;
>>  };
>>  
>>  /**
>> @@ -283,6 +287,8 @@ struct v4l2_ctrl_ref {
>>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>>   * @nr_of_buckets: Total number of buckets in the array.
>>   * @error:	The error code of the first failed control addition.
>> + * @req_obj:	The &struct media_request_object, used to link into a
>> + *		&struct media_request.
> 
> I would document that there is kref is inside @req_obj, as we can't
> add another kref here later.

OK.

Regards,

	Hans

> 
>>   */
>>  struct v4l2_ctrl_handler {
>>  	struct mutex _lock;
>> @@ -295,6 +301,7 @@ struct v4l2_ctrl_handler {
>>  	void *notify_priv;
>>  	u16 nr_of_buckets;
>>  	int error;
>> +	struct media_request_object req_obj;
>>  };
>>  
>>  /**
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Hans Verkuil]

On 05/07/2018 08:06 PM, Mauro Carvalho Chehab wrote:
> Em Thu,  3 May 2018 16:53:02 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Integrate the request support. This adds the v4l2_ctrl_request_complete
>> and v4l2_ctrl_request_setup functions to complete a request and (as a
>> helper function) to apply a request to the hardware.
>>
>> It takes care of queuing requests and correctly chaining control values
>> in the request queue.
>>
>> Note that when a request is marked completed it will copy control values
>> to the internal request state. This can be optimized in the future since
>> this is sub-optimal when dealing with large compound and/or array controls.
>>
>> For the initial 'stateless codec' use-case the current implementation is
>> sufficient.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
>>  include/media/v4l2-ctrls.h           |  23 ++
>>  2 files changed, 348 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
>> index da4cc1485dc4..56b986185463 100644
>> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
>> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
>> @@ -1647,6 +1647,13 @@ static int new_to_user(struct v4l2_ext_control *c,
>>  	return ptr_to_user(c, ctrl, ctrl->p_new);
>>  }
>>  
>> +/* Helper function: copy the request value back to the caller */
>> +static int req_to_user(struct v4l2_ext_control *c,
>> +		       struct v4l2_ctrl_ref *ref)
>> +{
>> +	return ptr_to_user(c, ref->ctrl, ref->p_req);
>> +}
>> +
>>  /* Helper function: copy the initial control value back to the caller */
>>  static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
>>  {
>> @@ -1766,6 +1773,26 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
>>  	ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
>>  }
>>  
>> +/* Copy the new value to the request value */
>> +static void new_to_req(struct v4l2_ctrl_ref *ref)
>> +{
>> +	if (!ref)
>> +		return;
>> +	ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
>> +	ref->req = ref;
>> +}
>> +
>> +/* Copy the request value to the new value */
>> +static void req_to_new(struct v4l2_ctrl_ref *ref)
>> +{
>> +	if (!ref)
>> +		return;
>> +	if (ref->req)
>> +		ptr_to_ptr(ref->ctrl, ref->req->p_req, ref->ctrl->p_new);
>> +	else
>> +		ptr_to_ptr(ref->ctrl, ref->ctrl->p_cur, ref->ctrl->p_new);
>> +}
>> +
>>  /* Return non-zero if one or more of the controls in the cluster has a new
>>     value that differs from the current value. */
>>  static int cluster_changed(struct v4l2_ctrl *master)
>> @@ -1875,6 +1902,9 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
>>  	lockdep_set_class_and_name(hdl->lock, key, name);
>>  	INIT_LIST_HEAD(&hdl->ctrls);
>>  	INIT_LIST_HEAD(&hdl->ctrl_refs);
>> +	INIT_LIST_HEAD(&hdl->requests);
>> +	INIT_LIST_HEAD(&hdl->requests_queued);
>> +	hdl->request_is_queued = false;
>>  	hdl->nr_of_buckets = 1 + nr_of_controls_hint / 8;
>>  	hdl->buckets = kvmalloc_array(hdl->nr_of_buckets,
>>  				      sizeof(hdl->buckets[0]),
>> @@ -1895,6 +1925,14 @@ void v4l2_ctrl_handler_free(struct v4l2_ctrl_handler *hdl)
>>  	if (hdl == NULL || hdl->buckets == NULL)
>>  		return;
>>  
>> +	if (!hdl->req_obj.req && !list_empty(&hdl->requests)) {
>> +		struct v4l2_ctrl_handler *req, *next_req;
>> +
>> +		list_for_each_entry_safe(req, next_req, &hdl->requests, requests) {
>> +			media_request_object_unbind(&req->req_obj);
>> +			media_request_object_put(&req->req_obj);
>> +		}
>> +	}
>>  	mutex_lock(hdl->lock);
>>  	/* Free all nodes */
>>  	list_for_each_entry_safe(ref, next_ref, &hdl->ctrl_refs, node) {
>> @@ -2816,6 +2854,128 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct v4l2_querymenu *qm)
>>  }
>>  EXPORT_SYMBOL(v4l2_querymenu);
>>  
>> +static int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
>> +				   const struct v4l2_ctrl_handler *from)
>> +{
>> +	struct v4l2_ctrl_ref *ref;
>> +	int err;
>> +
>> +	if (WARN_ON(!hdl || hdl == from))
>> +		return -EINVAL;
>> +
>> +	if (hdl->error)
>> +		return hdl->error;
>> +
>> +	WARN_ON(hdl->lock != &hdl->_lock);
>> +
>> +	mutex_lock(from->lock);
>> +	list_for_each_entry(ref, &from->ctrl_refs, node) {
>> +		struct v4l2_ctrl *ctrl = ref->ctrl;
>> +		struct v4l2_ctrl_ref *new_ref;
>> +
>> +		/* Skip refs inherited from other devices */
>> +		if (ref->from_other_dev)
>> +			continue;
>> +		/* And buttons */
>> +		if (ctrl->type == V4L2_CTRL_TYPE_BUTTON)
>> +			continue;
>> +		err = handler_new_ref(hdl, ctrl, &new_ref, false, true);
>> +		if (err) {
>> +			printk("%s: handler_new_ref on control %x (%s) returned %d\n", __func__, ctrl->id, ctrl->name, err);
>> +			err = 0;
>> +			continue;
>> +		}
>> +		if (err)
>> +			break;
>> +	}
>> +	mutex_unlock(from->lock);
>> +	return err;
>> +}
>> +
>> +static void v4l2_ctrl_request_queue(struct media_request_object *obj)
>> +{
>> +	struct v4l2_ctrl_handler *hdl =
>> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +	struct v4l2_ctrl_handler *main_hdl = obj->priv;
>> +	struct v4l2_ctrl_handler *prev_hdl = NULL;
>> +	struct v4l2_ctrl_ref *ref_ctrl, *ref_ctrl_prev = NULL;
>> +
>> +	if (list_empty(&main_hdl->requests_queued))
>> +		goto queue;
>> +
>> +	prev_hdl = list_last_entry(&main_hdl->requests_queued,
>> +				   struct v4l2_ctrl_handler, requests_queued);
>> +	/*
>> +	 * Note: prev_hdl and hdl must contain the same list of control
>> +	 * references, so if any differences are detected then that is a
>> +	 * driver bug and the WARN_ON is triggered.
>> +	 */
>> +	mutex_lock(prev_hdl->lock);
>> +	ref_ctrl_prev = list_first_entry(&prev_hdl->ctrl_refs,
>> +					 struct v4l2_ctrl_ref, node);
>> +	list_for_each_entry(ref_ctrl, &hdl->ctrl_refs, node) {
>> +		if (ref_ctrl->req)
>> +			continue;
>> +		while (ref_ctrl_prev->ctrl->id < ref_ctrl->ctrl->id) {
>> +			/* Should never happen, but just in case... */
>> +			if (list_is_last(&ref_ctrl_prev->node,
>> +					 &prev_hdl->ctrl_refs))
>> +				break;
>> +			ref_ctrl_prev = list_next_entry(ref_ctrl_prev, node);
>> +		}
>> +		if (WARN_ON(ref_ctrl_prev->ctrl->id != ref_ctrl->ctrl->id))
>> +			break;
> 
> Hmm... if the WARN_ON() is issued, should it still be queuing it? If
> there's a driver's bug, I suspect it would be better to do, instead:
> 
> 		if (WARN_ON(ref_ctrl_prev->ctrl->id != ref_ctrl->ctrl->id)) {
> 			mutex_unlock(prev_hdl->lock);
> 			return;
> 		}

Good point. I'll have to think about this a bit more.

> 
>> +		ref_ctrl->req = ref_ctrl_prev->req;
>> +	}
>> +	mutex_unlock(prev_hdl->lock);
>> +queue:
>> +	list_add_tail(&hdl->requests_queued, &main_hdl->requests_queued);
>> +	hdl->request_is_queued = true;
>> +}
>> +
>> +static void v4l2_ctrl_request_unbind(struct media_request_object *obj)
>> +{
>> +	struct v4l2_ctrl_handler *hdl =
>> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	list_del_init(&hdl->requests);
>> +	if (hdl->request_is_queued) {
>> +		list_del_init(&hdl->requests_queued);
>> +		hdl->request_is_queued = false;
>> +	}
>> +}
>> +
>> +static void v4l2_ctrl_request_release(struct media_request_object *obj)
>> +{
>> +	struct v4l2_ctrl_handler *hdl =
>> +		container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	v4l2_ctrl_handler_free(hdl);
>> +	kfree(hdl);
>> +}
>> +
>> +static const struct media_request_object_ops req_ops = {
>> +	.queue = v4l2_ctrl_request_queue,
>> +	.unbind = v4l2_ctrl_request_unbind,
>> +	.release = v4l2_ctrl_request_release,
>> +};
>> +
>> +static int v4l2_ctrl_request_bind(struct media_request *req,
>> +			   struct v4l2_ctrl_handler *hdl,
>> +			   struct v4l2_ctrl_handler *from)
>> +{
>> +	int ret;
>> +
>> +	ret = v4l2_ctrl_request_clone(hdl, from);
>> +
>> +	if (!ret) {
>> +		ret = media_request_object_bind(req, &req_ops,
>> +						from, &hdl->req_obj);
>> +		if (!ret)
>> +			list_add_tail(&hdl->requests, &from->requests);
>> +	}
>> +	return ret;
>> +}
>>  
>>  /* Some general notes on the atomic requirements of VIDIOC_G/TRY/S_EXT_CTRLS:
>>  
>> @@ -2877,6 +3037,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
>>  
>>  		if (cs->which &&
>>  		    cs->which != V4L2_CTRL_WHICH_DEF_VAL &&
>> +		    cs->which != V4L2_CTRL_WHICH_REQUEST_VAL &&
>>  		    V4L2_CTRL_ID2WHICH(id) != cs->which)
>>  			return -EINVAL;
>>  
>> @@ -2956,13 +3117,12 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
>>     whether there are any controls at all. */
>>  static int class_check(struct v4l2_ctrl_handler *hdl, u32 which)
>>  {
>> -	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL)
>> +	if (which == 0 || which == V4L2_CTRL_WHICH_DEF_VAL ||
>> +	    which == V4L2_CTRL_WHICH_REQUEST_VAL)
>>  		return 0;
>>  	return find_ref_lock(hdl, which | 1) ? 0 : -EINVAL;
>>  }
>>  
>> -
>> -
>>  /* Get extended controls. Allocates the helpers array if needed. */
>>  int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs)
>>  {
>> @@ -3028,8 +3188,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
>>  			u32 idx = i;
>>  
>>  			do {
>> -				ret = ctrl_to_user(cs->controls + idx,
>> -						   helpers[idx].ref->ctrl);
>> +				if (helpers[idx].ref->req)
>> +					ret = req_to_user(cs->controls + idx,
>> +						helpers[idx].ref->req);
>> +				else
>> +					ret = ctrl_to_user(cs->controls + idx,
>> +						helpers[idx].ref->ctrl);
>>  				idx = helpers[idx].next;
>>  			} while (!ret && idx);
>>  		}
>> @@ -3302,7 +3466,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
>>  		} while (!ret && idx);
>>  
>>  		if (!ret)
>> -			ret = try_or_set_cluster(fh, master, set, 0);
>> +			ret = try_or_set_cluster(fh, master,
>> +						 !hdl->req_obj.req && set, 0);
>> +		if (!ret && hdl->req_obj.req && set) {
>> +			for (j = 0; j < master->ncontrols; j++) {
>> +				struct v4l2_ctrl_ref *ref =
>> +					find_ref(hdl, master->cluster[j]->id);
>> +
>> +				new_to_req(ref);
>> +			}
>> +		}
>>  
>>  		/* Copy the new values back to userspace. */
>>  		if (!ret) {
>> @@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
>>  }
>>  EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
>>  
>> +void v4l2_ctrl_request_complete(struct media_request *req,
>> +				struct v4l2_ctrl_handler *main_hdl)
>> +{
>> +	struct media_request_object *obj;
>> +	struct v4l2_ctrl_handler *hdl;
>> +	struct v4l2_ctrl_ref *ref;
>> +
>> +	if (!req || !main_hdl)
>> +		return;
>> +
>> +	obj = media_request_object_find(req, &req_ops, main_hdl);
>> +	if (!obj)
>> +		return;
>> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
>> +		struct v4l2_ctrl *ctrl = ref->ctrl;
>> +		struct v4l2_ctrl *master = ctrl->cluster[0];
>> +		unsigned int i;
>> +
>> +		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
>> +			ref->req = ref;
>> +
>> +			v4l2_ctrl_lock(master);
>> +			/* g_volatile_ctrl will update the current control values */
>> +			for (i = 0; i < master->ncontrols; i++)
>> +				cur_to_new(master->cluster[i]);
>> +			call_op(master, g_volatile_ctrl);
>> +			new_to_req(ref);
>> +			v4l2_ctrl_unlock(master);
>> +			continue;
>> +		}
>> +		if (ref->req == ref)
>> +			continue;
>> +
>> +		v4l2_ctrl_lock(ctrl);
>> +		if (ref->req)
>> +			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
>> +		else
>> +			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
>> +		v4l2_ctrl_unlock(ctrl);
>> +	}
>> +
>> +	WARN_ON(!hdl->request_is_queued);
>> +	list_del_init(&hdl->requests_queued);
>> +	hdl->request_is_queued = false;
>> +	media_request_object_complete(obj);
>> +	media_request_object_put(obj);
>> +}
>> +EXPORT_SYMBOL(v4l2_ctrl_request_complete);
>> +
>> +void v4l2_ctrl_request_setup(struct media_request *req,
>> +			     struct v4l2_ctrl_handler *main_hdl)
>> +{
>> +	struct media_request_object *obj;
>> +	struct v4l2_ctrl_handler *hdl;
>> +	struct v4l2_ctrl_ref *ref;
>> +
>> +	if (!req || !main_hdl)
>> +		return;
>> +
>> +	obj = media_request_object_find(req, &req_ops, main_hdl);
>> +	if (!obj)
>> +		return;
>> +	if (obj->completed) {
>> +		media_request_object_put(obj);
>> +		return;
>> +	}
>> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	mutex_lock(hdl->lock);
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node)
>> +		ref->done = false;
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
>> +		struct v4l2_ctrl *ctrl = ref->ctrl;
>> +		struct v4l2_ctrl *master = ctrl->cluster[0];
>> +		bool have_new_data = false;
>> +		int i;
>> +
>> +		/*
>> +		 * Skip if this control was already handled by a cluster.
>> +		 * Skip button controls and read-only controls.
>> +		 */
>> +		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
>> +		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
>> +			continue;
>> +
>> +		v4l2_ctrl_lock(master);
>> +		for (i = 0; i < master->ncontrols; i++) {
>> +			if (master->cluster[i]) {
>> +				struct v4l2_ctrl_ref *r =
>> +					find_ref(hdl, master->cluster[i]->id);
>> +
>> +				if (r->req && r == r->req) {
>> +					have_new_data = true;
>> +					break;
>> +				}
>> +			}
>> +		}
>> +		if (!have_new_data) {
>> +			v4l2_ctrl_unlock(master);
>> +			continue;
>> +		}
>> +
>> +		for (i = 0; i < master->ncontrols; i++) {
>> +			if (master->cluster[i]) {
>> +				struct v4l2_ctrl_ref *r =
>> +					find_ref(hdl, master->cluster[i]->id);
>> +
>> +				req_to_new(r);
>> +				master->cluster[i]->is_new = 1;
>> +				r->done = true;
>> +			}
>> +		}
>> +		/*
>> +		 * For volatile autoclusters that are currently in auto mode
>> +		 * we need to discover if it will be set to manual mode.
>> +		 * If so, then we have to copy the current volatile values
>> +		 * first since those will become the new manual values (which
>> +		 * may be overwritten by explicit new values from this set
>> +		 * of controls).
>> +		 */
>> +		if (master->is_auto && master->has_volatiles &&
>> +		    !is_cur_manual(master)) {
>> +			s32 new_auto_val = *master->p_new.p_s32;
>> +
>> +			/*
>> +			 * If the new value == the manual value, then copy
>> +			 * the current volatile values.
>> +			 */
>> +			if (new_auto_val == master->manual_mode_value)
>> +				update_from_auto_cluster(master);
>> +		}
>> +
>> +		try_or_set_cluster(NULL, master, true, 0);
>> +
>> +		v4l2_ctrl_unlock(master);
>> +	}
>> +
>> +	mutex_unlock(hdl->lock);
>> +	media_request_object_put(obj);
>> +}
>> +EXPORT_SYMBOL(v4l2_ctrl_request_setup);
>> +
>>  void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
>>  {
>>  	if (ctrl == NULL)
>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
>> index 76352eb59f14..a0f7c38d1a90 100644
>> --- a/include/media/v4l2-ctrls.h
>> +++ b/include/media/v4l2-ctrls.h
>> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
>>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>>   * @from_other_dev: If true, then @ctrl was defined in another
>>   *		device than the &struct v4l2_ctrl_handler.
>> + * @done:	If true, then this control reference is part of a
>> + *		control cluster that was already set while applying
>> + *		the controls in this media request object.
> 
> Hmm... I would rename it for request_done or something similar, as it
> seems that this applies only for requests.

No, the variable name is correct (it serves the same purpose as the 'done'
field in struct v4l2_ctrl), but the description should be improved.

I also want to take another look at this: I wonder if it isn't possible to
use the v4l2_ctrl 'done' field for this instead of having to add a new field
here.

> 
>> + * @req:	If set, this refers to another request that sets this control.
>>   * @p_req:	The request value. Only used if the control handler
>>   *		is bound to a media request.
>>   *
>> @@ -263,6 +267,8 @@ struct v4l2_ctrl_ref {
>>  	struct v4l2_ctrl *ctrl;
>>  	struct v4l2_ctrl_helper *helper;
>>  	bool from_other_dev;
>> +	bool done;
>> +	struct v4l2_ctrl_ref *req;
>>  	union v4l2_ctrl_ptr p_req;
>>  };
>>  
>> @@ -287,6 +293,15 @@ struct v4l2_ctrl_ref {
>>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>>   * @nr_of_buckets: Total number of buckets in the array.
>>   * @error:	The error code of the first failed control addition.
>> + * @request_is_queued: True if the request was queued.
>> + * @requests:	List to keep track of open control handler request objects.
>> + *		For the parent control handler (@req_obj.req == NULL) this
>> + *		is the list header. When the parent control handler is
>> + *		removed, it has to unbind and put all these requests since
>> + *		they refer to the parent.
>> + * @requests_queued: List of the queued requests. This determines the order
>> + *		in which these controls are applied. Once the request is
>> + *		completed it is removed from this list.
>>   * @req_obj:	The &struct media_request_object, used to link into a
>>   *		&struct media_request.
>>   */
>> @@ -301,6 +316,9 @@ struct v4l2_ctrl_handler {
>>  	void *notify_priv;
>>  	u16 nr_of_buckets;
>>  	int error;
>> +	bool request_is_queued;
>> +	struct list_head requests;
>> +	struct list_head requests_queued;
>>  	struct media_request_object req_obj;
>>  };
>>  
>> @@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
>>   */
>>  __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
>>  
>> +void v4l2_ctrl_request_setup(struct media_request *req,
>> +			     struct v4l2_ctrl_handler *hdl);
>> +void v4l2_ctrl_request_complete(struct media_request *req,
>> +				struct v4l2_ctrl_handler *hdl);
>> +
>>  /* Helpers for ioctl_ops */
>>  
>>  /**
> 
> 
> 
> Thanks,
> Mauro
> 

Regards,

	Hans

Re: [PATCHv13 01/28] v4l2-device.h: always expose mdev

[Hans Verkuil]

On 05/07/2018 05:46 PM, Mauro Carvalho Chehab wrote:
> Em Fri, 4 May 2018 13:51:28 +0300
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
>> On Thu, May 03, 2018 at 04:52:51PM +0200, Hans Verkuil wrote:
>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>
>>> The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
>>> But since we will need to pass the media_device to vb2 and the
>>> control framework it is very convenient to just make this field
>>> available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
>>> then it will just be NULL.
>>>
>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>  
>>
>> Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>
> 
> This patch is no-brainer. It could be sent no matter what. However,
> the patch is too simple :-)
> 
> There are a number of places where if CONFIG_MEDIA_CONTROLLER
> (and for CONFIG_MEDIA_CONTROLLER_DVB - with is also optionally 
> added at DVB core) is tested just because the field may or may
> not be there.
> 
> If we're willing to always have it at the struct, then we should look
> on all #ifs for CONFIG_MEDIA_CONTROLLER and get rid of most (or all)
> of them, ensuring that function stubs will be enough for the code
> itself to do the right thing if !CONFIG_MEDIA_CONTROLLER.

I looked at this, and in all cases where v4l2_dev->mdev is checked the
following code always uses something that requires CONFIG_MEDIA_CONTROLLER
anyway (usually the entity).

Regards,

	Hans

> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 16/28] videobuf2-core: embed media_request_object

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:53:06 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Make vb2_buffer a request object.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  include/media/videobuf2-core.h | 3 +++
>  1 file changed, 3 insertions(+)
> 
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index 224c4820a044..3d54654c3cd4 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -17,6 +17,7 @@
>  #include <linux/poll.h>
>  #include <linux/dma-buf.h>
>  #include <linux/bitops.h>
> +#include <media/media-request.h>
>  
>  #define VB2_MAX_FRAME	(32)
>  #define VB2_MAX_PLANES	(8)
> @@ -238,6 +239,7 @@ struct vb2_queue;
>   * @num_planes:		number of planes in the buffer
>   *			on an internal driver queue.
>   * @timestamp:		frame timestamp in ns.
> + * @req_obj:		used to bind this buffer to a request
>   */
>  struct vb2_buffer {
>  	struct vb2_queue	*vb2_queue;
> @@ -246,6 +248,7 @@ struct vb2_buffer {
>  	unsigned int		memory;
>  	unsigned int		num_planes;
>  	u64			timestamp;
> +	struct media_request_object	req_obj;
>  
>  	/* private: internal use only
>  	 *

Hmm... this has a side effect of embedding a kref at struct vb2_buffer.
One struct can have just one kref.

I guess this is likely ok, but this is a big struct. I don't like
the idea of having a hidden kref indirectly embedded there, as the
lifetime of this struct will now be controlled outside vb2, with
looks weird.

Thanks,
Mauro

Re: [PATCHv13 03/28] media-request: implement media requests

[Mauro Carvalho Chehab]

Em Fri, 4 May 2018 15:27:50 +0300
Sakari Ailus <sakari.ailus@iki.fi> escreveu:

> Hi Hans,
> 
> I've read this patch a large number of times and I think also the details
> begin to seem sound. A few comments below.

I'm sending this after analyzing the other patches in this series,
as this is the core of the changes. So, although I wrote the comments
early, I wanted to read first all other patches before sending it.

> 
> On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:
> > From: Hans Verkuil <hans.verkuil@cisco.com>
> > 
> > Add initial media request support:
> > 
> > 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> > 2) Add struct media_request to store request objects.
> > 3) Add struct media_request_object to represent a request object.
> > 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> > 
> > Basic lifecycle: the application allocates a request, adds
> > objects to it, queues the request, polls until it is completed
> > and can then read the final values of the objects at the time
> > of completion. When it closes the file descriptor the request
> > memory will be freed (actually, when the last user of that request
> > releases the request).
> > 
> > Drivers will bind an object to a request (the 'adds objects to it'
> > phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> > validated (req_validate op), then queued (the req_queue op).
> > 
> > When done with an object it can either be unbound from the request
> > (e.g. when the driver has finished with a vb2 buffer) or marked as
> > completed (e.g. for controls associated with a buffer). When all
> > objects in the request are completed (or unbound), then the request
> > fd will signal an exception (poll).
> > 
> > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>

Hmm... As you're adding Copyrights from Intel/Google in this patch, that
indicates that part of the stuff you're adding here were authored by
others. So, you should use Co-developed-by: tag here, and get the SOBs
from the other developers that did part of the work[1].

[1] except if your work was sponsored by Cisco, Intel and Google, but
    I think this is not the case.

> > ---
> >  drivers/media/Makefile        |   3 +-
> >  drivers/media/media-device.c  |  14 ++
> >  drivers/media/media-request.c | 407 ++++++++++++++++++++++++++++++++++
> >  include/media/media-device.h  |  16 ++
> >  include/media/media-request.h | 244 ++++++++++++++++++++
> >  5 files changed, 683 insertions(+), 1 deletion(-)
> >  create mode 100644 drivers/media/media-request.c
> >  create mode 100644 include/media/media-request.h
> > 
> > diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> > index 594b462ddf0e..985d35ec6b29 100644
> > --- a/drivers/media/Makefile
> > +++ b/drivers/media/Makefile
> > @@ -3,7 +3,8 @@
> >  # Makefile for the kernel multimedia device drivers.
> >  #
> >  
> > -media-objs	:= media-device.o media-devnode.o media-entity.o
> > +media-objs	:= media-device.o media-devnode.o media-entity.o \
> > +		   media-request.o
> >  
> >  #
> >  # I2C drivers should come before other drivers, otherwise they'll fail
> > diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> > index 35e81f7c0d2f..bb6a64acd3f0 100644
> > --- a/drivers/media/media-device.c
> > +++ b/drivers/media/media-device.c
> > @@ -32,6 +32,7 @@
> >  #include <media/media-device.h>
> >  #include <media/media-devnode.h>
> >  #include <media/media-entity.h>
> > +#include <media/media-request.h>
> >  
> >  #ifdef CONFIG_MEDIA_CONTROLLER
> >  
> > @@ -366,6 +367,15 @@ static long media_device_get_topology(struct media_device *mdev,
> >  	return ret;
> >  }
> >  
> > +static long media_device_request_alloc(struct media_device *mdev,
> > +				       struct media_request_alloc *alloc)
> > +{
> > +	if (!mdev->ops || !mdev->ops->req_validate || !mdev->ops->req_queue)
> > +		return -ENOTTY;
> > +
> > +	return media_request_alloc(mdev, alloc);
> > +}
> > +
> >  static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
> >  {
> >  	/* All media IOCTLs are _IOWR() */
> > @@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
> >  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
> >  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
> >  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
> > +	MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
> >  };
> >  
> >  static long media_device_ioctl(struct file *filp, unsigned int cmd,
> > @@ -686,6 +697,8 @@ void media_device_init(struct media_device *mdev)
> >  	INIT_LIST_HEAD(&mdev->pads);
> >  	INIT_LIST_HEAD(&mdev->links);
> >  	INIT_LIST_HEAD(&mdev->entity_notify);
> > +
> > +	mutex_init(&mdev->req_queue_mutex);
> >  	mutex_init(&mdev->graph_mutex);
> >  	ida_init(&mdev->entity_internal_idx);
> >  
> > @@ -699,6 +712,7 @@ void media_device_cleanup(struct media_device *mdev)
> >  	mdev->entity_internal_idx_max = 0;
> >  	media_graph_walk_cleanup(&mdev->pm_count_walk);
> >  	mutex_destroy(&mdev->graph_mutex);
> > +	mutex_destroy(&mdev->req_queue_mutex);
> >  }
> >  EXPORT_SYMBOL_GPL(media_device_cleanup);
> >  
> > diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> > new file mode 100644
> > index 000000000000..c216c4ab628b
> > --- /dev/null
> > +++ b/drivers/media/media-request.c
> > @@ -0,0 +1,407 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * Media device request objects
> > + *
> > + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
> > + * Copyright (C) 2018 Intel Corporation
> > + * Copyright (C) 2018 Google, Inc.
> > + *
> > + * Author: Hans Verkuil <hans.verkuil@cisco.com>
> > + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
> > + */
> > +
> > +#include <linux/anon_inodes.h>
> > +#include <linux/file.h>
> > +
> > +#include <media/media-device.h>
> > +#include <media/media-request.h>
> > +
> > +static const char * const request_state[] = {
> > +	[MEDIA_REQUEST_STATE_IDLE]	 = "idle",
> > +	[MEDIA_REQUEST_STATE_VALIDATING] = "validating",
> > +	[MEDIA_REQUEST_STATE_QUEUED]	 = "queued",
> > +	[MEDIA_REQUEST_STATE_COMPLETE]	 = "complete",
> > +	[MEDIA_REQUEST_STATE_CLEANING]	 = "cleaning",
> > +};
> > +
> > +static const char *
> > +media_request_state_str(enum media_request_state state)
> > +{
> > +	if (WARN_ON(state >= ARRAY_SIZE(request_state)))
> > +		return "invalid";
> > +	return request_state[state];
> > +}
> > +
> > +static void media_request_clean(struct media_request *req)
> > +{
> > +	struct media_request_object *obj, *obj_safe;
> > +
> > +	WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_CLEANING);
> > +
> > +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
> > +		media_request_object_unbind(obj);
> > +		media_request_object_put(obj);
> > +	}
> > +
> > +	req->num_incomplete_objects = 0;  
> 
> The number of incomplete objects should be already zero here. I'd think
> that a different number would suggest that something has gone very wrong
> and should be complained about. How about adding
> WARN_ON(req->num_incomplete_objects) above this line? 
> 
> > +	wake_up_interruptible_all(&req->poll_wait);
> > +}
> > +
> > +static void media_request_release(struct kref *kref)
> > +{
> > +	struct media_request *req =
> > +		container_of(kref, struct media_request, kref);
> > +	struct media_device *mdev = req->mdev;
> > +
> > +	dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
> > +
> > +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
> > +
> > +	media_request_clean(req);
> > +
> > +	if (mdev->ops->req_free)
> > +		mdev->ops->req_free(req);
> > +	else
> > +		kfree(req);
> > +}
> > +
> > +void media_request_put(struct media_request *req)
> > +{
> > +	kref_put(&req->kref, media_request_release);
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_put);
> > +
> > +static int media_request_close(struct inode *inode, struct file *filp)
> > +{
> > +	struct media_request *req = filp->private_data;
> > +
> > +	media_request_put(req);
> > +	return 0;
> > +}
> > +
> > +static unsigned int media_request_poll(struct file *filp,
> > +				       struct poll_table_struct *wait)
> > +{
> > +	struct media_request *req = filp->private_data;
> > +	unsigned long flags;
> > +	unsigned int ret = 0;
> > +	enum media_request_state state;
> > +
> > +	if (!(poll_requested_events(wait) & POLLPRI))
> > +		return 0;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	state = atomic_read(&req->state);
> > +
> > +	if (state == MEDIA_REQUEST_STATE_COMPLETE) {
> > +		ret = POLLPRI;
> > +		goto unlock;
> > +	}
> > +	if (state != MEDIA_REQUEST_STATE_QUEUED) {
> > +		ret = POLLERR;
> > +		goto unlock;
> > +	}
> > +
> > +	poll_wait(filp, &req->poll_wait, wait);
> > +
> > +unlock:
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +	return ret;
> > +}
> > +
> > +static long media_request_ioctl_queue(struct media_request *req)
> > +{
> > +	struct media_device *mdev = req->mdev;
> > +	enum media_request_state state;
> > +	unsigned long flags;
> > +	int ret = 0;  
> 
> ret is unconditionally assigned below, no need to initialise here.
> 
> > +
> > +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
> > +
> > +	/*
> > +	 * Ensure the request that is validated will be the one that gets queued
> > +	 * next by serialising the queueing process. This mutex is also used
> > +	 * to serialize with canceling a vb2 queue and with setting values such
> > +	 * as controls in a request.
> > +	 */
> > +	mutex_lock(&mdev->req_queue_mutex);
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
> > +			       MEDIA_REQUEST_STATE_VALIDATING);
> > +	spin_unlock_irqrestore(&req->lock, flags);

It looks weird to serialize access to it with a mutex, a spin lock and 
an atomic call.

IMHO, locking is still an issue here. I would love to test the 
locks with some tool that would randomize syscalls, issuing close(),
poll() and read() at wrong times and inverting the order of some calls, 
in order to do some empiric test that all locks are at the right places.

Complex locking schemas like that usually tend to cause a lot of
troubles.

> > +	if (state != MEDIA_REQUEST_STATE_IDLE) {
> > +		dev_dbg(mdev->dev,
> > +			"request: unable to queue %s, request in state %s\n",
> > +			req->debug_str, media_request_state_str(state));
> > +		mutex_unlock(&mdev->req_queue_mutex);
> > +		return -EBUSY;
> > +	}
> > +
> > +	ret = mdev->ops->req_validate(req);
> > +
> > +	/*
> > +	 * If the req_validate was successful, then we mark the state as QUEUED
> > +	 * and call req_queue. The reason we set the state first is that this
> > +	 * allows req_queue to unbind or complete the queued objects in case
> > +	 * they are immediately 'consumed'. State changes from QUEUED to another
> > +	 * state can only happen if either the driver changes the state or if
> > +	 * the user cancels the vb2 queue. The driver can only change the state
> > +	 * after each object is queued through the req_queue op (and note that
> > +	 * that op cannot fail), so setting the state to QUEUED up front is
> > +	 * safe.
> > +	 *
> > +	 * The other reason for changing the state is if the vb2 queue is
> > +	 * canceled, and that uses the req_queue_mutex which is still locked
> > +	 * while req_queue is called, so that's safe as well.
> > +	 */
> > +	atomic_set(&req->state,
> > +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);

Why are you changing state also when ret fails?

Also, why you had to use a spin lock earlier in this function just 
to change the req->state but you don't need to use it here?

> > +
> > +	if (!ret)
> > +		mdev->ops->req_queue(req);
> > +
> > +	mutex_unlock(&mdev->req_queue_mutex);
> > +
> > +	if (ret)
> > +		dev_dbg(mdev->dev, "request: can't queue %s (%d)\n",
> > +			req->debug_str, ret);
> > +	else
> > +		media_request_get(req);
> > +
> > +	return ret;
> > +}
> > +
> > +static long media_request_ioctl_reinit(struct media_request *req)
> > +{
> > +	struct media_device *mdev = req->mdev;
> > +	unsigned long flags;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE &&
> > +	    atomic_read(&req->state) != MEDIA_REQUEST_STATE_COMPLETE) {
> > +		dev_dbg(mdev->dev,
> > +			"request: %s not in idle or complete state, cannot reinit\n",
> > +			req->debug_str);
> > +		spin_unlock_irqrestore(&req->lock, flags);
> > +		return -EBUSY;
> > +	}
> > +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +
> > +	media_request_clean(req);
> > +
> > +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
> > +
> > +	return 0;
> > +}
> > +
> > +static long media_request_ioctl(struct file *filp, unsigned int cmd,
> > +				unsigned long arg)
> > +{
> > +	struct media_request *req = filp->private_data;
> > +
> > +	switch (cmd) {
> > +	case MEDIA_REQUEST_IOC_QUEUE:
> > +		return media_request_ioctl_queue(req);
> > +	case MEDIA_REQUEST_IOC_REINIT:
> > +		return media_request_ioctl_reinit(req);
> > +	default:
> > +		return -ENOIOCTLCMD;
> > +	}
> > +}
> > +
> > +static const struct file_operations request_fops = {
> > +	.owner = THIS_MODULE,
> > +	.poll = media_request_poll,
> > +	.unlocked_ioctl = media_request_ioctl,
> > +	.release = media_request_close,
> > +};
> > +
> > +int media_request_alloc(struct media_device *mdev,
> > +			struct media_request_alloc *alloc)
> > +{
> > +	struct media_request *req;
> > +	struct file *filp;
> > +	char comm[TASK_COMM_LEN];
> > +	int fd;
> > +	int ret;
> > +
> > +	/* Either both are NULL or both are non-NULL */
> > +	if (WARN_ON(!mdev->ops->req_alloc ^ !mdev->ops->req_free))
> > +		return -ENOMEM;
> > +
> > +	fd = get_unused_fd_flags(O_CLOEXEC);
> > +	if (fd < 0)
> > +		return fd;
> > +
> > +	filp = anon_inode_getfile("request", &request_fops, NULL, O_CLOEXEC);
> > +	if (IS_ERR(filp)) {
> > +		ret = PTR_ERR(filp);
> > +		goto err_put_fd;
> > +	}
> > +
> > +	if (mdev->ops->req_alloc)
> > +		req = mdev->ops->req_alloc(mdev);
> > +	else
> > +		req = kzalloc(sizeof(*req), GFP_KERNEL);
> > +	if (!req) {
> > +		ret = -ENOMEM;
> > +		goto err_fput;
> > +	}
> > +
> > +	filp->private_data = req;
> > +	req->mdev = mdev;
> > +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
> > +	req->num_incomplete_objects = 0;
> > +	kref_init(&req->kref);
> > +	INIT_LIST_HEAD(&req->objects);
> > +	spin_lock_init(&req->lock);
> > +	init_waitqueue_head(&req->poll_wait);
> > +
> > +	alloc->fd = fd;
> > +
> > +	get_task_comm(comm, current);
> > +	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
> > +		 comm, fd);
> > +	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
> > +
> > +	fd_install(fd, filp);
> > +
> > +	return 0;
> > +
> > +err_fput:
> > +	fput(filp);
> > +
> > +err_put_fd:
> > +	put_unused_fd(fd);
> > +
> > +	return ret;
> > +}
> > +
> > +static void media_request_object_release(struct kref *kref)
> > +{
> > +	struct media_request_object *obj =
> > +		container_of(kref, struct media_request_object, kref);
> > +	struct media_request *req = obj->req;
> > +
> > +	if (req)
> > +		media_request_object_unbind(obj);
> > +	obj->ops->release(obj);
> > +}
> > +
> > +void media_request_object_put(struct media_request_object *obj)
> > +{
> > +	kref_put(&obj->kref, media_request_object_release);
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_put);
> > +
> > +void media_request_object_init(struct media_request_object *obj)
> > +{
> > +	obj->ops = NULL;
> > +	obj->req = NULL;
> > +	obj->priv = NULL;
> > +	obj->completed = false;
> > +	INIT_LIST_HEAD(&obj->list);
> > +	kref_init(&obj->kref);
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_init);
> > +
> > +int media_request_object_bind(struct media_request *req,
> > +			      const struct media_request_object_ops *ops,
> > +			      void *priv,
> > +			      struct media_request_object *obj)
> > +{
> > +	unsigned long flags;
> > +	int ret = -EBUSY;
> > +
> > +	if (WARN_ON(!ops->release))
> > +		return -EPERM;
> > +
> > +	obj->req = req;
> > +	obj->ops = ops;
> > +	obj->priv = priv;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +
> > +	if (WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE))
> > +		goto unlock;  
> 
> Is this worth a kernel warning, or rather how the drivers / other framework
> bits (e.g. VB2) prevent user from binding objects to non-idle requests?
> Even if you added a similar check to the caller, the request state could
> well change in the meantime.
> 
> Perhaps add __must_check to the return value?
> 
> > +
> > +	list_add_tail(&obj->list, &req->objects);
> > +	req->num_incomplete_objects++;
> > +	ret = 0;
> > +
> > +unlock:
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +	return ret;
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_bind);
> > +
> > +void media_request_object_unbind(struct media_request_object *obj)
> > +{
> > +	struct media_request *req = obj->req;
> > +	enum media_request_state state;
> > +	unsigned long flags;
> > +	bool completed = false;
> > +
> > +	if (WARN_ON(!req))
> > +		return;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	list_del(&obj->list);
> > +	obj->req = NULL;
> > +
> > +	state = atomic_read(&req->state);
> > +
> > +	if (state == MEDIA_REQUEST_STATE_COMPLETE ||
> > +	    state == MEDIA_REQUEST_STATE_CLEANING)
> > +		goto unlock;
> > +
> > +	if (WARN_ON(state == MEDIA_REQUEST_STATE_VALIDATING))
> > +		goto unlock;
> > +
> > +	if (WARN_ON(!req->num_incomplete_objects))
> > +		goto unlock;
> > +
> > +	req->num_incomplete_objects--;
> > +	if (state == MEDIA_REQUEST_STATE_QUEUED &&
> > +	    !req->num_incomplete_objects) {
> > +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
> > +		completed = true;
> > +		wake_up_interruptible_all(&req->poll_wait);
> > +	}
> > +
> > +unlock:
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +	if (obj->ops->unbind)
> > +		obj->ops->unbind(obj);
> > +	if (completed)
> > +		media_request_put(req);
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_unbind);
> > +
> > +void media_request_object_complete(struct media_request_object *obj)
> > +{
> > +	struct media_request *req = obj->req;
> > +	unsigned long flags;
> > +	bool completed = false;
> > +
> > +	spin_lock_irqsave(&req->lock, flags);
> > +	if (obj->completed)
> > +		goto unlock;
> > +	obj->completed = true;
> > +	if (WARN_ON(!req->num_incomplete_objects) ||
> > +	    WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_QUEUED))
> > +		goto unlock;
> > +
> > +	if (!--req->num_incomplete_objects) {
> > +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
> > +		wake_up_interruptible_all(&req->poll_wait);
> > +		completed = true;
> > +	}
> > +unlock:
> > +	spin_unlock_irqrestore(&req->lock, flags);
> > +	if (completed)
> > +		media_request_put(req);
> > +}
> > +EXPORT_SYMBOL_GPL(media_request_object_complete);
> > diff --git a/include/media/media-device.h b/include/media/media-device.h
> > index bcc6ec434f1f..7d855823341c 100644
> > --- a/include/media/media-device.h
> > +++ b/include/media/media-device.h
> > @@ -27,6 +27,7 @@
> >  
> >  struct ida;
> >  struct device;
> > +struct media_device;
> >  
> >  /**
> >   * struct media_entity_notify - Media Entity Notify
> > @@ -50,10 +51,21 @@ struct media_entity_notify {
> >   * struct media_device_ops - Media device operations
> >   * @link_notify: Link state change notification callback. This callback is
> >   *		 called with the graph_mutex held.
> > + * @req_alloc: Allocate a request
> > + * @req_free: Free a request

Please do place better descriptions there. First of all, either both
should be used or both should be NULL - as validated by
media_request_alloc() logic.

Second, it should be clearer that those are meant to be used when
the driver needs to allocate a bigger struct that embeds a 
struct media_request object on it, with should be destroyed at 
ops->req_free() call.

Also, such embed struct should not contain a kref (multiple krefs
at the same struct doesn't work).

> > + * @req_validate: Validate a request, but do not queue yet
> > + * @req_queue: Queue a validated request, cannot fail. If something goes
> > + *	       wrong when queueing this request then it should be marked
> > + *	       as such internally in the driver and any related buffers
> > + *	       must eventually return to vb2 with state VB2_BUF_STATE_ERROR.

Please describe what kind of req locks (if any) can/should
be used inside each callback.


> >   */
> >  struct media_device_ops {
> >  	int (*link_notify)(struct media_link *link, u32 flags,
> >  			   unsigned int notification);
> > +	struct media_request *(*req_alloc)(struct media_device *mdev);
> > +	void (*req_free)(struct media_request *req);
> > +	int (*req_validate)(struct media_request *req);
> > +	void (*req_queue)(struct media_request *req);
> >  };
> >  
> >  /**
> > @@ -88,6 +100,8 @@ struct media_device_ops {
> >   * @disable_source: Disable Source Handler function pointer
> >   *
> >   * @ops:	Operation handler callbacks
> > + * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
> > + *		     media device.

This description is incomplete as it doesn't match the explanation you
introduced at patch 00/18. Please add a complete locking description,
as patches 00 aren't stored anywhere at the git tree, and there are lots
of non-trivial assumptions on your locking schema.

> >   *
> >   * This structure represents an abstract high-level media device. It allows easy
> >   * access to entities and provides basic media device-level support. The
> > @@ -158,6 +172,8 @@ struct media_device {
> >  	void (*disable_source)(struct media_entity *entity);
> >  
> >  	const struct media_device_ops *ops;
> > +
> > +	struct mutex req_queue_mutex;
> >  };
> >  
> >  /* We don't need to include pci.h or usb.h here */
> > diff --git a/include/media/media-request.h b/include/media/media-request.h
> > new file mode 100644
> > index 000000000000..e39122dfd717
> > --- /dev/null
> > +++ b/include/media/media-request.h
> > @@ -0,0 +1,244 @@
> > +// SPDX-License-Identifier: GPL-2.0-only
> > +/*
> > + * Media device request objects
> > + *
> > + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
> > + * Copyright (C) 2018 Intel Corporation
> > + *
> > + * Author: Hans Verkuil <hans.verkuil@cisco.com>
> > + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
> > + */
> > +
> > +#ifndef MEDIA_REQUEST_H
> > +#define MEDIA_REQUEST_H
> > +
> > +#include <linux/list.h>
> > +#include <linux/slab.h>
> > +#include <linux/spinlock.h>
> > +#include <linux/atomic.h>
> > +
> > +#include <media/media-device.h>
> > +
> > +/**
> > + * enum media_request_state - media request state
> > + *
> > + * @MEDIA_REQUEST_STATE_IDLE:		Idle
> > + * @MEDIA_REQUEST_STATE_VALIDATING:	Validating the request, no state changes
> > + *					allowed
> > + * @MEDIA_REQUEST_STATE_QUEUED:		Queued
> > + * @MEDIA_REQUEST_STATE_COMPLETE:	Completed, the request is done
> > + * @MEDIA_REQUEST_STATE_CLEANING:	Cleaning, the request is being re-inited
> > + */
> > +enum media_request_state {
> > +	MEDIA_REQUEST_STATE_IDLE,
> > +	MEDIA_REQUEST_STATE_VALIDATING,
> > +	MEDIA_REQUEST_STATE_QUEUED,
> > +	MEDIA_REQUEST_STATE_COMPLETE,
> > +	MEDIA_REQUEST_STATE_CLEANING,
> > +};
> > +
> > +struct media_request_object;
> > +
> > +/**
> > + * struct media_request - Media device request
> > + * @mdev: Media device this request belongs to
> > + * @kref: Reference count
> > + * @debug_str: Prefix for debug messages (process name:fd)
> > + * @state: The state of the request
> > + * @objects: List of @struct media_request_object request objects
> > + * @num_objects: The number of objects in the request
> > + * @num_incompleted_objects: The number of incomplete objects in the request
> > + * @poll_wait: Wait queue for poll
> > + * @lock: Serializes access to this struct
> > + */
> > +struct media_request {
> > +	struct media_device *mdev;
> > +	struct kref kref;
> > +	char debug_str[TASK_COMM_LEN + 11];
> > +	atomic_t state;
> > +	struct list_head objects;
> > +	unsigned int num_incomplete_objects;
> > +	struct wait_queue_head poll_wait;
> > +	spinlock_t lock;
> > +};
> > +
> > +#ifdef CONFIG_MEDIA_CONTROLLER
> > +
> > +/**
> > + * media_request_get - Get the media request
> > + *
> > + * @req: The request
> > + *
> > + * Get the media request.
> > + */
> > +static inline void media_request_get(struct media_request *req)
> > +{
> > +	kref_get(&req->kref);
> > +}
> > +
> > +/**
> > + * media_request_put - Put the media request
> > + *
> > + * @req: The request
> > + *
> > + * Put the media request. The media request will be released
> > + * when the refcount reaches 0.
> > + */
> > +void media_request_put(struct media_request *req);
> > +
> > +/**
> > + * media_request_alloc - Allocate the media request
> > + *
> > + * @mdev: Media device this request belongs to
> > + * @alloc: Store the request's file descriptor in this struct
> > + *
> > + * Allocated the media request and put the fd in @alloc->fd.
> > + */
> > +int media_request_alloc(struct media_device *mdev,
> > +			struct media_request_alloc *alloc);
> > +
> > +#else
> > +
> > +static inline void media_request_get(struct media_request *req)
> > +{
> > +}
> > +
> > +static inline void media_request_put(struct media_request *req)
> > +{
> > +}
> > +
> > +#endif
> > +
> > +/**
> > + * struct media_request_object_ops - Media request object operations
> > + * @prepare: Validate and prepare the request object, optional.
> > + * @unprepare: Unprepare the request object, optional.
> > + * @queue: Queue the request object, optional.
> > + * @unbind: Unbind the request object, optional.
> > + * @release: Release the request object, required.
> > + */
> > +struct media_request_object_ops {
> > +	int (*prepare)(struct media_request_object *object);
> > +	void (*unprepare)(struct media_request_object *object);
> > +	void (*queue)(struct media_request_object *object);
> > +	void (*unbind)(struct media_request_object *object);
> > +	void (*release)(struct media_request_object *object);
> > +};
> > +
> > +/**
> > + * struct media_request_object - An opaque object that belongs to a media
> > + *				 request
> > + *
> > + * @ops: object's operations
> > + * @priv: object's priv pointer
> > + * @req: the request this object belongs to (can be NULL)
> > + * @list: List entry of the object for @struct media_request
> > + * @kref: Reference count of the object, acquire before releasing req->lock
> > + * @completed: If true, then this object was completed.
> > + *
> > + * An object related to the request. This struct is embedded in the
> > + * larger object data.

what do you mean by "the larger object data"? What struct is "the" struct?

> > + */
> > +struct media_request_object {
> > +	const struct media_request_object_ops *ops;
> > +	void *priv;
> > +	struct media_request *req;
> > +	struct list_head list;
> > +	struct kref kref;
> > +	bool completed;
> > +};
> > +
> > +#ifdef CONFIG_MEDIA_CONTROLLER
> > +
> > +/**
> > + * media_request_object_get - Get a media request object
> > + *
> > + * @obj: The object
> > + *
> > + * Get a media request object.
> > + */
> > +static inline void media_request_object_get(struct media_request_object *obj)
> > +{
> > +	kref_get(&obj->kref);
> > +}
> > +
> > +/**
> > + * media_request_object_put - Put a media request object
> > + *
> > + * @obj: The object
> > + *
> > + * Put a media request object. Once all references are gone, the
> > + * object's memory is released.
> > + */
> > +void media_request_object_put(struct media_request_object *obj);
> > +
> > +/**
> > + * media_request_object_init - Initialise a media request object
> > + *
> > + * Initialise a media request object. The object will be released using the
> > + * release callback of the ops once it has no references (this function
> > + * initialises references to one).
> > + */
> > +void media_request_object_init(struct media_request_object *obj);
> > +
> > +/**
> > + * media_request_object_bind - Bind a media request object to a request  
> 
> Argument documentation is missing.
> 
> I think you should also say that "every bound object must be unbound later
> on".
> 
> > + */
> > +int media_request_object_bind(struct media_request *req,
> > +			      const struct media_request_object_ops *ops,
> > +			      void *priv,
> > +			      struct media_request_object *obj);
> > +
> > +/**
> > + * media_request_object_unbind - Unbind a media request object
> > + *
> > + * @obj: The object
> > + *
> > + * Unbind the media request object from the request.
> > + */
> > +void media_request_object_unbind(struct media_request_object *obj);
> > +
> > +/**
> > + * media_request_object_complete - Mark the media request object as complete
> > + *
> > + * @obj: The object
> > + *
> > + * Mark the media request object as complete.  
> 
> Add:
> 
> Only bound request objects may be completed.
> 
> > + */
> > +void media_request_object_complete(struct media_request_object *obj);
> > +
> > +#else
> > +
> > +static inline void media_request_object_get(struct media_request_object *obj)
> > +{
> > +}
> > +
> > +static inline void media_request_object_put(struct media_request_object *obj)
> > +{
> > +}
> > +
> > +static inline void media_request_object_init(struct media_request_object *obj)
> > +{
> > +	obj->ops = NULL;
> > +	obj->req = NULL;
> > +}
> > +
> > +static inline int media_request_object_bind(struct media_request *req,
> > +			       const struct media_request_object_ops *ops,
> > +			       void *priv,
> > +			       struct media_request_object *obj)
> > +{
> > +	return 0;
> > +}
> > +
> > +static inline void media_request_object_unbind(struct media_request_object *obj)
> > +{
> > +}
> > +
> > +static inline void media_request_object_complete(struct media_request_object *obj)
> > +{
> > +}
> > +
> > +#endif
> > +
> > +#endif  
> 



Thanks,
Mauro

Re: [PATCHv13 00/28] Request API

[Mauro Carvalho Chehab]

Em Thu,  3 May 2018 16:52:50 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> From: Hans Verkuil <hans.verkuil@cisco.com>

> Regarding locking:
> 
> There are two request locks used: req_queue_mutex (part of media_device)
> ensures high-level serialization of queueing/reiniting and canceling
> requests. It serializes STREAMON/OFF, TRY/S_EXT_CTRLS, close() and
> MEDIA_REQUEST_IOC_QUEUE. This is the top-level lock and should be taken
> before any others.

Why it doesn't serialize poll()? 

> The lock spin_lock in struct media_request protects that structure and
> should be held for a short time only.
> 
> Note that VIDIOC_QBUF does not take this mutex: when
> a buffer is queued for a request it will add it to the
> request by calling media_request_object_bind(), and that returns an
> error if the request is in the wrong state. It is serialized via the
> spin_lock which in this specific case is sufficient.

It looks weird that some syscalls are not serialized. By not having
poll() and VIDIOC_QBUF serialized, I'm wandering if playing with
them and close() on separate threads won't cause bad locking. 

Thanks,
Mauro

Re: [PATCHv13 04/28] media-request: add media_request_get_by_fd

[Mauro Carvalho Chehab]

Em Tue, 8 May 2018 09:34:11 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On 05/07/2018 07:01 PM, Mauro Carvalho Chehab wrote:
> > Em Thu,  3 May 2018 16:52:54 +0200
> > Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> >   
> >> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>
> >> Add media_request_get_by_fd() to find a request based on the file
> >> descriptor.
> >>
> >> The caller has to call media_request_put() for the returned
> >> request since this function increments the refcount.
> >>
> >> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >> ---
> >>  drivers/media/media-request.c | 32 ++++++++++++++++++++++++++++++++
> >>  include/media/media-request.h | 24 ++++++++++++++++++++++++
> >>  2 files changed, 56 insertions(+)
> >>
> >> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> >> index c216c4ab628b..edc1c3af1959 100644
> >> --- a/drivers/media/media-request.c
> >> +++ b/drivers/media/media-request.c
> >> @@ -218,6 +218,38 @@ static const struct file_operations request_fops = {
> >>  	.release = media_request_close,
> >>  };
> >>  
> >> +struct media_request *
> >> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
> >> +{
> >> +	struct file *filp;
> >> +	struct media_request *req;
> >> +
> >> +	if (!mdev || !mdev->ops ||
> >> +	    !mdev->ops->req_validate || !mdev->ops->req_queue)
> >> +		return ERR_PTR(-EPERM);
> >> +
> >> +	filp = fget(request_fd);
> >> +	if (!filp)
> >> +		return ERR_PTR(-ENOENT);
> >> +
> >> +	if (filp->f_op != &request_fops)
> >> +		goto err_fput;
> >> +	req = filp->private_data;
> >> +	if (req->mdev != mdev)
> >> +		goto err_fput;
> >> +
> >> +	media_request_get(req);  
> > 
> > Hmm... this function changes the req struct (by calling kref_get) without
> > holding neither the mutex or the spin lock...  
> 
> kref_get is atomic, so why would you need to add additional locking?
> 
> Neither can the request be 'put' by another process before media_request_get()
> is called due to the fget() above: while the fd has a refcount > 0 the
> request refcount is also > 0. I'll add some comments to clarify this.

I see. Yeah, please add a comment here.

Thanks,
Mauro





> 
> >   
> >> +	fput(filp);
> >> +
> >> +	return req;
> >> +
> >> +err_fput:
> >> +	fput(filp);
> >> +
> >> +	return ERR_PTR(-ENOENT);
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_get_by_fd);
> >> +
> >>  int media_request_alloc(struct media_device *mdev,
> >>  			struct media_request_alloc *alloc)
> >>  {
> >> diff --git a/include/media/media-request.h b/include/media/media-request.h
> >> index e39122dfd717..997e096d7128 100644
> >> --- a/include/media/media-request.h
> >> +++ b/include/media/media-request.h
> >> @@ -86,6 +86,24 @@ static inline void media_request_get(struct media_request *req)
> >>   */
> >>  void media_request_put(struct media_request *req);
> >>  
> >> +/**
> >> + * media_request_get_by_fd - Get a media request by fd
> >> + *
> >> + * @mdev: Media device this request belongs to
> >> + * @request_fd: The file descriptor of the request
> >> + *
> >> + * Get the request represented by @request_fd that is owned
> >> + * by the media device.
> >> + *
> >> + * Return a -EPERM error pointer if requests are not supported
> >> + * by this driver. Return -ENOENT if the request was not found.
> >> + * Return the pointer to the request if found: the caller will
> >> + * have to call @media_request_put when it finished using the
> >> + * request.  
> > 
> > ... so, it should be said here how this should be serialized, in order
> > to avoid it to be destroyed by a task while some other task might be
> > trying to instantiate it.  
> 
> This doesn't need any serialization. If the request_fd is found, then
> it will return the request with the request refcount increased.
> 
> I also do not understand what you mean with "some other task might be
> trying to instantiate it".
> 
> I think there is some misunderstanding here.
> 
> Regards,
> 
> 	Hans
> 
> 
> >   
> >> + */
> >> +struct media_request *
> >> +media_request_get_by_fd(struct media_device *mdev, int request_fd);
> >> +
> >>  /**
> >>   * media_request_alloc - Allocate the media request
> >>   *
> >> @@ -107,6 +125,12 @@ static inline void media_request_put(struct media_request *req)
> >>  {
> >>  }
> >>  
> >> +static inline struct media_request *
> >> +media_request_get_by_fd(struct media_device *mdev, int request_fd)
> >> +{
> >> +	return ERR_PTR(-EPERM);
> >> +}
> >> +
> >>  #endif
> >>  
> >>  /**  
> > 
> > 
> > 
> > Thanks,
> > Mauro
> >   
> 



Thanks,
Mauro

Re: [PATCHv13 06/28] v4l2-dev: lock req_queue_mutex

[Mauro Carvalho Chehab]

Em Tue, 8 May 2018 09:45:27 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> On 05/07/2018 07:20 PM, Mauro Carvalho Chehab wrote:
> > Em Thu,  3 May 2018 16:52:56 +0200
> > Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> >   
> >> From: Hans Verkuil <hans.verkuil@cisco.com>
> >>
> >> We need to serialize streamon/off with queueing new requests.
> >> These ioctls may trigger the cancellation of a streaming
> >> operation, and that should not be mixed with queuing a new
> >> request at the same time.
> >>
> >> Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
> >> with MEDIA_REQUEST_IOC_QUEUE.
> >>
> >> Finally close() needs this lock since that too can trigger the
> >> cancellation of a streaming operation.
> >>
> >> We take the req_queue_mutex here before any other locks since
> >> it is a very high-level lock.
> >>
> >> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> >> ---
> >>  drivers/media/v4l2-core/v4l2-dev.c | 37 +++++++++++++++++++++++++++++-
> >>  1 file changed, 36 insertions(+), 1 deletion(-)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> >> index 1d0b2208e8fb..b1c9efc0ecc4 100644
> >> --- a/drivers/media/v4l2-core/v4l2-dev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> >> @@ -353,13 +353,36 @@ static long v4l2_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
> >>  
> >>  	if (vdev->fops->unlocked_ioctl) {
> >>  		struct mutex *lock = v4l2_ioctl_get_lock(vdev, cmd);
> >> +		struct mutex *queue_lock = NULL;
> >>  
> >> -		if (lock && mutex_lock_interruptible(lock))
> >> +		/*
> >> +		 * We need to serialize streamon/off with queueing new requests.
> >> +		 * These ioctls may trigger the cancellation of a streaming
> >> +		 * operation, and that should not be mixed with queueing a new
> >> +		 * request at the same time.
> >> +		 *
> >> +		 * Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
> >> +		 * with MEDIA_REQUEST_IOC_QUEUE.
> >> +		 */
> >> +		if (vdev->v4l2_dev->mdev &&
> >> +		    (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF ||
> >> +		     cmd == VIDIOC_S_EXT_CTRLS || cmd == VIDIOC_TRY_EXT_CTRLS))
> >> +			queue_lock = &vdev->v4l2_dev->mdev->req_queue_mutex;
> >> +
> >> +		if (queue_lock && mutex_lock_interruptible(queue_lock))
> >> +			return -ERESTARTSYS;  
> > 
> > Taking both locks seems risky. Here you're taking first v4l2 lock, returned
> > by v4l2_ioctl_get_lock(vdev, cmd), and then you're taking the req_queue lock.  
> 
> No,  v4l2_ioctl_get_lock() only returns a pointer to a mutex, it doesn't lock
> anything. I think you got confused there. I'll reorganize the code a bit so
> the call to  v4l2_ioctl_get_lock() happens after the queue_lock has been taken.

Yeah, I didn't actually look at the implementation of v4l2_ioctl_get_lock().

As we're using "_get" along this patch series to increment krefs (with is
a sort of locking), the name here confused me. IMHO, we should rename it
to v4l2_ioctl_return_lock() (or similar) on some future, in order to avoid
confusion.

> I'll also rename queue_lock to req_queue_lock (it's a bit more descriptive).

Agreed.

> 
> So we first take the high-level media_device req_queue_mutex if needed, and
> then the ioctl serialization lock. Doing it the other way around will indeed
> promptly deadlock (as I very quickly discovered after my initial implementation!).
> 
> So the order is:
> 
> 	req_queue_mutex (serialize request state changes from/to IDLE)
> 	ioctl lock (serialize ioctls)
> 	request->lock (spinlock)
> 
> The last is only held for short periods when updating the media_request struct.
> 
> > 
> > It is possible to call parts of the code that only handles req_queue
> > or v4l2 lock (for example, by mixing request API calls with non-requests
> > one). Worse than that, there are parts of the code where the request API
> > patches get both a mutex and a spin lock.
> > 
> > I didn't look too closely (nor ran any test), but I'm almost sure that
> > there are paths where it will end by leading into dead locks.  
> 
> I've done extensive testing with this and actually been very careful about
> the lock handling. It's also been tested with the cedrus driver.

I don't doubt it works using your apps, but real life can be messier:
people could be issuing ioctls at different orders, programs can abort
any time, closing file descriptors at random times, threads can be
used to paralelize ioctls, etc.

That not discarding the possibility of someone coming with some
ingenious code meant to cause machine hangups or to exposure some
other security flaws.



Thanks,
Mauro

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Mauro Carvalho Chehab]

Em Tue, 8 May 2018 10:07:22 +0200
Hans Verkuil <hverkuil@xs4all.nl> escreveu:

> >> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> >> index 76352eb59f14..a0f7c38d1a90 100644
> >> --- a/include/media/v4l2-ctrls.h
> >> +++ b/include/media/v4l2-ctrls.h
> >> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
> >>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
> >>   * @from_other_dev: If true, then @ctrl was defined in another
> >>   *		device than the &struct v4l2_ctrl_handler.
> >> + * @done:	If true, then this control reference is part of a
> >> + *		control cluster that was already set while applying
> >> + *		the controls in this media request object.  
> > 
> > Hmm... I would rename it for request_done or something similar, as it
> > seems that this applies only for requests.  
> 
> No, the variable name is correct (it serves the same purpose as the 'done'
> field in struct v4l2_ctrl), but the description should be improved.
> 
> I also want to take another look at this: I wonder if it isn't possible to
> use the v4l2_ctrl 'done' field for this instead of having to add a new field
> here.

If it can use v4l2_ctrl done, it would indeed be better. Otherwise, as
this new "done" is used only by requests, I really think that it shold
be renamed, to let it clearer that this is the "done" that should be used
when requests are used.

Thanks,
Mauro

Re: [PATCHv13 03/28] media-request: implement media requests

[Sakari Ailus]

Hi Mauro, Hans,

On Tue, May 08, 2018 at 07:21:16AM -0300, Mauro Carvalho Chehab wrote:
> Em Fri, 4 May 2018 15:27:50 +0300
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
> > Hi Hans,
> > 
> > I've read this patch a large number of times and I think also the details
> > begin to seem sound. A few comments below.
> 
> I'm sending this after analyzing the other patches in this series,
> as this is the core of the changes. So, although I wrote the comments
> early, I wanted to read first all other patches before sending it.
> 
> > 
> > On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:
> > > From: Hans Verkuil <hans.verkuil@cisco.com>
> > > 
> > > Add initial media request support:
> > > 
> > > 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> > > 2) Add struct media_request to store request objects.
> > > 3) Add struct media_request_object to represent a request object.
> > > 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> > > 
> > > Basic lifecycle: the application allocates a request, adds
> > > objects to it, queues the request, polls until it is completed
> > > and can then read the final values of the objects at the time
> > > of completion. When it closes the file descriptor the request
> > > memory will be freed (actually, when the last user of that request
> > > releases the request).
> > > 
> > > Drivers will bind an object to a request (the 'adds objects to it'
> > > phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> > > validated (req_validate op), then queued (the req_queue op).
> > > 
> > > When done with an object it can either be unbound from the request
> > > (e.g. when the driver has finished with a vb2 buffer) or marked as
> > > completed (e.g. for controls associated with a buffer). When all
> > > objects in the request are completed (or unbound), then the request
> > > fd will signal an exception (poll).
> > > 
> > > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Hmm... As you're adding Copyrights from Intel/Google in this patch, that
> indicates that part of the stuff you're adding here were authored by
> others. So, you should use Co-developed-by: tag here, and get the SOBs
> from the other developers that did part of the work[1].
> 
> [1] except if your work was sponsored by Cisco, Intel and Google, but
>     I think this is not the case.

I think this could be appropriate:

    Co-developed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
    Co-developed-by: Sakari Ailus <sakari.ailus@linux.intel.com>
    Co-developed-by: Alexandre Courbot <acourbot@chromium.org>

...

> > > +static long media_request_ioctl_queue(struct media_request *req)
> > > +{
> > > +	struct media_device *mdev = req->mdev;
> > > +	enum media_request_state state;
> > > +	unsigned long flags;
> > > +	int ret = 0;  
> > 
> > ret is unconditionally assigned below, no need to initialise here.
> > 
> > > +
> > > +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
> > > +
> > > +	/*
> > > +	 * Ensure the request that is validated will be the one that gets queued
> > > +	 * next by serialising the queueing process. This mutex is also used
> > > +	 * to serialize with canceling a vb2 queue and with setting values such
> > > +	 * as controls in a request.
> > > +	 */
> > > +	mutex_lock(&mdev->req_queue_mutex);
> > > +
> > > +	spin_lock_irqsave(&req->lock, flags);
> > > +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
> > > +			       MEDIA_REQUEST_STATE_VALIDATING);
> > > +	spin_unlock_irqrestore(&req->lock, flags);
> 
> It looks weird to serialize access to it with a mutex, a spin lock and 
> an atomic call.

req->lock is needed for state changes from idle to other states as also
other struct members need to be serialised with that, with the request
state. Which is kind of in line with my earlier point: there's little
if any benefit in making this field atomic.

The mutex is there to ensure only a single request remains in validating
state, i.e. we will be changing the state of the device request tip one
request at a time.

> 
> IMHO, locking is still an issue here. I would love to test the 
> locks with some tool that would randomize syscalls, issuing close(),
> poll() and read() at wrong times and inverting the order of some calls, 
> in order to do some empiric test that all locks are at the right places.
> 
> Complex locking schemas like that usually tend to cause a lot of
> troubles.
> 
> > > +	if (state != MEDIA_REQUEST_STATE_IDLE) {
> > > +		dev_dbg(mdev->dev,
> > > +			"request: unable to queue %s, request in state %s\n",
> > > +			req->debug_str, media_request_state_str(state));
> > > +		mutex_unlock(&mdev->req_queue_mutex);
> > > +		return -EBUSY;
> > > +	}
> > > +
> > > +	ret = mdev->ops->req_validate(req);
> > > +
> > > +	/*
> > > +	 * If the req_validate was successful, then we mark the state as QUEUED
> > > +	 * and call req_queue. The reason we set the state first is that this
> > > +	 * allows req_queue to unbind or complete the queued objects in case
> > > +	 * they are immediately 'consumed'. State changes from QUEUED to another
> > > +	 * state can only happen if either the driver changes the state or if
> > > +	 * the user cancels the vb2 queue. The driver can only change the state
> > > +	 * after each object is queued through the req_queue op (and note that
> > > +	 * that op cannot fail), so setting the state to QUEUED up front is
> > > +	 * safe.
> > > +	 *
> > > +	 * The other reason for changing the state is if the vb2 queue is
> > > +	 * canceled, and that uses the req_queue_mutex which is still locked
> > > +	 * while req_queue is called, so that's safe as well.
> > > +	 */
> > > +	atomic_set(&req->state,
> > > +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);
> 
> Why are you changing state also when ret fails?
> 
> Also, why you had to use a spin lock earlier in this function just 
> to change the req->state but you don't need to use it here?

The reason is subtle: the operations that need spinlock protection can take
place in request states other than "validating".

...

> > > +/**
> > > + * struct media_request_object_ops - Media request object operations
> > > + * @prepare: Validate and prepare the request object, optional.
> > > + * @unprepare: Unprepare the request object, optional.
> > > + * @queue: Queue the request object, optional.
> > > + * @unbind: Unbind the request object, optional.
> > > + * @release: Release the request object, required.
> > > + */
> > > +struct media_request_object_ops {
> > > +	int (*prepare)(struct media_request_object *object);
> > > +	void (*unprepare)(struct media_request_object *object);
> > > +	void (*queue)(struct media_request_object *object);
> > > +	void (*unbind)(struct media_request_object *object);
> > > +	void (*release)(struct media_request_object *object);
> > > +};
> > > +
> > > +/**
> > > + * struct media_request_object - An opaque object that belongs to a media
> > > + *				 request
> > > + *
> > > + * @ops: object's operations
> > > + * @priv: object's priv pointer
> > > + * @req: the request this object belongs to (can be NULL)
> > > + * @list: List entry of the object for @struct media_request
> > > + * @kref: Reference count of the object, acquire before releasing req->lock
> > > + * @completed: If true, then this object was completed.
> > > + *
> > > + * An object related to the request. This struct is embedded in the
> > > + * larger object data.
> 
> what do you mean by "the larger object data"? What struct is "the" struct?

There is no particular type: the API offers generic binding of objects to a
request. The objects are later retrieved when validating, queueing and
implementing that request.

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 05/28] media-request: add media_request_object_find

[Sakari Ailus]

On Fri, May 04, 2018 at 03:43:07PM +0300, Sakari Ailus wrote:
> > diff --git a/include/media/media-request.h b/include/media/media-request.h
> > index 997e096d7128..5367b4a2f91c 100644
> > --- a/include/media/media-request.h
> > +++ b/include/media/media-request.h
> > @@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
> >   */
> >  void media_request_object_put(struct media_request_object *obj);
> >  
> > +/**
> > + * media_request_object_find - Find an object in a request
> > + *
> > + * @ops: Find an object with this ops value
> > + * @priv: Find an object with this priv value
> > + *
> > + * Both @ops and @priv must be non-NULL.
> > + *
> > + * Returns NULL if not found or the object pointer. The caller must
> 
> I'd describe the successful case first. I.e. "Returns the object pointer or
> NULL it not found".

Oops... "Returns the object pointer or NULL if not found".

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 03/28] media-request: implement media requests

[Mauro Carvalho Chehab]

Em Tue, 8 May 2018 13:52:33 +0300
Sakari Ailus <sakari.ailus@iki.fi> escreveu:

> Hi Mauro, Hans,
> 
> On Tue, May 08, 2018 at 07:21:16AM -0300, Mauro Carvalho Chehab wrote:
> > Em Fri, 4 May 2018 15:27:50 +0300
> > Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> >   
> > > Hi Hans,
> > > 
> > > I've read this patch a large number of times and I think also the details
> > > begin to seem sound. A few comments below.  
> > 
> > I'm sending this after analyzing the other patches in this series,
> > as this is the core of the changes. So, although I wrote the comments
> > early, I wanted to read first all other patches before sending it.
> >   
> > > 
> > > On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:  
> > > > From: Hans Verkuil <hans.verkuil@cisco.com>
> > > > 
> > > > Add initial media request support:
> > > > 
> > > > 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> > > > 2) Add struct media_request to store request objects.
> > > > 3) Add struct media_request_object to represent a request object.
> > > > 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> > > > 
> > > > Basic lifecycle: the application allocates a request, adds
> > > > objects to it, queues the request, polls until it is completed
> > > > and can then read the final values of the objects at the time
> > > > of completion. When it closes the file descriptor the request
> > > > memory will be freed (actually, when the last user of that request
> > > > releases the request).
> > > > 
> > > > Drivers will bind an object to a request (the 'adds objects to it'
> > > > phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> > > > validated (req_validate op), then queued (the req_queue op).
> > > > 
> > > > When done with an object it can either be unbound from the request
> > > > (e.g. when the driver has finished with a vb2 buffer) or marked as
> > > > completed (e.g. for controls associated with a buffer). When all
> > > > objects in the request are completed (or unbound), then the request
> > > > fd will signal an exception (poll).
> > > > 
> > > > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>  
> > 
> > Hmm... As you're adding Copyrights from Intel/Google in this patch, that
> > indicates that part of the stuff you're adding here were authored by
> > others. So, you should use Co-developed-by: tag here, and get the SOBs
> > from the other developers that did part of the work[1].
> > 
> > [1] except if your work was sponsored by Cisco, Intel and Google, but
> >     I think this is not the case.  
> 
> I think this could be appropriate:
> 
>     Co-developed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
>     Co-developed-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>     Co-developed-by: Alexandre Courbot <acourbot@chromium.org>
> 
> ...
> 
> > > > +static long media_request_ioctl_queue(struct media_request *req)
> > > > +{
> > > > +	struct media_device *mdev = req->mdev;
> > > > +	enum media_request_state state;
> > > > +	unsigned long flags;
> > > > +	int ret = 0;    
> > > 
> > > ret is unconditionally assigned below, no need to initialise here.
> > >   
> > > > +
> > > > +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
> > > > +
> > > > +	/*
> > > > +	 * Ensure the request that is validated will be the one that gets queued
> > > > +	 * next by serialising the queueing process. This mutex is also used
> > > > +	 * to serialize with canceling a vb2 queue and with setting values such
> > > > +	 * as controls in a request.
> > > > +	 */
> > > > +	mutex_lock(&mdev->req_queue_mutex);
> > > > +
> > > > +	spin_lock_irqsave(&req->lock, flags);
> > > > +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
> > > > +			       MEDIA_REQUEST_STATE_VALIDATING);
> > > > +	spin_unlock_irqrestore(&req->lock, flags);  
> > 
> > It looks weird to serialize access to it with a mutex, a spin lock and 
> > an atomic call.  
> 
> req->lock is needed for state changes from idle to other states as also
> other struct members need to be serialised with that, with the request
> state. Which is kind of in line with my earlier point: there's little
> if any benefit in making this field atomic.

In this case, only req->state change is serialized - although atomic
also serializes it.

> The mutex is there to ensure only a single request remains in validating
> state, i.e. we will be changing the state of the device request tip one
> request at a time.

Better to add a comment explaining it.

> > 
> > IMHO, locking is still an issue here. I would love to test the 
> > locks with some tool that would randomize syscalls, issuing close(),
> > poll() and read() at wrong times and inverting the order of some calls, 
> > in order to do some empiric test that all locks are at the right places.
> > 
> > Complex locking schemas like that usually tend to cause a lot of
> > troubles.
> >   
> > > > +	if (state != MEDIA_REQUEST_STATE_IDLE) {
> > > > +		dev_dbg(mdev->dev,
> > > > +			"request: unable to queue %s, request in state %s\n",
> > > > +			req->debug_str, media_request_state_str(state));
> > > > +		mutex_unlock(&mdev->req_queue_mutex);
> > > > +		return -EBUSY;
> > > > +	}
> > > > +
> > > > +	ret = mdev->ops->req_validate(req);
> > > > +
> > > > +	/*
> > > > +	 * If the req_validate was successful, then we mark the state as QUEUED
> > > > +	 * and call req_queue. The reason we set the state first is that this
> > > > +	 * allows req_queue to unbind or complete the queued objects in case
> > > > +	 * they are immediately 'consumed'. State changes from QUEUED to another
> > > > +	 * state can only happen if either the driver changes the state or if
> > > > +	 * the user cancels the vb2 queue. The driver can only change the state
> > > > +	 * after each object is queued through the req_queue op (and note that
> > > > +	 * that op cannot fail), so setting the state to QUEUED up front is
> > > > +	 * safe.
> > > > +	 *
> > > > +	 * The other reason for changing the state is if the vb2 queue is
> > > > +	 * canceled, and that uses the req_queue_mutex which is still locked
> > > > +	 * while req_queue is called, so that's safe as well.
> > > > +	 */
> > > > +	atomic_set(&req->state,
> > > > +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);  
> > 
> > Why are you changing state also when ret fails?
> > 
> > Also, why you had to use a spin lock earlier in this function just 
> > to change the req->state but you don't need to use it here?  
> 
> The reason is subtle: the operations that need spinlock protection can take
> place in request states other than "validating".

Sorry but I didn't get. Assuming that only req->state changes, it
should *either* use the spin lock or not. Doing it on some places
and not doing on others seems wrong.

Ok, if there are very good reasons why doing that, it should be 
documented, as people will see this as a mistake.

> 
> ...
> 
> > > > +/**
> > > > + * struct media_request_object_ops - Media request object operations
> > > > + * @prepare: Validate and prepare the request object, optional.
> > > > + * @unprepare: Unprepare the request object, optional.
> > > > + * @queue: Queue the request object, optional.
> > > > + * @unbind: Unbind the request object, optional.
> > > > + * @release: Release the request object, required.
> > > > + */
> > > > +struct media_request_object_ops {
> > > > +	int (*prepare)(struct media_request_object *object);
> > > > +	void (*unprepare)(struct media_request_object *object);
> > > > +	void (*queue)(struct media_request_object *object);
> > > > +	void (*unbind)(struct media_request_object *object);
> > > > +	void (*release)(struct media_request_object *object);
> > > > +};
> > > > +
> > > > +/**
> > > > + * struct media_request_object - An opaque object that belongs to a media
> > > > + *				 request
> > > > + *
> > > > + * @ops: object's operations
> > > > + * @priv: object's priv pointer
> > > > + * @req: the request this object belongs to (can be NULL)
> > > > + * @list: List entry of the object for @struct media_request
> > > > + * @kref: Reference count of the object, acquire before releasing req->lock
> > > > + * @completed: If true, then this object was completed.
> > > > + *
> > > > + * An object related to the request. This struct is embedded in the
> > > > + * larger object data.  
> > 
> > what do you mean by "the larger object data"? What struct is "the" struct?  
> 
> There is no particular type: the API offers generic binding of objects to a
> request. The objects are later retrieved when validating, queueing and
> implementing that request.

it if can be any type, then it should be saying instead something like: 

	"An object related to the request. This struct should be embedded
	 into a larger object data."

Thanks,
Mauro

Re: [PATCHv13 03/28] media-request: implement media requests

[Sakari Ailus]

Hi Mauro,

On Tue, May 08, 2018 at 09:41:03AM -0300, Mauro Carvalho Chehab wrote:
> Em Tue, 8 May 2018 13:52:33 +0300
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
> > Hi Mauro, Hans,
> > 
> > On Tue, May 08, 2018 at 07:21:16AM -0300, Mauro Carvalho Chehab wrote:
> > > Em Fri, 4 May 2018 15:27:50 +0300
> > > Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> > >   
> > > > Hi Hans,
> > > > 
> > > > I've read this patch a large number of times and I think also the details
> > > > begin to seem sound. A few comments below.  
> > > 
> > > I'm sending this after analyzing the other patches in this series,
> > > as this is the core of the changes. So, although I wrote the comments
> > > early, I wanted to read first all other patches before sending it.
> > >   
> > > > 
> > > > On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:  
> > > > > From: Hans Verkuil <hans.verkuil@cisco.com>
> > > > > 
> > > > > Add initial media request support:
> > > > > 
> > > > > 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
> > > > > 2) Add struct media_request to store request objects.
> > > > > 3) Add struct media_request_object to represent a request object.
> > > > > 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
> > > > > 
> > > > > Basic lifecycle: the application allocates a request, adds
> > > > > objects to it, queues the request, polls until it is completed
> > > > > and can then read the final values of the objects at the time
> > > > > of completion. When it closes the file descriptor the request
> > > > > memory will be freed (actually, when the last user of that request
> > > > > releases the request).
> > > > > 
> > > > > Drivers will bind an object to a request (the 'adds objects to it'
> > > > > phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
> > > > > validated (req_validate op), then queued (the req_queue op).
> > > > > 
> > > > > When done with an object it can either be unbound from the request
> > > > > (e.g. when the driver has finished with a vb2 buffer) or marked as
> > > > > completed (e.g. for controls associated with a buffer). When all
> > > > > objects in the request are completed (or unbound), then the request
> > > > > fd will signal an exception (poll).
> > > > > 
> > > > > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>  
> > > 
> > > Hmm... As you're adding Copyrights from Intel/Google in this patch, that
> > > indicates that part of the stuff you're adding here were authored by
> > > others. So, you should use Co-developed-by: tag here, and get the SOBs
> > > from the other developers that did part of the work[1].
> > > 
> > > [1] except if your work was sponsored by Cisco, Intel and Google, but
> > >     I think this is not the case.  
> > 
> > I think this could be appropriate:
> > 
> >     Co-developed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
> >     Co-developed-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> >     Co-developed-by: Alexandre Courbot <acourbot@chromium.org>
> > 
> > ...
> > 
> > > > > +static long media_request_ioctl_queue(struct media_request *req)
> > > > > +{
> > > > > +	struct media_device *mdev = req->mdev;
> > > > > +	enum media_request_state state;
> > > > > +	unsigned long flags;
> > > > > +	int ret = 0;    
> > > > 
> > > > ret is unconditionally assigned below, no need to initialise here.
> > > >   
> > > > > +
> > > > > +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
> > > > > +
> > > > > +	/*
> > > > > +	 * Ensure the request that is validated will be the one that gets queued
> > > > > +	 * next by serialising the queueing process. This mutex is also used
> > > > > +	 * to serialize with canceling a vb2 queue and with setting values such
> > > > > +	 * as controls in a request.
> > > > > +	 */
> > > > > +	mutex_lock(&mdev->req_queue_mutex);
> > > > > +
> > > > > +	spin_lock_irqsave(&req->lock, flags);
> > > > > +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
> > > > > +			       MEDIA_REQUEST_STATE_VALIDATING);
> > > > > +	spin_unlock_irqrestore(&req->lock, flags);  
> > > 
> > > It looks weird to serialize access to it with a mutex, a spin lock and 
> > > an atomic call.  
> > 
> > req->lock is needed for state changes from idle to other states as also
> > other struct members need to be serialised with that, with the request
> > state. Which is kind of in line with my earlier point: there's little
> > if any benefit in making this field atomic.
> 
> In this case, only req->state change is serialized - although atomic
> also serializes it.

Correct. But other fields also need to be serialised --- together with the
state change. Therefore relying on atomic only is not an option.

> 
> > The mutex is there to ensure only a single request remains in validating
> > state, i.e. we will be changing the state of the device request tip one
> > request at a time.
> 
> Better to add a comment explaining it.

I agree. This should probably got to the documentation of the struct
itself. How about this:

@req_queue_mutex: Serialise validating and queueing requests in order to
guarantee exclusive access to the state of the device on the tip of the
request queue.

> 
> > > 
> > > IMHO, locking is still an issue here. I would love to test the 
> > > locks with some tool that would randomize syscalls, issuing close(),
> > > poll() and read() at wrong times and inverting the order of some calls, 
> > > in order to do some empiric test that all locks are at the right places.
> > > 
> > > Complex locking schemas like that usually tend to cause a lot of
> > > troubles.
> > >   
> > > > > +	if (state != MEDIA_REQUEST_STATE_IDLE) {
> > > > > +		dev_dbg(mdev->dev,
> > > > > +			"request: unable to queue %s, request in state %s\n",
> > > > > +			req->debug_str, media_request_state_str(state));
> > > > > +		mutex_unlock(&mdev->req_queue_mutex);
> > > > > +		return -EBUSY;
> > > > > +	}
> > > > > +
> > > > > +	ret = mdev->ops->req_validate(req);
> > > > > +
> > > > > +	/*
> > > > > +	 * If the req_validate was successful, then we mark the state as QUEUED
> > > > > +	 * and call req_queue. The reason we set the state first is that this
> > > > > +	 * allows req_queue to unbind or complete the queued objects in case
> > > > > +	 * they are immediately 'consumed'. State changes from QUEUED to another
> > > > > +	 * state can only happen if either the driver changes the state or if
> > > > > +	 * the user cancels the vb2 queue. The driver can only change the state
> > > > > +	 * after each object is queued through the req_queue op (and note that
> > > > > +	 * that op cannot fail), so setting the state to QUEUED up front is
> > > > > +	 * safe.
> > > > > +	 *
> > > > > +	 * The other reason for changing the state is if the vb2 queue is
> > > > > +	 * canceled, and that uses the req_queue_mutex which is still locked
> > > > > +	 * while req_queue is called, so that's safe as well.
> > > > > +	 */
> > > > > +	atomic_set(&req->state,
> > > > > +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);  
> > > 
> > > Why are you changing state also when ret fails?
> > > 
> > > Also, why you had to use a spin lock earlier in this function just 
> > > to change the req->state but you don't need to use it here?  
> > 
> > The reason is subtle: the operations that need spinlock protection can take
> > place in request states other than "validating".
> 
> Sorry but I didn't get. Assuming that only req->state changes, it
> should *either* use the spin lock or not. Doing it on some places
> and not doing on others seems wrong.
> 
> Ok, if there are very good reasons why doing that, it should be 
> documented, as people will see this as a mistake.

Documenting this better is an option. What would you think of making the
state an enum, and always requiring taking the spinlock to access it? There
would be no special cases anymore, and one way less to serialise access to
the data structures (atomic operations).

> 
> > 
> > ...
> > 
> > > > > +/**
> > > > > + * struct media_request_object_ops - Media request object operations
> > > > > + * @prepare: Validate and prepare the request object, optional.
> > > > > + * @unprepare: Unprepare the request object, optional.
> > > > > + * @queue: Queue the request object, optional.
> > > > > + * @unbind: Unbind the request object, optional.
> > > > > + * @release: Release the request object, required.
> > > > > + */
> > > > > +struct media_request_object_ops {
> > > > > +	int (*prepare)(struct media_request_object *object);
> > > > > +	void (*unprepare)(struct media_request_object *object);
> > > > > +	void (*queue)(struct media_request_object *object);
> > > > > +	void (*unbind)(struct media_request_object *object);
> > > > > +	void (*release)(struct media_request_object *object);
> > > > > +};
> > > > > +
> > > > > +/**
> > > > > + * struct media_request_object - An opaque object that belongs to a media
> > > > > + *				 request
> > > > > + *
> > > > > + * @ops: object's operations
> > > > > + * @priv: object's priv pointer
> > > > > + * @req: the request this object belongs to (can be NULL)
> > > > > + * @list: List entry of the object for @struct media_request
> > > > > + * @kref: Reference count of the object, acquire before releasing req->lock
> > > > > + * @completed: If true, then this object was completed.
> > > > > + *
> > > > > + * An object related to the request. This struct is embedded in the
> > > > > + * larger object data.  
> > > 
> > > what do you mean by "the larger object data"? What struct is "the" struct?  
> > 
> > There is no particular type: the API offers generic binding of objects to a
> > request. The objects are later retrieved when validating, queueing and
> > implementing that request.
> 
> it if can be any type, then it should be saying instead something like: 
> 
> 	"An object related to the request. This struct should be embedded
> 	 into a larger object data."

How about:

	"An object related to the request. This struct is embedded in a
	larger object that contains information related to a given
	request."

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 01/28] v4l2-device.h: always expose mdev

[Laurent Pinchart]

Hi Hans,

Thank you for the patch.

On Thursday, 3 May 2018 17:52:51 EEST Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> The mdev field is only present if CONFIG_MEDIA_CONTROLLER is set.
> But since we will need to pass the media_device to vb2 and the
> control framework it is very convenient to just make this field
> available all the time. If CONFIG_MEDIA_CONTROLLER is not set,
> then it will just be NULL.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>

Acked-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  include/media/v4l2-device.h | 4 +---
>  1 file changed, 1 insertion(+), 3 deletions(-)
> 
> diff --git a/include/media/v4l2-device.h b/include/media/v4l2-device.h
> index 0c9e4da55499..b330e4a08a6b 100644
> --- a/include/media/v4l2-device.h
> +++ b/include/media/v4l2-device.h
> @@ -33,7 +33,7 @@ struct v4l2_ctrl_handler;
>   * struct v4l2_device - main struct to for V4L2 device drivers
>   *
>   * @dev: pointer to struct device.
> - * @mdev: pointer to struct media_device
> + * @mdev: pointer to struct media_device, may be NULL.
>   * @subdevs: used to keep track of the registered subdevs
>   * @lock: lock this struct; can be used by the driver as well
>   *	if this struct is embedded into a larger struct.
> @@ -58,9 +58,7 @@ struct v4l2_ctrl_handler;
>   */
>  struct v4l2_device {
>  	struct device *dev;
> -#if defined(CONFIG_MEDIA_CONTROLLER)
>  	struct media_device *mdev;
> -#endif
>  	struct list_head subdevs;
>  	spinlock_t lock;
>  	char name[V4L2_DEVICE_NAME_SIZE];


-- 
Regards,

Laurent Pinchart

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Sakari Ailus]

Hi Hans,

On Thu, May 03, 2018 at 04:53:02PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Integrate the request support. This adds the v4l2_ctrl_request_complete
> and v4l2_ctrl_request_setup functions to complete a request and (as a
> helper function) to apply a request to the hardware.
> 
> It takes care of queuing requests and correctly chaining control values
> in the request queue.
> 
> Note that when a request is marked completed it will copy control values
> to the internal request state. This can be optimized in the future since
> this is sub-optimal when dealing with large compound and/or array controls.
> 
> For the initial 'stateless codec' use-case the current implementation is
> sufficient.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
>  include/media/v4l2-ctrls.h           |  23 ++
>  2 files changed, 348 insertions(+), 6 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> index da4cc1485dc4..56b986185463 100644
> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> @@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
>  }
>  EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
>  
> +void v4l2_ctrl_request_complete(struct media_request *req,
> +				struct v4l2_ctrl_handler *main_hdl)
> +{
> +	struct media_request_object *obj;
> +	struct v4l2_ctrl_handler *hdl;
> +	struct v4l2_ctrl_ref *ref;
> +
> +	if (!req || !main_hdl)
> +		return;
> +
> +	obj = media_request_object_find(req, &req_ops, main_hdl);
> +	if (!obj)
> +		return;
> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
> +		struct v4l2_ctrl *ctrl = ref->ctrl;
> +		struct v4l2_ctrl *master = ctrl->cluster[0];
> +		unsigned int i;
> +
> +		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
> +			ref->req = ref;
> +
> +			v4l2_ctrl_lock(master);
> +			/* g_volatile_ctrl will update the current control values */
> +			for (i = 0; i < master->ncontrols; i++)
> +				cur_to_new(master->cluster[i]);
> +			call_op(master, g_volatile_ctrl);
> +			new_to_req(ref);
> +			v4l2_ctrl_unlock(master);
> +			continue;
> +		}
> +		if (ref->req == ref)
> +			continue;
> +
> +		v4l2_ctrl_lock(ctrl);
> +		if (ref->req)
> +			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
> +		else
> +			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
> +		v4l2_ctrl_unlock(ctrl);
> +	}
> +
> +	WARN_ON(!hdl->request_is_queued);
> +	list_del_init(&hdl->requests_queued);
> +	hdl->request_is_queued = false;
> +	media_request_object_complete(obj);
> +	media_request_object_put(obj);
> +}
> +EXPORT_SYMBOL(v4l2_ctrl_request_complete);
> +
> +void v4l2_ctrl_request_setup(struct media_request *req,
> +			     struct v4l2_ctrl_handler *main_hdl)

Drivers are expected to use this function internally to make use of the
control values in the request. Is that your thinking as well?

The problem with this implementation is that once a driver eventually gets
a callback (s_ctrl), the callback doesn't have the information on the
request. That means the driver has no means to associate the control value
to the request anymore --- and that is against the very purpose of the
function.

Instead, I'd add a new argument to the callback function --- the request
--- or add another callback function to be used for applying control values
for requests. Or alternatively, provide an easy way to enumerate the
controls and their values in a control handler. For the driver must store
that value in the request itself to be able to use it: the current
implementation in vim2m is such that controls are simply set to the driver
as they'd arrive from the uAPI directly. This way also anyone having access
to the video device could set whatever control values that would end up
being used in processing the request.

Feel free to point out if I'm mistaken in my analysis.

> +{
> +	struct media_request_object *obj;
> +	struct v4l2_ctrl_handler *hdl;
> +	struct v4l2_ctrl_ref *ref;
> +
> +	if (!req || !main_hdl)
> +		return;
> +
> +	obj = media_request_object_find(req, &req_ops, main_hdl);
> +	if (!obj)
> +		return;
> +	if (obj->completed) {
> +		media_request_object_put(obj);
> +		return;
> +	}
> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
> +
> +	mutex_lock(hdl->lock);
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node)
> +		ref->done = false;
> +
> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
> +		struct v4l2_ctrl *ctrl = ref->ctrl;
> +		struct v4l2_ctrl *master = ctrl->cluster[0];
> +		bool have_new_data = false;
> +		int i;
> +
> +		/*
> +		 * Skip if this control was already handled by a cluster.
> +		 * Skip button controls and read-only controls.
> +		 */
> +		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
> +		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
> +			continue;
> +
> +		v4l2_ctrl_lock(master);
> +		for (i = 0; i < master->ncontrols; i++) {
> +			if (master->cluster[i]) {
> +				struct v4l2_ctrl_ref *r =
> +					find_ref(hdl, master->cluster[i]->id);
> +
> +				if (r->req && r == r->req) {
> +					have_new_data = true;
> +					break;
> +				}
> +			}
> +		}
> +		if (!have_new_data) {
> +			v4l2_ctrl_unlock(master);
> +			continue;
> +		}
> +
> +		for (i = 0; i < master->ncontrols; i++) {
> +			if (master->cluster[i]) {
> +				struct v4l2_ctrl_ref *r =
> +					find_ref(hdl, master->cluster[i]->id);
> +
> +				req_to_new(r);
> +				master->cluster[i]->is_new = 1;
> +				r->done = true;
> +			}
> +		}
> +		/*
> +		 * For volatile autoclusters that are currently in auto mode
> +		 * we need to discover if it will be set to manual mode.
> +		 * If so, then we have to copy the current volatile values
> +		 * first since those will become the new manual values (which
> +		 * may be overwritten by explicit new values from this set
> +		 * of controls).
> +		 */
> +		if (master->is_auto && master->has_volatiles &&
> +		    !is_cur_manual(master)) {
> +			s32 new_auto_val = *master->p_new.p_s32;
> +
> +			/*
> +			 * If the new value == the manual value, then copy
> +			 * the current volatile values.
> +			 */
> +			if (new_auto_val == master->manual_mode_value)
> +				update_from_auto_cluster(master);
> +		}
> +
> +		try_or_set_cluster(NULL, master, true, 0);
> +
> +		v4l2_ctrl_unlock(master);
> +	}
> +
> +	mutex_unlock(hdl->lock);
> +	media_request_object_put(obj);
> +}
> +EXPORT_SYMBOL(v4l2_ctrl_request_setup);
> +
>  void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
>  {
>  	if (ctrl == NULL)
> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
> index 76352eb59f14..a0f7c38d1a90 100644
> --- a/include/media/v4l2-ctrls.h
> +++ b/include/media/v4l2-ctrls.h
> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>   * @from_other_dev: If true, then @ctrl was defined in another
>   *		device than the &struct v4l2_ctrl_handler.
> + * @done:	If true, then this control reference is part of a
> + *		control cluster that was already set while applying
> + *		the controls in this media request object.
> + * @req:	If set, this refers to another request that sets this control.
>   * @p_req:	The request value. Only used if the control handler
>   *		is bound to a media request.
>   *
> @@ -263,6 +267,8 @@ struct v4l2_ctrl_ref {
>  	struct v4l2_ctrl *ctrl;
>  	struct v4l2_ctrl_helper *helper;
>  	bool from_other_dev;
> +	bool done;
> +	struct v4l2_ctrl_ref *req;
>  	union v4l2_ctrl_ptr p_req;
>  };
>  
> @@ -287,6 +293,15 @@ struct v4l2_ctrl_ref {
>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>   * @nr_of_buckets: Total number of buckets in the array.
>   * @error:	The error code of the first failed control addition.
> + * @request_is_queued: True if the request was queued.
> + * @requests:	List to keep track of open control handler request objects.
> + *		For the parent control handler (@req_obj.req == NULL) this
> + *		is the list header. When the parent control handler is
> + *		removed, it has to unbind and put all these requests since
> + *		they refer to the parent.
> + * @requests_queued: List of the queued requests. This determines the order
> + *		in which these controls are applied. Once the request is
> + *		completed it is removed from this list.
>   * @req_obj:	The &struct media_request_object, used to link into a
>   *		&struct media_request.
>   */
> @@ -301,6 +316,9 @@ struct v4l2_ctrl_handler {
>  	void *notify_priv;
>  	u16 nr_of_buckets;
>  	int error;
> +	bool request_is_queued;
> +	struct list_head requests;
> +	struct list_head requests_queued;
>  	struct media_request_object req_obj;
>  };
>  
> @@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
>   */
>  __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
>  
> +void v4l2_ctrl_request_setup(struct media_request *req,
> +			     struct v4l2_ctrl_handler *hdl);
> +void v4l2_ctrl_request_complete(struct media_request *req,
> +				struct v4l2_ctrl_handler *hdl);
> +
>  /* Helpers for ioctl_ops */
>  
>  /**
> -- 
> 2.17.0
> 

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Sakari Ailus]

On Wed, May 16, 2018 at 01:19:34PM +0300, Sakari Ailus wrote:
> Hi Hans,
> 
> On Thu, May 03, 2018 at 04:53:02PM +0200, Hans Verkuil wrote:
> > From: Hans Verkuil <hans.verkuil@cisco.com>
> > 
> > Integrate the request support. This adds the v4l2_ctrl_request_complete
> > and v4l2_ctrl_request_setup functions to complete a request and (as a
> > helper function) to apply a request to the hardware.
> > 
> > It takes care of queuing requests and correctly chaining control values
> > in the request queue.
> > 
> > Note that when a request is marked completed it will copy control values
> > to the internal request state. This can be optimized in the future since
> > this is sub-optimal when dealing with large compound and/or array controls.
> > 
> > For the initial 'stateless codec' use-case the current implementation is
> > sufficient.
> > 
> > Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
> >  include/media/v4l2-ctrls.h           |  23 ++
> >  2 files changed, 348 insertions(+), 6 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
> > index da4cc1485dc4..56b986185463 100644
> > --- a/drivers/media/v4l2-core/v4l2-ctrls.c
> > +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
> > @@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
> >  }
> >  EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
> >  
> > +void v4l2_ctrl_request_complete(struct media_request *req,
> > +				struct v4l2_ctrl_handler *main_hdl)
> > +{
> > +	struct media_request_object *obj;
> > +	struct v4l2_ctrl_handler *hdl;
> > +	struct v4l2_ctrl_ref *ref;
> > +
> > +	if (!req || !main_hdl)
> > +		return;
> > +
> > +	obj = media_request_object_find(req, &req_ops, main_hdl);
> > +	if (!obj)
> > +		return;
> > +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
> > +
> > +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
> > +		struct v4l2_ctrl *ctrl = ref->ctrl;
> > +		struct v4l2_ctrl *master = ctrl->cluster[0];
> > +		unsigned int i;
> > +
> > +		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
> > +			ref->req = ref;
> > +
> > +			v4l2_ctrl_lock(master);
> > +			/* g_volatile_ctrl will update the current control values */
> > +			for (i = 0; i < master->ncontrols; i++)
> > +				cur_to_new(master->cluster[i]);
> > +			call_op(master, g_volatile_ctrl);
> > +			new_to_req(ref);
> > +			v4l2_ctrl_unlock(master);
> > +			continue;
> > +		}
> > +		if (ref->req == ref)
> > +			continue;
> > +
> > +		v4l2_ctrl_lock(ctrl);
> > +		if (ref->req)
> > +			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
> > +		else
> > +			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
> > +		v4l2_ctrl_unlock(ctrl);
> > +	}
> > +
> > +	WARN_ON(!hdl->request_is_queued);
> > +	list_del_init(&hdl->requests_queued);
> > +	hdl->request_is_queued = false;
> > +	media_request_object_complete(obj);
> > +	media_request_object_put(obj);
> > +}
> > +EXPORT_SYMBOL(v4l2_ctrl_request_complete);
> > +
> > +void v4l2_ctrl_request_setup(struct media_request *req,
> > +			     struct v4l2_ctrl_handler *main_hdl)
> 
> Drivers are expected to use this function internally to make use of the
> control values in the request. Is that your thinking as well?
> 
> The problem with this implementation is that once a driver eventually gets
> a callback (s_ctrl), the callback doesn't have the information on the
> request. That means the driver has no means to associate the control value
> to the request anymore --- and that is against the very purpose of the
> function.
> 
> Instead, I'd add a new argument to the callback function --- the request
> --- or add another callback function to be used for applying control values
> for requests. Or alternatively, provide an easy way to enumerate the
> controls and their values in a control handler. For the driver must store

To address this fully --- using S_EXT_CTRLS on uAPI to the control handler
should likely be prevented as long as there are request objects related to
that handler. Or at least request objects that are not completed.

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Hans Verkuil]

On 05/16/18 12:19, Sakari Ailus wrote:
> Hi Hans,
> 
> On Thu, May 03, 2018 at 04:53:02PM +0200, Hans Verkuil wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Integrate the request support. This adds the v4l2_ctrl_request_complete
>> and v4l2_ctrl_request_setup functions to complete a request and (as a
>> helper function) to apply a request to the hardware.
>>
>> It takes care of queuing requests and correctly chaining control values
>> in the request queue.
>>
>> Note that when a request is marked completed it will copy control values
>> to the internal request state. This can be optimized in the future since
>> this is sub-optimal when dealing with large compound and/or array controls.
>>
>> For the initial 'stateless codec' use-case the current implementation is
>> sufficient.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/v4l2-core/v4l2-ctrls.c | 331 ++++++++++++++++++++++++++-
>>  include/media/v4l2-ctrls.h           |  23 ++
>>  2 files changed, 348 insertions(+), 6 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
>> index da4cc1485dc4..56b986185463 100644
>> --- a/drivers/media/v4l2-core/v4l2-ctrls.c
>> +++ b/drivers/media/v4l2-core/v4l2-ctrls.c
>> @@ -3429,6 +3602,152 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
>>  }
>>  EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
>>  
>> +void v4l2_ctrl_request_complete(struct media_request *req,
>> +				struct v4l2_ctrl_handler *main_hdl)
>> +{
>> +	struct media_request_object *obj;
>> +	struct v4l2_ctrl_handler *hdl;
>> +	struct v4l2_ctrl_ref *ref;
>> +
>> +	if (!req || !main_hdl)
>> +		return;
>> +
>> +	obj = media_request_object_find(req, &req_ops, main_hdl);
>> +	if (!obj)
>> +		return;
>> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
>> +		struct v4l2_ctrl *ctrl = ref->ctrl;
>> +		struct v4l2_ctrl *master = ctrl->cluster[0];
>> +		unsigned int i;
>> +
>> +		if (ctrl->flags & V4L2_CTRL_FLAG_VOLATILE) {
>> +			ref->req = ref;
>> +
>> +			v4l2_ctrl_lock(master);
>> +			/* g_volatile_ctrl will update the current control values */
>> +			for (i = 0; i < master->ncontrols; i++)
>> +				cur_to_new(master->cluster[i]);
>> +			call_op(master, g_volatile_ctrl);
>> +			new_to_req(ref);
>> +			v4l2_ctrl_unlock(master);
>> +			continue;
>> +		}
>> +		if (ref->req == ref)
>> +			continue;
>> +
>> +		v4l2_ctrl_lock(ctrl);
>> +		if (ref->req)
>> +			ptr_to_ptr(ctrl, ref->req->p_req, ref->p_req);
>> +		else
>> +			ptr_to_ptr(ctrl, ctrl->p_cur, ref->p_req);
>> +		v4l2_ctrl_unlock(ctrl);
>> +	}
>> +
>> +	WARN_ON(!hdl->request_is_queued);
>> +	list_del_init(&hdl->requests_queued);
>> +	hdl->request_is_queued = false;
>> +	media_request_object_complete(obj);
>> +	media_request_object_put(obj);
>> +}
>> +EXPORT_SYMBOL(v4l2_ctrl_request_complete);
>> +
>> +void v4l2_ctrl_request_setup(struct media_request *req,
>> +			     struct v4l2_ctrl_handler *main_hdl)
> 
> Drivers are expected to use this function internally to make use of the
> control values in the request. Is that your thinking as well?

Not quite. For simple drivers like stateless codecs this driver just applies
all the controls in the request to the hardware. However, this only works for
cases where you can do this just before you process the buffer.

More complex drivers are unlikely to be able to use this function. Instead
they will have to know about HW requirements like how long it will take for
a sensor to actually use a new value.

> The problem with this implementation is that once a driver eventually gets
> a callback (s_ctrl), the callback doesn't have the information on the
> request. That means the driver has no means to associate the control value
> to the request anymore --- and that is against the very purpose of the
> function.

s_ctrl just applies the value to the hardware, it shouldn't have to care
about the request. Scheduling when a control value in a request will have
to be applied to the hardware is something that the driver will have to
coordinate.

In any case, this v4l2_ctrl_request_setup() function is unsuitable for
such complex devices. We will need something smarter for that.

I have not looked at that since it is out of scope of stateless codecs.
The core problem here is how to schedule when you apply controls in a
request given the pending buffer queue.

Regards,

	Hans

> Instead, I'd add a new argument to the callback function --- the request
> --- or add another callback function to be used for applying control values
> for requests. Or alternatively, provide an easy way to enumerate the
> controls and their values in a control handler. For the driver must store
> that value in the request itself to be able to use it: the current
> implementation in vim2m is such that controls are simply set to the driver
> as they'd arrive from the uAPI directly. This way also anyone having access
> to the video device could set whatever control values that would end up
> being used in processing the request.
> 
> Feel free to point out if I'm mistaken in my analysis.
> 
>> +{
>> +	struct media_request_object *obj;
>> +	struct v4l2_ctrl_handler *hdl;
>> +	struct v4l2_ctrl_ref *ref;
>> +
>> +	if (!req || !main_hdl)
>> +		return;
>> +
>> +	obj = media_request_object_find(req, &req_ops, main_hdl);
>> +	if (!obj)
>> +		return;
>> +	if (obj->completed) {
>> +		media_request_object_put(obj);
>> +		return;
>> +	}
>> +	hdl = container_of(obj, struct v4l2_ctrl_handler, req_obj);
>> +
>> +	mutex_lock(hdl->lock);
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node)
>> +		ref->done = false;
>> +
>> +	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
>> +		struct v4l2_ctrl *ctrl = ref->ctrl;
>> +		struct v4l2_ctrl *master = ctrl->cluster[0];
>> +		bool have_new_data = false;
>> +		int i;
>> +
>> +		/*
>> +		 * Skip if this control was already handled by a cluster.
>> +		 * Skip button controls and read-only controls.
>> +		 */
>> +		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
>> +		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
>> +			continue;
>> +
>> +		v4l2_ctrl_lock(master);
>> +		for (i = 0; i < master->ncontrols; i++) {
>> +			if (master->cluster[i]) {
>> +				struct v4l2_ctrl_ref *r =
>> +					find_ref(hdl, master->cluster[i]->id);
>> +
>> +				if (r->req && r == r->req) {
>> +					have_new_data = true;
>> +					break;
>> +				}
>> +			}
>> +		}
>> +		if (!have_new_data) {
>> +			v4l2_ctrl_unlock(master);
>> +			continue;
>> +		}
>> +
>> +		for (i = 0; i < master->ncontrols; i++) {
>> +			if (master->cluster[i]) {
>> +				struct v4l2_ctrl_ref *r =
>> +					find_ref(hdl, master->cluster[i]->id);
>> +
>> +				req_to_new(r);
>> +				master->cluster[i]->is_new = 1;
>> +				r->done = true;
>> +			}
>> +		}
>> +		/*
>> +		 * For volatile autoclusters that are currently in auto mode
>> +		 * we need to discover if it will be set to manual mode.
>> +		 * If so, then we have to copy the current volatile values
>> +		 * first since those will become the new manual values (which
>> +		 * may be overwritten by explicit new values from this set
>> +		 * of controls).
>> +		 */
>> +		if (master->is_auto && master->has_volatiles &&
>> +		    !is_cur_manual(master)) {
>> +			s32 new_auto_val = *master->p_new.p_s32;
>> +
>> +			/*
>> +			 * If the new value == the manual value, then copy
>> +			 * the current volatile values.
>> +			 */
>> +			if (new_auto_val == master->manual_mode_value)
>> +				update_from_auto_cluster(master);
>> +		}
>> +
>> +		try_or_set_cluster(NULL, master, true, 0);
>> +
>> +		v4l2_ctrl_unlock(master);
>> +	}
>> +
>> +	mutex_unlock(hdl->lock);
>> +	media_request_object_put(obj);
>> +}
>> +EXPORT_SYMBOL(v4l2_ctrl_request_setup);
>> +
>>  void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
>>  {
>>  	if (ctrl == NULL)
>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
>> index 76352eb59f14..a0f7c38d1a90 100644
>> --- a/include/media/v4l2-ctrls.h
>> +++ b/include/media/v4l2-ctrls.h
>> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
>>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>>   * @from_other_dev: If true, then @ctrl was defined in another
>>   *		device than the &struct v4l2_ctrl_handler.
>> + * @done:	If true, then this control reference is part of a
>> + *		control cluster that was already set while applying
>> + *		the controls in this media request object.
>> + * @req:	If set, this refers to another request that sets this control.
>>   * @p_req:	The request value. Only used if the control handler
>>   *		is bound to a media request.
>>   *
>> @@ -263,6 +267,8 @@ struct v4l2_ctrl_ref {
>>  	struct v4l2_ctrl *ctrl;
>>  	struct v4l2_ctrl_helper *helper;
>>  	bool from_other_dev;
>> +	bool done;
>> +	struct v4l2_ctrl_ref *req;
>>  	union v4l2_ctrl_ptr p_req;
>>  };
>>  
>> @@ -287,6 +293,15 @@ struct v4l2_ctrl_ref {
>>   * @notify_priv: Passed as argument to the v4l2_ctrl notify callback.
>>   * @nr_of_buckets: Total number of buckets in the array.
>>   * @error:	The error code of the first failed control addition.
>> + * @request_is_queued: True if the request was queued.
>> + * @requests:	List to keep track of open control handler request objects.
>> + *		For the parent control handler (@req_obj.req == NULL) this
>> + *		is the list header. When the parent control handler is
>> + *		removed, it has to unbind and put all these requests since
>> + *		they refer to the parent.
>> + * @requests_queued: List of the queued requests. This determines the order
>> + *		in which these controls are applied. Once the request is
>> + *		completed it is removed from this list.
>>   * @req_obj:	The &struct media_request_object, used to link into a
>>   *		&struct media_request.
>>   */
>> @@ -301,6 +316,9 @@ struct v4l2_ctrl_handler {
>>  	void *notify_priv;
>>  	u16 nr_of_buckets;
>>  	int error;
>> +	bool request_is_queued;
>> +	struct list_head requests;
>> +	struct list_head requests_queued;
>>  	struct media_request_object req_obj;
>>  };
>>  
>> @@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
>>   */
>>  __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
>>  
>> +void v4l2_ctrl_request_setup(struct media_request *req,
>> +			     struct v4l2_ctrl_handler *hdl);
>> +void v4l2_ctrl_request_complete(struct media_request *req,
>> +				struct v4l2_ctrl_handler *hdl);
>> +
>>  /* Helpers for ioctl_ops */
>>  
>>  /**
>> -- 
>> 2.17.0
>>
> 

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Sakari Ailus]

On Thu, May 03, 2018 at 04:53:02PM +0200, Hans Verkuil wrote:
> @@ -1059,6 +1077,11 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
>   */
>  __poll_t v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
>  
> +void v4l2_ctrl_request_setup(struct media_request *req,
> +			     struct v4l2_ctrl_handler *hdl);
> +void v4l2_ctrl_request_complete(struct media_request *req,
> +				struct v4l2_ctrl_handler *hdl);
> +

The two kAPI functions appear to be without documentation.

-- 
Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 25/28] vim2m: support requests

[Sakari Ailus]

Hi Hans,

On Thu, May 03, 2018 at 04:53:15PM +0200, Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Add support for requests to vim2m.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  drivers/media/platform/vim2m.c | 34 ++++++++++++++++++++++++++++++++++
>  1 file changed, 34 insertions(+)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index a1b0bb08668d..7c86c711a4cd 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -387,8 +387,26 @@ static void device_run(void *priv)
>  	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
>  	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
>  
> +	/* Apply request controls if needed */
> +	if (src_buf->vb2_buf.req_obj.req)
> +		v4l2_ctrl_request_setup(src_buf->vb2_buf.req_obj.req,
> +					&ctx->hdl);
> +	if (dst_buf->vb2_buf.req_obj.req &&
> +	    dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
> +		v4l2_ctrl_request_setup(dst_buf->vb2_buf.req_obj.req,
> +					&ctx->hdl);

This seems wrong: you're making use of control values from two *different*,
*unrelated* requests.

I'd rather like that we didn't end up a request API for the mem2mem devices
and another for the rest... albeit the kernel implementation could be the
same or similar, it's still the uAPI that would have differing semantics
between the two.

> +
>  	device_process(ctx, src_buf, dst_buf);
>  
> +	/* Complete request controls if needed */
> +	if (src_buf->vb2_buf.req_obj.req)
> +		v4l2_ctrl_request_complete(src_buf->vb2_buf.req_obj.req,
> +					&ctx->hdl);
> +	if (dst_buf->vb2_buf.req_obj.req &&
> +	    dst_buf->vb2_buf.req_obj.req != src_buf->vb2_buf.req_obj.req)
> +		v4l2_ctrl_request_complete(dst_buf->vb2_buf.req_obj.req,
> +					&ctx->hdl);
> +
>  	/* Run a timer, which simulates a hardware irq  */
>  	schedule_irq(dev, ctx->transtime);
>  }
> @@ -823,12 +841,21 @@ static void vim2m_stop_streaming(struct vb2_queue *q)
>  			vbuf = v4l2_m2m_dst_buf_remove(ctx->fh.m2m_ctx);
>  		if (vbuf == NULL)
>  			return;
> +		v4l2_ctrl_request_complete(vbuf->vb2_buf.req_obj.req,
> +					   &ctx->hdl);
>  		spin_lock_irqsave(&ctx->dev->irqlock, flags);
>  		v4l2_m2m_buf_done(vbuf, VB2_BUF_STATE_ERROR);
>  		spin_unlock_irqrestore(&ctx->dev->irqlock, flags);
>  	}
>  }
>  
> +static void vim2m_buf_request_complete(struct vb2_buffer *vb)
> +{
> +	struct vim2m_ctx *ctx = vb2_get_drv_priv(vb->vb2_queue);
> +
> +	v4l2_ctrl_request_complete(vb->req_obj.req, &ctx->hdl);
> +}
> +
>  static const struct vb2_ops vim2m_qops = {
>  	.queue_setup	 = vim2m_queue_setup,
>  	.buf_prepare	 = vim2m_buf_prepare,
> @@ -837,6 +864,7 @@ static const struct vb2_ops vim2m_qops = {
>  	.stop_streaming  = vim2m_stop_streaming,
>  	.wait_prepare	 = vb2_ops_wait_prepare,
>  	.wait_finish	 = vb2_ops_wait_finish,
> +	.buf_request_complete = vim2m_buf_request_complete,
>  };
>  
>  static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq)
> @@ -1003,6 +1031,11 @@ static const struct v4l2_m2m_ops m2m_ops = {
>  	.job_abort	= job_abort,
>  };
>  
> +static const struct media_device_ops m2m_media_ops = {
> +	.req_validate = vb2_request_validate,
> +	.req_queue = vb2_m2m_request_queue,
> +};
> +
>  static int vim2m_probe(struct platform_device *pdev)
>  {
>  	struct vim2m_dev *dev;
> @@ -1027,6 +1060,7 @@ static int vim2m_probe(struct platform_device *pdev)
>  	dev->mdev.dev = &pdev->dev;
>  	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>  	media_device_init(&dev->mdev);
> +	dev->mdev.ops = &m2m_media_ops;
>  	dev->v4l2_dev.mdev = &dev->mdev;
>  	dev->pad[0].flags = MEDIA_PAD_FL_SINK;
>  	dev->pad[1].flags = MEDIA_PAD_FL_SOURCE;

-- 
Regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [PATCHv13 22/28] Documentation: v4l: document request API

[Laurent Pinchart]

Hi Hans,

Thank you for the patch.

On Thursday, 3 May 2018 17:53:12 EEST Hans Verkuil wrote:
> From: Alexandre Courbot <acourbot@chromium.org>
> 
> Document the request API for V4L2 devices, and amend the documentation
> of system calls influenced by it.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  .../media/uapi/mediactl/media-funcs.rst       |   3 +
>  .../uapi/mediactl/media-ioc-request-alloc.rst |  71 ++++++
>  .../uapi/mediactl/media-request-ioc-queue.rst |  46 ++++
>  .../mediactl/media-request-ioc-reinit.rst     |  51 +++++
>  Documentation/media/uapi/v4l/buffer.rst       |  18 +-
>  Documentation/media/uapi/v4l/common.rst       |   1 +
>  Documentation/media/uapi/v4l/request-api.rst  | 211 ++++++++++++++++++
>  .../media/uapi/v4l/vidioc-g-ext-ctrls.rst     |  25 ++-
>  Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
>  .../media/videodev2.h.rst.exceptions          |   1 +
>  10 files changed, 428 insertions(+), 6 deletions(-)
>  create mode 100644
> Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst create mode
> 100644 Documentation/media/uapi/mediactl/media-request-ioc-queue.rst create
> mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
> create mode 100644 Documentation/media/uapi/v4l/request-api.rst

I think I would have split documentation and included each part in the 
corresponding API definition patch to make it easier to review the definition 
and the documentation together, but that's not a big deal.

We might however still want to split this patch in two, with one part for the 
media controller API and one part for the V4L2 API, as the request API on MC 
isn't V4L2-dependent.

> diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst
> b/Documentation/media/uapi/mediactl/media-funcs.rst index
> 076856501cdb..f4da9b3e17ec 100644
> --- a/Documentation/media/uapi/mediactl/media-funcs.rst
> +++ b/Documentation/media/uapi/mediactl/media-funcs.rst
> @@ -16,3 +16,6 @@ Function Reference
>      media-ioc-enum-entities
>      media-ioc-enum-links
>      media-ioc-setup-link
> +    media-ioc-request-alloc
> +    media-request-ioc-queue
> +    media-request-ioc-reinit

Do we want to keep this alphabetically ordered ?

> diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
> b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst new file
> mode 100644
> index 000000000000..d45a94c9e23c
> --- /dev/null
> +++ b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
> @@ -0,0 +1,71 @@
> +.. SPDX-License-Identifier: GPL-2.0-only

Documentation/process/license-rules.rst doesn't specify GPL-2.0-only, this 
should be GPL-2.0. The comment applies to all other new files.

This being said, GPL isn't really a documentation license, shouldn't we pick a 
more appropriate license ?

> +
> +.. _media_ioc_request_alloc:
> +
> +*****************************
> +ioctl MEDIA_IOC_REQUEST_ALLOC
> +*****************************
> +
> +Name
> +====
> +
> +MEDIA_IOC_REQUEST_ALLOC - Allocate a request
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_ALLOC, struct
> media_request_alloc *argp )
> +    :name: MEDIA_IOC_REQUEST_ALLOC
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <media-func-open>`.
> +
> +``argp``
> +
> +
> +Description
> +===========
> +
> +If the media device supports :ref:`requests <media-request-api>`, then
> +this ioctl can be used to allocate a request. A request is accessed through
> +a file descriptor that is returned in struct
> :c:type:`media_request_alloc`.
> +
> +If the request was successfully allocated, then the request file descriptor
> +can be passed to :ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>`,
> +:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`,
> +:ref:`ioctl VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` and
> +:ref:`ioctl VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.

Conceptually speaking, shouldn't we refrain from referencing the V4L2 API from 
the MC API ?

> +In addition, the request can be queued by calling
> +:ref:`MEDIA_REQUEST_IOC_QUEUE` and re-initialized by calling
> +:ref:`MEDIA_REQUEST_IOC_REINIT`.
> +
> +Finally, the file descriptor can be polled to wait for the request to
> +complete.
> +
> +To free the request the file descriptor has to be closed.
> +
> +.. c:type:: media_request_alloc
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. flat-table:: struct media_request_alloc
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    *  -  __s32
> +       -  ``fd``
> +       -  The file descriptor of the request.
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
> b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst new file
> mode 100644
> index 000000000000..79f745a3d663
> --- /dev/null
> +++ b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
> @@ -0,0 +1,46 @@
> +.. SPDX-License-Identifier: GPL-2.0-only
> +
> +.. _media_request_ioc_queue:
> +
> +*****************************
> +ioctl MEDIA_REQUEST_IOC_QUEUE
> +*****************************
> +
> +Name
> +====
> +
> +MEDIA_REQUEST_IOC_QUEUE - Queue a request
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_QUEUE )
> +    :name: MEDIA_REQUEST_IOC_QUEUE
> +
> +
> +Arguments
> +=========
> +
> +``request_fd``
> +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
> +
> +
> +Description
> +===========
> +
> +If the media device supports :ref:`requests <media-request-api>`, then
> +this request ioctl can be used to queue a previously allocated request.
> +
> +If the request was successfully queued, then the file descriptor can be
> +polled to wait for the request to complete.

This says nothing about what queuing a request actually does :-) Neither does 
it document why it could fail.

> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EBUSY
> +    The request was already queued.
> diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
> b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst new file
> mode 100644
> index 000000000000..7cbb7eb5d73e
> --- /dev/null
> +++ b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
> @@ -0,0 +1,51 @@
> +.. SPDX-License-Identifier: GPL-2.0-only
> +
> +.. _media_request_ioc_reinit:
> +
> +******************************
> +ioctl MEDIA_REQUEST_IOC_REINIT
> +******************************
> +
> +Name
> +====
> +
> +MEDIA_REQUEST_IOC_REINIT - Re-initialize a request
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_REINIT )
> +    :name: MEDIA_REQUEST_IOC_REINIT
> +
> +
> +Arguments
> +=========
> +
> +``request_fd``
> +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.

I was expecting an argument to this ioctl, with a reference to another request 
(and I expected the same reference for the MEDIA_IOC_REQUEST_ALLOC ioctl). I 
think this topic has been discussed several times in the past, but possibly 
not in venues where everybody was present, so I'll detail the problem here.

The idea behind this version of the request API is to split request handling 
in three parts:

1. A request is allocated (MEDIA_IOC_REQUEST_ALLOC) or an existing request is 
re-initialized (MEDIA_REQUEST_IOC_REINIT). At that point the request contains 
no property (control, format, ...) to be modified.

2. The request is filled with properties to be modified using existing V4L2 
ioctls (VIDIOC_S_EXT_CTRLS, VIDIOC_SUBDEV_S_FMT, VIDIOC_SUBDEV_S_SELECTION, 
..). The current implementation supports controls only, with support for 
formats and selection rectangles to be added later.

3. The request is queued with MEDIA_REQUEST_IOC_QUEUE.

As usual the devil is in the details, and the details of the second step in 
particular.

The idea behind splitting request handling in those three steps is to reuse 
existing V4L2 ioctls as much as possible, and to keep the existing semantics 
of those ioctls. I believe this isn't possible with the proposed 
MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls, neither for 
controls nor for formats or selection rectangles, although the problem is more 
limited for controls. I will thus take formats and selection rectangles to 
explain where the problem lies.

The S_FMT and S_SELECTION semantics on subdevs (and, for that matter, on video 
nodes too) is that those ioctls don't fail when passed parameters invalid for 
the device, but return a valid configuration as close as possible to the 
requested one. This allows implementing negotiation of formats and selection 
rectangles through an iterative process that involves back-and-forth calls 
between user space and kernel space until the application finds a usable 
configuration accepted by the driver, or until it decides to fail because such 
a configuration doesn't exist. (As a reminder, such negotiation is needed 
because it was deemed impossible to create a proper API that would expose all 
device constraints in a static way.)

Formats and selection rectangles are propagated within subdevs from sink pads 
to source pads by the driver. Sink formats and selection rectangles thus 
influence the result of negotiation on source pads. For instance, on a scaler 
subdev that can't perform format conversion, the format on the source pad will 
be dictated by the format on the sink pad. In order to implement S_FMT on the 
source pad the kernel driver thus needs context information to get the format 
on the sink pad.

Without the request API this is easy. The context is either stored in the 
driver-specific device structure (for active formats) or in the file handle 
(for try formats). With the proposed request API, however, we have no such 
context: the requests are created empty (or re-initialized to be empty). 
Kernel drivers can access the active state of the device, or the state of 
previously queued requests, but can't access the state of prepared but not yet 
queued requests.

My proposal to solve this was to link requests by specifying the handle of a 
previous request to the MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT 
ioctls. That way kernel drivers would know in which order requests will be 
queued, and would be able to access the correct context information to 
implement the existing semantics of the S_FMT and S_SELECTION ioctls (as well 
as being able to implement the G_FMT and G_SELECTION ioctls). Note that the 
problem isn't limited to format and selection rectangles, similar problems can 
exist when multiple controls are related, such as auto and manual controls.

This would result in a more complex API and a more complex request lifetime 
management, as well as a more complex semantics of the request API. While I 
thought this was unavoidable, I came to reconsider my position on this topic, 
and I believe that the current request API proposal is acceptable in that 
regard, provided that we agree that the existing semantics of the S_FMT and 
S_SELECTION ioctls won't need to be implemented for requests as it won't be 
possible to do so. The reason that prompted me to change my mind is that I 
believe the API would be simpler to define and simpler to use if we dropped 
that semantics, and required applications to create requests that are fully 
valid instead of trying to mangle the content of a request the same way we do 
for format and selection rectangle negotiation. Obviously, negotiation is 
still needed in order for applications to find a valid configuration, but we 
have the V4L2_SUBDEV_FORMAT_TRY API for that that, in conjunction with the 
S_FMT and S_SELECTION ioctls, can be used to negotiate a configuration. The 
negotiated configuration could then be set in a request, and would be 
guaranteed to be valid.

To summarize this detailed explanation, I am fine with the 
MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls not providing a 
way to link requests, and thus making the API simpler, if we acknowledge that 
the S_FMT and S_SELECTION ioctls won't be possible to implement with their 
current semantics and agree not to implement format and selection rectangle 
negotiation on requests.

What I want to avoid is merging the MEDIA_IOC_REQUEST_ALLOC and 
MEDIA_REQUEST_IOC_REINIT ioctls as they are proposed today, and having to 
modify their semantics by adding request linking in the future. That should 
significantly change the semantics of the request API which could be very 
difficult to do without breaking userspace, and extending the API while 
keeping support for the current semantics would result in additional 
complexity that I'm not willing to accept. If anyone think that request 
linking, needed if we want to access entity state from the existing V4L2 
subdev ioctl handlers, is desired, then it needs to be built in the API from 
the start.

> +Description
> +===========
> +
> +If the media device supports :ref:`requests <media-request-api>`, then
> +this request ioctl can be used to re-initialize a previously allocated
> +request.
> +
> +Re-initializing a request will clear any existing data from the request.
> +This avoids having to close() a completed request and allocate a new
> +request. Instead the completed request can just be re-initialized and
> +it is ready to be used again.
> +
> +A request can only be re-initialized if it either has not been queued
> +yet, or if it was queued and completed.
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +EBUSY
> +    The request is queued but not yet completed.
> diff --git a/Documentation/media/uapi/v4l/buffer.rst
> b/Documentation/media/uapi/v4l/buffer.rst index e2c85ddc990b..8059fc0ed520
> 100644
> --- a/Documentation/media/uapi/v4l/buffer.rst
> +++ b/Documentation/media/uapi/v4l/buffer.rst
> @@ -306,10 +306,12 @@ struct v4l2_buffer
>        - A place holder for future extensions. Drivers and applications
>  	must set this to 0.
>      * - __u32
> -      - ``reserved``
> +      - ``request_fd``
>        -
> -      - A place holder for future extensions. Drivers and applications
> -	must set this to 0.
> +      - The file descriptor of the request to queue the buffer to. If
> specified
> +        and flag ``V4L2_BUF_FLAG_REQUEST_FD`` is set, then the buffer will
> be
> +	queued to that request. This is set by the user when calling
> +	:ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>` and ignored by other ioctls.

Do we need that flag ? Isn't enough to just set the request_fd field ?

> @@ -514,6 +516,11 @@ Buffer Flags
>  	streaming may continue as normal and the buffer may be reused
>  	normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
>  	called.
> +    * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
> +
> +      - ``V4L2_BUF_FLAG_IN_REQUEST``
> +      - 0x00000080
> +      - This buffer is part of a request that hasn't been queued yet.

What is this used for ? I don't see it being explained in the documentation.

>      * .. _`V4L2-BUF-FLAG-KEYFRAME`:
> 
>        - ``V4L2_BUF_FLAG_KEYFRAME``
> @@ -589,6 +596,11 @@ Buffer Flags
>  	the format. Any Any subsequent call to the
> 
>  	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
> 
>  	but return an ``EPIPE`` error code.
> +    * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
> +
> +      - ``V4L2_BUF_FLAG_REQUEST_FD``
> +      - 0x00800000
> +      - The ``request_fd`` field contains a valid file descriptor.
>      * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
> 
>        - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
> diff --git a/Documentation/media/uapi/v4l/common.rst
> b/Documentation/media/uapi/v4l/common.rst index 13f2ed3fc5a6..a4aa0059d45a
> 100644
> --- a/Documentation/media/uapi/v4l/common.rst
> +++ b/Documentation/media/uapi/v4l/common.rst
> @@ -44,3 +44,4 @@ applicable to all devices.
>      crop
>      selection-api
>      streaming-par
> +    request-api
> diff --git a/Documentation/media/uapi/v4l/request-api.rst
> b/Documentation/media/uapi/v4l/request-api.rst new file mode 100644
> index 000000000000..0ce75b254956
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/request-api.rst
> @@ -0,0 +1,211 @@
> +.. -*- coding: utf-8; mode: rst -*-
> +
> +.. _media-request-api:
> +
> +Request API
> +===========
> +
> +The Request API has been designed to allow V4L2 to deal with requirements
> of +modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android

I wouldn't call this MIPI cameras. In this context MIPI only defines the bus 
between the camera sensor and the ISP or bridge, so it's quite irrelevant. 
"ISP-based cameras" might be a better term, although some ISP-based cameras 
could still offer a high-level API that doesn't require or even benefit from 
the use of requests.

> Codec +v2). One such requirement is the ability for devices belonging to
> the same +pipeline to reconfigure and collaborate closely on a per-frame
> basis. Another is +efficient support of stateless codecs, which need
> per-frame controls to be set +asynchronously in order to be used
> efficiently.

Do you really mean asynchronously ? I thought the purpose of the request API 
with codecs was to set controls synchronously with frames.

> +Supporting these features without the Request API is possible but terribly
> +inefficient: user-space would have to flush all activity on the media
> pipeline, +reconfigure it for the next frame, queue the buffers to be
> processed with that +configuration, and wait until they are all available
> for dequeuing before +considering the next frame. This defeats the purpose
> of having buffer queues +since in practice only one buffer would be queued
> at a time.

It's not even always possible. For memory to memory devices such a behaviour 
might be possible, but for camera devices in general you don't have the option 
of stopping and reconfiguring the hardware as frames are continuously captured 
by the sensor.

> +The Request API allows a specific configuration of the pipeline (media
> +controller topology + controls for each device) to be associated with

Should we mention formats too ? Should we use "entity" instead of "device" as 
this documents an MC-based API ?

> specific +buffers. The parameters are applied by each participating device
> as buffers +associated to a request flow in. This allows user-space to
> schedule several +tasks ("requests") with different parameters in advance,
> knowing that the +parameters will be applied when needed to get the
> expected result. Control +values at the time of request completion are also
> available for reading.
> +
> +Usage
> +=====
> +
> +The Request API is used on top of standard media controller and V4L2 calls,
> +which are augmented with an extra ``request_fd`` parameter. Requests
> themselves +are allocated from the supporting media controller node.

I'd like it if we could document the request API in the MC documentation for 
the overall concept, and then have a V4L2-specific documentation that explains 
how the request API is used with V4L2.

> +
> +Request Allocation
> +------------------
> +
> +User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
> +for the media device node. This returns a file descriptor representing the
> +request. Typically, several such requests will be allocated.
> +
> +Request Preparation
> +-------------------
> +
> +Standard V4L2 ioctls can then receive a request file descriptor to express
> the +fact that the ioctl is part of said request, and is not to be applied
> +immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
> +support this.

As explained above in the documentation of MEDIA_IOC_REQUEST_ALLOC, I would 
rather list the ioctls explicitly here, as MEDIA_IOC_REQUEST_ALLOC is really 
not V4L2-specific.

> Controls set with a ``request_fd`` parameter are stored
> instead +of being immediately applied, and buffers queued to a request do
> not enter the +regular buffer queue until the request itself is queued.
> +
> +Request Submission
> +------------------
> +
> +Once the parameters and buffers of the request are specified, it can be
> +queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request FD.
> +This will make the buffers associated to the request available to their
> driver, +which can then apply the associated controls as buffers are
> processed. A queued +request cannot be modified anymore.
> +
> +If several devices are part of the request, individual drivers may

I'd write "If parameters for multiple entities are part of the request", or 
"If the request contains parameters for multiple entities".

> synchronize +so the requested pipeline's topology is applied before the
> buffers are +processed. This is at the discretion of media controller
> drivers and is not a +requirement.

One of the main purposes of the request API is to support atomic 
configuration. Saying that it's simply optional defeats that purpose. I agree 
that atomicity can't be achieved in all cases, and that there will often be a 
best-effort approach, but for hardware devices that support atomic 
configuration I think applications need to be able to rely on the driver 
implementing that feature. I would thus put a bit more pressure on drivers 
than just saying it's fully optional.

> +Buffers queued without an associated request after a request-bound buffer
> will +be processed using the state of the hardware at the time of the
> request +completion.

I'm not sure to follow that, could you explain in a bit more details ? Also, 
even more than for controls (see below), I really want to know about use cases 
for mixing buffers in and outside of requests for the same buffer queue before 
accepting it.

I also would like to see a detailed description of what happens when buffers 
are queued in a request for one queue and outside of requests for another 
queue (for example in the case of codecs), for both capture and output queues.

> All the same, controls set without a request are applied +immediately,
> regardless of whether a request is in use or not.

This will be tricky to implement, and will potentially generate side effects 
difficult to handle, and difficult to debug. I understand that this is a 
requirement for codecs, but I don't know the exact use case(s), so I can't 
comment on whether this is the right approach. While I don't necessarily 
reject this feature on principle, I first want to know more about the use 
case(s) before accepting it. If we end up needing it, we'll have to better 
document it, by clearly specifying what is allowed and what isn't (for 
instance, we might decide to allow setting control asynchronously only when 
they're not simultaneously modified through a queued request), as well as the 
exact behaviour that applications can expect in the allowed cases.

> +User-space can ``poll()`` a request FD in order to wait until the request
> +completes. A request is considered complete once all its associated buffers
> are +available for dequeuing and all the associated controls have been
> updated with +the values at the time of completion. Note that user-space
> does not need to wait +for the request to complete to dequeue its buffers:
> buffers that are available +halfway through a request can be dequeued
> independently of the request's state.
> +
> +A completed request contains the state of the request at the time of the
> +request completion. User-space can query that state by calling
> +:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request FD.
> +
> +Recycling and Destruction
> +-------------------------
> +
> +Finally, completed request can either be discarded or be reused. Calling
> +``close()`` on a request FD will make that FD unusable, freeing the request
> if +it is not referenced elsewhere.

Where else can it be referenced ? Does this only refer to the ability to 
duplicate FDs or share them with other processes ? If so, I would document 
that requests are freed when all file descriptors that refer to them are 
closed, not when close() is called on one file descriptor.

> The :ref:`MEDIA_REQUEST_IOC_REINIT`
> will +clear a request's state and make it available again. No state is
> retained by +this operation: the request is as if it had just been
> allocated.
> +
> +Example for a Memory-to-Memory Device
> +-------------------------------------
> +
> +M2M devices are single-node V4L2 devices providing one OUTPUT queue (for
> +user-space to provide input buffers) and one CAPTURE queue (to retrieve
> +processed data). These devices are commonly used for frame processors or
> +stateless codecs.
> +
> +In this use-case, the request API can be used to associate specific
> controls to +be applied by the driver before processing an OUTPUT buffer,

As commented below, I would explain that the controls are applied for the 
buffer, not before processing it.

> allowing user-space +to queue many such buffers in advance. It can also
> take advantage of requests' +ability to capture the state of controls when
> the request completes to read back +information that may be subject to
> change.
> +
> +Put into code, after obtaining a request, user-space can assign controls
> and one +OUTPUT buffer to it:
> +
> +.. code-block:: c
> +
> +	struct v4l2_buffer buf;
> +	struct v4l2_ext_controls ctrls;
> +	struct media_request_alloc alloc = { 0 };
> +	int req_fd;
> +	...
> +	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
> +	req_fd = alloc.fd;
> +	...
> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
> +	ctrls.request_fd = req_fd;
> +	ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
> +	...
> +	buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
> +	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
> +	buf.request_fd = req_fd;
> +	ioctl(codec_fd, VIDIOC_QBUF, &buf);
> +
> +Note that there is typically no need to use the Request API for CAPTURE
> buffers +since there are per-frame settings to report there.

Do you mean since there are no per-frame settings to report there ?

> +Once the request is fully prepared, it can be queued to the driver:
> +
> +.. code-block:: c
> +
> +	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
> +
> +User-space can then either wait for the request to complete by calling
> poll() on +its file descriptor, or start dequeuing CAPTURE buffers. Most
> likely, it will +want to get CAPTURE buffers as soon as possible and this
> can be done using a +regular DQBUF:
> +
> +.. code-block:: c
> +
> +	struct v4l2_buffer buf;
> +
> +	memset(&buf, 0, sizeof(buf));
> +	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	ioctl(codec_fd, VIDIOC_DQBUF, &buf);

The above explanation makes it sounds like waiting for the request to complete 
will ensure that a capture buffer is available, and that's not necessarily the 
case. There isn't always a guarantee that one output buffer will result in one 
and only one capture buffer being filled.

> +We can then, after ensuring that the request is completed via polling the
> +request FD, query control values at the time of its completion via an
> +annotated call to G_EXT_CTRLS. This is particularly useful for volatile
> controls +for which we want to query values as soon as the capture buffer
> is produced. +
> +.. code-block:: c
> +
> +	struct pollfd pfd = { .events = POLLPRI, .fd = request_fd };
> +	poll(&pfd, 1, -1);
> +	...
> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
> +	ctrls.request_fd = req_fd;
> +	ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls);
> +
> +Once we don't need the request anymore, we can either recycle it for reuse
> with +:ref:`MEDIA_REQUEST_IOC_REINIT`...
> +
> +.. code-block:: c
> +
> +	ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT);
> +
> +... or close its file descriptor to completely dispose of it.
> +
> +.. code-block:: c
> +
> +	close(req_fd);
> +
> +Example for a Simple Capture Device
> +-----------------------------------
> +
> +With a simple capture device, requests can be used to specify controls to
> apply +to a given CAPTURE buffer. The driver will apply these controls
> before producing +the marked CAPTURE buffer.

As explained below I think we should be a bit more precise with the wording 
here.

> +.. code-block:: c
> +
> +	struct v4l2_buffer buf;
> +	struct v4l2_ext_controls ctrls;
> +	struct media_request_alloc alloc = { 0 };
> +	int req_fd;
> +	...
> +	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
> +	req_fd = alloc.fd;
> +	...
> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
> +	ctrls.request_fd = req_fd;
> +	ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
> +	...
> +	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
> +	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
> +	buf.request_fd = req_fd;
> +	ioctl(camera_fd, VIDIOC_QBUF, &buf);
> +
> +Once the request is fully prepared, it can be queued to the driver:
> +
> +.. code-block:: c
> +
> +	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
> +
> +User-space can then dequeue buffers, wait for the request completion, query
> +controls and recycle the request as in the M2M example above.
> diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
> b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst index
> 2011c2b2ee67..8f788c444347 100644
> --- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
> @@ -95,6 +95,19 @@ appropriate. In the first case the new value is set in
> struct is inappropriate (e.g. the given menu index is not supported by the
> menu control), then this will also result in an ``EINVAL`` error code
> error.
> 
> +If ``request_fd`` is set to a not-submitted request file descriptor and
> +``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``, then the
> +controls are not applied immediately when calling
> +:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`, but instead are applied
> right +before the driver starts processing a buffer associated to the same
> request. +

Please see my related comments below for VIDIOC_QBUF.

What happens if which is set to V4L2_CTRL_WHICH_REQUEST_VAL and the request is 
already queued ?

> +If ``request_fd`` is specified and ``which`` is set to
> ``V4L2_CTRL_WHICH_REQUEST_VAL`` +during a call to :ref:`VIDIOC_G_EXT_CTRLS
> <VIDIOC_G_EXT_CTRLS>`, then the +returned values will be the values
> currently set for the request (or the +hardware value if none is set)

Do we really have a need for returning the hardware value ? That seems a bit 
random to me, wouldn't it be better to return an error instead ?

> if the request has not yet completed, or the +values of the controls at the
> time of request completion if it has already +completed.
> +
>  The driver will only set/get these controls if all control values are
>  correct. This prevents the situation where only some of the controls
>  were set/get. Only low-level errors (e. g. a failed i2c command) can
> @@ -209,8 +222,10 @@ still cause this situation.
>        - ``which``
>        - Which value of the control to get/set/try.
>  	``V4L2_CTRL_WHICH_CUR_VAL`` will return the current value of the
> -	control and ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
> -	value of the control.
> +	control, ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
> +	value of the control and ``V4L2_CTRL_WHICH_REQUEST_VAL`` indicates that
> +	these controls have to be retrieved from a request or tried/set for
> +	a request.
> 
>  	.. note::
> 
> @@ -272,8 +287,12 @@ still cause this situation.
>  	then you can call :ref:`VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` to try
> to discover the actual control that failed the validation step.
> Unfortunately,
>  	there is no ``TRY`` equivalent for :ref:`VIDIOC_G_EXT_CTRLS
> <VIDIOC_G_EXT_CTRLS>`. +    * - __s32
> +      - ``request_fd``
> +	File descriptor of the request to be used by this operation. Only
> +	valid if ``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``.
>      * - __u32
> -      - ``reserved``\ [2]
> +      - ``reserved``\ [1]
>        - Reserved for future extensions.
> 
>  	Drivers and applications must set the array to zero.
> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> b/Documentation/media/uapi/v4l/vidioc-qbuf.rst index
> 9e448a4aa3aa..c9f55d8340d1 100644
> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> @@ -98,6 +98,13 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF
> <VIDIOC_STREAMON>` or
>  :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
> 
>  device is closed.
> 
> +The ``request_fd`` field can be used when queuing to specify the file
> +descriptor of a request, if requests are in use. Setting it means that the
> +buffer will not be passed to the driver until the request itself is queued.
> +Also, the driver will apply any setting associated with the request before
> +processing the buffer.

I think we should be more precise with the wording, here and above in the 
documentation. From an API point of view, we need to guarantee that the 
settings from the request are used to process the image capture in the buffer, 
or the image output from the buffer. Drivers don't process buffers other than 
queuing them to the hardware, so saying that the driver will apply settings 
before processing the buffer is a bit vague in my opinion, and also possibly 
not very accurate depending on the hardware.

Saying that controls are applied right before the buffer is processed by the 
driver is inaccurate : for example exposure time might need several frames to 
take effect, and might thus need to be applied a few frames before the buffer 
is queued to the hardware for capture. The other way around is possible too : 
the hardware might support a buffer queue but have no queue for controls. 
Controls would then need to be applied a few frames after the buffer is queued 
to the hardware in order to take effect for the right buffer.

We're specifying an API that carries very strong semantics across multiple 
components and multiple objects, so I think it's very important to be as 
precise as possible.

> This field will be ignored unless the
> +``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
> +
>  Applications call the ``VIDIOC_DQBUF`` ioctl to dequeue a filled
>  (capturing) or displayed (output) buffer from the driver's outgoing
>  queue. They just set the ``type``, ``memory`` and ``reserved`` fields of
> diff --git a/Documentation/media/videodev2.h.rst.exceptions
> b/Documentation/media/videodev2.h.rst.exceptions index
> a5cb0a8686ac..5a5a1c772053 100644
> --- a/Documentation/media/videodev2.h.rst.exceptions
> +++ b/Documentation/media/videodev2.h.rst.exceptions
> @@ -514,6 +514,7 @@ ignore define V4L2_CTRL_DRIVER_PRIV
>  ignore define V4L2_CTRL_MAX_DIMS
>  ignore define V4L2_CTRL_WHICH_CUR_VAL
>  ignore define V4L2_CTRL_WHICH_DEF_VAL
> +ignore define V4L2_CTRL_WHICH_REQUEST_VAL
>  ignore define V4L2_OUT_CAP_CUSTOM_TIMINGS
>  ignore define V4L2_CID_MAX_CTRLS

-- 
Regards,

Laurent Pinchart

Re: [PATCHv13 02/28] uapi/linux/media.h: add request API

[Laurent Pinchart]

Hi Hans,

Thank you for the patch.

On Thursday, 3 May 2018 17:52:52 EEST Hans Verkuil wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Define the public request API.
> 
> This adds the new MEDIA_IOC_REQUEST_ALLOC ioctl to allocate a request
> and two ioctls that operate on a request in order to queue the
> contents of the request to the driver and to re-initialize the
> request.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> ---
>  include/uapi/linux/media.h | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index c7e9a5cba24e..32883d4d22b2 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -342,11 +342,23 @@ struct media_v2_topology {
> 
>  /* ioctls */
> 
> +struct __attribute__ ((packed)) media_request_alloc {
> +	__s32 fd;
> +};
> +
>  #define MEDIA_IOC_DEVICE_INFO	_IOWR('|', 0x00, struct media_device_info)
>  #define MEDIA_IOC_ENUM_ENTITIES	_IOWR('|', 0x01, struct media_entity_desc)
>  #define MEDIA_IOC_ENUM_LINKS	_IOWR('|', 0x02, struct media_links_enum)
>  #define MEDIA_IOC_SETUP_LINK	_IOWR('|', 0x03, struct media_link_desc)
>  #define MEDIA_IOC_G_TOPOLOGY	_IOWR('|', 0x04, struct media_v2_topology)
> +#define MEDIA_IOC_REQUEST_ALLOC	_IOWR('|', 0x05, struct
> media_request_alloc)
> +
> +/*
> + * These ioctls are called from the request file descriptor as returned

Maybe s/from/on/ ?

> + * by MEDIA_IOC_REQUEST_ALLOC.
> + */
> +#define MEDIA_REQUEST_IOC_QUEUE		_IO('|',  0x80)
> +#define MEDIA_REQUEST_IOC_REINIT	_IO('|',  0x81)
> 
>  #if !defined(__KERNEL__) || defined(__NEED_MEDIA_LEGACY_API)

This matches the documentation in patch 22/28, so as such,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

However, please note that I've sent comments to that patch that might result 
in changes to the API, in which case my Reviewed-by wouldn't be applicable 
anymore.

-- 
Regards,

Laurent Pinchart

Re: [PATCHv13 22/28] Documentation: v4l: document request API

[Tomasz Figa]

Hi Laurent,

Thanks for detailed review. Please let me add my thoughts inline as well.

On Fri, May 18, 2018 at 11:46 PM Laurent Pinchart <
laurent.pinchart@ideasonboard.com> wrote:

> Hi Hans,

> Thank you for the patch.

> On Thursday, 3 May 2018 17:53:12 EEST Hans Verkuil wrote:

[snip]

> > +Name
> > +====
> > +
> > +MEDIA_REQUEST_IOC_REINIT - Re-initialize a request
> > +
> > +
> > +Synopsis
> > +========
> > +
> > +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_REINIT )
> > +    :name: MEDIA_REQUEST_IOC_REINIT
> > +
> > +
> > +Arguments
> > +=========
> > +
> > +``request_fd``
> > +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.

> I was expecting an argument to this ioctl, with a reference to another
request
> (and I expected the same reference for the MEDIA_IOC_REQUEST_ALLOC
ioctl). I
> think this topic has been discussed several times in the past, but
possibly
> not in venues where everybody was present, so I'll detail the problem
here.

> The idea behind this version of the request API is to split request
handling
> in three parts:

> 1. A request is allocated (MEDIA_IOC_REQUEST_ALLOC) or an existing
request is
> re-initialized (MEDIA_REQUEST_IOC_REINIT). At that point the request
contains
> no property (control, format, ...) to be modified.

> 2. The request is filled with properties to be modified using existing
V4L2
> ioctls (VIDIOC_S_EXT_CTRLS, VIDIOC_SUBDEV_S_FMT,
VIDIOC_SUBDEV_S_SELECTION,
> ..). The current implementation supports controls only, with support for
> formats and selection rectangles to be added later.

> 3. The request is queued with MEDIA_REQUEST_IOC_QUEUE.

> As usual the devil is in the details, and the details of the second step
in
> particular.

> The idea behind splitting request handling in those three steps is to
reuse
> existing V4L2 ioctls as much as possible, and to keep the existing
semantics
> of those ioctls. I believe this isn't possible with the proposed
> MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls, neither for
> controls nor for formats or selection rectangles, although the problem is
more
> limited for controls. I will thus take formats and selection rectangles to
> explain where the problem lies.

> The S_FMT and S_SELECTION semantics on subdevs (and, for that matter, on
video
> nodes too) is that those ioctls don't fail when passed parameters invalid
for
> the device, but return a valid configuration as close as possible to the
> requested one. This allows implementing negotiation of formats and
selection
> rectangles through an iterative process that involves back-and-forth calls
> between user space and kernel space until the application finds a usable
> configuration accepted by the driver, or until it decides to fail because
such
> a configuration doesn't exist. (As a reminder, such negotiation is needed
> because it was deemed impossible to create a proper API that would expose
all
> device constraints in a static way.)

> Formats and selection rectangles are propagated within subdevs from sink
pads
> to source pads by the driver. Sink formats and selection rectangles thus
> influence the result of negotiation on source pads. For instance, on a
scaler
> subdev that can't perform format conversion, the format on the source pad
will
> be dictated by the format on the sink pad. In order to implement S_FMT on
the
> source pad the kernel driver thus needs context information to get the
format
> on the sink pad.

> Without the request API this is easy. The context is either stored in the
> driver-specific device structure (for active formats) or in the file
handle
> (for try formats). With the proposed request API, however, we have no such
> context: the requests are created empty (or re-initialized to be empty).
> Kernel drivers can access the active state of the device, or the state of
> previously queued requests, but can't access the state of prepared but
not yet
> queued requests.

> My proposal to solve this was to link requests by specifying the handle
of a
> previous request to the MEDIA_IOC_REQUEST_ALLOC and
MEDIA_REQUEST_IOC_REINIT
> ioctls. That way kernel drivers would know in which order requests will be
> queued, and would be able to access the correct context information to
> implement the existing semantics of the S_FMT and S_SELECTION ioctls (as
well
> as being able to implement the G_FMT and G_SELECTION ioctls). Note that
the
> problem isn't limited to format and selection rectangles, similar
problems can
> exist when multiple controls are related, such as auto and manual
controls.

> This would result in a more complex API and a more complex request
lifetime
> management, as well as a more complex semantics of the request API. While
I
> thought this was unavoidable, I came to reconsider my position on this
topic,
> and I believe that the current request API proposal is acceptable in that
> regard, provided that we agree that the existing semantics of the S_FMT
and
> S_SELECTION ioctls won't need to be implemented for requests as it won't
be
> possible to do so. The reason that prompted me to change my mind is that I
> believe the API would be simpler to define and simpler to use if we
dropped
> that semantics, and required applications to create requests that are
fully
> valid instead of trying to mangle the content of a request the same way
we do
> for format and selection rectangle negotiation. Obviously, negotiation is
> still needed in order for applications to find a valid configuration, but
we
> have the V4L2_SUBDEV_FORMAT_TRY API for that that, in conjunction with the
> S_FMT and S_SELECTION ioctls, can be used to negotiate a configuration.
The
> negotiated configuration could then be set in a request, and would be
> guaranteed to be valid.

I really like the idea of requiring the requests to be valid. I believe
this is something similar to DRM atomic API and it avoids the state
negotiation hell in the drivers and userspace, where the drivers would be
adjusting settings back and forth and eventually arriving to some values
not matching any of the ones originally given.

[snip]

> > Codec +v2). One such requirement is the ability for devices belonging to
> > the same +pipeline to reconfigure and collaborate closely on a per-frame
> > basis. Another is +efficient support of stateless codecs, which need
> > per-frame controls to be set +asynchronously in order to be used
> > efficiently.

> Do you really mean asynchronously ? I thought the purpose of the request
API
> with codecs was to set controls synchronously with frames.

Set controls synchronized with frames, but submit frames (with matching
controls) to be processed asynchronously, as compared to no request API,
when you would have to: - set controls, - queue buffer, - wait for
decoding, -set controls, - queue buffer, and so on, all synchronously.

[snip]

> > synchronize +so the requested pipeline's topology is applied before the
> > buffers are +processed. This is at the discretion of media controller
> > drivers and is not a +requirement.

> One of the main purposes of the request API is to support atomic
> configuration. Saying that it's simply optional defeats that purpose. I
agree
> that atomicity can't be achieved in all cases, and that there will often
be a
> best-effort approach, but for hardware devices that support atomic
> configuration I think applications need to be able to rely on the driver
> implementing that feature. I would thus put a bit more pressure on drivers
> than just saying it's fully optional.

I think we should say that drivers should make their best effort to apply
the state atomically. However, I'd want to avoid userspace (or users)
blindly relying on the API as guarantying atomicity, since this really
depends on hardware capability.


> > +Buffers queued without an associated request after a request-bound
buffer
> > will +be processed using the state of the hardware at the time of the
> > request +completion.

> I'm not sure to follow that, could you explain in a bit more details ?
Also,
> even more than for controls (see below), I really want to know about use
cases
> for mixing buffers in and outside of requests for the same buffer queue
before
> accepting it.

AFAIR this was defined just to avoid making the API have undefined
behavior. I can't think of any real use case for mixing requests with
non-request processing (at least within the same buffer queue)...

[snip]

> > All the same, controls set without a request are applied +immediately,
> > regardless of whether a request is in use or not.

> This will be tricky to implement, and will potentially generate side
effects
> difficult to handle, and difficult to debug. I understand that this is a
> requirement for codecs, but I don't know the exact use case(s), so I can't
> comment on whether this is the right approach.

I don't think this is a requirement for codecs. Not for stateless ones for
sure, because for them the requirement is to actually use the requests for
setting controls.

As above, I think this was defined to avoid making the API have undefined
behavior, but I can also see some classes of controls which should be
handled regardless of buffers flow, e.g. autofocus, where the userspace
just wants the motor to start moving instantly after the user clicks the
screen (tap-to-focus).

> While I don't necessarily
> reject this feature on principle, I first want to know more about the use
> case(s) before accepting it. If we end up needing it, we'll have to better
> document it, by clearly specifying what is allowed and what isn't (for
> instance, we might decide to allow setting control asynchronously only
when
> they're not simultaneously modified through a queued request), as well as
the
> exact behaviour that applications can expect in the allowed cases.

[snip]

> > +If ``request_fd`` is specified and ``which`` is set to
> > ``V4L2_CTRL_WHICH_REQUEST_VAL`` +during a call to
:ref:`VIDIOC_G_EXT_CTRLS
> > <VIDIOC_G_EXT_CTRLS>`, then the +returned values will be the values
> > currently set for the request (or the +hardware value if none is set)

> Do we really have a need for returning the hardware value ? That seems a
bit
> random to me, wouldn't it be better to return an error instead ?

As per my autofocus example above and also things like various statistics
from camera sensor that don't really depend on other state (e.g. absolute
light intensity), we want to read the values instantly, for example to
drive the userspace 3A loop.

[snip]

> > diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> > b/Documentation/media/uapi/v4l/vidioc-qbuf.rst index
> > 9e448a4aa3aa..c9f55d8340d1 100644
> > --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> > +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> > @@ -98,6 +98,13 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF
> > <VIDIOC_STREAMON>` or
> >  :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
> >
> >  device is closed.
> >
> > +The ``request_fd`` field can be used when queuing to specify the file
> > +descriptor of a request, if requests are in use. Setting it means that
the
> > +buffer will not be passed to the driver until the request itself is
queued.
> > +Also, the driver will apply any setting associated with the request
before
> > +processing the buffer.

> I think we should be more precise with the wording, here and above in the
> documentation. From an API point of view, we need to guarantee that the
> settings from the request are used to process the image capture in the
buffer,
> or the image output from the buffer.

Making the API guarantee that would essentially eliminate usage of this API
from any hardware that can't guarantee 100% atomicity of all the state
within request, which I'd say would be actually most of existing hardware.

> Drivers don't process buffers other than
> queuing them to the hardware, so saying that the driver will apply
settings
> before processing the buffer is a bit vague in my opinion, and also
possibly
> not very accurate depending on the hardware.

The question is whether we can guarantee anything more precise.


> Saying that controls are applied right before the buffer is processed by
the
> driver is inaccurate : for example exposure time might need several
frames to
> take effect, and might thus need to be applied a few frames before the
buffer
> is queued to the hardware for capture.

I believe this would be typically accounted for the userspace queuing such
exposure setting several requests before. Hardware that would be able to
handle this on its own, if such exists, would be treated as 0 frame delay
of exposure setting. (We might need a way to report that for drivers,
though. Right now we can just have configuration files for userspace.)

> The other way around is possible too :
> the hardware might support a buffer queue but have no queue for controls.
> Controls would then need to be applied a few frames after the buffer is
queued
> to the hardware in order to take effect for the right buffer.

> We're specifying an API that carries very strong semantics across multiple
> components and multiple objects, so I think it's very important to be as
> precise as possible.

Or, do you just mean, s/before processing the buffer/before hardware starts
processing the buffer/?

Best regards,
Tomasz

Re: [PATCHv13 22/28] Documentation: v4l: document request API

[Hans Verkuil]

On 24/05/18 06:32, Tomasz Figa wrote:
> Hi Laurent,
> 
> Thanks for detailed review. Please let me add my thoughts inline as well.
> 
> On Fri, May 18, 2018 at 11:46 PM Laurent Pinchart <
> laurent.pinchart@ideasonboard.com> wrote:
> 
>> Hi Hans,
> 
>> Thank you for the patch.
> 
>> On Thursday, 3 May 2018 17:53:12 EEST Hans Verkuil wrote:
> 
> [snip]
> 
>>> +Name
>>> +====
>>> +
>>> +MEDIA_REQUEST_IOC_REINIT - Re-initialize a request
>>> +
>>> +
>>> +Synopsis
>>> +========
>>> +
>>> +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_REINIT )
>>> +    :name: MEDIA_REQUEST_IOC_REINIT
>>> +
>>> +
>>> +Arguments
>>> +=========
>>> +
>>> +``request_fd``
>>> +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
> 
>> I was expecting an argument to this ioctl, with a reference to another
> request
>> (and I expected the same reference for the MEDIA_IOC_REQUEST_ALLOC
> ioctl). I
>> think this topic has been discussed several times in the past, but
> possibly
>> not in venues where everybody was present, so I'll detail the problem
> here.
> 
>> The idea behind this version of the request API is to split request
> handling
>> in three parts:
> 
>> 1. A request is allocated (MEDIA_IOC_REQUEST_ALLOC) or an existing
> request is
>> re-initialized (MEDIA_REQUEST_IOC_REINIT). At that point the request
> contains
>> no property (control, format, ...) to be modified.
> 
>> 2. The request is filled with properties to be modified using existing
> V4L2
>> ioctls (VIDIOC_S_EXT_CTRLS, VIDIOC_SUBDEV_S_FMT,
> VIDIOC_SUBDEV_S_SELECTION,
>> ..). The current implementation supports controls only, with support for
>> formats and selection rectangles to be added later.
> 
>> 3. The request is queued with MEDIA_REQUEST_IOC_QUEUE.
> 
>> As usual the devil is in the details, and the details of the second step
> in
>> particular.
> 
>> The idea behind splitting request handling in those three steps is to
> reuse
>> existing V4L2 ioctls as much as possible, and to keep the existing
> semantics
>> of those ioctls. I believe this isn't possible with the proposed
>> MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls, neither for
>> controls nor for formats or selection rectangles, although the problem is
> more
>> limited for controls. I will thus take formats and selection rectangles to
>> explain where the problem lies.
> 
>> The S_FMT and S_SELECTION semantics on subdevs (and, for that matter, on
> video
>> nodes too) is that those ioctls don't fail when passed parameters invalid
> for
>> the device, but return a valid configuration as close as possible to the
>> requested one. This allows implementing negotiation of formats and
> selection
>> rectangles through an iterative process that involves back-and-forth calls
>> between user space and kernel space until the application finds a usable
>> configuration accepted by the driver, or until it decides to fail because
> such
>> a configuration doesn't exist. (As a reminder, such negotiation is needed
>> because it was deemed impossible to create a proper API that would expose
> all
>> device constraints in a static way.)
> 
>> Formats and selection rectangles are propagated within subdevs from sink
> pads
>> to source pads by the driver. Sink formats and selection rectangles thus
>> influence the result of negotiation on source pads. For instance, on a
> scaler
>> subdev that can't perform format conversion, the format on the source pad
> will
>> be dictated by the format on the sink pad. In order to implement S_FMT on
> the
>> source pad the kernel driver thus needs context information to get the
> format
>> on the sink pad.
> 
>> Without the request API this is easy. The context is either stored in the
>> driver-specific device structure (for active formats) or in the file
> handle
>> (for try formats). With the proposed request API, however, we have no such
>> context: the requests are created empty (or re-initialized to be empty).
>> Kernel drivers can access the active state of the device, or the state of
>> previously queued requests, but can't access the state of prepared but
> not yet
>> queued requests.
> 
>> My proposal to solve this was to link requests by specifying the handle
> of a
>> previous request to the MEDIA_IOC_REQUEST_ALLOC and
> MEDIA_REQUEST_IOC_REINIT
>> ioctls. That way kernel drivers would know in which order requests will be
>> queued, and would be able to access the correct context information to
>> implement the existing semantics of the S_FMT and S_SELECTION ioctls (as
> well
>> as being able to implement the G_FMT and G_SELECTION ioctls). Note that
> the
>> problem isn't limited to format and selection rectangles, similar
> problems can
>> exist when multiple controls are related, such as auto and manual
> controls.
> 
>> This would result in a more complex API and a more complex request
> lifetime
>> management, as well as a more complex semantics of the request API. While
> I
>> thought this was unavoidable, I came to reconsider my position on this
> topic,
>> and I believe that the current request API proposal is acceptable in that
>> regard, provided that we agree that the existing semantics of the S_FMT
> and
>> S_SELECTION ioctls won't need to be implemented for requests as it won't
> be
>> possible to do so. The reason that prompted me to change my mind is that I
>> believe the API would be simpler to define and simpler to use if we
> dropped
>> that semantics, and required applications to create requests that are
> fully
>> valid instead of trying to mangle the content of a request the same way
> we do
>> for format and selection rectangle negotiation. Obviously, negotiation is
>> still needed in order for applications to find a valid configuration, but
> we
>> have the V4L2_SUBDEV_FORMAT_TRY API for that that, in conjunction with the
>> S_FMT and S_SELECTION ioctls, can be used to negotiate a configuration.
> The
>> negotiated configuration could then be set in a request, and would be
>> guaranteed to be valid.
> 
> I really like the idea of requiring the requests to be valid. I believe
> this is something similar to DRM atomic API and it avoids the state
> negotiation hell in the drivers and userspace, where the drivers would be
> adjusting settings back and forth and eventually arriving to some values
> not matching any of the ones originally given.

I agree. If there is a desire to 'try' a request and let the driver modify
incompatible values, then we can make a new ioctl for that, specifically
for that purpose. But I'm not sure if that's really needed.

> 
> [snip]
> 
>>> Codec +v2). One such requirement is the ability for devices belonging to
>>> the same +pipeline to reconfigure and collaborate closely on a per-frame
>>> basis. Another is +efficient support of stateless codecs, which need
>>> per-frame controls to be set +asynchronously in order to be used
>>> efficiently.
> 
>> Do you really mean asynchronously ? I thought the purpose of the request
> API
>> with codecs was to set controls synchronously with frames.
> 
> Set controls synchronized with frames, but submit frames (with matching
> controls) to be processed asynchronously, as compared to no request API,
> when you would have to: - set controls, - queue buffer, - wait for
> decoding, -set controls, - queue buffer, and so on, all synchronously.
> 
> [snip]
> 
>>> synchronize +so the requested pipeline's topology is applied before the
>>> buffers are +processed. This is at the discretion of media controller
>>> drivers and is not a +requirement.
> 
>> One of the main purposes of the request API is to support atomic
>> configuration. Saying that it's simply optional defeats that purpose. I
> agree
>> that atomicity can't be achieved in all cases, and that there will often
> be a
>> best-effort approach, but for hardware devices that support atomic
>> configuration I think applications need to be able to rely on the driver
>> implementing that feature. I would thus put a bit more pressure on drivers
>> than just saying it's fully optional.
> 
> I think we should say that drivers should make their best effort to apply
> the state atomically. However, I'd want to avoid userspace (or users)
> blindly relying on the API as guarantying atomicity, since this really
> depends on hardware capability.

VIDIOC_S_EXT_CTRLS works like that as well: it does a best-effort to ensure
atomicity, but it is constrained by HW capabilities.

> 
> 
>>> +Buffers queued without an associated request after a request-bound
> buffer
>>> will +be processed using the state of the hardware at the time of the
>>> request +completion.
> 
>> I'm not sure to follow that, could you explain in a bit more details ?
> Also,
>> even more than for controls (see below), I really want to know about use
> cases
>> for mixing buffers in and outside of requests for the same buffer queue
> before
>> accepting it.
> 
> AFAIR this was defined just to avoid making the API have undefined
> behavior. I can't think of any real use case for mixing requests with
> non-request processing (at least within the same buffer queue)...

In my upcoming v15 of this patch series I prohibit mixing of request
and non-request buffers: once the first request buffer is queued to a
vb2_queue all following buffers must also be part of a request until the
queue is canceled.

This is a fair restriction in my opinion.

> 
> [snip]
> 
>>> All the same, controls set without a request are applied +immediately,
>>> regardless of whether a request is in use or not.
> 
>> This will be tricky to implement, and will potentially generate side
> effects
>> difficult to handle, and difficult to debug. I understand that this is a
>> requirement for codecs, but I don't know the exact use case(s), so I can't
>> comment on whether this is the right approach.
> 
> I don't think this is a requirement for codecs. Not for stateless ones for
> sure, because for them the requirement is to actually use the requests for
> setting controls.
> 
> As above, I think this was defined to avoid making the API have undefined
> behavior, but I can also see some classes of controls which should be
> handled regardless of buffers flow, e.g. autofocus, where the userspace
> just wants the motor to start moving instantly after the user clicks the
> screen (tap-to-focus).

I implemented the same for VIDIOC_S_EXT_CTRLS: once a queue switches to
using requests it is no longer allowed to set controls outside of requests.
You can still read the current HW state, though.

I'm not sure yet whether this is what I want. I think there are controls
where using requests do not necessarily make sense. E.g. setting the
powerline frequency (50 vs 60 Hz): really not something you need a request
for.

I see the following options here:

1) Allow setting controls directly or via requests as implemented in v13:
   it is up to userspace to not do anything stupid.
2) Never allow mixing (as is currently implemented in v15)
3) Have drivers explicitly mark controls that should only be set via
   requests (new control flag). Thus drivers can prevent mixing on a
   per-control basis.
4) Once a control was made part of a request then the core marks it
   as 'request-only' and will refuse setting it directly. Basically the
   same as 3, but done automatically.

Even though I implemented 2 for v15, I am not actually happy about it.
I like 1 or 3 better. Option 4, while relatively easy to implement, leads
to unexpected side-effects (you add a control to a request and suddenly
you can't set it directly anymore).

Personally I do not see a problem with 1. It makes debugging easier
(you can still set controls while streaming to test behavior) and I think
it is not unreasonable to expect userspace to do sane things. Adding a
fat notice to the spec that says that setting the same control both directly
and through requests can lead to undefined results seems sufficient to me.

I might actually switch back to option 1 before I post v15, I'm doing a
good job of convincing myself :-)

Regards,

	Hans

Re: [PATCHv13 03/28] media-request: implement media requests

[Hans Verkuil]

On 04/05/18 14:27, Sakari Ailus wrote:
> Hi Hans,
> 
> I've read this patch a large number of times and I think also the details
> begin to seem sound. A few comments below.
> 
> On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Add initial media request support:
>>
>> 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
>> 2) Add struct media_request to store request objects.
>> 3) Add struct media_request_object to represent a request object.
>> 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
>>
>> Basic lifecycle: the application allocates a request, adds
>> objects to it, queues the request, polls until it is completed
>> and can then read the final values of the objects at the time
>> of completion. When it closes the file descriptor the request
>> memory will be freed (actually, when the last user of that request
>> releases the request).
>>
>> Drivers will bind an object to a request (the 'adds objects to it'
>> phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
>> validated (req_validate op), then queued (the req_queue op).
>>
>> When done with an object it can either be unbound from the request
>> (e.g. when the driver has finished with a vb2 buffer) or marked as
>> completed (e.g. for controls associated with a buffer). When all
>> objects in the request are completed (or unbound), then the request
>> fd will signal an exception (poll).
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/Makefile        |   3 +-
>>  drivers/media/media-device.c  |  14 ++
>>  drivers/media/media-request.c | 407 ++++++++++++++++++++++++++++++++++
>>  include/media/media-device.h  |  16 ++
>>  include/media/media-request.h | 244 ++++++++++++++++++++
>>  5 files changed, 683 insertions(+), 1 deletion(-)
>>  create mode 100644 drivers/media/media-request.c
>>  create mode 100644 include/media/media-request.h
>>
>> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
>> index 594b462ddf0e..985d35ec6b29 100644
>> --- a/drivers/media/Makefile
>> +++ b/drivers/media/Makefile
>> @@ -3,7 +3,8 @@
>>  # Makefile for the kernel multimedia device drivers.
>>  #
>>  
>> -media-objs	:= media-device.o media-devnode.o media-entity.o
>> +media-objs	:= media-device.o media-devnode.o media-entity.o \
>> +		   media-request.o
>>  
>>  #
>>  # I2C drivers should come before other drivers, otherwise they'll fail
>> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
>> index 35e81f7c0d2f..bb6a64acd3f0 100644
>> --- a/drivers/media/media-device.c
>> +++ b/drivers/media/media-device.c
>> @@ -32,6 +32,7 @@
>>  #include <media/media-device.h>
>>  #include <media/media-devnode.h>
>>  #include <media/media-entity.h>
>> +#include <media/media-request.h>
>>  
>>  #ifdef CONFIG_MEDIA_CONTROLLER
>>  
>> @@ -366,6 +367,15 @@ static long media_device_get_topology(struct media_device *mdev,
>>  	return ret;
>>  }
>>  
>> +static long media_device_request_alloc(struct media_device *mdev,
>> +				       struct media_request_alloc *alloc)
>> +{
>> +	if (!mdev->ops || !mdev->ops->req_validate || !mdev->ops->req_queue)
>> +		return -ENOTTY;
>> +
>> +	return media_request_alloc(mdev, alloc);
>> +}
>> +
>>  static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
>>  {
>>  	/* All media IOCTLs are _IOWR() */
>> @@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>>  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>>  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>>  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
>> +	MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
>>  };
>>  
>>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
>> @@ -686,6 +697,8 @@ void media_device_init(struct media_device *mdev)
>>  	INIT_LIST_HEAD(&mdev->pads);
>>  	INIT_LIST_HEAD(&mdev->links);
>>  	INIT_LIST_HEAD(&mdev->entity_notify);
>> +
>> +	mutex_init(&mdev->req_queue_mutex);
>>  	mutex_init(&mdev->graph_mutex);
>>  	ida_init(&mdev->entity_internal_idx);
>>  
>> @@ -699,6 +712,7 @@ void media_device_cleanup(struct media_device *mdev)
>>  	mdev->entity_internal_idx_max = 0;
>>  	media_graph_walk_cleanup(&mdev->pm_count_walk);
>>  	mutex_destroy(&mdev->graph_mutex);
>> +	mutex_destroy(&mdev->req_queue_mutex);
>>  }
>>  EXPORT_SYMBOL_GPL(media_device_cleanup);
>>  
>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>> new file mode 100644
>> index 000000000000..c216c4ab628b
>> --- /dev/null
>> +++ b/drivers/media/media-request.c
>> @@ -0,0 +1,407 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Media device request objects
>> + *
>> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
>> + * Copyright (C) 2018 Intel Corporation
>> + * Copyright (C) 2018 Google, Inc.
>> + *
>> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
>> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
>> + */
>> +
>> +#include <linux/anon_inodes.h>
>> +#include <linux/file.h>
>> +
>> +#include <media/media-device.h>
>> +#include <media/media-request.h>
>> +
>> +static const char * const request_state[] = {
>> +	[MEDIA_REQUEST_STATE_IDLE]	 = "idle",
>> +	[MEDIA_REQUEST_STATE_VALIDATING] = "validating",
>> +	[MEDIA_REQUEST_STATE_QUEUED]	 = "queued",
>> +	[MEDIA_REQUEST_STATE_COMPLETE]	 = "complete",
>> +	[MEDIA_REQUEST_STATE_CLEANING]	 = "cleaning",
>> +};
>> +
>> +static const char *
>> +media_request_state_str(enum media_request_state state)
>> +{
>> +	if (WARN_ON(state >= ARRAY_SIZE(request_state)))
>> +		return "invalid";
>> +	return request_state[state];
>> +}
>> +
>> +static void media_request_clean(struct media_request *req)
>> +{
>> +	struct media_request_object *obj, *obj_safe;
>> +
>> +	WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_CLEANING);
>> +
>> +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
>> +		media_request_object_unbind(obj);
>> +		media_request_object_put(obj);
>> +	}
>> +
>> +	req->num_incomplete_objects = 0;
> 
> The number of incomplete objects should be already zero here. I'd think
> that a different number would suggest that something has gone very wrong
> and should be complained about. How about adding
> WARN_ON(req->num_incomplete_objects) above this line? 

Makes sense. Added.

> 
>> +	wake_up_interruptible_all(&req->poll_wait);
>> +}
>> +
>> +static void media_request_release(struct kref *kref)
>> +{
>> +	struct media_request *req =
>> +		container_of(kref, struct media_request, kref);
>> +	struct media_device *mdev = req->mdev;
>> +
>> +	dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
>> +
>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
>> +
>> +	media_request_clean(req);
>> +
>> +	if (mdev->ops->req_free)
>> +		mdev->ops->req_free(req);
>> +	else
>> +		kfree(req);
>> +}
>> +
>> +void media_request_put(struct media_request *req)
>> +{
>> +	kref_put(&req->kref, media_request_release);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_put);
>> +
>> +static int media_request_close(struct inode *inode, struct file *filp)
>> +{
>> +	struct media_request *req = filp->private_data;
>> +
>> +	media_request_put(req);
>> +	return 0;
>> +}
>> +
>> +static unsigned int media_request_poll(struct file *filp,
>> +				       struct poll_table_struct *wait)
>> +{
>> +	struct media_request *req = filp->private_data;
>> +	unsigned long flags;
>> +	unsigned int ret = 0;
>> +	enum media_request_state state;
>> +
>> +	if (!(poll_requested_events(wait) & POLLPRI))
>> +		return 0;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	state = atomic_read(&req->state);
>> +
>> +	if (state == MEDIA_REQUEST_STATE_COMPLETE) {
>> +		ret = POLLPRI;
>> +		goto unlock;
>> +	}
>> +	if (state != MEDIA_REQUEST_STATE_QUEUED) {
>> +		ret = POLLERR;
>> +		goto unlock;
>> +	}
>> +
>> +	poll_wait(filp, &req->poll_wait, wait);
>> +
>> +unlock:
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	return ret;
>> +}
>> +
>> +static long media_request_ioctl_queue(struct media_request *req)
>> +{
>> +	struct media_device *mdev = req->mdev;
>> +	enum media_request_state state;
>> +	unsigned long flags;
>> +	int ret = 0;
> 
> ret is unconditionally assigned below, no need to initialise here.

Fixed.

> 
>> +
>> +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
>> +
>> +	/*
>> +	 * Ensure the request that is validated will be the one that gets queued
>> +	 * next by serialising the queueing process. This mutex is also used
>> +	 * to serialize with canceling a vb2 queue and with setting values such
>> +	 * as controls in a request.
>> +	 */
>> +	mutex_lock(&mdev->req_queue_mutex);
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
>> +			       MEDIA_REQUEST_STATE_VALIDATING);
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	if (state != MEDIA_REQUEST_STATE_IDLE) {
>> +		dev_dbg(mdev->dev,
>> +			"request: unable to queue %s, request in state %s\n",
>> +			req->debug_str, media_request_state_str(state));
>> +		mutex_unlock(&mdev->req_queue_mutex);
>> +		return -EBUSY;
>> +	}
>> +
>> +	ret = mdev->ops->req_validate(req);
>> +
>> +	/*
>> +	 * If the req_validate was successful, then we mark the state as QUEUED
>> +	 * and call req_queue. The reason we set the state first is that this
>> +	 * allows req_queue to unbind or complete the queued objects in case
>> +	 * they are immediately 'consumed'. State changes from QUEUED to another
>> +	 * state can only happen if either the driver changes the state or if
>> +	 * the user cancels the vb2 queue. The driver can only change the state
>> +	 * after each object is queued through the req_queue op (and note that
>> +	 * that op cannot fail), so setting the state to QUEUED up front is
>> +	 * safe.
>> +	 *
>> +	 * The other reason for changing the state is if the vb2 queue is
>> +	 * canceled, and that uses the req_queue_mutex which is still locked
>> +	 * while req_queue is called, so that's safe as well.
>> +	 */
>> +	atomic_set(&req->state,
>> +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);
>> +
>> +	if (!ret)
>> +		mdev->ops->req_queue(req);
>> +
>> +	mutex_unlock(&mdev->req_queue_mutex);
>> +
>> +	if (ret)
>> +		dev_dbg(mdev->dev, "request: can't queue %s (%d)\n",
>> +			req->debug_str, ret);
>> +	else
>> +		media_request_get(req);
>> +
>> +	return ret;
>> +}
>> +
>> +static long media_request_ioctl_reinit(struct media_request *req)
>> +{
>> +	struct media_device *mdev = req->mdev;
>> +	unsigned long flags;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE &&
>> +	    atomic_read(&req->state) != MEDIA_REQUEST_STATE_COMPLETE) {
>> +		dev_dbg(mdev->dev,
>> +			"request: %s not in idle or complete state, cannot reinit\n",
>> +			req->debug_str);
>> +		spin_unlock_irqrestore(&req->lock, flags);
>> +		return -EBUSY;
>> +	}
>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +
>> +	media_request_clean(req);
>> +
>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
>> +
>> +	return 0;
>> +}
>> +
>> +static long media_request_ioctl(struct file *filp, unsigned int cmd,
>> +				unsigned long arg)
>> +{
>> +	struct media_request *req = filp->private_data;
>> +
>> +	switch (cmd) {
>> +	case MEDIA_REQUEST_IOC_QUEUE:
>> +		return media_request_ioctl_queue(req);
>> +	case MEDIA_REQUEST_IOC_REINIT:
>> +		return media_request_ioctl_reinit(req);
>> +	default:
>> +		return -ENOIOCTLCMD;
>> +	}
>> +}
>> +
>> +static const struct file_operations request_fops = {
>> +	.owner = THIS_MODULE,
>> +	.poll = media_request_poll,
>> +	.unlocked_ioctl = media_request_ioctl,
>> +	.release = media_request_close,
>> +};
>> +
>> +int media_request_alloc(struct media_device *mdev,
>> +			struct media_request_alloc *alloc)
>> +{
>> +	struct media_request *req;
>> +	struct file *filp;
>> +	char comm[TASK_COMM_LEN];
>> +	int fd;
>> +	int ret;
>> +
>> +	/* Either both are NULL or both are non-NULL */
>> +	if (WARN_ON(!mdev->ops->req_alloc ^ !mdev->ops->req_free))
>> +		return -ENOMEM;
>> +
>> +	fd = get_unused_fd_flags(O_CLOEXEC);
>> +	if (fd < 0)
>> +		return fd;
>> +
>> +	filp = anon_inode_getfile("request", &request_fops, NULL, O_CLOEXEC);
>> +	if (IS_ERR(filp)) {
>> +		ret = PTR_ERR(filp);
>> +		goto err_put_fd;
>> +	}
>> +
>> +	if (mdev->ops->req_alloc)
>> +		req = mdev->ops->req_alloc(mdev);
>> +	else
>> +		req = kzalloc(sizeof(*req), GFP_KERNEL);
>> +	if (!req) {
>> +		ret = -ENOMEM;
>> +		goto err_fput;
>> +	}
>> +
>> +	filp->private_data = req;
>> +	req->mdev = mdev;
>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
>> +	req->num_incomplete_objects = 0;
>> +	kref_init(&req->kref);
>> +	INIT_LIST_HEAD(&req->objects);
>> +	spin_lock_init(&req->lock);
>> +	init_waitqueue_head(&req->poll_wait);
>> +
>> +	alloc->fd = fd;
>> +
>> +	get_task_comm(comm, current);
>> +	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
>> +		 comm, fd);
>> +	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
>> +
>> +	fd_install(fd, filp);
>> +
>> +	return 0;
>> +
>> +err_fput:
>> +	fput(filp);
>> +
>> +err_put_fd:
>> +	put_unused_fd(fd);
>> +
>> +	return ret;
>> +}
>> +
>> +static void media_request_object_release(struct kref *kref)
>> +{
>> +	struct media_request_object *obj =
>> +		container_of(kref, struct media_request_object, kref);
>> +	struct media_request *req = obj->req;
>> +
>> +	if (req)
>> +		media_request_object_unbind(obj);
>> +	obj->ops->release(obj);
>> +}
>> +
>> +void media_request_object_put(struct media_request_object *obj)
>> +{
>> +	kref_put(&obj->kref, media_request_object_release);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_put);
>> +
>> +void media_request_object_init(struct media_request_object *obj)
>> +{
>> +	obj->ops = NULL;
>> +	obj->req = NULL;
>> +	obj->priv = NULL;
>> +	obj->completed = false;
>> +	INIT_LIST_HEAD(&obj->list);
>> +	kref_init(&obj->kref);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_init);
>> +
>> +int media_request_object_bind(struct media_request *req,
>> +			      const struct media_request_object_ops *ops,
>> +			      void *priv,
>> +			      struct media_request_object *obj)
>> +{
>> +	unsigned long flags;
>> +	int ret = -EBUSY;
>> +
>> +	if (WARN_ON(!ops->release))
>> +		return -EPERM;
>> +
>> +	obj->req = req;
>> +	obj->ops = ops;
>> +	obj->priv = priv;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +
>> +	if (WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE))
>> +		goto unlock;
> 
> Is this worth a kernel warning, or rather how the drivers / other framework
> bits (e.g. VB2) prevent user from binding objects to non-idle requests?
> Even if you added a similar check to the caller, the request state could
> well change in the meantime.
> 
> Perhaps add __must_check to the return value?

This has already been addressed in v14 with media_request_lock_for_update().

> 
>> +
>> +	list_add_tail(&obj->list, &req->objects);
>> +	req->num_incomplete_objects++;
>> +	ret = 0;
>> +
>> +unlock:
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_bind);
>> +
>> +void media_request_object_unbind(struct media_request_object *obj)
>> +{
>> +	struct media_request *req = obj->req;
>> +	enum media_request_state state;
>> +	unsigned long flags;
>> +	bool completed = false;
>> +
>> +	if (WARN_ON(!req))
>> +		return;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	list_del(&obj->list);
>> +	obj->req = NULL;
>> +
>> +	state = atomic_read(&req->state);
>> +
>> +	if (state == MEDIA_REQUEST_STATE_COMPLETE ||
>> +	    state == MEDIA_REQUEST_STATE_CLEANING)
>> +		goto unlock;
>> +
>> +	if (WARN_ON(state == MEDIA_REQUEST_STATE_VALIDATING))
>> +		goto unlock;
>> +
>> +	if (WARN_ON(!req->num_incomplete_objects))
>> +		goto unlock;
>> +
>> +	req->num_incomplete_objects--;
>> +	if (state == MEDIA_REQUEST_STATE_QUEUED &&
>> +	    !req->num_incomplete_objects) {
>> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
>> +		completed = true;
>> +		wake_up_interruptible_all(&req->poll_wait);
>> +	}
>> +
>> +unlock:
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	if (obj->ops->unbind)
>> +		obj->ops->unbind(obj);
>> +	if (completed)
>> +		media_request_put(req);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_unbind);
>> +
>> +void media_request_object_complete(struct media_request_object *obj)
>> +{
>> +	struct media_request *req = obj->req;
>> +	unsigned long flags;
>> +	bool completed = false;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	if (obj->completed)
>> +		goto unlock;
>> +	obj->completed = true;
>> +	if (WARN_ON(!req->num_incomplete_objects) ||
>> +	    WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_QUEUED))
>> +		goto unlock;
>> +
>> +	if (!--req->num_incomplete_objects) {
>> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
>> +		wake_up_interruptible_all(&req->poll_wait);
>> +		completed = true;
>> +	}
>> +unlock:
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	if (completed)
>> +		media_request_put(req);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_complete);
>> diff --git a/include/media/media-device.h b/include/media/media-device.h
>> index bcc6ec434f1f..7d855823341c 100644
>> --- a/include/media/media-device.h
>> +++ b/include/media/media-device.h
>> @@ -27,6 +27,7 @@
>>  
>>  struct ida;
>>  struct device;
>> +struct media_device;
>>  
>>  /**
>>   * struct media_entity_notify - Media Entity Notify
>> @@ -50,10 +51,21 @@ struct media_entity_notify {
>>   * struct media_device_ops - Media device operations
>>   * @link_notify: Link state change notification callback. This callback is
>>   *		 called with the graph_mutex held.
>> + * @req_alloc: Allocate a request
>> + * @req_free: Free a request
>> + * @req_validate: Validate a request, but do not queue yet
>> + * @req_queue: Queue a validated request, cannot fail. If something goes
>> + *	       wrong when queueing this request then it should be marked
>> + *	       as such internally in the driver and any related buffers
>> + *	       must eventually return to vb2 with state VB2_BUF_STATE_ERROR.
>>   */
>>  struct media_device_ops {
>>  	int (*link_notify)(struct media_link *link, u32 flags,
>>  			   unsigned int notification);
>> +	struct media_request *(*req_alloc)(struct media_device *mdev);
>> +	void (*req_free)(struct media_request *req);
>> +	int (*req_validate)(struct media_request *req);
>> +	void (*req_queue)(struct media_request *req);
>>  };
>>  
>>  /**
>> @@ -88,6 +100,8 @@ struct media_device_ops {
>>   * @disable_source: Disable Source Handler function pointer
>>   *
>>   * @ops:	Operation handler callbacks
>> + * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
>> + *		     media device.
>>   *
>>   * This structure represents an abstract high-level media device. It allows easy
>>   * access to entities and provides basic media device-level support. The
>> @@ -158,6 +172,8 @@ struct media_device {
>>  	void (*disable_source)(struct media_entity *entity);
>>  
>>  	const struct media_device_ops *ops;
>> +
>> +	struct mutex req_queue_mutex;
>>  };
>>  
>>  /* We don't need to include pci.h or usb.h here */
>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>> new file mode 100644
>> index 000000000000..e39122dfd717
>> --- /dev/null
>> +++ b/include/media/media-request.h
>> @@ -0,0 +1,244 @@
>> +// SPDX-License-Identifier: GPL-2.0-only
>> +/*
>> + * Media device request objects
>> + *
>> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
>> + * Copyright (C) 2018 Intel Corporation
>> + *
>> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
>> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
>> + */
>> +
>> +#ifndef MEDIA_REQUEST_H
>> +#define MEDIA_REQUEST_H
>> +
>> +#include <linux/list.h>
>> +#include <linux/slab.h>
>> +#include <linux/spinlock.h>
>> +#include <linux/atomic.h>
>> +
>> +#include <media/media-device.h>
>> +
>> +/**
>> + * enum media_request_state - media request state
>> + *
>> + * @MEDIA_REQUEST_STATE_IDLE:		Idle
>> + * @MEDIA_REQUEST_STATE_VALIDATING:	Validating the request, no state changes
>> + *					allowed
>> + * @MEDIA_REQUEST_STATE_QUEUED:		Queued
>> + * @MEDIA_REQUEST_STATE_COMPLETE:	Completed, the request is done
>> + * @MEDIA_REQUEST_STATE_CLEANING:	Cleaning, the request is being re-inited
>> + */
>> +enum media_request_state {
>> +	MEDIA_REQUEST_STATE_IDLE,
>> +	MEDIA_REQUEST_STATE_VALIDATING,
>> +	MEDIA_REQUEST_STATE_QUEUED,
>> +	MEDIA_REQUEST_STATE_COMPLETE,
>> +	MEDIA_REQUEST_STATE_CLEANING,
>> +};
>> +
>> +struct media_request_object;
>> +
>> +/**
>> + * struct media_request - Media device request
>> + * @mdev: Media device this request belongs to
>> + * @kref: Reference count
>> + * @debug_str: Prefix for debug messages (process name:fd)
>> + * @state: The state of the request
>> + * @objects: List of @struct media_request_object request objects
>> + * @num_objects: The number of objects in the request
>> + * @num_incompleted_objects: The number of incomplete objects in the request
>> + * @poll_wait: Wait queue for poll
>> + * @lock: Serializes access to this struct
>> + */
>> +struct media_request {
>> +	struct media_device *mdev;
>> +	struct kref kref;
>> +	char debug_str[TASK_COMM_LEN + 11];
>> +	atomic_t state;
>> +	struct list_head objects;
>> +	unsigned int num_incomplete_objects;
>> +	struct wait_queue_head poll_wait;
>> +	spinlock_t lock;
>> +};
>> +
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +
>> +/**
>> + * media_request_get - Get the media request
>> + *
>> + * @req: The request
>> + *
>> + * Get the media request.
>> + */
>> +static inline void media_request_get(struct media_request *req)
>> +{
>> +	kref_get(&req->kref);
>> +}
>> +
>> +/**
>> + * media_request_put - Put the media request
>> + *
>> + * @req: The request
>> + *
>> + * Put the media request. The media request will be released
>> + * when the refcount reaches 0.
>> + */
>> +void media_request_put(struct media_request *req);
>> +
>> +/**
>> + * media_request_alloc - Allocate the media request
>> + *
>> + * @mdev: Media device this request belongs to
>> + * @alloc: Store the request's file descriptor in this struct
>> + *
>> + * Allocated the media request and put the fd in @alloc->fd.
>> + */
>> +int media_request_alloc(struct media_device *mdev,
>> +			struct media_request_alloc *alloc);
>> +
>> +#else
>> +
>> +static inline void media_request_get(struct media_request *req)
>> +{
>> +}
>> +
>> +static inline void media_request_put(struct media_request *req)
>> +{
>> +}
>> +
>> +#endif
>> +
>> +/**
>> + * struct media_request_object_ops - Media request object operations
>> + * @prepare: Validate and prepare the request object, optional.
>> + * @unprepare: Unprepare the request object, optional.
>> + * @queue: Queue the request object, optional.
>> + * @unbind: Unbind the request object, optional.
>> + * @release: Release the request object, required.
>> + */
>> +struct media_request_object_ops {
>> +	int (*prepare)(struct media_request_object *object);
>> +	void (*unprepare)(struct media_request_object *object);
>> +	void (*queue)(struct media_request_object *object);
>> +	void (*unbind)(struct media_request_object *object);
>> +	void (*release)(struct media_request_object *object);
>> +};
>> +
>> +/**
>> + * struct media_request_object - An opaque object that belongs to a media
>> + *				 request
>> + *
>> + * @ops: object's operations
>> + * @priv: object's priv pointer
>> + * @req: the request this object belongs to (can be NULL)
>> + * @list: List entry of the object for @struct media_request
>> + * @kref: Reference count of the object, acquire before releasing req->lock
>> + * @completed: If true, then this object was completed.
>> + *
>> + * An object related to the request. This struct is embedded in the
>> + * larger object data.
>> + */
>> +struct media_request_object {
>> +	const struct media_request_object_ops *ops;
>> +	void *priv;
>> +	struct media_request *req;
>> +	struct list_head list;
>> +	struct kref kref;
>> +	bool completed;
>> +};
>> +
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +
>> +/**
>> + * media_request_object_get - Get a media request object
>> + *
>> + * @obj: The object
>> + *
>> + * Get a media request object.
>> + */
>> +static inline void media_request_object_get(struct media_request_object *obj)
>> +{
>> +	kref_get(&obj->kref);
>> +}
>> +
>> +/**
>> + * media_request_object_put - Put a media request object
>> + *
>> + * @obj: The object
>> + *
>> + * Put a media request object. Once all references are gone, the
>> + * object's memory is released.
>> + */
>> +void media_request_object_put(struct media_request_object *obj);
>> +
>> +/**
>> + * media_request_object_init - Initialise a media request object
>> + *
>> + * Initialise a media request object. The object will be released using the
>> + * release callback of the ops once it has no references (this function
>> + * initialises references to one).
>> + */
>> +void media_request_object_init(struct media_request_object *obj);
>> +
>> +/**
>> + * media_request_object_bind - Bind a media request object to a request
> 
> Argument documentation is missing.
> 
> I think you should also say that "every bound object must be unbound later
> on".

Done.

> 
>> + */
>> +int media_request_object_bind(struct media_request *req,
>> +			      const struct media_request_object_ops *ops,
>> +			      void *priv,
>> +			      struct media_request_object *obj);
>> +
>> +/**
>> + * media_request_object_unbind - Unbind a media request object
>> + *
>> + * @obj: The object
>> + *
>> + * Unbind the media request object from the request.
>> + */
>> +void media_request_object_unbind(struct media_request_object *obj);
>> +
>> +/**
>> + * media_request_object_complete - Mark the media request object as complete
>> + *
>> + * @obj: The object
>> + *
>> + * Mark the media request object as complete.
> 
> Add:
> 
> Only bound request objects may be completed.

Done.

> 
>> + */
>> +void media_request_object_complete(struct media_request_object *obj);
>> +
>> +#else
>> +
>> +static inline void media_request_object_get(struct media_request_object *obj)
>> +{
>> +}
>> +
>> +static inline void media_request_object_put(struct media_request_object *obj)
>> +{
>> +}
>> +
>> +static inline void media_request_object_init(struct media_request_object *obj)
>> +{
>> +	obj->ops = NULL;
>> +	obj->req = NULL;
>> +}
>> +
>> +static inline int media_request_object_bind(struct media_request *req,
>> +			       const struct media_request_object_ops *ops,
>> +			       void *priv,
>> +			       struct media_request_object *obj)
>> +{
>> +	return 0;
>> +}
>> +
>> +static inline void media_request_object_unbind(struct media_request_object *obj)
>> +{
>> +}
>> +
>> +static inline void media_request_object_complete(struct media_request_object *obj)
>> +{
>> +}
>> +
>> +#endif
>> +
>> +#endif
> 

Regards,

	Hans

Re: [PATCHv13 05/28] media-request: add media_request_object_find

[Hans Verkuil]

On 04/05/18 14:43, Sakari Ailus wrote:
> On Thu, May 03, 2018 at 04:52:55PM +0200, Hans Verkuil wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> Add media_request_object_find to find a request object inside a
>> request based on ops and/or priv values.
>>
>> Objects of the same type (vb2 buffer, control handler) will have
>> the same ops value. And objects that refer to the same 'parent'
>> object (e.g. the v4l2_ctrl_handler that has the current driver
>> state) will have the same priv value.
>>
>> The caller has to call media_request_object_put() for the returned
>> object since this function increments the refcount.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  drivers/media/media-request.c | 25 +++++++++++++++++++++++++
>>  include/media/media-request.h | 24 ++++++++++++++++++++++++
>>  2 files changed, 49 insertions(+)
>>
>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>> index edc1c3af1959..c7e11e816e27 100644
>> --- a/drivers/media/media-request.c
>> +++ b/drivers/media/media-request.c
>> @@ -322,6 +322,31 @@ static void media_request_object_release(struct kref *kref)
>>  	obj->ops->release(obj);
>>  }
>>  
>> +struct media_request_object *
>> +media_request_object_find(struct media_request *req,
>> +			  const struct media_request_object_ops *ops,
>> +			  void *priv)
>> +{
>> +	struct media_request_object *obj;
>> +	struct media_request_object *found = NULL;
>> +	unsigned long flags;
>> +
>> +	if (WARN_ON(!ops || !priv))
>> +		return NULL;
>> +
>> +	spin_lock_irqsave(&req->lock, flags);
>> +	list_for_each_entry(obj, &req->objects, list) {
>> +		if (obj->ops == ops && obj->priv == priv) {
>> +			media_request_object_get(obj);
>> +			found = obj;
>> +			break;
>> +		}
>> +	}
>> +	spin_unlock_irqrestore(&req->lock, flags);
>> +	return found;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_object_find);
>> +
>>  void media_request_object_put(struct media_request_object *obj)
>>  {
>>  	kref_put(&obj->kref, media_request_object_release);
>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>> index 997e096d7128..5367b4a2f91c 100644
>> --- a/include/media/media-request.h
>> +++ b/include/media/media-request.h
>> @@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
>>   */
>>  void media_request_object_put(struct media_request_object *obj);
>>  
>> +/**
>> + * media_request_object_find - Find an object in a request
>> + *
>> + * @ops: Find an object with this ops value
>> + * @priv: Find an object with this priv value
>> + *
>> + * Both @ops and @priv must be non-NULL.
>> + *
>> + * Returns NULL if not found or the object pointer. The caller must
> 
> I'd describe the successful case first. I.e. "Returns the object pointer or
> NULL it not found".

Done.

> 
> Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>

Thanks!

	Hans

> 
>> + * call media_request_object_put() once it finished using the object.
>> + */
>> +struct media_request_object *
>> +media_request_object_find(struct media_request *req,
>> +			  const struct media_request_object_ops *ops,
>> +			  void *priv);
>> +
>>  /**
>>   * media_request_object_init - Initialise a media request object
>>   *
>> @@ -241,6 +257,14 @@ static inline void media_request_object_put(struct media_request_object *obj)
>>  {
>>  }
>>  
>> +static inline struct media_request_object *
>> +media_request_object_find(struct media_request *req,
>> +			  const struct media_request_object_ops *ops,
>> +			  void *priv)
>> +{
>> +	return NULL;
>> +}
>> +
>>  static inline void media_request_object_init(struct media_request_object *obj)
>>  {
>>  	obj->ops = NULL;
> 

Re: [PATCHv13 05/28] media-request: add media_request_object_find

[Hans Verkuil]

On 07/05/18 19:05, Mauro Carvalho Chehab wrote:
> Em Fri, 4 May 2018 15:43:07 +0300
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
>> On Thu, May 03, 2018 at 04:52:55PM +0200, Hans Verkuil wrote:
>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>
>>> Add media_request_object_find to find a request object inside a
>>> request based on ops and/or priv values.
>>>
>>> Objects of the same type (vb2 buffer, control handler) will have
>>> the same ops value. And objects that refer to the same 'parent'
>>> object (e.g. the v4l2_ctrl_handler that has the current driver
>>> state) will have the same priv value.
>>>
>>> The caller has to call media_request_object_put() for the returned
>>> object since this function increments the refcount.
>>>
>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>> ---
>>>  drivers/media/media-request.c | 25 +++++++++++++++++++++++++
>>>  include/media/media-request.h | 24 ++++++++++++++++++++++++
>>>  2 files changed, 49 insertions(+)
>>>
>>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>>> index edc1c3af1959..c7e11e816e27 100644
>>> --- a/drivers/media/media-request.c
>>> +++ b/drivers/media/media-request.c
>>> @@ -322,6 +322,31 @@ static void media_request_object_release(struct kref *kref)
>>>  	obj->ops->release(obj);
>>>  }
>>>  
>>> +struct media_request_object *
>>> +media_request_object_find(struct media_request *req,
>>> +			  const struct media_request_object_ops *ops,
>>> +			  void *priv)
>>> +{
>>> +	struct media_request_object *obj;
>>> +	struct media_request_object *found = NULL;
>>> +	unsigned long flags;
>>> +
>>> +	if (WARN_ON(!ops || !priv))
>>> +		return NULL;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	list_for_each_entry(obj, &req->objects, list) {
>>> +		if (obj->ops == ops && obj->priv == priv) {
>>> +			media_request_object_get(obj);
>>> +			found = obj;
>>> +			break;
>>> +		}
>>> +	}
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +	return found;
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_find);
>>> +
>>>  void media_request_object_put(struct media_request_object *obj)
>>>  {
>>>  	kref_put(&obj->kref, media_request_object_release);
>>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>>> index 997e096d7128..5367b4a2f91c 100644
>>> --- a/include/media/media-request.h
>>> +++ b/include/media/media-request.h
>>> @@ -196,6 +196,22 @@ static inline void media_request_object_get(struct media_request_object *obj)
>>>   */
>>>  void media_request_object_put(struct media_request_object *obj);
>>>  
>>> +/**
>>> + * media_request_object_find - Find an object in a request
>>> + *
>>> + * @ops: Find an object with this ops value
>>> + * @priv: Find an object with this priv value
>>> + *
>>> + * Both @ops and @priv must be non-NULL.
>>> + *
>>> + * Returns NULL if not found or the object pointer. The caller must  
>>
>> I'd describe the successful case first. I.e. "Returns the object pointer or
>> NULL it not found".
> 
> It would be good to also tell that this routine internally uses the
> spin lock.

Done.

Regards,

	Hans

> 
>>
>> Acked-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>
>>> + * call media_request_object_put() once it finished using the object.
>>> + */
>>> +struct media_request_object *
>>> +media_request_object_find(struct media_request *req,
>>> +			  const struct media_request_object_ops *ops,
>>> +			  void *priv);
>>> +
>>>  /**
>>>   * media_request_object_init - Initialise a media request object
>>>   *
>>> @@ -241,6 +257,14 @@ static inline void media_request_object_put(struct media_request_object *obj)
>>>  {
>>>  }
>>>  
>>> +static inline struct media_request_object *
>>> +media_request_object_find(struct media_request *req,
>>> +			  const struct media_request_object_ops *ops,
>>> +			  void *priv)
>>> +{
>>> +	return NULL;
>>> +}
>>> +
>>>  static inline void media_request_object_init(struct media_request_object *obj)
>>>  {
>>>  	obj->ops = NULL;  
>>
> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 06/28] v4l2-dev: lock req_queue_mutex

[Hans Verkuil]

On 08/05/18 12:45, Mauro Carvalho Chehab wrote:
> Em Tue, 8 May 2018 09:45:27 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>> On 05/07/2018 07:20 PM, Mauro Carvalho Chehab wrote:
>>> Em Thu,  3 May 2018 16:52:56 +0200
>>> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
>>>   
>>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>>
>>>> We need to serialize streamon/off with queueing new requests.
>>>> These ioctls may trigger the cancellation of a streaming
>>>> operation, and that should not be mixed with queuing a new
>>>> request at the same time.
>>>>
>>>> Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
>>>> with MEDIA_REQUEST_IOC_QUEUE.
>>>>
>>>> Finally close() needs this lock since that too can trigger the
>>>> cancellation of a streaming operation.
>>>>
>>>> We take the req_queue_mutex here before any other locks since
>>>> it is a very high-level lock.
>>>>
>>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>>>> ---
>>>>  drivers/media/v4l2-core/v4l2-dev.c | 37 +++++++++++++++++++++++++++++-
>>>>  1 file changed, 36 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>>>> index 1d0b2208e8fb..b1c9efc0ecc4 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>>>> @@ -353,13 +353,36 @@ static long v4l2_ioctl(struct file *filp, unsigned int cmd, unsigned long arg)
>>>>  
>>>>  	if (vdev->fops->unlocked_ioctl) {
>>>>  		struct mutex *lock = v4l2_ioctl_get_lock(vdev, cmd);
>>>> +		struct mutex *queue_lock = NULL;
>>>>  
>>>> -		if (lock && mutex_lock_interruptible(lock))
>>>> +		/*
>>>> +		 * We need to serialize streamon/off with queueing new requests.
>>>> +		 * These ioctls may trigger the cancellation of a streaming
>>>> +		 * operation, and that should not be mixed with queueing a new
>>>> +		 * request at the same time.
>>>> +		 *
>>>> +		 * Also TRY/S_EXT_CTRLS needs this lock to correctly serialize
>>>> +		 * with MEDIA_REQUEST_IOC_QUEUE.
>>>> +		 */
>>>> +		if (vdev->v4l2_dev->mdev &&
>>>> +		    (cmd == VIDIOC_STREAMON || cmd == VIDIOC_STREAMOFF ||
>>>> +		     cmd == VIDIOC_S_EXT_CTRLS || cmd == VIDIOC_TRY_EXT_CTRLS))
>>>> +			queue_lock = &vdev->v4l2_dev->mdev->req_queue_mutex;
>>>> +
>>>> +		if (queue_lock && mutex_lock_interruptible(queue_lock))
>>>> +			return -ERESTARTSYS;  
>>>
>>> Taking both locks seems risky. Here you're taking first v4l2 lock, returned
>>> by v4l2_ioctl_get_lock(vdev, cmd), and then you're taking the req_queue lock.  
>>
>> No,  v4l2_ioctl_get_lock() only returns a pointer to a mutex, it doesn't lock
>> anything. I think you got confused there. I'll reorganize the code a bit so
>> the call to  v4l2_ioctl_get_lock() happens after the queue_lock has been taken.
> 
> Yeah, I didn't actually look at the implementation of v4l2_ioctl_get_lock().
> 
> As we're using "_get" along this patch series to increment krefs (with is
> a sort of locking), the name here confused me. IMHO, we should rename it
> to v4l2_ioctl_return_lock() (or similar) on some future, in order to avoid
> confusion.

I think in the near future the v4l2_ioctl_get_lock() function can disappear.
My patch that pushes taking the ioctl serialization lock down into v4l2-ioctl.c
helps, and the gspca patch series removes the disable_locking bitarray.

Once that's all in I think the remaining code of v4l2_ioctl_get_lock can just
be moved to __video_do_ioctl() where it really belongs.

> 
>> I'll also rename queue_lock to req_queue_lock (it's a bit more descriptive).
> 
> Agreed.
> 
>>
>> So we first take the high-level media_device req_queue_mutex if needed, and
>> then the ioctl serialization lock. Doing it the other way around will indeed
>> promptly deadlock (as I very quickly discovered after my initial implementation!).
>>
>> So the order is:
>>
>> 	req_queue_mutex (serialize request state changes from/to IDLE)
>> 	ioctl lock (serialize ioctls)
>> 	request->lock (spinlock)
>>
>> The last is only held for short periods when updating the media_request struct.
>>
>>>
>>> It is possible to call parts of the code that only handles req_queue
>>> or v4l2 lock (for example, by mixing request API calls with non-requests
>>> one). Worse than that, there are parts of the code where the request API
>>> patches get both a mutex and a spin lock.
>>>
>>> I didn't look too closely (nor ran any test), but I'm almost sure that
>>> there are paths where it will end by leading into dead locks.  
>>
>> I've done extensive testing with this and actually been very careful about
>> the lock handling. It's also been tested with the cedrus driver.
> 
> I don't doubt it works using your apps, but real life can be messier:
> people could be issuing ioctls at different orders, programs can abort
> any time, closing file descriptors at random times, threads can be
> used to paralelize ioctls, etc.
> 
> That not discarding the possibility of someone coming with some
> ingenious code meant to cause machine hangups or to exposure some
> other security flaws.

Which is why the best place for this is in the core. I think v15 is more
readable in this respect.

Regards,

	Hans

> 
> 
> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 12/28] v4l2-ctrls: add core request support

[Hans Verkuil]

On 08/05/18 12:49, Mauro Carvalho Chehab wrote:
> Em Tue, 8 May 2018 10:07:22 +0200
> Hans Verkuil <hverkuil@xs4all.nl> escreveu:
> 
>>>> diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
>>>> index 76352eb59f14..a0f7c38d1a90 100644
>>>> --- a/include/media/v4l2-ctrls.h
>>>> +++ b/include/media/v4l2-ctrls.h
>>>> @@ -250,6 +250,10 @@ struct v4l2_ctrl {
>>>>   *		``prepare_ext_ctrls`` function at ``v4l2-ctrl.c``.
>>>>   * @from_other_dev: If true, then @ctrl was defined in another
>>>>   *		device than the &struct v4l2_ctrl_handler.
>>>> + * @done:	If true, then this control reference is part of a
>>>> + *		control cluster that was already set while applying
>>>> + *		the controls in this media request object.  
>>>
>>> Hmm... I would rename it for request_done or something similar, as it
>>> seems that this applies only for requests.  
>>
>> No, the variable name is correct (it serves the same purpose as the 'done'
>> field in struct v4l2_ctrl), but the description should be improved.
>>
>> I also want to take another look at this: I wonder if it isn't possible to
>> use the v4l2_ctrl 'done' field for this instead of having to add a new field
>> here.

Unfortunately, that isn't possible.

> 
> If it can use v4l2_ctrl done, it would indeed be better. Otherwise, as
> this new "done" is used only by requests, I really think that it shold
> be renamed, to let it clearer that this is the "done" that should be used
> when requests are used.

I've renamed 'done' to req_done and improved (I hope) the comment.

Regards,

	Hans

> 
> Thanks,
> Mauro
> 

Re: [PATCHv13 03/28] media-request: implement media requests

[Hans Verkuil]

On 08/05/18 12:21, Mauro Carvalho Chehab wrote:
> Em Fri, 4 May 2018 15:27:50 +0300
> Sakari Ailus <sakari.ailus@iki.fi> escreveu:
> 
>> Hi Hans,
>>
>> I've read this patch a large number of times and I think also the details
>> begin to seem sound. A few comments below.
> 
> I'm sending this after analyzing the other patches in this series,
> as this is the core of the changes. So, although I wrote the comments
> early, I wanted to read first all other patches before sending it.
> 
>>
>> On Thu, May 03, 2018 at 04:52:53PM +0200, Hans Verkuil wrote:
>>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>>
>>> Add initial media request support:
>>>
>>> 1) Add MEDIA_IOC_REQUEST_ALLOC ioctl support to media-device.c
>>> 2) Add struct media_request to store request objects.
>>> 3) Add struct media_request_object to represent a request object.
>>> 4) Add MEDIA_REQUEST_IOC_QUEUE/REINIT ioctl support.
>>>
>>> Basic lifecycle: the application allocates a request, adds
>>> objects to it, queues the request, polls until it is completed
>>> and can then read the final values of the objects at the time
>>> of completion. When it closes the file descriptor the request
>>> memory will be freed (actually, when the last user of that request
>>> releases the request).
>>>
>>> Drivers will bind an object to a request (the 'adds objects to it'
>>> phase), when MEDIA_REQUEST_IOC_QUEUE is called the request is
>>> validated (req_validate op), then queued (the req_queue op).
>>>
>>> When done with an object it can either be unbound from the request
>>> (e.g. when the driver has finished with a vb2 buffer) or marked as
>>> completed (e.g. for controls associated with a buffer). When all
>>> objects in the request are completed (or unbound), then the request
>>> fd will signal an exception (poll).
>>>
>>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Hmm... As you're adding Copyrights from Intel/Google in this patch, that
> indicates that part of the stuff you're adding here were authored by
> others. So, you should use Co-developed-by: tag here, and get the SOBs
> from the other developers that did part of the work[1].
> 
> [1] except if your work was sponsored by Cisco, Intel and Google, but
>     I think this is not the case.

I'll check this with Sakari and Google.

> 
>>> ---
>>>  drivers/media/Makefile        |   3 +-
>>>  drivers/media/media-device.c  |  14 ++
>>>  drivers/media/media-request.c | 407 ++++++++++++++++++++++++++++++++++
>>>  include/media/media-device.h  |  16 ++
>>>  include/media/media-request.h | 244 ++++++++++++++++++++
>>>  5 files changed, 683 insertions(+), 1 deletion(-)
>>>  create mode 100644 drivers/media/media-request.c
>>>  create mode 100644 include/media/media-request.h
>>>
>>> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
>>> index 594b462ddf0e..985d35ec6b29 100644
>>> --- a/drivers/media/Makefile
>>> +++ b/drivers/media/Makefile
>>> @@ -3,7 +3,8 @@
>>>  # Makefile for the kernel multimedia device drivers.
>>>  #
>>>  
>>> -media-objs	:= media-device.o media-devnode.o media-entity.o
>>> +media-objs	:= media-device.o media-devnode.o media-entity.o \
>>> +		   media-request.o
>>>  
>>>  #
>>>  # I2C drivers should come before other drivers, otherwise they'll fail
>>> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
>>> index 35e81f7c0d2f..bb6a64acd3f0 100644
>>> --- a/drivers/media/media-device.c
>>> +++ b/drivers/media/media-device.c
>>> @@ -32,6 +32,7 @@
>>>  #include <media/media-device.h>
>>>  #include <media/media-devnode.h>
>>>  #include <media/media-entity.h>
>>> +#include <media/media-request.h>
>>>  
>>>  #ifdef CONFIG_MEDIA_CONTROLLER
>>>  
>>> @@ -366,6 +367,15 @@ static long media_device_get_topology(struct media_device *mdev,
>>>  	return ret;
>>>  }
>>>  
>>> +static long media_device_request_alloc(struct media_device *mdev,
>>> +				       struct media_request_alloc *alloc)
>>> +{
>>> +	if (!mdev->ops || !mdev->ops->req_validate || !mdev->ops->req_queue)
>>> +		return -ENOTTY;
>>> +
>>> +	return media_request_alloc(mdev, alloc);
>>> +}
>>> +
>>>  static long copy_arg_from_user(void *karg, void __user *uarg, unsigned int cmd)
>>>  {
>>>  	/* All media IOCTLs are _IOWR() */
>>> @@ -414,6 +424,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>>>  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>>>  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>>>  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
>>> +	MEDIA_IOC(REQUEST_ALLOC, media_device_request_alloc, 0),
>>>  };
>>>  
>>>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
>>> @@ -686,6 +697,8 @@ void media_device_init(struct media_device *mdev)
>>>  	INIT_LIST_HEAD(&mdev->pads);
>>>  	INIT_LIST_HEAD(&mdev->links);
>>>  	INIT_LIST_HEAD(&mdev->entity_notify);
>>> +
>>> +	mutex_init(&mdev->req_queue_mutex);
>>>  	mutex_init(&mdev->graph_mutex);
>>>  	ida_init(&mdev->entity_internal_idx);
>>>  
>>> @@ -699,6 +712,7 @@ void media_device_cleanup(struct media_device *mdev)
>>>  	mdev->entity_internal_idx_max = 0;
>>>  	media_graph_walk_cleanup(&mdev->pm_count_walk);
>>>  	mutex_destroy(&mdev->graph_mutex);
>>> +	mutex_destroy(&mdev->req_queue_mutex);
>>>  }
>>>  EXPORT_SYMBOL_GPL(media_device_cleanup);
>>>  
>>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>>> new file mode 100644
>>> index 000000000000..c216c4ab628b
>>> --- /dev/null
>>> +++ b/drivers/media/media-request.c
>>> @@ -0,0 +1,407 @@
>>> +// SPDX-License-Identifier: GPL-2.0-only
>>> +/*
>>> + * Media device request objects
>>> + *
>>> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
>>> + * Copyright (C) 2018 Intel Corporation
>>> + * Copyright (C) 2018 Google, Inc.
>>> + *
>>> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
>>> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> + */
>>> +
>>> +#include <linux/anon_inodes.h>
>>> +#include <linux/file.h>
>>> +
>>> +#include <media/media-device.h>
>>> +#include <media/media-request.h>
>>> +
>>> +static const char * const request_state[] = {
>>> +	[MEDIA_REQUEST_STATE_IDLE]	 = "idle",
>>> +	[MEDIA_REQUEST_STATE_VALIDATING] = "validating",
>>> +	[MEDIA_REQUEST_STATE_QUEUED]	 = "queued",
>>> +	[MEDIA_REQUEST_STATE_COMPLETE]	 = "complete",
>>> +	[MEDIA_REQUEST_STATE_CLEANING]	 = "cleaning",
>>> +};
>>> +
>>> +static const char *
>>> +media_request_state_str(enum media_request_state state)
>>> +{
>>> +	if (WARN_ON(state >= ARRAY_SIZE(request_state)))
>>> +		return "invalid";
>>> +	return request_state[state];
>>> +}
>>> +
>>> +static void media_request_clean(struct media_request *req)
>>> +{
>>> +	struct media_request_object *obj, *obj_safe;
>>> +
>>> +	WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_CLEANING);
>>> +
>>> +	list_for_each_entry_safe(obj, obj_safe, &req->objects, list) {
>>> +		media_request_object_unbind(obj);
>>> +		media_request_object_put(obj);
>>> +	}
>>> +
>>> +	req->num_incomplete_objects = 0;  
>>
>> The number of incomplete objects should be already zero here. I'd think
>> that a different number would suggest that something has gone very wrong
>> and should be complained about. How about adding
>> WARN_ON(req->num_incomplete_objects) above this line? 
>>
>>> +	wake_up_interruptible_all(&req->poll_wait);
>>> +}
>>> +
>>> +static void media_request_release(struct kref *kref)
>>> +{
>>> +	struct media_request *req =
>>> +		container_of(kref, struct media_request, kref);
>>> +	struct media_device *mdev = req->mdev;
>>> +
>>> +	dev_dbg(mdev->dev, "request: release %s\n", req->debug_str);
>>> +
>>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
>>> +
>>> +	media_request_clean(req);
>>> +
>>> +	if (mdev->ops->req_free)
>>> +		mdev->ops->req_free(req);
>>> +	else
>>> +		kfree(req);
>>> +}
>>> +
>>> +void media_request_put(struct media_request *req)
>>> +{
>>> +	kref_put(&req->kref, media_request_release);
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_put);
>>> +
>>> +static int media_request_close(struct inode *inode, struct file *filp)
>>> +{
>>> +	struct media_request *req = filp->private_data;
>>> +
>>> +	media_request_put(req);
>>> +	return 0;
>>> +}
>>> +
>>> +static unsigned int media_request_poll(struct file *filp,
>>> +				       struct poll_table_struct *wait)
>>> +{
>>> +	struct media_request *req = filp->private_data;
>>> +	unsigned long flags;
>>> +	unsigned int ret = 0;
>>> +	enum media_request_state state;
>>> +
>>> +	if (!(poll_requested_events(wait) & POLLPRI))
>>> +		return 0;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	state = atomic_read(&req->state);
>>> +
>>> +	if (state == MEDIA_REQUEST_STATE_COMPLETE) {
>>> +		ret = POLLPRI;
>>> +		goto unlock;
>>> +	}
>>> +	if (state != MEDIA_REQUEST_STATE_QUEUED) {
>>> +		ret = POLLERR;
>>> +		goto unlock;
>>> +	}
>>> +
>>> +	poll_wait(filp, &req->poll_wait, wait);
>>> +
>>> +unlock:
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +	return ret;
>>> +}
>>> +
>>> +static long media_request_ioctl_queue(struct media_request *req)
>>> +{
>>> +	struct media_device *mdev = req->mdev;
>>> +	enum media_request_state state;
>>> +	unsigned long flags;
>>> +	int ret = 0;  
>>
>> ret is unconditionally assigned below, no need to initialise here.
>>
>>> +
>>> +	dev_dbg(mdev->dev, "request: queue %s\n", req->debug_str);
>>> +
>>> +	/*
>>> +	 * Ensure the request that is validated will be the one that gets queued
>>> +	 * next by serialising the queueing process. This mutex is also used
>>> +	 * to serialize with canceling a vb2 queue and with setting values such
>>> +	 * as controls in a request.
>>> +	 */
>>> +	mutex_lock(&mdev->req_queue_mutex);
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	state = atomic_cmpxchg(&req->state, MEDIA_REQUEST_STATE_IDLE,
>>> +			       MEDIA_REQUEST_STATE_VALIDATING);
>>> +	spin_unlock_irqrestore(&req->lock, flags);
> 
> It looks weird to serialize access to it with a mutex, a spin lock and 
> an atomic call.
> 
> IMHO, locking is still an issue here. I would love to test the 
> locks with some tool that would randomize syscalls, issuing close(),
> poll() and read() at wrong times and inverting the order of some calls, 
> in order to do some empiric test that all locks are at the right places.
> 
> Complex locking schemas like that usually tend to cause a lot of
> troubles.

Reqv15 reverts back to using spin locks only for this, so this is easier
to understand in v15. Mixing spin locks and atomic was not a good idea.

> 
>>> +	if (state != MEDIA_REQUEST_STATE_IDLE) {
>>> +		dev_dbg(mdev->dev,
>>> +			"request: unable to queue %s, request in state %s\n",
>>> +			req->debug_str, media_request_state_str(state));
>>> +		mutex_unlock(&mdev->req_queue_mutex);
>>> +		return -EBUSY;
>>> +	}
>>> +
>>> +	ret = mdev->ops->req_validate(req);
>>> +
>>> +	/*
>>> +	 * If the req_validate was successful, then we mark the state as QUEUED
>>> +	 * and call req_queue. The reason we set the state first is that this
>>> +	 * allows req_queue to unbind or complete the queued objects in case
>>> +	 * they are immediately 'consumed'. State changes from QUEUED to another
>>> +	 * state can only happen if either the driver changes the state or if
>>> +	 * the user cancels the vb2 queue. The driver can only change the state
>>> +	 * after each object is queued through the req_queue op (and note that
>>> +	 * that op cannot fail), so setting the state to QUEUED up front is
>>> +	 * safe.
>>> +	 *
>>> +	 * The other reason for changing the state is if the vb2 queue is
>>> +	 * canceled, and that uses the req_queue_mutex which is still locked
>>> +	 * while req_queue is called, so that's safe as well.
>>> +	 */
>>> +	atomic_set(&req->state,
>>> +		   ret ? MEDIA_REQUEST_STATE_IDLE : MEDIA_REQUEST_STATE_QUEUED);
> 
> Why are you changing state also when ret fails?

Because the state during the validation was set to STATE_VALIDATING. On
failure that state has to revert back to IDLE.

> 
> Also, why you had to use a spin lock earlier in this function just 
> to change the req->state but you don't need to use it here?
> 
>>> +
>>> +	if (!ret)
>>> +		mdev->ops->req_queue(req);
>>> +
>>> +	mutex_unlock(&mdev->req_queue_mutex);
>>> +
>>> +	if (ret)
>>> +		dev_dbg(mdev->dev, "request: can't queue %s (%d)\n",
>>> +			req->debug_str, ret);
>>> +	else
>>> +		media_request_get(req);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static long media_request_ioctl_reinit(struct media_request *req)
>>> +{
>>> +	struct media_device *mdev = req->mdev;
>>> +	unsigned long flags;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	if (atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE &&
>>> +	    atomic_read(&req->state) != MEDIA_REQUEST_STATE_COMPLETE) {
>>> +		dev_dbg(mdev->dev,
>>> +			"request: %s not in idle or complete state, cannot reinit\n",
>>> +			req->debug_str);
>>> +		spin_unlock_irqrestore(&req->lock, flags);
>>> +		return -EBUSY;
>>> +	}
>>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_CLEANING);
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +
>>> +	media_request_clean(req);
>>> +
>>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
>>> +
>>> +	return 0;
>>> +}
>>> +
>>> +static long media_request_ioctl(struct file *filp, unsigned int cmd,
>>> +				unsigned long arg)
>>> +{
>>> +	struct media_request *req = filp->private_data;
>>> +
>>> +	switch (cmd) {
>>> +	case MEDIA_REQUEST_IOC_QUEUE:
>>> +		return media_request_ioctl_queue(req);
>>> +	case MEDIA_REQUEST_IOC_REINIT:
>>> +		return media_request_ioctl_reinit(req);
>>> +	default:
>>> +		return -ENOIOCTLCMD;
>>> +	}
>>> +}
>>> +
>>> +static const struct file_operations request_fops = {
>>> +	.owner = THIS_MODULE,
>>> +	.poll = media_request_poll,
>>> +	.unlocked_ioctl = media_request_ioctl,
>>> +	.release = media_request_close,
>>> +};
>>> +
>>> +int media_request_alloc(struct media_device *mdev,
>>> +			struct media_request_alloc *alloc)
>>> +{
>>> +	struct media_request *req;
>>> +	struct file *filp;
>>> +	char comm[TASK_COMM_LEN];
>>> +	int fd;
>>> +	int ret;
>>> +
>>> +	/* Either both are NULL or both are non-NULL */
>>> +	if (WARN_ON(!mdev->ops->req_alloc ^ !mdev->ops->req_free))
>>> +		return -ENOMEM;
>>> +
>>> +	fd = get_unused_fd_flags(O_CLOEXEC);
>>> +	if (fd < 0)
>>> +		return fd;
>>> +
>>> +	filp = anon_inode_getfile("request", &request_fops, NULL, O_CLOEXEC);
>>> +	if (IS_ERR(filp)) {
>>> +		ret = PTR_ERR(filp);
>>> +		goto err_put_fd;
>>> +	}
>>> +
>>> +	if (mdev->ops->req_alloc)
>>> +		req = mdev->ops->req_alloc(mdev);
>>> +	else
>>> +		req = kzalloc(sizeof(*req), GFP_KERNEL);
>>> +	if (!req) {
>>> +		ret = -ENOMEM;
>>> +		goto err_fput;
>>> +	}
>>> +
>>> +	filp->private_data = req;
>>> +	req->mdev = mdev;
>>> +	atomic_set(&req->state, MEDIA_REQUEST_STATE_IDLE);
>>> +	req->num_incomplete_objects = 0;
>>> +	kref_init(&req->kref);
>>> +	INIT_LIST_HEAD(&req->objects);
>>> +	spin_lock_init(&req->lock);
>>> +	init_waitqueue_head(&req->poll_wait);
>>> +
>>> +	alloc->fd = fd;
>>> +
>>> +	get_task_comm(comm, current);
>>> +	snprintf(req->debug_str, sizeof(req->debug_str), "%s:%d",
>>> +		 comm, fd);
>>> +	dev_dbg(mdev->dev, "request: allocated %s\n", req->debug_str);
>>> +
>>> +	fd_install(fd, filp);
>>> +
>>> +	return 0;
>>> +
>>> +err_fput:
>>> +	fput(filp);
>>> +
>>> +err_put_fd:
>>> +	put_unused_fd(fd);
>>> +
>>> +	return ret;
>>> +}
>>> +
>>> +static void media_request_object_release(struct kref *kref)
>>> +{
>>> +	struct media_request_object *obj =
>>> +		container_of(kref, struct media_request_object, kref);
>>> +	struct media_request *req = obj->req;
>>> +
>>> +	if (req)
>>> +		media_request_object_unbind(obj);
>>> +	obj->ops->release(obj);
>>> +}
>>> +
>>> +void media_request_object_put(struct media_request_object *obj)
>>> +{
>>> +	kref_put(&obj->kref, media_request_object_release);
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_put);
>>> +
>>> +void media_request_object_init(struct media_request_object *obj)
>>> +{
>>> +	obj->ops = NULL;
>>> +	obj->req = NULL;
>>> +	obj->priv = NULL;
>>> +	obj->completed = false;
>>> +	INIT_LIST_HEAD(&obj->list);
>>> +	kref_init(&obj->kref);
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_init);
>>> +
>>> +int media_request_object_bind(struct media_request *req,
>>> +			      const struct media_request_object_ops *ops,
>>> +			      void *priv,
>>> +			      struct media_request_object *obj)
>>> +{
>>> +	unsigned long flags;
>>> +	int ret = -EBUSY;
>>> +
>>> +	if (WARN_ON(!ops->release))
>>> +		return -EPERM;
>>> +
>>> +	obj->req = req;
>>> +	obj->ops = ops;
>>> +	obj->priv = priv;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +
>>> +	if (WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_IDLE))
>>> +		goto unlock;  
>>
>> Is this worth a kernel warning, or rather how the drivers / other framework
>> bits (e.g. VB2) prevent user from binding objects to non-idle requests?
>> Even if you added a similar check to the caller, the request state could
>> well change in the meantime.
>>
>> Perhaps add __must_check to the return value?
>>
>>> +
>>> +	list_add_tail(&obj->list, &req->objects);
>>> +	req->num_incomplete_objects++;
>>> +	ret = 0;
>>> +
>>> +unlock:
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +	return ret;
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_bind);
>>> +
>>> +void media_request_object_unbind(struct media_request_object *obj)
>>> +{
>>> +	struct media_request *req = obj->req;
>>> +	enum media_request_state state;
>>> +	unsigned long flags;
>>> +	bool completed = false;
>>> +
>>> +	if (WARN_ON(!req))
>>> +		return;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	list_del(&obj->list);
>>> +	obj->req = NULL;
>>> +
>>> +	state = atomic_read(&req->state);
>>> +
>>> +	if (state == MEDIA_REQUEST_STATE_COMPLETE ||
>>> +	    state == MEDIA_REQUEST_STATE_CLEANING)
>>> +		goto unlock;
>>> +
>>> +	if (WARN_ON(state == MEDIA_REQUEST_STATE_VALIDATING))
>>> +		goto unlock;
>>> +
>>> +	if (WARN_ON(!req->num_incomplete_objects))
>>> +		goto unlock;
>>> +
>>> +	req->num_incomplete_objects--;
>>> +	if (state == MEDIA_REQUEST_STATE_QUEUED &&
>>> +	    !req->num_incomplete_objects) {
>>> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
>>> +		completed = true;
>>> +		wake_up_interruptible_all(&req->poll_wait);
>>> +	}
>>> +
>>> +unlock:
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +	if (obj->ops->unbind)
>>> +		obj->ops->unbind(obj);
>>> +	if (completed)
>>> +		media_request_put(req);
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_unbind);
>>> +
>>> +void media_request_object_complete(struct media_request_object *obj)
>>> +{
>>> +	struct media_request *req = obj->req;
>>> +	unsigned long flags;
>>> +	bool completed = false;
>>> +
>>> +	spin_lock_irqsave(&req->lock, flags);
>>> +	if (obj->completed)
>>> +		goto unlock;
>>> +	obj->completed = true;
>>> +	if (WARN_ON(!req->num_incomplete_objects) ||
>>> +	    WARN_ON(atomic_read(&req->state) != MEDIA_REQUEST_STATE_QUEUED))
>>> +		goto unlock;
>>> +
>>> +	if (!--req->num_incomplete_objects) {
>>> +		atomic_set(&req->state, MEDIA_REQUEST_STATE_COMPLETE);
>>> +		wake_up_interruptible_all(&req->poll_wait);
>>> +		completed = true;
>>> +	}
>>> +unlock:
>>> +	spin_unlock_irqrestore(&req->lock, flags);
>>> +	if (completed)
>>> +		media_request_put(req);
>>> +}
>>> +EXPORT_SYMBOL_GPL(media_request_object_complete);
>>> diff --git a/include/media/media-device.h b/include/media/media-device.h
>>> index bcc6ec434f1f..7d855823341c 100644
>>> --- a/include/media/media-device.h
>>> +++ b/include/media/media-device.h
>>> @@ -27,6 +27,7 @@
>>>  
>>>  struct ida;
>>>  struct device;
>>> +struct media_device;
>>>  
>>>  /**
>>>   * struct media_entity_notify - Media Entity Notify
>>> @@ -50,10 +51,21 @@ struct media_entity_notify {
>>>   * struct media_device_ops - Media device operations
>>>   * @link_notify: Link state change notification callback. This callback is
>>>   *		 called with the graph_mutex held.
>>> + * @req_alloc: Allocate a request
>>> + * @req_free: Free a request
> 
> Please do place better descriptions there. First of all, either both
> should be used or both should be NULL - as validated by
> media_request_alloc() logic.
> 
> Second, it should be clearer that those are meant to be used when
> the driver needs to allocate a bigger struct that embeds a 
> struct media_request object on it, with should be destroyed at 
> ops->req_free() call.
> 
> Also, such embed struct should not contain a kref (multiple krefs
> at the same struct doesn't work).

Improved the comments.

> 
>>> + * @req_validate: Validate a request, but do not queue yet
>>> + * @req_queue: Queue a validated request, cannot fail. If something goes
>>> + *	       wrong when queueing this request then it should be marked
>>> + *	       as such internally in the driver and any related buffers
>>> + *	       must eventually return to vb2 with state VB2_BUF_STATE_ERROR.
> 
> Please describe what kind of req locks (if any) can/should
> be used inside each callback.

I've mentioned that req_queue_mutex is held when calling these two ops.

> 
> 
>>>   */
>>>  struct media_device_ops {
>>>  	int (*link_notify)(struct media_link *link, u32 flags,
>>>  			   unsigned int notification);
>>> +	struct media_request *(*req_alloc)(struct media_device *mdev);
>>> +	void (*req_free)(struct media_request *req);
>>> +	int (*req_validate)(struct media_request *req);
>>> +	void (*req_queue)(struct media_request *req);
>>>  };
>>>  
>>>  /**
>>> @@ -88,6 +100,8 @@ struct media_device_ops {
>>>   * @disable_source: Disable Source Handler function pointer
>>>   *
>>>   * @ops:	Operation handler callbacks
>>> + * @req_queue_mutex: Serialise the MEDIA_REQUEST_IOC_QUEUE ioctl w.r.t. this
>>> + *		     media device.
> 
> This description is incomplete as it doesn't match the explanation you
> introduced at patch 00/18. Please add a complete locking description,
> as patches 00 aren't stored anywhere at the git tree, and there are lots
> of non-trivial assumptions on your locking schema.

True, updated the description. It was horrible outdated :-)

> 
>>>   *
>>>   * This structure represents an abstract high-level media device. It allows easy
>>>   * access to entities and provides basic media device-level support. The
>>> @@ -158,6 +172,8 @@ struct media_device {
>>>  	void (*disable_source)(struct media_entity *entity);
>>>  
>>>  	const struct media_device_ops *ops;
>>> +
>>> +	struct mutex req_queue_mutex;
>>>  };
>>>  
>>>  /* We don't need to include pci.h or usb.h here */
>>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>>> new file mode 100644
>>> index 000000000000..e39122dfd717
>>> --- /dev/null
>>> +++ b/include/media/media-request.h
>>> @@ -0,0 +1,244 @@
>>> +// SPDX-License-Identifier: GPL-2.0-only
>>> +/*
>>> + * Media device request objects
>>> + *
>>> + * Copyright 2018 Cisco Systems, Inc. and/or its affiliates. All rights reserved.
>>> + * Copyright (C) 2018 Intel Corporation
>>> + *
>>> + * Author: Hans Verkuil <hans.verkuil@cisco.com>
>>> + * Author: Sakari Ailus <sakari.ailus@linux.intel.com>
>>> + */
>>> +
>>> +#ifndef MEDIA_REQUEST_H
>>> +#define MEDIA_REQUEST_H
>>> +
>>> +#include <linux/list.h>
>>> +#include <linux/slab.h>
>>> +#include <linux/spinlock.h>
>>> +#include <linux/atomic.h>
>>> +
>>> +#include <media/media-device.h>
>>> +
>>> +/**
>>> + * enum media_request_state - media request state
>>> + *
>>> + * @MEDIA_REQUEST_STATE_IDLE:		Idle
>>> + * @MEDIA_REQUEST_STATE_VALIDATING:	Validating the request, no state changes
>>> + *					allowed
>>> + * @MEDIA_REQUEST_STATE_QUEUED:		Queued
>>> + * @MEDIA_REQUEST_STATE_COMPLETE:	Completed, the request is done
>>> + * @MEDIA_REQUEST_STATE_CLEANING:	Cleaning, the request is being re-inited
>>> + */
>>> +enum media_request_state {
>>> +	MEDIA_REQUEST_STATE_IDLE,
>>> +	MEDIA_REQUEST_STATE_VALIDATING,
>>> +	MEDIA_REQUEST_STATE_QUEUED,
>>> +	MEDIA_REQUEST_STATE_COMPLETE,
>>> +	MEDIA_REQUEST_STATE_CLEANING,
>>> +};
>>> +
>>> +struct media_request_object;
>>> +
>>> +/**
>>> + * struct media_request - Media device request
>>> + * @mdev: Media device this request belongs to
>>> + * @kref: Reference count
>>> + * @debug_str: Prefix for debug messages (process name:fd)
>>> + * @state: The state of the request
>>> + * @objects: List of @struct media_request_object request objects
>>> + * @num_objects: The number of objects in the request
>>> + * @num_incompleted_objects: The number of incomplete objects in the request
>>> + * @poll_wait: Wait queue for poll
>>> + * @lock: Serializes access to this struct
>>> + */
>>> +struct media_request {
>>> +	struct media_device *mdev;
>>> +	struct kref kref;
>>> +	char debug_str[TASK_COMM_LEN + 11];
>>> +	atomic_t state;
>>> +	struct list_head objects;
>>> +	unsigned int num_incomplete_objects;
>>> +	struct wait_queue_head poll_wait;
>>> +	spinlock_t lock;
>>> +};
>>> +
>>> +#ifdef CONFIG_MEDIA_CONTROLLER
>>> +
>>> +/**
>>> + * media_request_get - Get the media request
>>> + *
>>> + * @req: The request
>>> + *
>>> + * Get the media request.
>>> + */
>>> +static inline void media_request_get(struct media_request *req)
>>> +{
>>> +	kref_get(&req->kref);
>>> +}
>>> +
>>> +/**
>>> + * media_request_put - Put the media request
>>> + *
>>> + * @req: The request
>>> + *
>>> + * Put the media request. The media request will be released
>>> + * when the refcount reaches 0.
>>> + */
>>> +void media_request_put(struct media_request *req);
>>> +
>>> +/**
>>> + * media_request_alloc - Allocate the media request
>>> + *
>>> + * @mdev: Media device this request belongs to
>>> + * @alloc: Store the request's file descriptor in this struct
>>> + *
>>> + * Allocated the media request and put the fd in @alloc->fd.
>>> + */
>>> +int media_request_alloc(struct media_device *mdev,
>>> +			struct media_request_alloc *alloc);
>>> +
>>> +#else
>>> +
>>> +static inline void media_request_get(struct media_request *req)
>>> +{
>>> +}
>>> +
>>> +static inline void media_request_put(struct media_request *req)
>>> +{
>>> +}
>>> +
>>> +#endif
>>> +
>>> +/**
>>> + * struct media_request_object_ops - Media request object operations
>>> + * @prepare: Validate and prepare the request object, optional.
>>> + * @unprepare: Unprepare the request object, optional.
>>> + * @queue: Queue the request object, optional.
>>> + * @unbind: Unbind the request object, optional.
>>> + * @release: Release the request object, required.
>>> + */
>>> +struct media_request_object_ops {
>>> +	int (*prepare)(struct media_request_object *object);
>>> +	void (*unprepare)(struct media_request_object *object);
>>> +	void (*queue)(struct media_request_object *object);
>>> +	void (*unbind)(struct media_request_object *object);
>>> +	void (*release)(struct media_request_object *object);
>>> +};
>>> +
>>> +/**
>>> + * struct media_request_object - An opaque object that belongs to a media
>>> + *				 request
>>> + *
>>> + * @ops: object's operations
>>> + * @priv: object's priv pointer
>>> + * @req: the request this object belongs to (can be NULL)
>>> + * @list: List entry of the object for @struct media_request
>>> + * @kref: Reference count of the object, acquire before releasing req->lock
>>> + * @completed: If true, then this object was completed.
>>> + *
>>> + * An object related to the request. This struct is embedded in the
>>> + * larger object data.
> 
> what do you mean by "the larger object data"? What struct is "the" struct?

Improved the text to clarify this.

> 
>>> + */
>>> +struct media_request_object {
>>> +	const struct media_request_object_ops *ops;
>>> +	void *priv;
>>> +	struct media_request *req;
>>> +	struct list_head list;
>>> +	struct kref kref;
>>> +	bool completed;
>>> +};
>>> +
>>> +#ifdef CONFIG_MEDIA_CONTROLLER
>>> +
>>> +/**
>>> + * media_request_object_get - Get a media request object
>>> + *
>>> + * @obj: The object
>>> + *
>>> + * Get a media request object.
>>> + */
>>> +static inline void media_request_object_get(struct media_request_object *obj)
>>> +{
>>> +	kref_get(&obj->kref);
>>> +}
>>> +
>>> +/**
>>> + * media_request_object_put - Put a media request object
>>> + *
>>> + * @obj: The object
>>> + *
>>> + * Put a media request object. Once all references are gone, the
>>> + * object's memory is released.
>>> + */
>>> +void media_request_object_put(struct media_request_object *obj);
>>> +
>>> +/**
>>> + * media_request_object_init - Initialise a media request object
>>> + *
>>> + * Initialise a media request object. The object will be released using the
>>> + * release callback of the ops once it has no references (this function
>>> + * initialises references to one).
>>> + */
>>> +void media_request_object_init(struct media_request_object *obj);
>>> +
>>> +/**
>>> + * media_request_object_bind - Bind a media request object to a request  
>>
>> Argument documentation is missing.
>>
>> I think you should also say that "every bound object must be unbound later
>> on".
>>
>>> + */
>>> +int media_request_object_bind(struct media_request *req,
>>> +			      const struct media_request_object_ops *ops,
>>> +			      void *priv,
>>> +			      struct media_request_object *obj);
>>> +
>>> +/**
>>> + * media_request_object_unbind - Unbind a media request object
>>> + *
>>> + * @obj: The object
>>> + *
>>> + * Unbind the media request object from the request.
>>> + */
>>> +void media_request_object_unbind(struct media_request_object *obj);
>>> +
>>> +/**
>>> + * media_request_object_complete - Mark the media request object as complete
>>> + *
>>> + * @obj: The object
>>> + *
>>> + * Mark the media request object as complete.  
>>
>> Add:
>>
>> Only bound request objects may be completed.
>>
>>> + */
>>> +void media_request_object_complete(struct media_request_object *obj);
>>> +
>>> +#else
>>> +
>>> +static inline void media_request_object_get(struct media_request_object *obj)
>>> +{
>>> +}
>>> +
>>> +static inline void media_request_object_put(struct media_request_object *obj)
>>> +{
>>> +}
>>> +
>>> +static inline void media_request_object_init(struct media_request_object *obj)
>>> +{
>>> +	obj->ops = NULL;
>>> +	obj->req = NULL;
>>> +}
>>> +
>>> +static inline int media_request_object_bind(struct media_request *req,
>>> +			       const struct media_request_object_ops *ops,
>>> +			       void *priv,
>>> +			       struct media_request_object *obj)
>>> +{
>>> +	return 0;
>>> +}
>>> +
>>> +static inline void media_request_object_unbind(struct media_request_object *obj)
>>> +{
>>> +}
>>> +
>>> +static inline void media_request_object_complete(struct media_request_object *obj)
>>> +{
>>> +}
>>> +
>>> +#endif
>>> +
>>> +#endif  
>>
> 
> 
> 
> Thanks,
> Mauro
> 

Thank you for all the feedback!

Regards,

	Hans

Re: [PATCHv13 22/28] Documentation: v4l: document request API

[Hans Verkuil]

On 18/05/18 16:46, Laurent Pinchart wrote:
> Hi Hans,
> 
> Thank you for the patch.
> 
> On Thursday, 3 May 2018 17:53:12 EEST Hans Verkuil wrote:
>> From: Alexandre Courbot <acourbot@chromium.org>
>>
>> Document the request API for V4L2 devices, and amend the documentation
>> of system calls influenced by it.
>>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> ---
>>  .../media/uapi/mediactl/media-funcs.rst       |   3 +
>>  .../uapi/mediactl/media-ioc-request-alloc.rst |  71 ++++++
>>  .../uapi/mediactl/media-request-ioc-queue.rst |  46 ++++
>>  .../mediactl/media-request-ioc-reinit.rst     |  51 +++++
>>  Documentation/media/uapi/v4l/buffer.rst       |  18 +-
>>  Documentation/media/uapi/v4l/common.rst       |   1 +
>>  Documentation/media/uapi/v4l/request-api.rst  | 211 ++++++++++++++++++
>>  .../media/uapi/v4l/vidioc-g-ext-ctrls.rst     |  25 ++-
>>  Documentation/media/uapi/v4l/vidioc-qbuf.rst  |   7 +
>>  .../media/videodev2.h.rst.exceptions          |   1 +
>>  10 files changed, 428 insertions(+), 6 deletions(-)
>>  create mode 100644
>> Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst create mode
>> 100644 Documentation/media/uapi/mediactl/media-request-ioc-queue.rst create
>> mode 100644 Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
>> create mode 100644 Documentation/media/uapi/v4l/request-api.rst
> 
> I think I would have split documentation and included each part in the 
> corresponding API definition patch to make it easier to review the definition 
> and the documentation together, but that's not a big deal.
> 
> We might however still want to split this patch in two, with one part for the 
> media controller API and one part for the V4L2 API, as the request API on MC 
> isn't V4L2-dependent.
> 
>> diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst
>> b/Documentation/media/uapi/mediactl/media-funcs.rst index
>> 076856501cdb..f4da9b3e17ec 100644
>> --- a/Documentation/media/uapi/mediactl/media-funcs.rst
>> +++ b/Documentation/media/uapi/mediactl/media-funcs.rst
>> @@ -16,3 +16,6 @@ Function Reference
>>      media-ioc-enum-entities
>>      media-ioc-enum-links
>>      media-ioc-setup-link
>> +    media-ioc-request-alloc
>> +    media-request-ioc-queue
>> +    media-request-ioc-reinit
> 
> Do we want to keep this alphabetically ordered ?

It isn't today. It is more ordered in the sequence you would use the
ioctls. I have no strong preference one way or another.

> 
>> diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
>> b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst new file
>> mode 100644
>> index 000000000000..d45a94c9e23c
>> --- /dev/null
>> +++ b/Documentation/media/uapi/mediactl/media-ioc-request-alloc.rst
>> @@ -0,0 +1,71 @@
>> +.. SPDX-License-Identifier: GPL-2.0-only
> 
> Documentation/process/license-rules.rst doesn't specify GPL-2.0-only, this 
> should be GPL-2.0. The comment applies to all other new files.

GPL-2.0-only should be a valid license in 4.18 (patches were posted to add this).
It's confusing since SPDX has deprecated GPL-2.0 in favor of GPL-2.0-only.

I'll keep this unless it is still illegal in 4.18.

> 
> This being said, GPL isn't really a documentation license, shouldn't we pick a 
> more appropriate license ?

All our docs are under GPL. I don't want to introduce something else.

> 
>> +
>> +.. _media_ioc_request_alloc:
>> +
>> +*****************************
>> +ioctl MEDIA_IOC_REQUEST_ALLOC
>> +*****************************
>> +
>> +Name
>> +====
>> +
>> +MEDIA_IOC_REQUEST_ALLOC - Allocate a request
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_ALLOC, struct
>> media_request_alloc *argp )
>> +    :name: MEDIA_IOC_REQUEST_ALLOC
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <media-func-open>`.
>> +
>> +``argp``
>> +
>> +
>> +Description
>> +===========
>> +
>> +If the media device supports :ref:`requests <media-request-api>`, then
>> +this ioctl can be used to allocate a request. A request is accessed through
>> +a file descriptor that is returned in struct
>> :c:type:`media_request_alloc`.
>> +
>> +If the request was successfully allocated, then the request file descriptor
>> +can be passed to :ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>`,
>> +:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`,
>> +:ref:`ioctl VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` and
>> +:ref:`ioctl VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
> 
> Conceptually speaking, shouldn't we refrain from referencing the V4L2 API from 
> the MC API ?

Strictly speaking, yes, but I think it makes sense to be practical here and
give the reader links to the ioctls that can actually be used with requests.
That way there is no need to hunt around for this information.

I'm keeping this for now.

> 
>> +In addition, the request can be queued by calling
>> +:ref:`MEDIA_REQUEST_IOC_QUEUE` and re-initialized by calling
>> +:ref:`MEDIA_REQUEST_IOC_REINIT`.
>> +
>> +Finally, the file descriptor can be polled to wait for the request to
>> +complete.
>> +
>> +To free the request the file descriptor has to be closed.
>> +
>> +.. c:type:: media_request_alloc
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>> +
>> +.. flat-table:: struct media_request_alloc
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    *  -  __s32
>> +       -  ``fd``
>> +       -  The file descriptor of the request.
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
>> b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst new file
>> mode 100644
>> index 000000000000..79f745a3d663
>> --- /dev/null
>> +++ b/Documentation/media/uapi/mediactl/media-request-ioc-queue.rst
>> @@ -0,0 +1,46 @@
>> +.. SPDX-License-Identifier: GPL-2.0-only
>> +
>> +.. _media_request_ioc_queue:
>> +
>> +*****************************
>> +ioctl MEDIA_REQUEST_IOC_QUEUE
>> +*****************************
>> +
>> +Name
>> +====
>> +
>> +MEDIA_REQUEST_IOC_QUEUE - Queue a request
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_QUEUE )
>> +    :name: MEDIA_REQUEST_IOC_QUEUE
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``request_fd``
>> +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +If the media device supports :ref:`requests <media-request-api>`, then
>> +this request ioctl can be used to queue a previously allocated request.
>> +
>> +If the request was successfully queued, then the file descriptor can be
>> +polled to wait for the request to complete.
> 
> This says nothing about what queuing a request actually does :-) Neither does 
> it document why it could fail.

It links to the main request API chapter, which explains this in detail.
I don't think it is useful to repeat that information.

I added a bit about the error codes.

> 
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +EBUSY
>> +    The request was already queued.
>> diff --git a/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
>> b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst new file
>> mode 100644
>> index 000000000000..7cbb7eb5d73e
>> --- /dev/null
>> +++ b/Documentation/media/uapi/mediactl/media-request-ioc-reinit.rst
>> @@ -0,0 +1,51 @@
>> +.. SPDX-License-Identifier: GPL-2.0-only
>> +
>> +.. _media_request_ioc_reinit:
>> +
>> +******************************
>> +ioctl MEDIA_REQUEST_IOC_REINIT
>> +******************************
>> +
>> +Name
>> +====
>> +
>> +MEDIA_REQUEST_IOC_REINIT - Re-initialize a request
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int request_fd, MEDIA_REQUEST_IOC_REINIT )
>> +    :name: MEDIA_REQUEST_IOC_REINIT
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``request_fd``
>> +    File descriptor returned by :ref:`MEDIA_IOC_REQUEST_ALLOC`.
> 
> I was expecting an argument to this ioctl, with a reference to another request 
> (and I expected the same reference for the MEDIA_IOC_REQUEST_ALLOC ioctl). I 
> think this topic has been discussed several times in the past, but possibly 
> not in venues where everybody was present, so I'll detail the problem here.
> 
> The idea behind this version of the request API is to split request handling 
> in three parts:
> 
> 1. A request is allocated (MEDIA_IOC_REQUEST_ALLOC) or an existing request is 
> re-initialized (MEDIA_REQUEST_IOC_REINIT). At that point the request contains 
> no property (control, format, ...) to be modified.
> 
> 2. The request is filled with properties to be modified using existing V4L2 
> ioctls (VIDIOC_S_EXT_CTRLS, VIDIOC_SUBDEV_S_FMT, VIDIOC_SUBDEV_S_SELECTION, 
> ..). The current implementation supports controls only, with support for 
> formats and selection rectangles to be added later.
> 
> 3. The request is queued with MEDIA_REQUEST_IOC_QUEUE.
> 
> As usual the devil is in the details, and the details of the second step in 
> particular.
> 
> The idea behind splitting request handling in those three steps is to reuse 
> existing V4L2 ioctls as much as possible, and to keep the existing semantics 
> of those ioctls. I believe this isn't possible with the proposed 
> MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls, neither for 
> controls nor for formats or selection rectangles, although the problem is more 
> limited for controls. I will thus take formats and selection rectangles to 
> explain where the problem lies.
> 
> The S_FMT and S_SELECTION semantics on subdevs (and, for that matter, on video 
> nodes too) is that those ioctls don't fail when passed parameters invalid for 
> the device, but return a valid configuration as close as possible to the 
> requested one. This allows implementing negotiation of formats and selection 
> rectangles through an iterative process that involves back-and-forth calls 
> between user space and kernel space until the application finds a usable 
> configuration accepted by the driver, or until it decides to fail because such 
> a configuration doesn't exist. (As a reminder, such negotiation is needed 
> because it was deemed impossible to create a proper API that would expose all 
> device constraints in a static way.)
> 
> Formats and selection rectangles are propagated within subdevs from sink pads 
> to source pads by the driver. Sink formats and selection rectangles thus 
> influence the result of negotiation on source pads. For instance, on a scaler 
> subdev that can't perform format conversion, the format on the source pad will 
> be dictated by the format on the sink pad. In order to implement S_FMT on the 
> source pad the kernel driver thus needs context information to get the format 
> on the sink pad.
> 
> Without the request API this is easy. The context is either stored in the 
> driver-specific device structure (for active formats) or in the file handle 
> (for try formats). With the proposed request API, however, we have no such 
> context: the requests are created empty (or re-initialized to be empty). 
> Kernel drivers can access the active state of the device, or the state of 
> previously queued requests, but can't access the state of prepared but not yet 
> queued requests.

Indeed, this is also the case for controls. When you first bind a control
handler to a request it will contain the current HW values for any controls
there were never explicitly set for the request. When you queue the request
it will link to the first request in the existing queue that sets the control
or the current HW value if there wasn't any.

I did think of using the handle of a previous request, but then you have to
make a full clone of all the data in the request you passed in, but that
might be a lot of data and that is something I'd like to avoid.

> My proposal to solve this was to link requests by specifying the handle of a 
> previous request to the MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT 
> ioctls. That way kernel drivers would know in which order requests will be 
> queued, and would be able to access the correct context information to 
> implement the existing semantics of the S_FMT and S_SELECTION ioctls (as well 
> as being able to implement the G_FMT and G_SELECTION ioctls). Note that the 
> problem isn't limited to format and selection rectangles, similar problems can 
> exist when multiple controls are related, such as auto and manual controls.
> 
> This would result in a more complex API and a more complex request lifetime 
> management, as well as a more complex semantics of the request API. While I 
> thought this was unavoidable, I came to reconsider my position on this topic, 
> and I believe that the current request API proposal is acceptable in that 
> regard, provided that we agree that the existing semantics of the S_FMT and 
> S_SELECTION ioctls won't need to be implemented for requests as it won't be 
> possible to do so. The reason that prompted me to change my mind is that I 
> believe the API would be simpler to define and simpler to use if we dropped 
> that semantics, and required applications to create requests that are fully 
> valid instead of trying to mangle the content of a request the same way we do 
> for format and selection rectangle negotiation. Obviously, negotiation is 
> still needed in order for applications to find a valid configuration, but we 
> have the V4L2_SUBDEV_FORMAT_TRY API for that that, in conjunction with the 
> S_FMT and S_SELECTION ioctls, can be used to negotiate a configuration. The 
> negotiated configuration could then be set in a request, and would be 
> guaranteed to be valid.

One other option might be to implement a MEDIA_REQUEST_IOC_TRY ioctl. It will
'try' the request against the current tip of the request queue and the driver
can modify the data. If OK, then the application can call IOC_QUEUE to
actually queue the request. Internally IOC_TRY would just call req_validate
with a 'bool try' argument set to true.

I'm not sure if it worth the effort, but it would at least be possible.

> 
> To summarize this detailed explanation, I am fine with the 
> MEDIA_IOC_REQUEST_ALLOC and MEDIA_REQUEST_IOC_REINIT ioctls not providing a 
> way to link requests, and thus making the API simpler, if we acknowledge that 
> the S_FMT and S_SELECTION ioctls won't be possible to implement with their 
> current semantics and agree not to implement format and selection rectangle 
> negotiation on requests.

I agree.

> What I want to avoid is merging the MEDIA_IOC_REQUEST_ALLOC and 
> MEDIA_REQUEST_IOC_REINIT ioctls as they are proposed today, and having to 
> modify their semantics by adding request linking in the future. That should 
> significantly change the semantics of the request API which could be very 
> difficult to do without breaking userspace, and extending the API while 
> keeping support for the current semantics would result in additional 
> complexity that I'm not willing to accept. If anyone think that request 
> linking, needed if we want to access entity state from the existing V4L2 
> subdev ioctl handlers, is desired, then it needs to be built in the API from 
> the start.
> 
>> +Description
>> +===========
>> +
>> +If the media device supports :ref:`requests <media-request-api>`, then
>> +this request ioctl can be used to re-initialize a previously allocated
>> +request.
>> +
>> +Re-initializing a request will clear any existing data from the request.
>> +This avoids having to close() a completed request and allocate a new
>> +request. Instead the completed request can just be re-initialized and
>> +it is ready to be used again.
>> +
>> +A request can only be re-initialized if it either has not been queued
>> +yet, or if it was queued and completed.
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +EBUSY
>> +    The request is queued but not yet completed.
>> diff --git a/Documentation/media/uapi/v4l/buffer.rst
>> b/Documentation/media/uapi/v4l/buffer.rst index e2c85ddc990b..8059fc0ed520
>> 100644
>> --- a/Documentation/media/uapi/v4l/buffer.rst
>> +++ b/Documentation/media/uapi/v4l/buffer.rst
>> @@ -306,10 +306,12 @@ struct v4l2_buffer
>>        - A place holder for future extensions. Drivers and applications
>>  	must set this to 0.
>>      * - __u32
>> -      - ``reserved``
>> +      - ``request_fd``
>>        -
>> -      - A place holder for future extensions. Drivers and applications
>> -	must set this to 0.
>> +      - The file descriptor of the request to queue the buffer to. If
>> specified
>> +        and flag ``V4L2_BUF_FLAG_REQUEST_FD`` is set, then the buffer will
>> be
>> +	queued to that request. This is set by the user when calling
>> +	:ref:`ioctl VIDIOC_QBUF <VIDIOC_QBUF>` and ignored by other ioctls.
> 
> Do we need that flag ? Isn't enough to just set the request_fd field ?

Yes, it is needed. 0 is a valid fd, and applications currently set this
field to 0. So we need a flag to signal the presence of an fd.

> 
>> @@ -514,6 +516,11 @@ Buffer Flags
>>  	streaming may continue as normal and the buffer may be reused
>>  	normally. Drivers set this flag when the ``VIDIOC_DQBUF`` ioctl is
>>  	called.
>> +    * .. _`V4L2-BUF-FLAG-IN-REQUEST`:
>> +
>> +      - ``V4L2_BUF_FLAG_IN_REQUEST``
>> +      - 0x00000080
>> +      - This buffer is part of a request that hasn't been queued yet.
> 
> What is this used for ? I don't see it being explained in the documentation.

I'll improve this. It is set if the buffer was added to a request but
the request itself hasn't been queued yet. It's a status flag.

That said, I need to make some changes to vb2 since there are some problems
with combining VIDIOC_BUF_PREPARE and requests.

> 
>>      * .. _`V4L2-BUF-FLAG-KEYFRAME`:
>>
>>        - ``V4L2_BUF_FLAG_KEYFRAME``
>> @@ -589,6 +596,11 @@ Buffer Flags
>>  	the format. Any Any subsequent call to the
>>
>>  	:ref:`VIDIOC_DQBUF <VIDIOC_QBUF>` ioctl will not block anymore,
>>
>>  	but return an ``EPIPE`` error code.
>> +    * .. _`V4L2-BUF-FLAG-REQUEST-FD`:
>> +
>> +      - ``V4L2_BUF_FLAG_REQUEST_FD``
>> +      - 0x00800000
>> +      - The ``request_fd`` field contains a valid file descriptor.
>>      * .. _`V4L2-BUF-FLAG-TIMESTAMP-MASK`:
>>
>>        - ``V4L2_BUF_FLAG_TIMESTAMP_MASK``
>> diff --git a/Documentation/media/uapi/v4l/common.rst
>> b/Documentation/media/uapi/v4l/common.rst index 13f2ed3fc5a6..a4aa0059d45a
>> 100644
>> --- a/Documentation/media/uapi/v4l/common.rst
>> +++ b/Documentation/media/uapi/v4l/common.rst
>> @@ -44,3 +44,4 @@ applicable to all devices.
>>      crop
>>      selection-api
>>      streaming-par
>> +    request-api
>> diff --git a/Documentation/media/uapi/v4l/request-api.rst
>> b/Documentation/media/uapi/v4l/request-api.rst new file mode 100644
>> index 000000000000..0ce75b254956
>> --- /dev/null
>> +++ b/Documentation/media/uapi/v4l/request-api.rst
>> @@ -0,0 +1,211 @@
>> +.. -*- coding: utf-8; mode: rst -*-
>> +
>> +.. _media-request-api:
>> +
>> +Request API
>> +===========
>> +
>> +The Request API has been designed to allow V4L2 to deal with requirements
>> of +modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android
> 
> I wouldn't call this MIPI cameras. In this context MIPI only defines the bus 
> between the camera sensor and the ISP or bridge, so it's quite irrelevant. 
> "ISP-based cameras" might be a better term, although some ISP-based cameras 
> could still offer a high-level API that doesn't require or even benefit from 
> the use of requests.
> 
>> Codec +v2). One such requirement is the ability for devices belonging to
>> the same +pipeline to reconfigure and collaborate closely on a per-frame
>> basis. Another is +efficient support of stateless codecs, which need
>> per-frame controls to be set +asynchronously in order to be used
>> efficiently.
> 
> Do you really mean asynchronously ? I thought the purpose of the request API 
> with codecs was to set controls synchronously with frames.

Right.

> 
>> +Supporting these features without the Request API is possible but terribly
>> +inefficient: user-space would have to flush all activity on the media
>> pipeline, +reconfigure it for the next frame, queue the buffers to be
>> processed with that +configuration, and wait until they are all available
>> for dequeuing before +considering the next frame. This defeats the purpose
>> of having buffer queues +since in practice only one buffer would be queued
>> at a time.
> 
> It's not even always possible. For memory to memory devices such a behaviour 
> might be possible, but for camera devices in general you don't have the option 
> of stopping and reconfiguring the hardware as frames are continuously captured 
> by the sensor.
> 
>> +The Request API allows a specific configuration of the pipeline (media
>> +controller topology + controls for each device) to be associated with
> 
> Should we mention formats too ?

Not supported at the moment. But yes, this will have to be added in the
future.

 Should we use "entity" instead of "device" as
> this documents an MC-based API ?

Yes, that's better.

> 
>> specific +buffers. The parameters are applied by each participating device
>> as buffers +associated to a request flow in. This allows user-space to
>> schedule several +tasks ("requests") with different parameters in advance,
>> knowing that the +parameters will be applied when needed to get the
>> expected result. Control +values at the time of request completion are also
>> available for reading.
>> +
>> +Usage
>> +=====
>> +
>> +The Request API is used on top of standard media controller and V4L2 calls,
>> +which are augmented with an extra ``request_fd`` parameter. Requests
>> themselves +are allocated from the supporting media controller node.
> 
> I'd like it if we could document the request API in the MC documentation for 
> the overall concept, and then have a V4L2-specific documentation that explains 
> how the request API is used with V4L2.

I agree, it should be moved.

> 
>> +
>> +Request Allocation
>> +------------------
>> +
>> +User-space allocates requests using :ref:`MEDIA_IOC_REQUEST_ALLOC`
>> +for the media device node. This returns a file descriptor representing the
>> +request. Typically, several such requests will be allocated.
>> +
>> +Request Preparation
>> +-------------------
>> +
>> +Standard V4L2 ioctls can then receive a request file descriptor to express
>> the +fact that the ioctl is part of said request, and is not to be applied
>> +immediately. See :ref:`MEDIA_IOC_REQUEST_ALLOC` for a list of ioctls that
>> +support this.
> 
> As explained above in the documentation of MEDIA_IOC_REQUEST_ALLOC, I would 
> rather list the ioctls explicitly here, as MEDIA_IOC_REQUEST_ALLOC is really 
> not V4L2-specific.
> 
>> Controls set with a ``request_fd`` parameter are stored
>> instead +of being immediately applied, and buffers queued to a request do
>> not enter the +regular buffer queue until the request itself is queued.
>> +
>> +Request Submission
>> +------------------
>> +
>> +Once the parameters and buffers of the request are specified, it can be
>> +queued by calling :ref:`MEDIA_REQUEST_IOC_QUEUE` on the request FD.
>> +This will make the buffers associated to the request available to their
>> driver, +which can then apply the associated controls as buffers are
>> processed. A queued +request cannot be modified anymore.
>> +
>> +If several devices are part of the request, individual drivers may
> 
> I'd write "If parameters for multiple entities are part of the request", or 
> "If the request contains parameters for multiple entities".
> 
>> synchronize +so the requested pipeline's topology is applied before the
>> buffers are +processed. This is at the discretion of media controller
>> drivers and is not a +requirement.
> 
> One of the main purposes of the request API is to support atomic 
> configuration. Saying that it's simply optional defeats that purpose. I agree 
> that atomicity can't be achieved in all cases, and that there will often be a 
> best-effort approach, but for hardware devices that support atomic 
> configuration I think applications need to be able to rely on the driver 
> implementing that feature. I would thus put a bit more pressure on drivers 
> than just saying it's fully optional.

I agree, I made the change ('best effort implementation').

> 
>> +Buffers queued without an associated request after a request-bound buffer
>> will +be processed using the state of the hardware at the time of the
>> request +completion.
> 
> I'm not sure to follow that, could you explain in a bit more details ? Also, 
> even more than for controls (see below), I really want to know about use cases 
> for mixing buffers in and outside of requests for the same buffer queue before 
> accepting it.

In v15 I prohibit queueing buffers directly once requests are used. I see no
use-case for mixing these modes and it is too messy.

> 
> I also would like to see a detailed description of what happens when buffers 
> are queued in a request for one queue and outside of requests for another 
> queue (for example in the case of codecs), for both capture and output queues.

See my RFC (specific to m2m devices at the moment):

https://www.mail-archive.com/linux-media@vger.kernel.org/msg132202.html

> 
>> All the same, controls set without a request are applied +immediately,
>> regardless of whether a request is in use or not.
> 
> This will be tricky to implement, and will potentially generate side effects 
> difficult to handle, and difficult to debug. I understand that this is a 
> requirement for codecs, but I don't know the exact use case(s), so I can't 
> comment on whether this is the right approach. While I don't necessarily 
> reject this feature on principle, I first want to know more about the use 
> case(s) before accepting it. If we end up needing it, we'll have to better 
> document it, by clearly specifying what is allowed and what isn't (for 
> instance, we might decide to allow setting control asynchronously only when 
> they're not simultaneously modified through a queued request), as well as the 
> exact behaviour that applications can expect in the allowed cases.

I discussed this in my previous reply.

> 
>> +User-space can ``poll()`` a request FD in order to wait until the request
>> +completes. A request is considered complete once all its associated buffers
>> are +available for dequeuing and all the associated controls have been
>> updated with +the values at the time of completion. Note that user-space
>> does not need to wait +for the request to complete to dequeue its buffers:
>> buffers that are available +halfway through a request can be dequeued
>> independently of the request's state.
>> +
>> +A completed request contains the state of the request at the time of the
>> +request completion. User-space can query that state by calling
>> +:ref:`ioctl VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` with the request FD.
>> +
>> +Recycling and Destruction
>> +-------------------------
>> +
>> +Finally, completed request can either be discarded or be reused. Calling
>> +``close()`` on a request FD will make that FD unusable, freeing the request
>> if +it is not referenced elsewhere.
> 
> Where else can it be referenced ? Does this only refer to the ability to 
> duplicate FDs or share them with other processes ? If so, I would document 
> that requests are freed when all file descriptors that refer to them are 
> closed, not when close() is called on one file descriptor.

I rephrased this. It referred to the kernel still having a reference to it.
I.e. if you queue a request, then close it, then the kernel still is using
it until the request is completed. Only then is it freed.

> 
>> The :ref:`MEDIA_REQUEST_IOC_REINIT`
>> will +clear a request's state and make it available again. No state is
>> retained by +this operation: the request is as if it had just been
>> allocated.
>> +
>> +Example for a Memory-to-Memory Device
>> +-------------------------------------
>> +
>> +M2M devices are single-node V4L2 devices providing one OUTPUT queue (for
>> +user-space to provide input buffers) and one CAPTURE queue (to retrieve
>> +processed data). These devices are commonly used for frame processors or
>> +stateless codecs.
>> +
>> +In this use-case, the request API can be used to associate specific
>> controls to +be applied by the driver before processing an OUTPUT buffer,
> 
> As commented below, I would explain that the controls are applied for the 
> buffer, not before processing it.

Done.

> 
>> allowing user-space +to queue many such buffers in advance. It can also
>> take advantage of requests' +ability to capture the state of controls when
>> the request completes to read back +information that may be subject to
>> change.
>> +
>> +Put into code, after obtaining a request, user-space can assign controls
>> and one +OUTPUT buffer to it:
>> +
>> +.. code-block:: c
>> +
>> +	struct v4l2_buffer buf;
>> +	struct v4l2_ext_controls ctrls;
>> +	struct media_request_alloc alloc = { 0 };
>> +	int req_fd;
>> +	...
>> +	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
>> +	req_fd = alloc.fd;
>> +	...
>> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
>> +	ctrls.request_fd = req_fd;
>> +	ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
>> +	...
>> +	buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
>> +	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
>> +	buf.request_fd = req_fd;
>> +	ioctl(codec_fd, VIDIOC_QBUF, &buf);
>> +
>> +Note that there is typically no need to use the Request API for CAPTURE
>> buffers +since there are per-frame settings to report there.
> 
> Do you mean since there are no per-frame settings to report there ?

Indeed. Fixed.

> 
>> +Once the request is fully prepared, it can be queued to the driver:
>> +
>> +.. code-block:: c
>> +
>> +	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
>> +
>> +User-space can then either wait for the request to complete by calling
>> poll() on +its file descriptor, or start dequeuing CAPTURE buffers. Most
>> likely, it will +want to get CAPTURE buffers as soon as possible and this
>> can be done using a +regular DQBUF:
>> +
>> +.. code-block:: c
>> +
>> +	struct v4l2_buffer buf;
>> +
>> +	memset(&buf, 0, sizeof(buf));
>> +	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +	ioctl(codec_fd, VIDIOC_DQBUF, &buf);
> 
> The above explanation makes it sounds like waiting for the request to complete 
> will ensure that a capture buffer is available, and that's not necessarily the 
> case. There isn't always a guarantee that one output buffer will result in one 
> and only one capture buffer being filled.

I've made a note of this in the documentation.

> 
>> +We can then, after ensuring that the request is completed via polling the
>> +request FD, query control values at the time of its completion via an
>> +annotated call to G_EXT_CTRLS. This is particularly useful for volatile
>> controls +for which we want to query values as soon as the capture buffer
>> is produced. +
>> +.. code-block:: c
>> +
>> +	struct pollfd pfd = { .events = POLLPRI, .fd = request_fd };
>> +	poll(&pfd, 1, -1);
>> +	...
>> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
>> +	ctrls.request_fd = req_fd;
>> +	ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls);
>> +
>> +Once we don't need the request anymore, we can either recycle it for reuse
>> with +:ref:`MEDIA_REQUEST_IOC_REINIT`...
>> +
>> +.. code-block:: c
>> +
>> +	ioctl(req_fd, MEDIA_REQUEST_IOC_REINIT);
>> +
>> +... or close its file descriptor to completely dispose of it.
>> +
>> +.. code-block:: c
>> +
>> +	close(req_fd);
>> +
>> +Example for a Simple Capture Device
>> +-----------------------------------
>> +
>> +With a simple capture device, requests can be used to specify controls to
>> apply +to a given CAPTURE buffer. The driver will apply these controls
>> before producing +the marked CAPTURE buffer.
> 
> As explained below I think we should be a bit more precise with the wording 
> here.
> 
>> +.. code-block:: c
>> +
>> +	struct v4l2_buffer buf;
>> +	struct v4l2_ext_controls ctrls;
>> +	struct media_request_alloc alloc = { 0 };
>> +	int req_fd;
>> +	...
>> +	ioctl(media_fd, MEDIA_IOC_REQUEST_ALLOC, &alloc);
>> +	req_fd = alloc.fd;
>> +	...
>> +	ctrls.which = V4L2_CTRL_WHICH_REQUEST_VAL;
>> +	ctrls.request_fd = req_fd;
>> +	ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
>> +	...
>> +	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
>> +	buf.flags |= V4L2_BUF_FLAG_REQUEST_FD;
>> +	buf.request_fd = req_fd;
>> +	ioctl(camera_fd, VIDIOC_QBUF, &buf);
>> +
>> +Once the request is fully prepared, it can be queued to the driver:
>> +
>> +.. code-block:: c
>> +
>> +	ioctl(req_fd, MEDIA_REQUEST_IOC_QUEUE);
>> +
>> +User-space can then dequeue buffers, wait for the request completion, query
>> +controls and recycle the request as in the M2M example above.
>> diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
>> b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst index
>> 2011c2b2ee67..8f788c444347 100644
>> --- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
>> +++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
>> @@ -95,6 +95,19 @@ appropriate. In the first case the new value is set in
>> struct is inappropriate (e.g. the given menu index is not supported by the
>> menu control), then this will also result in an ``EINVAL`` error code
>> error.
>>
>> +If ``request_fd`` is set to a not-submitted request file descriptor and
>> +``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``, then the
>> +controls are not applied immediately when calling
>> +:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`, but instead are applied
>> right +before the driver starts processing a buffer associated to the same
>> request. +
> 
> Please see my related comments below for VIDIOC_QBUF.
> 
> What happens if which is set to V4L2_CTRL_WHICH_REQUEST_VAL and the request is 
> already queued ?

EBUSY will be returned.

> 
>> +If ``request_fd`` is specified and ``which`` is set to
>> ``V4L2_CTRL_WHICH_REQUEST_VAL`` +during a call to :ref:`VIDIOC_G_EXT_CTRLS
>> <VIDIOC_G_EXT_CTRLS>`, then the +returned values will be the values
>> currently set for the request (or the +hardware value if none is set)
> 
> Do we really have a need for returning the hardware value ? That seems a bit 
> random to me, wouldn't it be better to return an error instead ?

It gets very confusing if you have to remember which controls you set in the
request and which you didn't. This simply returns the value that will be set
when this request is applied. If none of the queued requests ever set this
control, then it will fall back to the hardware value.

> 
>> if the request has not yet completed, or the +values of the controls at the
>> time of request completion if it has already +completed.
>> +
>>  The driver will only set/get these controls if all control values are
>>  correct. This prevents the situation where only some of the controls
>>  were set/get. Only low-level errors (e. g. a failed i2c command) can
>> @@ -209,8 +222,10 @@ still cause this situation.
>>        - ``which``
>>        - Which value of the control to get/set/try.
>>  	``V4L2_CTRL_WHICH_CUR_VAL`` will return the current value of the
>> -	control and ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
>> -	value of the control.
>> +	control, ``V4L2_CTRL_WHICH_DEF_VAL`` will return the default
>> +	value of the control and ``V4L2_CTRL_WHICH_REQUEST_VAL`` indicates that
>> +	these controls have to be retrieved from a request or tried/set for
>> +	a request.
>>
>>  	.. note::
>>
>> @@ -272,8 +287,12 @@ still cause this situation.
>>  	then you can call :ref:`VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` to try
>> to discover the actual control that failed the validation step.
>> Unfortunately,
>>  	there is no ``TRY`` equivalent for :ref:`VIDIOC_G_EXT_CTRLS
>> <VIDIOC_G_EXT_CTRLS>`. +    * - __s32
>> +      - ``request_fd``
>> +	File descriptor of the request to be used by this operation. Only
>> +	valid if ``which`` is set to ``V4L2_CTRL_WHICH_REQUEST_VAL``.
>>      * - __u32
>> -      - ``reserved``\ [2]
>> +      - ``reserved``\ [1]
>>        - Reserved for future extensions.
>>
>>  	Drivers and applications must set the array to zero.
>> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> b/Documentation/media/uapi/v4l/vidioc-qbuf.rst index
>> 9e448a4aa3aa..c9f55d8340d1 100644
>> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
>> @@ -98,6 +98,13 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF
>> <VIDIOC_STREAMON>` or
>>  :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
>>
>>  device is closed.
>>
>> +The ``request_fd`` field can be used when queuing to specify the file
>> +descriptor of a request, if requests are in use. Setting it means that the
>> +buffer will not be passed to the driver until the request itself is queued.
>> +Also, the driver will apply any setting associated with the request before
>> +processing the buffer.
> 
> I think we should be more precise with the wording, here and above in the 
> documentation. From an API point of view, we need to guarantee that the 
> settings from the request are used to process the image capture in the buffer, 
> or the image output from the buffer. Drivers don't process buffers other than 
> queuing them to the hardware, so saying that the driver will apply settings 
> before processing the buffer is a bit vague in my opinion, and also possibly 
> not very accurate depending on the hardware.
> 
> Saying that controls are applied right before the buffer is processed by the 
> driver is inaccurate : for example exposure time might need several frames to 
> take effect, and might thus need to be applied a few frames before the buffer 
> is queued to the hardware for capture. The other way around is possible too : 
> the hardware might support a buffer queue but have no queue for controls. 
> Controls would then need to be applied a few frames after the buffer is queued 
> to the hardware in order to take effect for the right buffer.
> 
> We're specifying an API that carries very strong semantics across multiple 
> components and multiple objects, so I think it's very important to be as 
> precise as possible.

I've rephrased it to say:

"the driver will apply any settings associated with the request for this buffer."

It leaves the 'when' out of it, just that it has be applied 'for' this buffer.

I think your suggestion for this phrase makes a lot of sense.

> 
>> This field will be ignored unless the
>> +``V4L2_BUF_FLAG_REQUEST_FD`` flag is set.
>> +
>>  Applications call the ``VIDIOC_DQBUF`` ioctl to dequeue a filled
>>  (capturing) or displayed (output) buffer from the driver's outgoing
>>  queue. They just set the ``type``, ``memory`` and ``reserved`` fields of
>> diff --git a/Documentation/media/videodev2.h.rst.exceptions
>> b/Documentation/media/videodev2.h.rst.exceptions index
>> a5cb0a8686ac..5a5a1c772053 100644
>> --- a/Documentation/media/videodev2.h.rst.exceptions
>> +++ b/Documentation/media/videodev2.h.rst.exceptions
>> @@ -514,6 +514,7 @@ ignore define V4L2_CTRL_DRIVER_PRIV
>>  ignore define V4L2_CTRL_MAX_DIMS
>>  ignore define V4L2_CTRL_WHICH_CUR_VAL
>>  ignore define V4L2_CTRL_WHICH_DEF_VAL
>> +ignore define V4L2_CTRL_WHICH_REQUEST_VAL
>>  ignore define V4L2_OUT_CAP_CUSTOM_TIMINGS
>>  ignore define V4L2_CID_MAX_CTRLS
> 

Thanks for all the feedback!

Regards,

	Hans