[RFCv3 00/17] Request API, take three

[RFCv3 00/17] Request API, take three

[Alexandre Courbot]

As discussed yesterday, here is a rebase on the media master branch. A
few minor fixes for typos have also slept in, but otherwise this is
equivalent to v2. I expect to have the buffer queueing behavior fixed in
the next version.

Alexandre Courbot (9):
  media: add request API core and UAPI
  media: videobuf2: add support for requests
  media: vb2: add support for requests in QBUF ioctl
  v4l2: add request API support
  videodev2.h: add request_fd field to v4l2_ext_controls
  v4l2-ctrls: support requests in EXT_CTRLS ioctls
  v4l2: document the request API interface
  media: vim2m: add media device
  media: vim2m: add request support

Hans Verkuil (7):
  videodev2.h: Add request_fd field to v4l2_buffer
  v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev
  v4l2-ctrls: prepare internal structs for request API
  v4l2-ctrls: add core request API
  v4l2-ctrls: use ref in helper instead of ctrl
  v4l2-ctrls: support g/s_ext_ctrls for requests
  v4l2-ctrls: add v4l2_ctrl_request_setup

Laurent Pinchart (1):
  media: Document the media request API

 Documentation/media/uapi/mediactl/media-funcs.rst  |   1 +
 .../media/uapi/mediactl/media-ioc-request-cmd.rst  | 142 ++++++++++
 Documentation/media/uapi/v4l/buffer.rst            |  10 +-
 Documentation/media/uapi/v4l/common.rst            |   1 +
 Documentation/media/uapi/v4l/request-api.rst       | 236 ++++++++++++++++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst          |  16 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  21 ++
 drivers/media/Makefile                             |   3 +-
 drivers/media/common/videobuf2/videobuf2-core.c    | 133 ++++++++-
 drivers/media/common/videobuf2/videobuf2-v4l2.c    |  31 +-
 drivers/media/dvb-frontends/rtl2832_sdr.c          |   5 +-
 drivers/media/media-device.c                       |   7 +
 drivers/media/media-request-mgr.c                  | 105 +++++++
 drivers/media/media-request.c                      | 311 +++++++++++++++++++++
 drivers/media/pci/bt8xx/bttv-driver.c              |   2 +-
 drivers/media/pci/cx23885/cx23885-417.c            |   2 +-
 drivers/media/pci/cx88/cx88-blackbird.c            |   2 +-
 drivers/media/pci/cx88/cx88-video.c                |   2 +-
 drivers/media/pci/saa7134/saa7134-empress.c        |   4 +-
 drivers/media/pci/saa7134/saa7134-video.c          |   2 +-
 drivers/media/platform/exynos4-is/fimc-capture.c   |   2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c        |   3 +-
 drivers/media/platform/rcar_drif.c                 |   2 +-
 drivers/media/platform/soc_camera/soc_camera.c     |   3 +-
 drivers/media/platform/vim2m.c                     |  79 ++++++
 drivers/media/platform/vivid/vivid-ctrls.c         |  46 +--
 drivers/media/usb/cpia2/cpia2_v4l.c                |   2 +-
 drivers/media/usb/cx231xx/cx231xx-417.c            |   2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c          |   4 +-
 drivers/media/usb/msi2500/msi2500.c                |   2 +-
 drivers/media/usb/tm6000/tm6000-video.c            |   2 +-
 drivers/media/v4l2-core/Makefile                   |   2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c      |   9 +-
 drivers/media/v4l2-core/v4l2-ctrls.c               | 238 ++++++++++++++--
 drivers/media/v4l2-core/v4l2-device.c              |   3 +-
 drivers/media/v4l2-core/v4l2-ioctl.c               | 125 ++++++++-
 drivers/media/v4l2-core/v4l2-request.c             |  54 ++++
 drivers/staging/media/imx/imx-media-dev.c          |   2 +-
 drivers/staging/media/imx/imx-media-fim.c          |   2 +-
 include/media/media-device.h                       |   3 +
 include/media/media-entity.h                       |   9 +
 include/media/media-request-mgr.h                  |  73 +++++
 include/media/media-request.h                      | 186 ++++++++++++
 include/media/v4l2-ctrls.h                         |  17 +-
 include/media/v4l2-request.h                       |  34 +++
 include/media/videobuf2-core.h                     |  15 +-
 include/media/videobuf2-v4l2.h                     |   2 +
 include/uapi/linux/media.h                         |  10 +
 include/uapi/linux/videodev2.h                     |   6 +-
 49 files changed, 1886 insertions(+), 87 deletions(-)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst
 create mode 100644 drivers/media/media-request-mgr.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/media-request-mgr.h
 create mode 100644 include/media/media-request.h
 create mode 100644 include/media/v4l2-request.h

-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 01/17] media: add request API core and UAPI

[Alexandre Courbot]

The request API provides a way to group buffers and device parameters
into units of work to be queued and executed. This patch introduces the
UAPI and core framework.

This patch is based on the previous work by Laurent Pinchart. The core
has changed considerably, but the UAPI is mostly untouched.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/Makefile               |   3 +-
 drivers/media/media-device.c         |   7 +
 drivers/media/media-request-mgr.c    | 105 ++++++++++++
 drivers/media/media-request.c        | 311 +++++++++++++++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
 include/media/media-device.h         |   3 +
 include/media/media-entity.h         |   9 +
 include/media/media-request-mgr.h    |  73 ++++++++
 include/media/media-request.h        | 186 +++++++++++++++++++++
 include/uapi/linux/media.h           |  10 ++
 10 files changed, 707 insertions(+), 2 deletions(-)
 create mode 100644 drivers/media/media-request-mgr.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request-mgr.h
 create mode 100644 include/media/media-request.h

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..06c43ddb52ea 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -3,7 +3,8 @@
 # Makefile for the kernel multimedia device drivers.
 #
 
-media-objs	:= media-device.o media-devnode.o media-entity.o
+media-objs	:= media-device.o media-devnode.o media-entity.o \
+		   media-request.o media-request-mgr.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index e79f72b8b858..024ee81a8334 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,8 @@
 #include <media/media-device.h>
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
+#include <media/media-request.h>
+#include <media/media-request-mgr.h>
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -407,6 +409,7 @@ static const struct media_ioctl_info ioctl_info[] = {
 	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
+	MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
@@ -688,6 +691,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
 
 void media_device_cleanup(struct media_device *mdev)
 {
+	if (mdev->req_mgr) {
+		media_request_mgr_free(mdev->req_mgr);
+		mdev->req_mgr = NULL;
+	}
 	ida_destroy(&mdev->entity_internal_idx);
 	mdev->entity_internal_idx_max = 0;
 	media_graph_walk_cleanup(&mdev->pm_count_walk);
diff --git a/drivers/media/media-request-mgr.c b/drivers/media/media-request-mgr.c
new file mode 100644
index 000000000000..686e877a884b
--- /dev/null
+++ b/drivers/media/media-request-mgr.c
@@ -0,0 +1,105 @@
+/*
+ * Generic request manager implementation.
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/device.h>
+#include <linux/slab.h>
+
+#include <media/media-device.h>
+#include <media/media-request.h>
+#include <media/media-request-mgr.h>
+
+static struct media_request *
+media_request_alloc(struct media_request_mgr *mgr)
+{
+	struct media_request *req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		return ERR_PTR(-ENOMEM);
+
+	req->mgr = mgr;
+	req->state = MEDIA_REQUEST_STATE_IDLE;
+	kref_init(&req->kref);
+	INIT_LIST_HEAD(&req->data);
+	init_waitqueue_head(&req->complete_wait);
+	ATOMIC_INIT_NOTIFIER_HEAD(&req->submit_notif);
+	mutex_init(&req->lock);
+
+	mutex_lock(&mgr->mutex);
+	req->id = ++mgr->req_id;
+	list_add_tail(&req->list, &mgr->requests);
+	mutex_unlock(&mgr->mutex);
+
+	return req;
+}
+
+static void media_request_free(struct media_request *req)
+{
+	struct media_request_mgr *mgr = req->mgr;
+	struct media_request_entity_data *data, *next;
+
+	mutex_lock(&mgr->mutex);
+	list_del(&req->list);
+	mutex_unlock(&mgr->mutex);
+
+	list_for_each_entry_safe(data, next, &req->data, list) {
+		list_del(&data->list);
+		data->entity->req_ops->data_free(data);
+	}
+
+	kfree(req);
+}
+
+void media_request_mgr_free(struct media_request_mgr *mgr)
+{
+	struct media_device *mdev = mgr->mdev;
+
+	/* Just a sanity check - we should have no remaining requests */
+	while (!list_empty(&mgr->requests)) {
+		struct media_request *req;
+
+		req = list_first_entry(&mgr->requests, typeof(*req), list);
+		dev_warn(mdev->dev,
+			"%s: request %u still referenced, deleting forcibly!\n",
+			__func__, req->id);
+		mgr->ops->req_free(req);
+	}
+
+	kfree(mgr);
+}
+EXPORT_SYMBOL_GPL(media_request_mgr_free);
+
+static const struct media_request_mgr_ops request_mgr_generic_ops = {
+	.req_alloc = media_request_alloc,
+	.req_free = media_request_free,
+};
+
+struct media_request_mgr *
+media_request_mgr_alloc(struct media_device *mdev)
+{
+	struct media_request_mgr *mgr;
+
+	mgr = kzalloc(sizeof(*mgr), GFP_KERNEL);
+	if (!mgr)
+		return ERR_PTR(-ENOMEM);
+
+	mgr->mdev = mdev;
+	mutex_init(&mgr->mutex);
+	INIT_LIST_HEAD(&mgr->requests);
+	mgr->ops = &request_mgr_generic_ops;
+
+	return mgr;
+}
+EXPORT_SYMBOL_GPL(media_request_mgr_alloc);
diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
new file mode 100644
index 000000000000..30a23235b019
--- /dev/null
+++ b/drivers/media/media-request.c
@@ -0,0 +1,311 @@
+/*
+ * Request base implementation
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/anon_inodes.h>
+#include <linux/media.h>
+#include <linux/fs.h>
+#include <linux/file.h>
+#include <linux/slab.h>
+#include <linux/list.h>
+
+#include <media/media-device.h>
+#include <media/media-request.h>
+#include <media/media-request-mgr.h>
+
+const struct file_operations request_fops;
+
+struct media_request *media_request_get(struct media_request *req)
+{
+	kref_get(&req->kref);
+	return req;
+}
+EXPORT_SYMBOL_GPL(media_request_get);
+
+struct media_request *
+media_request_get_from_fd(int fd)
+{
+	struct file *f;
+	struct media_request *req;
+
+	f = fget(fd);
+	if (!f)
+		return NULL;
+
+	/* Not a request FD? */
+	if (f->f_op != &request_fops) {
+		fput(f);
+		return NULL;
+	}
+
+	req = f->private_data;
+	media_request_get(req);
+	fput(f);
+
+	return req;
+}
+EXPORT_SYMBOL_GPL(media_request_get_from_fd);
+
+static void media_request_release(struct kref *kref)
+{
+	struct media_request *req =
+		container_of(kref, typeof(*req), kref);
+
+	req->mgr->ops->req_free(req);
+}
+
+void media_request_put(struct media_request *req)
+{
+	if (WARN_ON(req == NULL))
+		return;
+
+	kref_put(&req->kref, media_request_release);
+}
+EXPORT_SYMBOL_GPL(media_request_put);
+
+struct media_request_entity_data *
+media_request_get_entity_data(struct media_request *req,
+			      struct media_entity *entity, void *fh)
+{
+	struct media_request_entity_data *data;
+
+	mutex_lock(&req->lock);
+
+	/* First look whether we already have entity data */
+	list_for_each_entry(data, &req->data, list) {
+		if (data->entity == entity) {
+			/*
+			 * If so, then the fh must match otherwise this means
+			 * we are trying to set the same entity through
+			 * different handles
+			 */
+			if (data->fh != fh)
+				data = ERR_PTR(-EINVAL);
+			goto done;
+		}
+	}
+
+	/* No entity data found, let's create it */
+	data = entity->req_ops->data_alloc(entity, fh);
+	if (IS_ERR(data))
+		goto done;
+	data->fh = fh;
+	data->entity = entity;
+	list_add_tail(&data->list, &req->data);
+
+done:
+	mutex_unlock(&req->lock);
+	return data;
+}
+EXPORT_SYMBOL_GPL(media_request_get_entity_data);
+
+static const char * const media_request_states[] __maybe_unused = {
+	"IDLE",
+	"SUBMITTED",
+	"COMPLETED",
+	"DELETED",
+};
+
+static const char *media_request_state(enum media_request_state state)
+{
+	return state < ARRAY_SIZE(media_request_states) ?
+		media_request_states[state] : "INVALID";
+}
+
+static int media_device_request_close(struct inode *inode, struct file *filp)
+{
+	struct media_request *req = filp->private_data;
+
+	if (req == NULL)
+		return 0;
+
+	media_request_put(req);
+
+	return 0;
+}
+
+static unsigned int media_request_poll(struct file *file, poll_table *wait)
+{
+	struct media_request *req = file->private_data;
+
+	poll_wait(file, &req->complete_wait, wait);
+
+	if (req->state == MEDIA_REQUEST_STATE_COMPLETED)
+		return POLLIN | POLLRDNORM;
+
+	return 0;
+}
+
+const struct file_operations request_fops = {
+	.owner = THIS_MODULE,
+	.poll = media_request_poll,
+	.release = media_device_request_close,
+};
+
+void media_request_complete(struct media_request *req)
+{
+	struct media_request_mgr *mgr = req->mgr;
+	struct media_device *mdev = mgr->mdev;
+
+	mutex_lock(&req->lock);
+
+	if (WARN_ON(req->state != MEDIA_REQUEST_STATE_SUBMITTED)) {
+		dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
+			__func__, req->id, media_request_state(req->state));
+		mutex_unlock(&req->lock);
+		return;
+	}
+
+	req->state = MEDIA_REQUEST_STATE_COMPLETED;
+
+	if (mgr->ops->req_complete)
+		mgr->ops->req_complete(req);
+
+	wake_up_interruptible(&req->complete_wait);
+
+	mutex_unlock(&req->lock);
+
+	/* Release the reference acquired when we submitted the request */
+	media_request_put(req);
+}
+EXPORT_SYMBOL_GPL(media_request_complete);
+
+/*
+ * Process the MEDIA_REQ_CMD_ALLOC command
+ */
+static int media_request_cmd_alloc(struct media_request_mgr *mgr,
+				   struct media_request_cmd *cmd)
+{
+	struct media_request *req;
+	int fd;
+
+	req = mgr->ops->req_alloc(mgr);
+	if (!req)
+		return -ENOMEM;
+
+	fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
+	if (fd < 0)
+		return fd;
+
+	cmd->fd = fd;
+
+	return 0;
+}
+
+/*
+ * Process the MEDIA_REQ_CMD_SUBMIT command
+ */
+static int media_request_cmd_submit(struct media_request *req)
+{
+	mutex_lock(&req->lock);
+
+	if (req->state != MEDIA_REQUEST_STATE_IDLE) {
+		dev_dbg(req->mgr->mdev->dev,
+			"%s: unable to submit request in state %s\n",
+			__func__, media_request_state(req->state));
+		mutex_unlock(&req->lock);
+		return -EINVAL;
+	}
+
+	if (atomic_read(&req->buf_cpt) == 0) {
+		dev_dbg(req->mgr->mdev->dev,
+			"%s: request has no buffers!\n", __func__);
+		mutex_unlock(&req->lock);
+		return -EINVAL;
+	}
+
+	req->state = MEDIA_REQUEST_STATE_SUBMITTED;
+
+	/* Hold a reference to the request until it is completed */
+	media_request_get(req);
+
+	mutex_unlock(&req->lock);
+
+	atomic_notifier_call_chain(&req->submit_notif, req->state, req);
+
+	return 0;
+}
+
+static int media_request_cmd_reinit(struct media_request *req)
+{
+	struct media_request_entity_data *data, *next;
+
+	mutex_lock(&req->lock);
+
+	if (req->state == MEDIA_REQUEST_STATE_SUBMITTED) {
+		dev_dbg(req->mgr->mdev->dev,
+			"%s: unable to reinit submitted request\n", __func__);
+		mutex_unlock(&req->lock);
+		return -EINVAL;
+	}
+
+	/* delete all entity data */
+	list_for_each_entry_safe(data, next, &req->data, list) {
+		list_del(&data->list);
+		data->entity->req_ops->data_free(data);
+	}
+
+	/* reinitialize request to idle state */
+	req->state = MEDIA_REQUEST_STATE_IDLE;
+	atomic_set(&req->buf_cpt, 0);
+
+	mutex_unlock(&req->lock);
+
+	return 0;
+}
+
+long media_device_request_cmd(struct media_device *mdev,
+			      struct media_request_cmd *cmd)
+{
+	struct media_request *req = NULL;
+	int ret;
+
+	if (!mdev->req_mgr)
+		return -ENOTTY;
+
+	if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
+		req = media_request_get_from_fd(cmd->fd);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+
+		/* requests must belong to this media device's manager */
+		if (req->mgr != mdev->req_mgr) {
+			media_request_put(req);
+			return -EINVAL;
+		}
+	}
+
+	switch (cmd->cmd) {
+	case MEDIA_REQ_CMD_ALLOC:
+		ret = media_request_cmd_alloc(mdev->req_mgr, cmd);
+		break;
+
+	case MEDIA_REQ_CMD_SUBMIT:
+		ret = media_request_cmd_submit(req);
+		break;
+
+	case MEDIA_REQ_CMD_REINIT:
+		ret = media_request_cmd_reinit(req);
+		break;
+
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+	if (req)
+		media_request_put(req);
+
+	return ret;
+}
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 260288ca4f55..e5109e5b8bf5 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -870,7 +870,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
 	__u32 i;
 
 	/* zero the reserved fields */
-	c->reserved[0] = c->reserved[1] = 0;
+	c->reserved[0] = 0;
 	for (i = 0; i < c->count; i++)
 		c->controls[i].reserved2[0] = 0;
 
diff --git a/include/media/media-device.h b/include/media/media-device.h
index bcc6ec434f1f..f2d471b8a53f 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -27,6 +27,7 @@
 
 struct ida;
 struct device;
+struct media_request_mgr;
 
 /**
  * struct media_entity_notify - Media Entity Notify
@@ -158,6 +159,8 @@ struct media_device {
 	void (*disable_source)(struct media_entity *entity);
 
 	const struct media_device_ops *ops;
+
+	struct media_request_mgr *req_mgr;
 };
 
 /* We don't need to include pci.h or usb.h here */
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a732af1dbba0..39f5e88e2b23 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -172,6 +172,8 @@ struct media_pad {
 	unsigned long flags;
 };
 
+struct media_request;
+struct v4l2_ext_controls;
 /**
  * struct media_entity_operations - Media entity operations
  * @get_fwnode_pad:	Return the pad number based on a fwnode endpoint or
@@ -197,6 +199,11 @@ struct media_entity_operations {
 	int (*link_validate)(struct media_link *link);
 };
 
+struct media_entity_request_ops {
+	struct media_request_entity_data *(*data_alloc)(struct media_entity *entity, void *fh);
+	void (*data_free)(struct media_request_entity_data *data);
+};
+
 /**
  * enum media_entity_type - Media entity type
  *
@@ -281,6 +288,8 @@ struct media_entity {
 
 	const struct media_entity_operations *ops;
 
+	const struct media_entity_request_ops *req_ops;
+
 	int stream_count;
 	int use_count;
 
diff --git a/include/media/media-request-mgr.h b/include/media/media-request-mgr.h
new file mode 100644
index 000000000000..a3161fa2add0
--- /dev/null
+++ b/include/media/media-request-mgr.h
@@ -0,0 +1,73 @@
+/*
+ * Generic request manager.
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_MGR_H
+#define _MEDIA_REQUEST_MGR_H
+
+#include <linux/mutex.h>
+
+struct media_request_mgr;
+
+/**
+ * struct media_request_mgr_ops - request manager operations
+ *
+ * @req_alloc:	allocate a request
+ * @req_free:	free a previously allocated request
+ * @req_complete: callback invoked when a request has completed
+ *
+ */
+struct media_request_mgr_ops {
+	struct media_request *(*req_alloc)(struct media_request_mgr *mgr);
+	void (*req_free)(struct media_request *req);
+	void (*req_complete)(struct media_request *req);
+};
+
+/**
+ * struct media_request_mgr - requests manager
+ *
+ * @mdev:	media_device owning this manager
+ * @ops:	implementation of the manager
+ * @mutex:	protects requests, active_request, req_id, and all members of
+ *		struct media_request
+ * @requests:	list of alive requests produced by this manager
+ * @req_id:	counter used to identify requests for debugging purposes
+ */
+struct media_request_mgr {
+	struct media_device *mdev;
+	const struct media_request_mgr_ops *ops;
+
+	struct mutex mutex;
+	struct list_head requests;
+	u32 req_id;
+};
+
+/**
+ * media_request_mgr_alloc() - return an instance of the generic manager
+ *
+ * @mdev:	owning media device
+ */
+struct media_request_mgr *
+media_request_mgr_alloc(struct media_device *mdev);
+
+/**
+ * media_request_mgr_free() - free a media manager
+ *
+ * This should only be called when all requests produced by this manager
+ * has been destroyed. Will warn if that is not the case.
+ *
+ */
+void media_request_mgr_free(struct media_request_mgr *mgr);
+
+#endif
diff --git a/include/media/media-request.h b/include/media/media-request.h
new file mode 100644
index 000000000000..817df13ef6e3
--- /dev/null
+++ b/include/media/media-request.h
@@ -0,0 +1,186 @@
+/*
+ * Media requests.
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_H
+#define _MEDIA_REQUEST_H
+
+#include <linux/kref.h>
+#include <linux/list.h>
+#include <linux/notifier.h>
+#include <linux/wait.h>
+
+struct media_device;
+struct media_entity;
+struct media_request_cmd;
+struct media_request_entity_data;
+struct media_request_mgr;
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+
+enum media_request_state {
+	MEDIA_REQUEST_STATE_IDLE,
+	MEDIA_REQUEST_STATE_SUBMITTED,
+	MEDIA_REQUEST_STATE_COMPLETED,
+	MEDIA_REQUEST_STATE_DELETED,
+};
+
+/**
+ * struct media_request - Media request base structure
+ * @id:		request id, used internally for debugging
+ * @mgr:	manager this request belongs to
+ * @kref:	reference count
+ * @list:	list entry in the media device requests list
+ * @lock:	protects internal state against concurrent accesses
+ * @state:	current state of the request
+ * @data:	per-entity data list
+ * @complete_wait:	wait queue that signals once the request has completed
+ * @submit_notif:	notification list to call when the request is submitted
+ * @buf_cpt:	counter of queued/completed buffers used to decide when the
+ *		request is completed
+ */
+struct media_request {
+	u32 id;
+	struct media_request_mgr *mgr;
+	struct kref kref;
+	struct list_head list;
+
+	struct mutex lock;
+	enum media_request_state state;
+	struct list_head data;
+	wait_queue_head_t complete_wait;
+	struct atomic_notifier_head submit_notif;
+	atomic_t buf_cpt;
+};
+
+/**
+ * media_request_get() - increment the reference counter of a request
+ *
+ * The calling context must call media_request_put() once it does not need
+ * the reference to the request anymore.
+ *
+ * Returns the request that has been passed as argument.
+ *
+ * @req:	request to acquire a reference of
+ */
+struct media_request *media_request_get(struct media_request *req);
+
+/**
+ * media_request_get_from_fd() - lookup request by fd and acquire a reference.
+ *
+ * Look a request up from its fd, acquire a reference and return a pointer to
+ * the request. As for media_request_get(), media_request_put() must be called
+ * once the reference is not used anymore.
+ *
+ * @req:	request to lookup and acquire.
+ *
+ */
+struct media_request *media_request_get_from_fd(int fd);
+
+/**
+ * media_request_put() - decrement the reference counter of a request
+ *
+ * Mirror function of media_request_get() and media_request_get_from_fd(). Will
+ * free the request if this was the last valid reference.
+ *
+ * @req:	request to release.
+ */
+void media_request_put(struct media_request *req);
+
+/**
+ * media_request_get_entity_data() - get per-entity data for a request
+ * @req:	request to get entity data from
+ * @entity:	entity to get data of
+ * @fh:		cookie identifying the handle from which the entity is accessed
+ *
+ * Search and return the entity data associated associated to the request. If no
+ * such data exists, it is allocated as per the entity operations.
+ *
+ * The fh arguments serves as a cookie to make sure the same entity is not
+ * accessed through different opened file handles. The same handle must be
+ * passed to all calls of this function for the same entity. Failure to do so
+ * will return an error.
+ *
+ * Returns the per-entity data, or an error code if a problem happened. -EINVAL
+ * means that data for the entity already existed, but has been allocated under
+ * a different cookie.
+ */
+struct media_request_entity_data *
+media_request_get_entity_data(struct media_request *req,
+			      struct media_entity *entity, void *fh);
+
+
+/**
+ * media_request_complete() - to be invoked when the request is complete
+ *
+ * @req:	request which has completed
+ */
+void media_request_complete(struct media_request *req);
+
+/**
+ * struct media_request_entity_data - per-entity request data
+ *
+ * Base structure used to store request state data. To be extended by actual
+ * implementation.
+ *
+ * @entity:	entity this data belongs to
+ * @fh:		subsystem-dependent. For V4L2, the v4l2_fh of the opened device
+ * @list:	entry in media_request::data
+ * @applied:	whether this request has already been applied for this entity
+ */
+struct media_request_entity_data {
+	struct media_entity *entity;
+	void *fh;
+	struct list_head list;
+	bool applied;
+};
+
+/**
+ * media_device_request_cmd() - perform the REQUEST_CMD ioctl
+ *
+ * @mdev:	media device the ioctl has been called on
+ * @cmd:	user-space request command structure
+ */
+long media_device_request_cmd(struct media_device *mdev,
+			      struct media_request_cmd *cmd);
+
+#else /* CONFIG_MEDIA_CONTROLLER */
+
+static inline media_request_get(struct media_request *req)
+{
+}
+
+static inline struct media_request *media_request_get_from_fd(int fd)
+{
+	return ERR_PTR(-ENOSYS);
+}
+
+static inline void media_request_put(struct media_request *req)
+{
+}
+
+static inline struct media_request *media_request_get_entity_data(
+					  struct media_request *req,
+					  struct media_entity *entity, void *fh)
+{
+	return ERR_PTR(-ENOSYS);
+}
+
+static inline void media_request_entity_complete(struct media_request *req)
+{
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
+#endif
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index b9b9446095e9..6e20ac9848b5 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -406,6 +406,15 @@ struct media_v2_topology {
 	__u64 ptr_links;
 } __attribute__ ((packed));
 
+#define MEDIA_REQ_CMD_ALLOC		0
+#define MEDIA_REQ_CMD_SUBMIT		1
+#define MEDIA_REQ_CMD_REINIT		2
+
+struct media_request_cmd {
+	__u32 cmd;
+	__s32 fd;
+} __attribute__ ((packed));
+
 /* ioctls */
 
 #define MEDIA_IOC_DEVICE_INFO		_IOWR('|', 0x00, struct media_device_info)
@@ -413,5 +422,6 @@ struct media_v2_topology {
 #define MEDIA_IOC_ENUM_LINKS		_IOWR('|', 0x02, struct media_links_enum)
 #define MEDIA_IOC_SETUP_LINK		_IOWR('|', 0x03, struct media_link_desc)
 #define MEDIA_IOC_G_TOPOLOGY		_IOWR('|', 0x04, struct media_v2_topology)
+#define MEDIA_IOC_REQUEST_CMD		_IOWR('|', 0x05, struct media_request_cmd)
 
 #endif /* __LINUX_MEDIA_H */
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 02/17] videodev2.h: Add request_fd field to v4l2_buffer

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

When queuing buffers allow for passing the request that should
be associated with this buffer.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
[acourbot@chromium.org: make request ID 32-bit]
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-v4l2.c | 3 ++-
 drivers/media/usb/cpia2/cpia2_v4l.c             | 2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c   | 9 ++++++---
 drivers/media/v4l2-core/v4l2-ioctl.c            | 4 ++--
 include/media/videobuf2-v4l2.h                  | 2 ++
 include/uapi/linux/videodev2.h                  | 3 ++-
 6 files changed, 15 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index fac3cd6f901d..0034f4d190f2 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
+	b->request_fd = vbuf->request_fd;
 	b->reserved = 0;
 
 	if (q->is_multiplanar) {
@@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
 	}
 	vb->timestamp = 0;
 	vbuf->sequence = 0;
+	vbuf->request_fd = b->request_fd;
 
 	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
 		if (b->memory == VB2_MEMORY_USERPTR) {
diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c b/drivers/media/usb/cpia2/cpia2_v4l.c
index a1c59f19cf2d..54c5aa0ecd26 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
 	buf->sequence = cam->buffers[buf->index].seq;
 	buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
 	buf->length = cam->frame_size;
-	buf->reserved2 = 0;
+	buf->request_fd = 0;
 	buf->reserved = 0;
 	memset(&buf->timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 5198c9eeb348..32bf47489a2e 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -386,7 +386,7 @@ struct v4l2_buffer32 {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__s32			request_fd;
 	__u32			reserved;
 };
 
@@ -486,6 +486,7 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
 {
 	u32 type;
 	u32 length;
+	s32 request_fd;
 	enum v4l2_memory memory;
 	struct v4l2_plane32 __user *uplane32;
 	struct v4l2_plane __user *uplane;
@@ -500,7 +501,9 @@ static int get_v4l2_buffer32(struct v4l2_buffer __user *kp,
 	    get_user(memory, &up->memory) ||
 	    put_user(memory, &kp->memory) ||
 	    get_user(length, &up->length) ||
-	    put_user(length, &kp->length))
+	    put_user(length, &kp->length) ||
+	    get_user(request_fd, &up->request_fd) ||
+	    put_user(request_fd, &kp->request_fd))
 		return -EFAULT;
 
 	if (V4L2_TYPE_IS_OUTPUT(type))
@@ -604,7 +607,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer __user *kp,
 	    assign_in_user(&up->timestamp.tv_usec, &kp->timestamp.tv_usec) ||
 	    copy_in_user(&up->timecode, &kp->timecode, sizeof(kp->timecode)) ||
 	    assign_in_user(&up->sequence, &kp->sequence) ||
-	    assign_in_user(&up->reserved2, &kp->reserved2) ||
+	    assign_in_user(&up->request_fd, &kp->request_fd) ||
 	    assign_in_user(&up->reserved, &kp->reserved) ||
 	    get_user(length, &kp->length) ||
 	    put_user(length, &up->length))
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index e5109e5b8bf5..2f40ac0cdf6e 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 	const struct v4l2_plane *plane;
 	int i;
 
-	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s",
+	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request_fd=%u, flags=0x%08x, field=%s, sequence=%d, memory=%s",
 			p->timestamp.tv_sec / 3600,
 			(int)(p->timestamp.tv_sec / 60) % 60,
 			(int)(p->timestamp.tv_sec % 60),
 			(long)p->timestamp.tv_usec,
 			p->index,
-			prt_names(p->type, v4l2_type_names),
+			prt_names(p->type, v4l2_type_names), p->request_fd,
 			p->flags, prt_names(p->field, v4l2_field_names),
 			p->sequence, prt_names(p->memory, v4l2_memory_names));
 
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 126cf559d4ce..a0eef2d42f0d 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -32,6 +32,7 @@
  *		&enum v4l2_field.
  * @timecode:	frame timecode.
  * @sequence:	sequence count of this frame.
+ * @request_fd:	fd of the request used by the buffer.
  *
  * Should contain enough information to be able to cover all the fields
  * of &struct v4l2_buffer at ``videodev2.h``.
@@ -43,6 +44,7 @@ struct vb2_v4l2_buffer {
 	__u32			field;
 	struct v4l2_timecode	timecode;
 	__u32			sequence;
+	__s32			request_fd;
 };
 
 /*
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 982718965180..4fd46ae8fad5 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -909,6 +909,7 @@ struct v4l2_plane {
  * @length:	size in bytes of the buffer (NOT its payload) for single-plane
  *		buffers (when type != *_MPLANE); number of elements in the
  *		planes array for multi-plane buffers
+ * @request_fd: fd of the request that this buffer should use
  *
  * Contains data exchanged by application and driver using one of the Streaming
  * I/O methods.
@@ -932,7 +933,7 @@ struct v4l2_buffer {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__s32			request_fd;
 	__u32			reserved;
 };
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 03/17] media: videobuf2: add support for requests

[Alexandre Courbot]

Make vb2 aware of requests. Drivers can specify whether a given queue
can accept requests or not. Queues that accept requests will block on a
buffer that is part of a request until that request is submitted.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/common/videobuf2/videobuf2-core.c | 133 ++++++++++++++++++++++--
 drivers/media/common/videobuf2/videobuf2-v4l2.c |  28 ++++-
 include/media/videobuf2-core.h                  |  15 ++-
 3 files changed, 168 insertions(+), 8 deletions(-)

diff --git a/drivers/media/common/videobuf2/videobuf2-core.c b/drivers/media/common/videobuf2/videobuf2-core.c
index f7109f827f6e..c1b9ccbdecb3 100644
--- a/drivers/media/common/videobuf2/videobuf2-core.c
+++ b/drivers/media/common/videobuf2/videobuf2-core.c
@@ -28,6 +28,7 @@
 
 #include <media/videobuf2-core.h>
 #include <media/v4l2-mc.h>
+#include <media/media-request.h>
 
 #include <trace/events/vb2.h>
 
@@ -930,6 +931,17 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		vb->state = state;
 	}
 	atomic_dec(&q->owned_by_drv_count);
+	if (vb->request) {
+		struct media_request *req = vb->request;
+
+		if (atomic_dec_and_test(&req->buf_cpt))
+			media_request_complete(vb->request);
+
+		/* release reference acquired during qbuf */
+		vb->request = NULL;
+		media_request_put(req);
+	}
+
 	spin_unlock_irqrestore(&q->done_lock, flags);
 
 	trace_vb2_buf_done(q, vb);
@@ -1306,6 +1318,53 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+/*
+ * vb2_check_buf_req_status() - Validate request state of a buffer
+ * @vb:		buffer to check
+ *
+ * Returns true if a buffer is ready to be passed to the driver request-wise.
+ * This means that neither this buffer nor any previously-queued buffer is
+ * associated to a request that is not yet submitted.
+ *
+ * If this function returns false, then the buffer shall not be passed to its
+ * driver since the request state is not completely built yet. In that case,
+ * this function will register a notifier to be called when the request is
+ * submitted and the queue can be unblocked.
+ *
+ * This function must be called with req_lock held.
+ */
+static bool vb2_check_buf_req_status(struct vb2_buffer *vb)
+{
+	struct media_request *req = vb->request;
+	struct vb2_queue *q = vb->vb2_queue;
+	int ret = false;
+
+	mutex_lock(&q->req_lock);
+
+	if (!req) {
+		ret = !q->waiting_req;
+		goto done;
+	}
+
+	mutex_lock(&req->lock);
+	if (req->state == MEDIA_REQUEST_STATE_SUBMITTED) {
+		mutex_unlock(&req->lock);
+		ret = !q->waiting_req;
+		goto done;
+	}
+
+	if (!q->waiting_req) {
+		q->waiting_req = true;
+		atomic_notifier_chain_register(&req->submit_notif,
+					       &q->req_blk);
+	}
+	mutex_unlock(&req->lock);
+
+done:
+	mutex_unlock(&q->req_lock);
+	return ret;
+}
+
 /*
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1326,8 +1385,11 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	 * If any buffers were queued before streamon,
 	 * we can now pass them to driver for processing.
 	 */
-	list_for_each_entry(vb, &q->queued_list, queued_entry)
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (!vb2_check_buf_req_status(vb))
+			break;
 		__enqueue_in_driver(vb);
+	}
 
 	/* Tell the driver to start streaming */
 	q->start_streaming_called = 1;
@@ -1369,7 +1431,46 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+/*
+ * vb2_unblock_requests() - unblock a queue waiting for a request submission
+ * @nb:		notifier block that has been registered
+ * @action:	unused
+ * @data:	request that has been submitted
+ *
+ * This is a callback function that is registered when
+ * vb2_check_buf_req_status() returns false. It is invoked when the request
+ * blocking the queue has been submitted. This means its buffers (and all
+ * following valid buffers) can be passed to drivers.
+ */
+static int vb2_unblock_requests(struct notifier_block *nb, unsigned long action,
+				void *data)
+{
+	struct vb2_queue *q = container_of(nb, struct vb2_queue, req_blk);
+	struct media_request *req = data;
+	struct vb2_buffer *vb;
+	bool found_request = false;
+
+	mutex_lock(&q->req_lock);
+	atomic_notifier_chain_unregister(&req->submit_notif, &q->req_blk);
+	q->waiting_req = false;
+	mutex_unlock(&q->req_lock);
+
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		/* All buffers before our request are already passed to the driver */
+		if (!found_request && vb->request != req)
+			continue;
+		found_request = true;
+
+		if (!vb2_check_buf_req_status(vb))
+			break;
+		__enqueue_in_driver(vb);
+	}
+
+	return 0;
+}
+
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
+		  struct media_request *req, void *pb)
 {
 	struct vb2_buffer *vb;
 	int ret;
@@ -1392,6 +1493,24 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 		return -EINVAL;
 	}
 
+	vb->request = req;
+	if (req) {
+		struct vb2_buffer *_vb;
+
+		/* does the queue support requests? */
+		if (!q->allow_requests)
+			return -EINVAL;
+
+		/* do we already have a buffer for this request in the queue? */
+		list_for_each_entry(_vb, &q->queued_list, queued_entry)
+			if (_vb->request == req)
+				return -EBUSY;
+
+		/* make sure the request stays alive as long as we need */
+		media_request_get(req);
+		atomic_inc(&req->buf_cpt);
+	}
+
 	/*
 	 * Add to the queued buffers list, a buffer will stay on it until
 	 * dequeued in dqbuf.
@@ -1410,7 +1529,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	 * If already streaming, give the buffer to driver for processing.
 	 * If not, the buffer will be given to driver on next streamon.
 	 */
-	if (q->start_streaming_called)
+	if (q->start_streaming_called && vb2_check_buf_req_status(vb))
 		__enqueue_in_driver(vb);
 
 	/* Fill buffer information for the userspace */
@@ -2004,6 +2123,8 @@ int vb2_core_queue_init(struct vb2_queue *q)
 	spin_lock_init(&q->done_lock);
 	mutex_init(&q->mmap_lock);
 	init_waitqueue_head(&q->done_wq);
+	mutex_init(&q->req_lock);
+	q->req_blk.notifier_call = vb2_unblock_requests;
 
 	q->memory = VB2_MEMORY_UNKNOWN;
 
@@ -2255,7 +2376,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2434,7 +2555,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2537,7 +2658,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/common/videobuf2/videobuf2-v4l2.c b/drivers/media/common/videobuf2/videobuf2-v4l2.c
index 0034f4d190f2..69286e7f9dd0 100644
--- a/drivers/media/common/videobuf2/videobuf2-v4l2.c
+++ b/drivers/media/common/videobuf2/videobuf2-v4l2.c
@@ -30,6 +30,7 @@
 #include <media/v4l2-common.h>
 
 #include <media/videobuf2-v4l2.h>
+#include <media/media-request.h>
 
 static int debug;
 module_param(debug, int, 0644);
@@ -563,6 +564,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	struct media_request *req = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -570,8 +572,32 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		return -EBUSY;
 	}
 
+	/*
+	 * The caller should have validated that the request is valid,
+	 * so we just need to look it up without further checking
+	 */
+	if (b->request_fd > 0) {
+		req = media_request_get_from_fd(b->request_fd);
+		if (!req)
+			return -EINVAL;
+
+		mutex_lock(&req->lock);
+		if (req->state != MEDIA_REQUEST_STATE_IDLE) {
+			mutex_unlock(&req->lock);
+			media_request_put(req);
+			return -EINVAL;
+		}
+		mutex_unlock(&req->lock);
+	}
+
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	if (!ret)
+		ret = vb2_core_qbuf(q, b->index, req, b);
+
+	if (req)
+		media_request_put(req);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index f3ee4c7c2fb3..ddd793e73022 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -238,6 +238,7 @@ struct vb2_queue;
  * @num_planes:		number of planes in the buffer
  *			on an internal driver queue.
  * @timestamp:		frame timestamp in ns.
+ * @request:		request the buffer belongs to, if any.
  */
 struct vb2_buffer {
 	struct vb2_queue	*vb2_queue;
@@ -246,6 +247,7 @@ struct vb2_buffer {
 	unsigned int		memory;
 	unsigned int		num_planes;
 	u64			timestamp;
+	struct media_request	*request;
 
 	/* private: internal use only
 	 *
@@ -446,6 +448,7 @@ struct vb2_buf_ops {
  * @quirk_poll_must_check_waiting_for_buffers: Return %POLLERR at poll when QBUF
  *              has not been called. This is a vb1 idiom that has been adopted
  *              also by vb2.
+ * @allow_requests:	whether requests are supported on this queue.
  * @lock:	pointer to a mutex that protects the &struct vb2_queue. The
  *		driver can set this to a mutex to let the v4l2 core serialize
  *		the queuing ioctls. If the driver wants to handle locking
@@ -502,6 +505,9 @@ struct vb2_buf_ops {
  *		when a buffer with the %V4L2_BUF_FLAG_LAST is dequeued.
  * @fileio:	file io emulator internal data, used only if emulator is active
  * @threadio:	thread io internal data, used only if thread is active
+ * @req_lock:	protects req_blk and waiting_req
+ * @req_blk:	notifier to be called when waiting for a request to be submitted
+ * @waiting_req:whether this queue is currently waiting on a request submission
  */
 struct vb2_queue {
 	unsigned int			type;
@@ -513,6 +519,7 @@ struct vb2_queue {
 	unsigned			fileio_write_immediately:1;
 	unsigned			allow_zero_bytesused:1;
 	unsigned		   quirk_poll_must_check_waiting_for_buffers:1;
+	unsigned			allow_requests:1;
 
 	struct mutex			*lock;
 	void				*owner;
@@ -556,6 +563,10 @@ struct vb2_queue {
 	struct vb2_fileio_data		*fileio;
 	struct vb2_threadio_data	*threadio;
 
+	struct mutex			req_lock;
+	struct notifier_block		req_blk;
+	bool				waiting_req;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these queue-related ops are
@@ -743,6 +754,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * @q:		pointer to &struct vb2_queue with videobuf2 queue.
  * @index:	id number of the buffer
+ * @req:	request this buffer belongs to, if any
  * @pb:		buffer structure passed from userspace to
  *		v4l2_ioctl_ops->vidioc_qbuf handler in driver
  *
@@ -759,7 +771,8 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  *
  * Return: returns zero on success; an error code otherwise.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
+		  struct media_request *req, void *pb);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 04/17] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

Support the request argument of the QBUF ioctl.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 83 +++++++++++++++++++++++++++++++++++-
 1 file changed, 82 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2f40ac0cdf6e..235acdde3111 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -29,6 +29,7 @@
 #include <media/v4l2-device.h>
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mc.h>
+#include <media/media-request.h>
 
 #include <trace/events/v4l2.h>
 
@@ -968,6 +969,71 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 	return -EINVAL;
 }
 
+/*
+ * Validate that a given request can be used during an ioctl.
+ *
+ * When using the request API, request file descriptors must be matched against
+ * the actual request object. User-space can pass any file descriptor, so we
+ * need to make sure the call is valid before going further.
+ *
+ * This function looks up the request and associated data and performs the
+ * following sanity checks:
+ *
+ * * Make sure that the entity belongs to the media_device managing the passed
+ *   request,
+ * * Make sure that the entity data (if any) is associated to the current file
+ *   handler.
+ *
+ * This function returns a pointer to the valid request, or and error code in
+ * case of failure. When successful, a reference to the request is acquired and
+ * must be properly released by the caller. If _data is not NULL, then it is
+ * filled with a pointer to the corresponding entity data.
+ */
+#ifdef CONFIG_MEDIA_CONTROLLER
+static struct media_request *
+check_request(int request, struct file *file, void *fh, struct media_request_entity_data **_data)
+{
+	struct media_request *req = NULL;
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
+	struct media_entity *entity = &vfd->entity;
+	struct media_request_entity_data *data;
+
+	if (!entity)
+		return ERR_PTR(-EINVAL);
+
+	req = media_request_get_from_fd(request);
+	if (!req)
+		return ERR_PTR(-EINVAL);
+
+	/* Validate that the entity belongs to the correct media_device */
+	if (vfd->v4l2_dev->mdev->req_mgr != req->mgr) {
+		media_request_put(req);
+		return ERR_PTR(-EINVAL);
+	}
+
+	/* Validate that the entity's data belongs to the correct fh */
+	data = media_request_get_entity_data(req, entity, vfh);
+	if (IS_ERR(data)) {
+		media_request_put(req);
+		return ERR_PTR(PTR_ERR(data));
+	}
+
+	if (_data)
+		*_data = data;
+
+	return req;
+}
+#else /* CONFIG_MEDIA_CONTROLLER */
+static struct media_request *
+check_request(int request, struct file *file, void *fh)
+{
+	return ERR_PTR(-ENOSYS);
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 static void v4l_sanitize_format(struct v4l2_format *fmt)
 {
 	unsigned int offset;
@@ -1878,10 +1944,25 @@ static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
+	struct media_request *req = NULL;
 	struct v4l2_buffer *p = arg;
 	int ret = check_fmt(file, p->type);
 
-	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+	if (ret)
+		return ret;
+
+	if (p->request_fd > 0) {
+		req = check_request(p->request_fd, file, fh, NULL);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+	}
+
+	ret = ops->vidioc_qbuf(file, fh, p);
+
+	if (req)
+		media_request_put(req);
+
+	return ret;
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 05/17] media: Document the media request API

[Alexandre Courbot]

From: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

The media request API is made of a new ioctl to implement request
management. Document it.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
[acourbot@chromium.org: adapt for newest API]
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 Documentation/media/uapi/mediactl/media-funcs.rst  |   1 +
 .../media/uapi/mediactl/media-ioc-request-cmd.rst  | 142 +++++++++++++++++++++
 2 files changed, 143 insertions(+)
 create mode 100644 Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst

diff --git a/Documentation/media/uapi/mediactl/media-funcs.rst b/Documentation/media/uapi/mediactl/media-funcs.rst
index 076856501cdb..e3a45d82ffcb 100644
--- a/Documentation/media/uapi/mediactl/media-funcs.rst
+++ b/Documentation/media/uapi/mediactl/media-funcs.rst
@@ -15,4 +15,5 @@ Function Reference
     media-ioc-g-topology
     media-ioc-enum-entities
     media-ioc-enum-links
+    media-ioc-request-cmd
     media-ioc-setup-link
diff --git a/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
new file mode 100644
index 000000000000..ced76ff3498d
--- /dev/null
+++ b/Documentation/media/uapi/mediactl/media-ioc-request-cmd.rst
@@ -0,0 +1,142 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media_ioc_request_cmd:
+
+***************************
+ioctl MEDIA_IOC_REQUEST_CMD
+***************************
+
+Name
+====
+
+MEDIA_IOC_REQUEST_CMD - Manage media device requests
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, MEDIA_IOC_REQUEST_CMD, struct media_request_cmd *argp )
+    :name: MEDIA_IOC_REQUEST_CMD
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <media-func-open>`.
+
+``argp``
+
+
+Description
+===========
+
+The MEDIA_IOC_REQUEST_CMD ioctl allows applications to manage media device
+requests. A request is an object that can group media device configuration
+parameters, including subsystem-specific parameters, in order to apply all the
+parameters atomically. Applications are responsible for allocating and
+deleting requests, filling them with configuration parameters and submitting
+them.
+
+Request operations are performed by calling the MEDIA_IOC_REQUEST_CMD ioctl
+with a pointer to a struct :c:type:`media_request_cmd` with the cmd field set
+to the appropriate command. :ref:`media-request-command` lists the commands
+supported by the ioctl.
+
+The struct :c:type:`media_request_cmd` request field contains the file
+descriptor of the request on which the command operates. For the
+``MEDIA_REQ_CMD_ALLOC`` command the field is set to zero by applications and
+filled by the driver. For all other commands the field is set by applications
+and left untouched by the driver.
+
+To allocate a new request applications use the ``MEDIA_REQ_CMD_ALLOC``
+command. The driver will allocate a new request and return its FD in the
+request field. After allocation, the request is "empty", which means that it
+does not hold any state of its own, and that the hardware's state will not be
+affected by it unless it is passed as argument to V4L2 or media controller
+commands.
+
+Requests are reference-counted. A newly allocated request is referenced
+by the returned file descriptor, and can be later referenced by
+subsystem-specific operations. Requests will thus be automatically deleted
+when they're no longer used after the returned file descriptor is closed.
+
+If a request isn't needed applications can delete it by calling ``close()``
+on it. The driver will drop the file handle reference. The request will not
+be usable through the MEDIA_IOC_REQUEST_CMD ioctl anymore, but will only be
+deleted when the last reference is released. If no other reference exists when
+``close()`` is invoked the request will be deleted immediately.
+
+After creating a request applications should fill it with configuration
+parameters. This is performed through subsystem-specific request APIs outside
+the scope of the media controller API. See the appropriate subsystem APIs for
+more information, including how they interact with the MEDIA_IOC_REQUEST_CMD
+ioctl.
+
+Once a request contains all the desired configuration parameters it can be
+submitted using the ``MEDIA_REQ_CMD_SUBMIT`` command. This will let the
+buffers queued for the request be passed to their respective drivers, which
+will then apply the request's parameters before processing them.
+
+Once a request has been queued applications are not allowed to modify its
+configuration parameters until the request has been fully processed. Any
+attempt to do so will result in the related subsystem API returning an error.
+The application that submitted the request can wait for its completion by
+polling on the request's file descriptor.
+
+Once a request has completed, it can be reused. The ``MEDIA_REQ_CMD_REINIT``
+command will bring it back to its initial state, so it can be prepared and
+submitted again.
+
+.. c:type:: media_request_cmd
+
+.. flat-table:: struct media_request_cmd
+    :header-rows:  0
+    :stub-columns: 0
+    :widths: 1 2 8
+
+    * - __u32
+      - ``cmd``
+      - Command, set by the application. See below for the list of supported
+        commands.
+    * - __u32
+      - ``fd``
+      - Request FD, set by the driver for the MEDIA_REQ_CMD_ALLOC command and
+        by the application for all other commands.
+
+
+.. _media-request-command:
+
+.. cssclass:: longtable
+
+.. flat-table:: Media request commands
+    :header-rows:  0
+    :stub-columns: 0
+
+    * .. _MEDIA-REQ-CMD-ALLOC:
+
+      - ``MEDIA_REQ_CMD_ALLOC``
+      - Allocate a new request.
+    * .. _MEDIA-REQ-CMD-SUBMIT:
+
+      - ``MEDIA_REQ_CMD_SUBMIT``
+      - Submit a request to be processed.
+    * .. _MEDIA-REQ-CMD-QUEUE:
+
+      - ``MEDIA_REQ_CMD_REINIT``
+      - Reinitialize a completed request.
+
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+EINVAL
+    The struct :c:type:`media_request_cmd` specifies an invalid command or
+    references a non-existing request.
+
+ENOSYS
+    Request API is not available on this device.
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 06/17] v4l2-ctrls: v4l2_ctrl_add_handler: add from_other_dev

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add a 'bool from_other_dev' argument: set to true if the two
handlers refer to different devices (e.g. it is true when
inheriting controls from a subdev into a main v4l2 bridge
driver).

This will be used later when implementing support for the
request API since we need to skip such controls.

TODO: check drivers/staging/media/imx/imx-media-fim.c change.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/dvb-frontends/rtl2832_sdr.c        |  5 +--
 drivers/media/pci/bt8xx/bttv-driver.c            |  2 +-
 drivers/media/pci/cx23885/cx23885-417.c          |  2 +-
 drivers/media/pci/cx88/cx88-blackbird.c          |  2 +-
 drivers/media/pci/cx88/cx88-video.c              |  2 +-
 drivers/media/pci/saa7134/saa7134-empress.c      |  4 +--
 drivers/media/pci/saa7134/saa7134-video.c        |  2 +-
 drivers/media/platform/exynos4-is/fimc-capture.c |  2 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c      |  3 +-
 drivers/media/platform/rcar_drif.c               |  2 +-
 drivers/media/platform/soc_camera/soc_camera.c   |  3 +-
 drivers/media/platform/vivid/vivid-ctrls.c       | 46 ++++++++++++------------
 drivers/media/usb/cx231xx/cx231xx-417.c          |  2 +-
 drivers/media/usb/cx231xx/cx231xx-video.c        |  4 +--
 drivers/media/usb/msi2500/msi2500.c              |  2 +-
 drivers/media/usb/tm6000/tm6000-video.c          |  2 +-
 drivers/media/v4l2-core/v4l2-ctrls.c             | 11 +++---
 drivers/media/v4l2-core/v4l2-device.c            |  3 +-
 drivers/staging/media/imx/imx-media-dev.c        |  2 +-
 drivers/staging/media/imx/imx-media-fim.c        |  2 +-
 include/media/v4l2-ctrls.h                       |  4 ++-
 21 files changed, 58 insertions(+), 49 deletions(-)

diff --git a/drivers/media/dvb-frontends/rtl2832_sdr.c b/drivers/media/dvb-frontends/rtl2832_sdr.c
index c6e78d870ccd..6064d28224e8 100644
--- a/drivers/media/dvb-frontends/rtl2832_sdr.c
+++ b/drivers/media/dvb-frontends/rtl2832_sdr.c
@@ -1394,7 +1394,8 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
 	case RTL2832_SDR_TUNER_E4000:
 		v4l2_ctrl_handler_init(&dev->hdl, 9);
 		if (subdev)
-			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler, NULL);
+			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler,
+					      NULL, true);
 		break;
 	case RTL2832_SDR_TUNER_R820T:
 	case RTL2832_SDR_TUNER_R828D:
@@ -1423,7 +1424,7 @@ static int rtl2832_sdr_probe(struct platform_device *pdev)
 		v4l2_ctrl_handler_init(&dev->hdl, 2);
 		if (subdev)
 			v4l2_ctrl_add_handler(&dev->hdl, subdev->ctrl_handler,
-					      NULL);
+					      NULL, true);
 		break;
 	default:
 		v4l2_ctrl_handler_init(&dev->hdl, 0);
diff --git a/drivers/media/pci/bt8xx/bttv-driver.c b/drivers/media/pci/bt8xx/bttv-driver.c
index b366a7e1d976..91874f775d37 100644
--- a/drivers/media/pci/bt8xx/bttv-driver.c
+++ b/drivers/media/pci/bt8xx/bttv-driver.c
@@ -4211,7 +4211,7 @@ static int bttv_probe(struct pci_dev *dev, const struct pci_device_id *pci_id)
 	/* register video4linux + input */
 	if (!bttv_tvcards[btv->c.type].no_video) {
 		v4l2_ctrl_add_handler(&btv->radio_ctrl_handler, hdl,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, false);
 		if (btv->radio_ctrl_handler.error) {
 			result = btv->radio_ctrl_handler.error;
 			goto fail2;
diff --git a/drivers/media/pci/cx23885/cx23885-417.c b/drivers/media/pci/cx23885/cx23885-417.c
index a71f3c7569ce..762823871c78 100644
--- a/drivers/media/pci/cx23885/cx23885-417.c
+++ b/drivers/media/pci/cx23885/cx23885-417.c
@@ -1527,7 +1527,7 @@ int cx23885_417_register(struct cx23885_dev *dev)
 	dev->cxhdl.priv = dev;
 	dev->cxhdl.func = cx23885_api_func;
 	cx2341x_handler_set_50hz(&dev->cxhdl, tsport->height == 576);
-	v4l2_ctrl_add_handler(&dev->ctrl_handler, &dev->cxhdl.hdl, NULL);
+	v4l2_ctrl_add_handler(&dev->ctrl_handler, &dev->cxhdl.hdl, NULL, false);
 
 	/* Allocate and initialize V4L video device */
 	dev->v4l_device = cx23885_video_dev_alloc(tsport,
diff --git a/drivers/media/pci/cx88/cx88-blackbird.c b/drivers/media/pci/cx88/cx88-blackbird.c
index 0e0952e60795..39f69d89a663 100644
--- a/drivers/media/pci/cx88/cx88-blackbird.c
+++ b/drivers/media/pci/cx88/cx88-blackbird.c
@@ -1183,7 +1183,7 @@ static int cx8802_blackbird_probe(struct cx8802_driver *drv)
 	err = cx2341x_handler_init(&dev->cxhdl, 36);
 	if (err)
 		goto fail_core;
-	v4l2_ctrl_add_handler(&dev->cxhdl.hdl, &core->video_hdl, NULL);
+	v4l2_ctrl_add_handler(&dev->cxhdl.hdl, &core->video_hdl, NULL, false);
 
 	/* blackbird stuff */
 	pr_info("cx23416 based mpeg encoder (blackbird reference design)\n");
diff --git a/drivers/media/pci/cx88/cx88-video.c b/drivers/media/pci/cx88/cx88-video.c
index 9be682cdb644..e35bfa03a1e2 100644
--- a/drivers/media/pci/cx88/cx88-video.c
+++ b/drivers/media/pci/cx88/cx88-video.c
@@ -1378,7 +1378,7 @@ static int cx8800_initdev(struct pci_dev *pci_dev,
 		if (vc->id == V4L2_CID_CHROMA_AGC)
 			core->chroma_agc = vc;
 	}
-	v4l2_ctrl_add_handler(&core->video_hdl, &core->audio_hdl, NULL);
+	v4l2_ctrl_add_handler(&core->video_hdl, &core->audio_hdl, NULL, false);
 
 	/* load and configure helper modules */
 
diff --git a/drivers/media/pci/saa7134/saa7134-empress.c b/drivers/media/pci/saa7134/saa7134-empress.c
index 66acfd35ffc6..fc75ce00dbf8 100644
--- a/drivers/media/pci/saa7134/saa7134-empress.c
+++ b/drivers/media/pci/saa7134/saa7134-empress.c
@@ -265,9 +265,9 @@ static int empress_init(struct saa7134_dev *dev)
 		 "%s empress (%s)", dev->name,
 		 saa7134_boards[dev->board].name);
 	v4l2_ctrl_handler_init(hdl, 21);
-	v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler, empress_ctrl_filter);
+	v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler, empress_ctrl_filter, false);
 	if (dev->empress_sd)
-		v4l2_ctrl_add_handler(hdl, dev->empress_sd->ctrl_handler, NULL);
+		v4l2_ctrl_add_handler(hdl, dev->empress_sd->ctrl_handler, NULL, true);
 	if (hdl->error) {
 		video_device_release(dev->empress_dev);
 		return hdl->error;
diff --git a/drivers/media/pci/saa7134/saa7134-video.c b/drivers/media/pci/saa7134/saa7134-video.c
index 052e101d898c..22c27d95c1bf 100644
--- a/drivers/media/pci/saa7134/saa7134-video.c
+++ b/drivers/media/pci/saa7134/saa7134-video.c
@@ -2136,7 +2136,7 @@ int saa7134_video_init1(struct saa7134_dev *dev)
 		hdl = &dev->radio_ctrl_handler;
 		v4l2_ctrl_handler_init(hdl, 2);
 		v4l2_ctrl_add_handler(hdl, &dev->ctrl_handler,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, false);
 		if (hdl->error)
 			return hdl->error;
 	}
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index ed9302caa004..64f3ab3584d7 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -1421,7 +1421,7 @@ static int fimc_link_setup(struct media_entity *entity,
 		return 0;
 
 	return v4l2_ctrl_add_handler(&vc->ctx->ctrls.handler,
-				     sensor->ctrl_handler, NULL);
+				     sensor->ctrl_handler, NULL, true);
 }
 
 static const struct media_entity_operations fimc_sd_media_ops = {
diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index b479b882da12..90246113fa03 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -900,7 +900,8 @@ int rvin_v4l2_probe(struct rvin_dev *vin)
 	if (ret < 0)
 		return ret;
 
-	ret = v4l2_ctrl_add_handler(&vin->ctrl_handler, sd->ctrl_handler, NULL);
+	ret = v4l2_ctrl_add_handler(&vin->ctrl_handler, sd->ctrl_handler,
+				    NULL, true);
 	if (ret < 0)
 		return ret;
 
diff --git a/drivers/media/platform/rcar_drif.c b/drivers/media/platform/rcar_drif.c
index b2e080ef5391..70cef10c81a6 100644
--- a/drivers/media/platform/rcar_drif.c
+++ b/drivers/media/platform/rcar_drif.c
@@ -1167,7 +1167,7 @@ static int rcar_drif_notify_complete(struct v4l2_async_notifier *notifier)
 	}
 
 	ret = v4l2_ctrl_add_handler(&sdr->ctrl_hdl,
-				    sdr->ep.subdev->ctrl_handler, NULL);
+				    sdr->ep.subdev->ctrl_handler, NULL, true);
 	if (ret) {
 		rdrif_err(sdr, "failed: ctrl add hdlr ret %d\n", ret);
 		goto error;
diff --git a/drivers/media/platform/soc_camera/soc_camera.c b/drivers/media/platform/soc_camera/soc_camera.c
index d13e2c5fb06f..aeff51afc418 100644
--- a/drivers/media/platform/soc_camera/soc_camera.c
+++ b/drivers/media/platform/soc_camera/soc_camera.c
@@ -1180,7 +1180,8 @@ static int soc_camera_probe_finish(struct soc_camera_device *icd)
 
 	v4l2_subdev_call(sd, video, g_tvnorms, &icd->vdev->tvnorms);
 
-	ret = v4l2_ctrl_add_handler(&icd->ctrl_handler, sd->ctrl_handler, NULL);
+	ret = v4l2_ctrl_add_handler(&icd->ctrl_handler, sd->ctrl_handler,
+				    NULL, true);
 	if (ret < 0)
 		return ret;
 
diff --git a/drivers/media/platform/vivid/vivid-ctrls.c b/drivers/media/platform/vivid/vivid-ctrls.c
index 3f9d354827af..6a72b2cf284f 100644
--- a/drivers/media/platform/vivid/vivid-ctrls.c
+++ b/drivers/media/platform/vivid/vivid-ctrls.c
@@ -1672,59 +1672,59 @@ int vivid_create_controls(struct vivid_dev *dev, bool show_ccs_cap,
 		v4l2_ctrl_auto_cluster(2, &dev->autogain, 0, true);
 
 	if (dev->has_vid_cap) {
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_vid, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_aud, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_sdtv_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_loop_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_fb, NULL);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_vid, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_user_aud, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_sdtv_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_loop_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_cap, hdl_fb, NULL, false);
 		if (hdl_vid_cap->error)
 			return hdl_vid_cap->error;
 		dev->vid_cap_dev.ctrl_handler = hdl_vid_cap;
 	}
 	if (dev->has_vid_out) {
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_aud, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vid_out, hdl_fb, NULL);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_user_aud, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vid_out, hdl_fb, NULL, false);
 		if (hdl_vid_out->error)
 			return hdl_vid_out->error;
 		dev->vid_out_dev.ctrl_handler = hdl_vid_out;
 	}
 	if (dev->has_vbi_cap) {
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_streaming, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_sdtv_cap, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_loop_cap, NULL);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_streaming, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_sdtv_cap, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_cap, hdl_loop_cap, NULL, false);
 		if (hdl_vbi_cap->error)
 			return hdl_vbi_cap->error;
 		dev->vbi_cap_dev.ctrl_handler = hdl_vbi_cap;
 	}
 	if (dev->has_vbi_out) {
-		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_streaming, NULL);
+		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_vbi_out, hdl_streaming, NULL, false);
 		if (hdl_vbi_out->error)
 			return hdl_vbi_out->error;
 		dev->vbi_out_dev.ctrl_handler = hdl_vbi_out;
 	}
 	if (dev->has_radio_rx) {
-		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_aud, NULL);
+		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_radio_rx, hdl_user_aud, NULL, false);
 		if (hdl_radio_rx->error)
 			return hdl_radio_rx->error;
 		dev->radio_rx_dev.ctrl_handler = hdl_radio_rx;
 	}
 	if (dev->has_radio_tx) {
-		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_aud, NULL);
+		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_radio_tx, hdl_user_aud, NULL, false);
 		if (hdl_radio_tx->error)
 			return hdl_radio_tx->error;
 		dev->radio_tx_dev.ctrl_handler = hdl_radio_tx;
 	}
 	if (dev->has_sdr_cap) {
-		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_user_gen, NULL);
-		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_streaming, NULL);
+		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_user_gen, NULL, false);
+		v4l2_ctrl_add_handler(hdl_sdr_cap, hdl_streaming, NULL, false);
 		if (hdl_sdr_cap->error)
 			return hdl_sdr_cap->error;
 		dev->sdr_cap_dev.ctrl_handler = hdl_sdr_cap;
diff --git a/drivers/media/usb/cx231xx/cx231xx-417.c b/drivers/media/usb/cx231xx/cx231xx-417.c
index d538fa407742..ba8a4072633a 100644
--- a/drivers/media/usb/cx231xx/cx231xx-417.c
+++ b/drivers/media/usb/cx231xx/cx231xx-417.c
@@ -1991,7 +1991,7 @@ int cx231xx_417_register(struct cx231xx *dev)
 	dev->mpeg_ctrl_handler.ops = &cx231xx_ops;
 	if (dev->sd_cx25840)
 		v4l2_ctrl_add_handler(&dev->mpeg_ctrl_handler.hdl,
-				dev->sd_cx25840->ctrl_handler, NULL);
+				dev->sd_cx25840->ctrl_handler, NULL, false);
 	if (dev->mpeg_ctrl_handler.hdl.error) {
 		err = dev->mpeg_ctrl_handler.hdl.error;
 		dprintk(3, "%s: can't add cx25840 controls\n", dev->name);
diff --git a/drivers/media/usb/cx231xx/cx231xx-video.c b/drivers/media/usb/cx231xx/cx231xx-video.c
index 0b6ac509fdb1..0f25e58760da 100644
--- a/drivers/media/usb/cx231xx/cx231xx-video.c
+++ b/drivers/media/usb/cx231xx/cx231xx-video.c
@@ -2204,10 +2204,10 @@ int cx231xx_register_analog_devices(struct cx231xx *dev)
 
 	if (dev->sd_cx25840) {
 		v4l2_ctrl_add_handler(&dev->ctrl_handler,
-				dev->sd_cx25840->ctrl_handler, NULL);
+				dev->sd_cx25840->ctrl_handler, NULL, true);
 		v4l2_ctrl_add_handler(&dev->radio_ctrl_handler,
 				dev->sd_cx25840->ctrl_handler,
-				v4l2_ctrl_radio_filter);
+				v4l2_ctrl_radio_filter, true);
 	}
 
 	if (dev->ctrl_handler.error)
diff --git a/drivers/media/usb/msi2500/msi2500.c b/drivers/media/usb/msi2500/msi2500.c
index 65ef755adfdc..4aacd77a5d58 100644
--- a/drivers/media/usb/msi2500/msi2500.c
+++ b/drivers/media/usb/msi2500/msi2500.c
@@ -1278,7 +1278,7 @@ static int msi2500_probe(struct usb_interface *intf,
 	}
 
 	/* currently all controls are from subdev */
-	v4l2_ctrl_add_handler(&dev->hdl, sd->ctrl_handler, NULL);
+	v4l2_ctrl_add_handler(&dev->hdl, sd->ctrl_handler, NULL, true);
 
 	dev->v4l2_dev.ctrl_handler = &dev->hdl;
 	dev->vdev.v4l2_dev = &dev->v4l2_dev;
diff --git a/drivers/media/usb/tm6000/tm6000-video.c b/drivers/media/usb/tm6000/tm6000-video.c
index df040558997e..39b76244ab06 100644
--- a/drivers/media/usb/tm6000/tm6000-video.c
+++ b/drivers/media/usb/tm6000/tm6000-video.c
@@ -1623,7 +1623,7 @@ int tm6000_v4l2_register(struct tm6000_core *dev)
 	v4l2_ctrl_new_std(&dev->ctrl_handler, &tm6000_ctrl_ops,
 			V4L2_CID_HUE, -128, 127, 1, 0);
 	v4l2_ctrl_add_handler(&dev->ctrl_handler,
-			&dev->radio_ctrl_handler, NULL);
+			&dev->radio_ctrl_handler, NULL, false);
 
 	if (dev->radio_ctrl_handler.error)
 		ret = dev->radio_ctrl_handler.error;
diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index cbb2ef43945f..2e58381444d1 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1876,7 +1876,8 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
-			   struct v4l2_ctrl *ctrl)
+			   struct v4l2_ctrl *ctrl,
+			   bool from_other_dev)
 {
 	struct v4l2_ctrl_ref *ref;
 	struct v4l2_ctrl_ref *new_ref;
@@ -1900,6 +1901,7 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	if (!new_ref)
 		return handler_set_err(hdl, -ENOMEM);
 	new_ref->ctrl = ctrl;
+	new_ref->from_other_dev = from_other_dev;
 	if (ctrl->handler == hdl) {
 		/* By default each control starts in a cluster of its own.
 		   new_ref->ctrl is basically a cluster array with one
@@ -2080,7 +2082,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
 		ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
 	}
 
-	if (handler_new_ref(hdl, ctrl)) {
+	if (handler_new_ref(hdl, ctrl, false)) {
 		kvfree(ctrl);
 		return NULL;
 	}
@@ -2249,7 +2251,8 @@ EXPORT_SYMBOL(v4l2_ctrl_new_int_menu);
 /* Add the controls from another handler to our own. */
 int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 			  struct v4l2_ctrl_handler *add,
-			  bool (*filter)(const struct v4l2_ctrl *ctrl))
+			  bool (*filter)(const struct v4l2_ctrl *ctrl),
+			  bool from_other_dev)
 {
 	struct v4l2_ctrl_ref *ref;
 	int ret = 0;
@@ -2272,7 +2275,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 		/* Filter any unwanted controls */
 		if (filter && !filter(ctrl))
 			continue;
-		ret = handler_new_ref(hdl, ctrl);
+		ret = handler_new_ref(hdl, ctrl, from_other_dev);
 		if (ret)
 			break;
 	}
diff --git a/drivers/media/v4l2-core/v4l2-device.c b/drivers/media/v4l2-core/v4l2-device.c
index 937c6de85606..8391a7f0895b 100644
--- a/drivers/media/v4l2-core/v4l2-device.c
+++ b/drivers/media/v4l2-core/v4l2-device.c
@@ -178,7 +178,8 @@ int v4l2_device_register_subdev(struct v4l2_device *v4l2_dev,
 
 	sd->v4l2_dev = v4l2_dev;
 	/* This just returns 0 if either of the two args is NULL */
-	err = v4l2_ctrl_add_handler(v4l2_dev->ctrl_handler, sd->ctrl_handler, NULL);
+	err = v4l2_ctrl_add_handler(v4l2_dev->ctrl_handler, sd->ctrl_handler,
+				    NULL, true);
 	if (err)
 		goto error_module;
 
diff --git a/drivers/staging/media/imx/imx-media-dev.c b/drivers/staging/media/imx/imx-media-dev.c
index 289d775c4820..08799beaea42 100644
--- a/drivers/staging/media/imx/imx-media-dev.c
+++ b/drivers/staging/media/imx/imx-media-dev.c
@@ -391,7 +391,7 @@ static int imx_media_inherit_controls(struct imx_media_dev *imxmd,
 
 		ret = v4l2_ctrl_add_handler(vfd->ctrl_handler,
 					    sd->ctrl_handler,
-					    NULL);
+					    NULL, true);
 		if (ret)
 			return ret;
 	}
diff --git a/drivers/staging/media/imx/imx-media-fim.c b/drivers/staging/media/imx/imx-media-fim.c
index 6df189135db8..8cf773eef9da 100644
--- a/drivers/staging/media/imx/imx-media-fim.c
+++ b/drivers/staging/media/imx/imx-media-fim.c
@@ -463,7 +463,7 @@ int imx_media_fim_add_controls(struct imx_media_fim *fim)
 {
 	/* add the FIM controls to the calling subdev ctrl handler */
 	return v4l2_ctrl_add_handler(fim->sd->ctrl_handler,
-				     &fim->ctrl_handler, NULL);
+				     &fim->ctrl_handler, NULL, false);
 }
 EXPORT_SYMBOL_GPL(imx_media_fim_add_controls);
 
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 5253b5471897..a4750a8090e3 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -257,6 +257,7 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl_ref *next;
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
+	bool from_other_dev;
 };
 
 /**
@@ -642,7 +643,8 @@ typedef bool (*v4l2_ctrl_filter)(const struct v4l2_ctrl *ctrl);
  */
 int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 			  struct v4l2_ctrl_handler *add,
-			  v4l2_ctrl_filter filter);
+			  v4l2_ctrl_filter filter,
+			  bool from_other_dev);
 
 /**
  * v4l2_ctrl_radio_filter() - Standard filter for radio controls.
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 07/17] v4l2-ctrls: prepare internal structs for request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add a refcount and is_request bool to struct v4l2_ctrl_handler:
this is used to refcount a handler that represents a request.

Add a p_req field to struct v4l2_ctrl_ref that will store the
request value.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 1 +
 include/media/v4l2-ctrls.h           | 4 ++++
 2 files changed, 5 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 2e58381444d1..1ff8fc59fff5 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1761,6 +1761,7 @@ int v4l2_ctrl_handler_init_class(struct v4l2_ctrl_handler *hdl,
 				      sizeof(hdl->buckets[0]),
 				      GFP_KERNEL | __GFP_ZERO);
 	hdl->error = hdl->buckets ? 0 : -ENOMEM;
+	hdl->is_request = false;
 	return hdl->error;
 }
 EXPORT_SYMBOL(v4l2_ctrl_handler_init_class);
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index a4750a8090e3..22b6be78057f 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -18,6 +18,7 @@
 #define _V4L2_CTRLS_H
 
 #include <linux/list.h>
+#include <linux/kref.h>
 #include <linux/mutex.h>
 #include <linux/videodev2.h>
 
@@ -257,6 +258,7 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl_ref *next;
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
+	union v4l2_ctrl_ptr p_req;
 	bool from_other_dev;
 };
 
@@ -292,7 +294,9 @@ struct v4l2_ctrl_handler {
 	v4l2_ctrl_notify_fnc notify;
 	void *notify_priv;
 	u16 nr_of_buckets;
+	bool is_request;
 	int error;
+	struct kref ref;
 };
 
 /**
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 08/17] v4l2-ctrls: add core request API

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add the four core request functions:

v4l2_ctrl_request_init() initializes a new (empty) request.
v4l2_ctrl_request_clone() resets a request based on another request
(or clears it if that request is NULL).
v4l2_ctrl_request_get(): increase refcount
v4l2_ctrl_request_put(): decrease refcount and delete if it reaches 0.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
[acourbot@chromium.org: turn v4l2_ctrl_request_alloc into init function]
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 106 ++++++++++++++++++++++++++++++++++-
 include/media/v4l2-ctrls.h           |   7 +++
 2 files changed, 110 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 1ff8fc59fff5..c692a6d925c6 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1878,6 +1878,7 @@ EXPORT_SYMBOL(v4l2_ctrl_find);
 /* Allocate a new v4l2_ctrl_ref and hook it into the handler. */
 static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 			   struct v4l2_ctrl *ctrl,
+			   struct v4l2_ctrl_ref **ctrl_ref,
 			   bool from_other_dev)
 {
 	struct v4l2_ctrl_ref *ref;
@@ -1885,6 +1886,10 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	u32 id = ctrl->id;
 	u32 class_ctrl = V4L2_CTRL_ID2WHICH(id) | 1;
 	int bucket = id % hdl->nr_of_buckets;	/* which bucket to use */
+	unsigned int sz_extra = 0;
+
+	if (ctrl_ref)
+		*ctrl_ref = NULL;
 
 	/*
 	 * Automatically add the control class if it is not yet present and
@@ -1898,11 +1903,16 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	if (hdl->error)
 		return hdl->error;
 
-	new_ref = kzalloc(sizeof(*new_ref), GFP_KERNEL);
+	if (hdl->is_request)
+		sz_extra = ctrl->elems * ctrl->elem_size;
+	new_ref = kzalloc(sizeof(*new_ref) + sz_extra, GFP_KERNEL);
 	if (!new_ref)
 		return handler_set_err(hdl, -ENOMEM);
 	new_ref->ctrl = ctrl;
 	new_ref->from_other_dev = from_other_dev;
+	if (sz_extra)
+		new_ref->p_req.p = &new_ref[1];
+
 	if (ctrl->handler == hdl) {
 		/* By default each control starts in a cluster of its own.
 		   new_ref->ctrl is basically a cluster array with one
@@ -1942,6 +1952,8 @@ static int handler_new_ref(struct v4l2_ctrl_handler *hdl,
 	/* Insert the control node in the hash */
 	new_ref->next = hdl->buckets[bucket];
 	hdl->buckets[bucket] = new_ref;
+	if (ctrl_ref)
+		*ctrl_ref = new_ref;
 
 unlock:
 	mutex_unlock(hdl->lock);
@@ -2083,7 +2095,7 @@ static struct v4l2_ctrl *v4l2_ctrl_new(struct v4l2_ctrl_handler *hdl,
 		ctrl->type_ops->init(ctrl, idx, ctrl->p_new);
 	}
 
-	if (handler_new_ref(hdl, ctrl, false)) {
+	if (handler_new_ref(hdl, ctrl, NULL, false)) {
 		kvfree(ctrl);
 		return NULL;
 	}
@@ -2276,7 +2288,7 @@ int v4l2_ctrl_add_handler(struct v4l2_ctrl_handler *hdl,
 		/* Filter any unwanted controls */
 		if (filter && !filter(ctrl))
 			continue;
-		ret = handler_new_ref(hdl, ctrl, from_other_dev);
+		ret = handler_new_ref(hdl, ctrl, NULL, from_other_dev);
 		if (ret)
 			break;
 	}
@@ -2685,6 +2697,94 @@ int v4l2_querymenu(struct v4l2_ctrl_handler *hdl, struct v4l2_querymenu *qm)
 }
 EXPORT_SYMBOL(v4l2_querymenu);
 
+int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl)
+{
+	int err;
+
+	err = v4l2_ctrl_handler_init(hdl, 0);
+	if (err)
+		return err;
+	hdl->is_request = true;
+	kref_init(&hdl->ref);
+
+	return 0;
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_init);
+
+int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+			    const struct v4l2_ctrl_handler *from,
+			    bool (*filter)(const struct v4l2_ctrl *ctrl))
+{
+	struct v4l2_ctrl_ref *ref;
+	int err;
+
+	if (WARN_ON(!hdl || hdl == from))
+		return -EINVAL;
+
+	if (hdl->error)
+		return hdl->error;
+
+	WARN_ON(hdl->lock != &hdl->_lock);
+	v4l2_ctrl_handler_free(hdl);
+	err = v4l2_ctrl_handler_init(hdl, (from->nr_of_buckets - 1) * 8);
+	hdl->is_request = true;
+	if (err)
+		return err;
+	if (!from)
+		return 0;
+
+	mutex_lock(from->lock);
+	list_for_each_entry(ref, &from->ctrl_refs, node) {
+		struct v4l2_ctrl *ctrl = ref->ctrl;
+		struct v4l2_ctrl_ref *new_ref;
+
+		/* Skip refs inherited from other devices */
+		if (ref->from_other_dev)
+			continue;
+		/* And buttons and control classes */
+		if (ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+		    ctrl->type == V4L2_CTRL_TYPE_CTRL_CLASS)
+			continue;
+		/* Filter any unwanted controls */
+		if (filter && !filter(ctrl))
+			continue;
+		err = handler_new_ref(hdl, ctrl, &new_ref, false);
+		if (err)
+			break;
+		if (from->is_request)
+			ptr_to_ptr(ctrl, ref->p_req, new_ref->p_req);
+		else
+			ptr_to_ptr(ctrl, ctrl->p_cur, new_ref->p_req);
+	}
+	mutex_unlock(from->lock);
+	return err;
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_clone);
+
+void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl)
+{
+	if (WARN_ON(!hdl->is_request))
+		return;
+	kref_get(&hdl->ref);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_get);
+
+static void v4l2_ctrl_request_release(struct kref *kref)
+{
+	struct v4l2_ctrl_handler *hdl =
+		container_of(kref, struct v4l2_ctrl_handler, ref);
+
+	v4l2_ctrl_handler_free(hdl);
+	kfree(hdl);
+}
+
+void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl)
+{
+	if (WARN_ON(!hdl->is_request))
+		return;
+	kref_put(&hdl->ref, v4l2_ctrl_request_release);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_put);
 
 /* Some general notes on the atomic requirements of VIDIOC_G/TRY/S_EXT_CTRLS:
 
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index 22b6be78057f..bddceb794961 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -1052,6 +1052,13 @@ int v4l2_ctrl_subscribe_event(struct v4l2_fh *fh,
  */
 unsigned int v4l2_ctrl_poll(struct file *file, struct poll_table_struct *wait);
 
+int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl);
+int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
+			    const struct v4l2_ctrl_handler *from,
+			    bool (*filter)(const struct v4l2_ctrl *ctrl));
+void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl);
+void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl);
+
 /* Helpers for ioctl_ops */
 
 /**
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 09/17] v4l2-ctrls: use ref in helper instead of ctrl

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

The next patch needs the reference to a control instead of the
control itself, so change struct v4l2_ctrl_helper accordingly.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 18 +++++++++---------
 1 file changed, 9 insertions(+), 9 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index c692a6d925c6..9090a49eef91 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -37,8 +37,8 @@
 struct v4l2_ctrl_helper {
 	/* Pointer to the control reference of the master control */
 	struct v4l2_ctrl_ref *mref;
-	/* The control corresponding to the v4l2_ext_control ID field. */
-	struct v4l2_ctrl *ctrl;
+	/* The control ref corresponding to the v4l2_ext_control ID field. */
+	struct v4l2_ctrl_ref *ref;
 	/* v4l2_ext_control index of the next control belonging to the
 	   same cluster, or 0 if there isn't any. */
 	u32 next;
@@ -2856,6 +2856,7 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 		ref = find_ref_lock(hdl, id);
 		if (ref == NULL)
 			return -EINVAL;
+		h->ref = ref;
 		ctrl = ref->ctrl;
 		if (ctrl->flags & V4L2_CTRL_FLAG_DISABLED)
 			return -EINVAL;
@@ -2878,7 +2879,6 @@ static int prepare_ext_ctrls(struct v4l2_ctrl_handler *hdl,
 		}
 		/* Store the ref to the master control of the cluster */
 		h->mref = ref;
-		h->ctrl = ctrl;
 		/* Initially set next to 0, meaning that there is no other
 		   control in this helper array belonging to the same
 		   cluster */
@@ -2963,7 +2963,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 	cs->error_idx = cs->count;
 
 	for (i = 0; !ret && i < cs->count; i++)
-		if (helpers[i].ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
+		if (helpers[i].ref->ctrl->flags & V4L2_CTRL_FLAG_WRITE_ONLY)
 			ret = -EACCES;
 
 	for (i = 0; !ret && i < cs->count; i++) {
@@ -2998,7 +2998,7 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 
 			do {
 				ret = ctrl_to_user(cs->controls + idx,
-						   helpers[idx].ctrl);
+						   helpers[idx].ref->ctrl);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
@@ -3137,7 +3137,7 @@ static int validate_ctrls(struct v4l2_ext_controls *cs,
 
 	cs->error_idx = cs->count;
 	for (i = 0; i < cs->count; i++) {
-		struct v4l2_ctrl *ctrl = helpers[i].ctrl;
+		struct v4l2_ctrl *ctrl = helpers[i].ref->ctrl;
 		union v4l2_ctrl_ptr p_new;
 
 		cs->error_idx = i;
@@ -3249,7 +3249,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 			do {
 				/* Check if the auto control is part of the
 				   list, and remember the new value. */
-				if (helpers[tmp_idx].ctrl == master)
+				if (helpers[tmp_idx].ref->ctrl == master)
 					new_auto_val = cs->controls[tmp_idx].value;
 				tmp_idx = helpers[tmp_idx].next;
 			} while (tmp_idx);
@@ -3262,7 +3262,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 		/* Copy the new caller-supplied control values.
 		   user_to_new() sets 'is_new' to 1. */
 		do {
-			struct v4l2_ctrl *ctrl = helpers[idx].ctrl;
+			struct v4l2_ctrl *ctrl = helpers[idx].ref->ctrl;
 
 			ret = user_to_new(cs->controls + idx, ctrl);
 			if (!ret && ctrl->is_ptr)
@@ -3278,7 +3278,7 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 			idx = i;
 			do {
 				ret = new_to_user(cs->controls + idx,
-						helpers[idx].ctrl);
+						helpers[idx].ref->ctrl);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 10/17] v4l2-ctrls: support g/s_ext_ctrls for requests

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

The v4l2_g/s_ext_ctrls functions now support control handlers that
represent requests.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 37 ++++++++++++++++++++++++++++++++----
 1 file changed, 33 insertions(+), 4 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 9090a49eef91..4fa7adef7531 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1528,6 +1528,13 @@ static int new_to_user(struct v4l2_ext_control *c,
 	return ptr_to_user(c, ctrl, ctrl->p_new);
 }
 
+/* Helper function: copy the request value back to the caller */
+static int req_to_user(struct v4l2_ext_control *c,
+		       struct v4l2_ctrl_ref *ref)
+{
+	return ptr_to_user(c, ref->ctrl, ref->p_req);
+}
+
 /* Helper function: copy the initial control value back to the caller */
 static int def_to_user(struct v4l2_ext_control *c, struct v4l2_ctrl *ctrl)
 {
@@ -1647,6 +1654,14 @@ static void cur_to_new(struct v4l2_ctrl *ctrl)
 	ptr_to_ptr(ctrl, ctrl->p_cur, ctrl->p_new);
 }
 
+/* Copy the new value to the request value */
+static void new_to_req(struct v4l2_ctrl_ref *ref)
+{
+	if (!ref)
+		return;
+	ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
    value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -2971,7 +2986,8 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 				    struct v4l2_ctrl *ctrl);
 		struct v4l2_ctrl *master;
 
-		ctrl_to_user = def_value ? def_to_user : cur_to_user;
+		ctrl_to_user = def_value ? def_to_user :
+			       (hdl->is_request ? NULL : cur_to_user);
 
 		if (helpers[i].mref == NULL)
 			continue;
@@ -2997,8 +3013,12 @@ int v4l2_g_ext_ctrls(struct v4l2_ctrl_handler *hdl, struct v4l2_ext_controls *cs
 			u32 idx = i;
 
 			do {
-				ret = ctrl_to_user(cs->controls + idx,
-						   helpers[idx].ref->ctrl);
+				if (ctrl_to_user)
+					ret = ctrl_to_user(cs->controls + idx,
+						helpers[idx].ref->ctrl);
+				else
+					ret = req_to_user(cs->controls + idx,
+						helpers[idx].ref);
 				idx = helpers[idx].next;
 			} while (!ret && idx);
 		}
@@ -3271,7 +3291,16 @@ static int try_set_ext_ctrls(struct v4l2_fh *fh, struct v4l2_ctrl_handler *hdl,
 		} while (!ret && idx);
 
 		if (!ret)
-			ret = try_or_set_cluster(fh, master, set, 0);
+			ret = try_or_set_cluster(fh, master,
+						 !hdl->is_request && set, 0);
+		if (!ret && hdl->is_request && set) {
+			for (j = 0; j < master->ncontrols; j++) {
+				struct v4l2_ctrl_ref *ref =
+					find_ref(hdl, master->cluster[j]->id);
+
+				new_to_req(ref);
+			}
+		}
 
 		/* Copy the new values back to userspace. */
 		if (!ret) {
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 11/17] v4l2-ctrls: add v4l2_ctrl_request_setup

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

Add a helper function that can set controls from a request.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ctrls.c | 71 ++++++++++++++++++++++++++++++++++++
 include/media/v4l2-ctrls.h           |  2 +
 2 files changed, 73 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ctrls.c b/drivers/media/v4l2-core/v4l2-ctrls.c
index 4fa7adef7531..dccc27968df1 100644
--- a/drivers/media/v4l2-core/v4l2-ctrls.c
+++ b/drivers/media/v4l2-core/v4l2-ctrls.c
@@ -1662,6 +1662,14 @@ static void new_to_req(struct v4l2_ctrl_ref *ref)
 	ptr_to_ptr(ref->ctrl, ref->ctrl->p_new, ref->p_req);
 }
 
+/* Copy the request value to the new value */
+static void req_to_new(struct v4l2_ctrl_ref *ref)
+{
+	if (!ref)
+		return;
+	ptr_to_ptr(ref->ctrl, ref->p_req, ref->ctrl->p_new);
+}
+
 /* Return non-zero if one or more of the controls in the cluster has a new
    value that differs from the current value. */
 static int cluster_changed(struct v4l2_ctrl *master)
@@ -3427,6 +3435,69 @@ int __v4l2_ctrl_s_ctrl_string(struct v4l2_ctrl *ctrl, const char *s)
 }
 EXPORT_SYMBOL(__v4l2_ctrl_s_ctrl_string);
 
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl)
+{
+	struct v4l2_ctrl_ref *ref;
+
+	if (!hdl)
+		return;
+
+	mutex_lock(hdl->lock);
+
+	list_for_each_entry(ref, &hdl->ctrl_refs, node)
+		ref->done = false;
+
+	list_for_each_entry(ref, &hdl->ctrl_refs, node) {
+		struct v4l2_ctrl *ctrl = ref->ctrl;
+		struct v4l2_ctrl *master = ctrl->cluster[0];
+		int i;
+
+		/* Skip if this control was already handled by a cluster. */
+		/* Skip button controls and read-only controls. */
+		if (ref->done || ctrl->type == V4L2_CTRL_TYPE_BUTTON ||
+		    (ctrl->flags & V4L2_CTRL_FLAG_READ_ONLY))
+			continue;
+
+		v4l2_ctrl_lock(master);
+		for (i = 0; i < master->ncontrols; i++) {
+			if (master->cluster[i]) {
+				struct v4l2_ctrl_ref *r =
+					find_ref(hdl, master->cluster[i]->id);
+
+				req_to_new(r);
+				master->cluster[i]->is_new = 1;
+				r->done = true;
+			}
+		}
+		/*
+		 * For volatile autoclusters that are currently in auto mode
+		 * we need to discover if it will be set to manual mode.
+		 * If so, then we have to copy the current volatile values
+		 * first since those will become the new manual values (which
+		 * may be overwritten by explicit new values from this set
+		 * of controls).
+		 */
+		if (master->is_auto && master->has_volatiles &&
+		    !is_cur_manual(master)) {
+			s32 new_auto_val = *master->p_new.p_s32;
+
+			/*
+			 * If the new value == the manual value, then copy
+			 * the current volatile values.
+			 */
+			if (new_auto_val == master->manual_mode_value)
+				update_from_auto_cluster(master);
+		}
+
+		try_or_set_cluster(NULL, master, true, 0);
+
+		v4l2_ctrl_unlock(master);
+	}
+
+	mutex_unlock(hdl->lock);
+}
+EXPORT_SYMBOL(v4l2_ctrl_request_setup);
+
 void v4l2_ctrl_notify(struct v4l2_ctrl *ctrl, v4l2_ctrl_notify_fnc notify, void *priv)
 {
 	if (ctrl == NULL)
diff --git a/include/media/v4l2-ctrls.h b/include/media/v4l2-ctrls.h
index bddceb794961..922824dbd7db 100644
--- a/include/media/v4l2-ctrls.h
+++ b/include/media/v4l2-ctrls.h
@@ -259,6 +259,7 @@ struct v4l2_ctrl_ref {
 	struct v4l2_ctrl *ctrl;
 	struct v4l2_ctrl_helper *helper;
 	union v4l2_ctrl_ptr p_req;
+	bool done;
 	bool from_other_dev;
 };
 
@@ -1056,6 +1057,7 @@ int v4l2_ctrl_request_init(struct v4l2_ctrl_handler *hdl);
 int v4l2_ctrl_request_clone(struct v4l2_ctrl_handler *hdl,
 			    const struct v4l2_ctrl_handler *from,
 			    bool (*filter)(const struct v4l2_ctrl *ctrl));
+void v4l2_ctrl_request_setup(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_get(struct v4l2_ctrl_handler *hdl);
 void v4l2_ctrl_request_put(struct v4l2_ctrl_handler *hdl);
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 12/17] v4l2: add request API support

[Alexandre Courbot]

Add a v4l2 request entity data structure that takes care of storing the
request-related state of a V4L2 device ; in this case, its controls.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/Makefile       |  2 +-
 drivers/media/v4l2-core/v4l2-request.c | 54 ++++++++++++++++++++++++++++++++++
 include/media/v4l2-request.h           | 34 +++++++++++++++++++++
 3 files changed, 89 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/v4l2-core/v4l2-request.c
 create mode 100644 include/media/v4l2-request.h

diff --git a/drivers/media/v4l2-core/Makefile b/drivers/media/v4l2-core/Makefile
index 80de2cb9c476..1113dea1f4f9 100644
--- a/drivers/media/v4l2-core/Makefile
+++ b/drivers/media/v4l2-core/Makefile
@@ -15,7 +15,7 @@ obj-$(CONFIG_V4L2_FWNODE) += v4l2-fwnode.o
 ifeq ($(CONFIG_TRACEPOINTS),y)
   videodev-objs += vb2-trace.o v4l2-trace.o
 endif
-videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o
+videodev-$(CONFIG_MEDIA_CONTROLLER) += v4l2-mc.o v4l2-request.o
 
 obj-$(CONFIG_VIDEO_V4L2) += videodev.o
 obj-$(CONFIG_VIDEO_V4L2) += v4l2-common.o
diff --git a/drivers/media/v4l2-core/v4l2-request.c b/drivers/media/v4l2-core/v4l2-request.c
new file mode 100644
index 000000000000..7bc29d3cc332
--- /dev/null
+++ b/drivers/media/v4l2-core/v4l2-request.c
@@ -0,0 +1,54 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/slab.h>
+
+#include <media/v4l2-request.h>
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+	struct v4l2_ctrl_handler *hdl)
+{
+	struct media_request_v4l2_entity_data *data;
+	int ret;
+
+	data = kzalloc(sizeof(*data), GFP_KERNEL);
+
+	ret = v4l2_ctrl_request_init(&data->ctrls);
+	if (ret) {
+		kfree(data);
+		return ERR_PTR(ret);
+	}
+
+	ret = v4l2_ctrl_request_clone(&data->ctrls, hdl, NULL);
+	if (ret) {
+		kfree(data);
+		return ERR_PTR(ret);
+	}
+
+	return &data->base;
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_alloc);
+
+void
+media_request_v4l2_entity_data_free(struct media_request_entity_data *_data)
+{
+	struct media_request_v4l2_entity_data *data;
+
+	data = to_v4l2_entity_data(_data);
+
+	v4l2_ctrl_handler_free(&data->ctrls);
+	kfree(data);
+}
+EXPORT_SYMBOL_GPL(media_request_v4l2_entity_data_free);
diff --git a/include/media/v4l2-request.h b/include/media/v4l2-request.h
new file mode 100644
index 000000000000..db38dc5fc460
--- /dev/null
+++ b/include/media/v4l2-request.h
@@ -0,0 +1,34 @@
+/*
+ * Media requests support for V4L2
+ *
+ * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_V4L2_H
+#define _MEDIA_REQUEST_V4L2_H
+
+#include <media/media-request.h>
+#include <media/v4l2-ctrls.h>
+
+struct media_request_v4l2_entity_data {
+	struct media_request_entity_data base;
+
+	struct v4l2_ctrl_handler ctrls;
+};
+#define to_v4l2_entity_data(d) \
+	container_of(d, struct media_request_v4l2_entity_data, base)
+
+struct media_request_entity_data *media_request_v4l2_entity_data_alloc(
+	struct v4l2_ctrl_handler *hdl);
+void media_request_v4l2_entity_data_free(struct media_request_entity_data *data);
+
+#endif
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 13/17] videodev2.h: add request_fd field to v4l2_ext_controls

[Alexandre Courbot]

Allow to specify a request to be used with the S_EXT_CTRLS and
G_EXT_CTRLS operations.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 include/uapi/linux/videodev2.h | 3 ++-
 1 file changed, 2 insertions(+), 1 deletion(-)

diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 4fd46ae8fad5..91cfe0cbd5c5 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -1592,7 +1592,8 @@ struct v4l2_ext_controls {
 	};
 	__u32 count;
 	__u32 error_idx;
-	__u32 reserved[2];
+	__s32 request_fd;
+	__u32 reserved[1];
 	struct v4l2_ext_control *controls;
 };
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 14/17] v4l2-ctrls: support requests in EXT_CTRLS ioctls

[Alexandre Courbot]

Read and use the request_fd field of struct v4l2_ext_controls to apply
VIDIOC_G_EXT_CTRLS or VIDIOC_S_EXT_CTRLS to a request when asked by
userspace.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 36 ++++++++++++++++++++++++++++++++++++
 1 file changed, 36 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 235acdde3111..cbaefcad9694 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -30,6 +30,7 @@
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mc.h>
 #include <media/media-request.h>
+#include <media/v4l2-request.h>
 
 #include <trace/events/v4l2.h>
 
@@ -2158,6 +2159,24 @@ static int v4l_g_ext_ctrls(const struct v4l2_ioctl_ops *ops,
 		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
 
 	p->error_idx = p->count;
+
+	if (p->request_fd > 0) {
+		struct media_request *req = NULL;
+		struct media_request_entity_data *_data;
+		struct media_request_v4l2_entity_data *data;
+		int ret;
+
+		req = check_request(p->request_fd, file, fh, &_data);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+		data = to_v4l2_entity_data(_data);
+
+		ret = v4l2_g_ext_ctrls(&data->ctrls, p);
+
+		media_request_put(req);
+		return ret;
+	}
+
 	if (vfh && vfh->ctrl_handler)
 		return v4l2_g_ext_ctrls(vfh->ctrl_handler, p);
 	if (vfd->ctrl_handler)
@@ -2177,6 +2196,23 @@ static int v4l_s_ext_ctrls(const struct v4l2_ioctl_ops *ops,
 		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
 
 	p->error_idx = p->count;
+	if (p->request_fd > 0) {
+		struct media_request *req = NULL;
+		struct media_request_entity_data *_data;
+		struct media_request_v4l2_entity_data *data;
+		int ret;
+
+		req = check_request(p->request_fd, file, fh, &_data);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+		data = to_v4l2_entity_data(_data);
+
+		ret = v4l2_s_ext_ctrls(vfh, &data->ctrls, p);
+
+		media_request_put(req);
+		return ret;
+	}
+
 	if (vfh && vfh->ctrl_handler)
 		return v4l2_s_ext_ctrls(vfh, vfh->ctrl_handler, p);
 	if (vfd->ctrl_handler)
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 15/17] v4l2: document the request API interface

[Alexandre Courbot]

Document how the request API can be used along with the existing V4L2
interface.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 Documentation/media/uapi/v4l/buffer.rst            |  10 +-
 Documentation/media/uapi/v4l/common.rst            |   1 +
 Documentation/media/uapi/v4l/request-api.rst       | 236 +++++++++++++++++++++
 .../media/uapi/v4l/vidioc-g-ext-ctrls.rst          |  16 +-
 Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  21 ++
 5 files changed, 280 insertions(+), 4 deletions(-)
 create mode 100644 Documentation/media/uapi/v4l/request-api.rst

diff --git a/Documentation/media/uapi/v4l/buffer.rst b/Documentation/media/uapi/v4l/buffer.rst
index ae6ee73f151c..9d082784081d 100644
--- a/Documentation/media/uapi/v4l/buffer.rst
+++ b/Documentation/media/uapi/v4l/buffer.rst
@@ -301,10 +301,14 @@ struct v4l2_buffer
 	elements in the ``planes`` array. The driver will fill in the
 	actual number of valid elements in that array.
     * - __u32
-      - ``reserved2``
+      - ``request_fd``
       -
-      - A place holder for future extensions. Drivers and applications
-	must set this to 0.
+      - The file descriptor of the request associated with this buffer.
+	user-space can set this when calling :ref:`VIDIOC_QBUF`, and drivers
+	will return the request used when processing a buffer (if any) upon
+	:ref:`VIDIOC_DQBUF`.
+
+	A value of 0 means the buffer is not associated with any request.
     * - __u32
       - ``reserved``
       -
diff --git a/Documentation/media/uapi/v4l/common.rst b/Documentation/media/uapi/v4l/common.rst
index 13f2ed3fc5a6..a4aa0059d45a 100644
--- a/Documentation/media/uapi/v4l/common.rst
+++ b/Documentation/media/uapi/v4l/common.rst
@@ -44,3 +44,4 @@ applicable to all devices.
     crop
     selection-api
     streaming-par
+    request-api
diff --git a/Documentation/media/uapi/v4l/request-api.rst b/Documentation/media/uapi/v4l/request-api.rst
new file mode 100644
index 000000000000..4c61a0dbe3a9
--- /dev/null
+++ b/Documentation/media/uapi/v4l/request-api.rst
@@ -0,0 +1,236 @@
+.. -*- coding: utf-8; mode: rst -*-
+
+.. _media-request-api:
+
+Request API
+===========
+
+The Request API has been designed to allow V4L2 to deal with requirements of
+modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
+v2). One such requirement is the ability for devices belonging to the same
+pipeline to reconfigure and collaborate closely on a per-frame basis. Another is
+efficient support of stateless codecs, which need per-frame controls to be set
+asynchronously in order to be efficiently used.
+
+Supporting these features without the Request API is possible but terribly
+inefficient: user-space would have to flush all activity on the media pipeline,
+reconfigure it for the next frame, queue the buffers to be processed with that
+configuration, and wait until they are all available for dequeing before
+considering the next frame. This defeats the purpose of having buffer queues
+since in practice only one buffer would be queued at a time.
+
+The Request API allows a specific configuration of the pipeline (media
+controller topology + controls for each device) to be associated with specific
+buffers. The parameters are applied by each participating device as buffers
+associated to a request flow in. This allows user-space to schedule several
+tasks ("requests") with different parameters in advance, knowing that the
+parameters will be applied when needed to get the expected result. Controls
+values at the time of request completion are also available for reading.
+
+Usage
+=====
+
+The Request API is used on top of standard media controller and V4L2 calls,
+which are augmented with an extra ``request_fd`` parameter. All operations on
+requests themselves are performed using the command parameter of the
+:c:func:`MEDIA_IOC_REQUEST_CMD` ioctl.
+
+Request Allocation
+------------------
+
+User-space allocates requests using the ``MEDIA_REQ_CMD_ALLOC`` command on
+an opened media device. This returns a file descriptor representing the
+request. Typically, several such requests will be allocated.
+
+Request Preparation
+-------------------
+
+Standard V4L2 ioctls can then receive a request file descriptor to express the
+fact that the ioctl is part of said request, and is not to be applied
+immediately. V4L2 ioctls supporting this are :c:func:`VIDIOC_S_EXT_CTRLS` and
+:c:func:`VIDIOC_QBUF`. Controls set with a request parameter are stored instead
+of being immediately applied, and queued buffers will block the buffer queue
+until the request becomes active.
+
+RFC Note: currently several buffers can be queued to the same queue with the
+same request. The request parameters will be only be applied when processing
+the first buffer. Does it make more sense to allow at most one buffer per
+request per queue instead?
+
+Request Submission
+------------------
+
+Once the parameters and buffers of the request are specified, it can be
+submitted with the ``MEDIA_REQ_CMD_SUBMIT`` command. This will make the buffers
+associated to the request available to their driver, which can then apply the
+saved controls as buffers are processed. A submitted request cannot be modified
+anymore.
+
+If several devices are part of the request, individual drivers may synchronize
+so the requested pipeline's topology is applied before the buffers are
+processed. This is at the discretion of the drivers and is not a requirement.
+
+Buffers queued without an associated request after a request-bound buffer will
+be processed using the state of the hardware at the time of the request
+completion. All the same, controls set without a request are applied
+immediately, regardless of whether a request is in use or not.
+
+User-space can ``poll()`` a request FD in order to wait until the request
+completes. A request is considered complete once all its associated buffers are
+available for dequeing. Note that user-space does not need to wait for the
+request to complete to dequeue its buffers: buffers that are available halfway
+through a request can be dequeued independently of the request's state.
+
+A completed request includes the state of all devices that had queued buffers
+associated with it at the time of the request completion. User-space can query
+that state by calling :c:func:`VIDIOC_G_EXT_CTRLS` with the request FD.
+
+RFC Note: requests are currently processed in buffer order, meaning that they
+do not necessarily start being processed in the order of their submission. Also,
+requests do not need to complete by processing order: a driver can decide to
+hold the buffer associated to a request in order to process the few next ones,
+which will result in the first request completing after the others.
+
+Recycling and Destruction
+-------------------------
+
+Finally, completed request can either be discarded or be reused. Calling
+``close()`` on a request FD will make that FD unusable, freeing the request if
+it is not referenced elsewhere. The ``MEDIA_REQ_CMD_REINIT`` command will clear
+a request's state and make it available again. No state is retained by this
+operation: the request is as if it had just been allocated via
+``MEDIA_REQ_CMD_ALLOC``.
+
+RFC Note: Since requests are represented by FDs themselves, the
+``MEDIA_REQ_CMD_SUBMIT`` and ``MEDIA_REQ_CMD_REININT`` commands can be performed
+on the request FD instead of the media device. This means the media device would
+only need to manage ``MEDIA_REQ_CMD_ALLOC``, which could be turned into an
+ioctl, while ``MEDIA_REQ_CMD_SUBMIT`` and ``MEDIA_REQ_CMD_REININT`` would
+become ioctls on the request itself. This has the advantage of allowing
+receivers of a request FD to submit the request, and also decouples requests
+from the media device - a scenario that makes sense for stateless codecs where
+the media device is not really useful.
+
+Example for a Codec Device
+--------------------------
+
+Codecs are single-node V4L2 devices providing one OUTPUT queue (for user-space
+to provide input buffers) and one CAPTURE queue (to retrieve processed data).
+
+In this use-case, request API is used to associate specific controls to be
+applied by the driver before processing a buffer from the OUTPUT queue. When
+retrieving a buffer from a capture queue, user-space can then identify which
+request, if any, produced it by looking at the request_fd field of the dequeued
+v4l2_buffer.
+
+Put into code, after obtaining a request, user-space can assign controls and one
+OUTPUT buffer to it:
+
+	struct v4l2_buf buf;
+	struct v4l2_ext_controls ctrls;
+	struct media_request_cmd cmd;
+	int req_fd;
+	...
+	cmd.cmd = MEDIA_REQ_CMD_ALLOC;
+	ioctl(media_fd, MEDIA_IOC_REQUEST_CMD, &cmd);
+	req_fd = cmd.fd;
+	...
+	ctrls.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
+	...
+	buf.type = V4L2_BUF_TYPE_VIDEO_OUTPUT;
+	buf.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_QBUF, &buf);
+
+Note that request_fd shall not be specified for CAPTURE buffers: these are
+queued normally, and will be stamped with the request that produced them.
+
+Once the request is fully prepared, it can be submitted to the driver:
+
+	struct media_request_cmd cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.cmd = MEDIA_REQ_CMD_SUBMIT;
+	cmd.fd = request_fd;
+	ioctl(media_fd, MEDIA_IOC_REQUEST_CMD, &cmd);
+
+User-space can then lookup the request_fd field of dequeued capture buffers to
+confirm which one has been produced by the request.
+
+	struct v4l2_buf buf;
+
+	memset(&buf, 0, sizeof(buf));
+	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	ioctl(codec_fd, VIDIOC_DQBUF, &buf);
+
+	if (buf.request_fd == request_fd) {
+		...
+
+We can then, after ensuring that the request is completed via polling the
+request FD, query control values at the time the request completed via an
+annotated call to G_EXT_CTRLS. This is particularly useful for volatile controls
+for which we want to query values as soon as the capture buffer is produced.
+
+	struct pollfd pfd = { .events = POLLIN, .fd = request_fd };
+	poll(&pfd, 1, -1);
+	...
+	ctrls.request_fd = req_fd;
+	ioctl(codec_fd, VIDIOC_G_EXT_CTRLS, &ctrls);
+
+Once we don't need the request anymore, we can either recycle it for reuse with
+MEDIA_REQ_CMD_REINIT...
+
+	cmd.cmd = MEDIA_REQ_CMD_REINIT;
+	cmd.fd = request_fd;
+	ioctl(media_fd, MEDIA_IOC_REQUEST_CMD, &cmd);
+
+... or close its file descriptor to completely dispose of it.
+
+	close(request_fd);
+
+Example for a Simple Capture Device
+-----------------------------------
+
+With a simple capture device, requests can be used to specify controls to apply
+to a given CAPTURE buffer. The driver will apply these controls before producing
+the marked CAPTURE buffer.
+
+	struct v4l2_buf buf;
+	struct v4l2_ext_controls ctrls;
+	struct media_request_cmd cmd;
+	int req_fd;
+	...
+	cmd.cmd = MEDIA_REQ_CMD_ALLOC;
+	ioctl(media_fd, MEDIA_IOC_REQUEST_CMD, &cmd);
+	req_fd = cmd.fd;
+	...
+	ctrls.request_fd = req_fd;
+	ioctl(camera_fd, VIDIOC_S_EXT_CTRLS, &ctrls);
+	...
+	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	buf.request_fd = req_fd;
+	ioctl(camera_fd, VIDIOC_QBUF, &buf);
+
+Once the request is fully prepared, it can be submitted to the driver:
+
+	struct media_request_cmd cmd;
+
+	memset(&cmd, 0, sizeof(cmd));
+	cmd.cmd = MEDIA_REQ_CMD_SUBMIT;
+	cmd.fd = request_fd;
+	ioctl(media_fd, MEDIA_IOC_REQUEST_CMD, &cmd);
+
+User-space can then lookup the request_fd field of dequeued capture buffers to
+confirm which one has been produced by the request.
+
+	struct v4l2_buf buf;
+
+	memset(&buf, 0, sizeof(buf));
+	buf.type = V4L2_BUF_TYPE_VIDEO_CAPTURE;
+	ioctl(camera_fd, VIDIOC_DQBUF, &buf);
+
+	if (buf.request_fd == request_fd) {
+		...
+
+Use of G_EXT_CTRLS, request reinitialization and closing works as with the codec
+example above.
diff --git a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
index 2011c2b2ee67..b0a9d0e89ba0 100644
--- a/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
+++ b/Documentation/media/uapi/v4l/vidioc-g-ext-ctrls.rst
@@ -95,6 +95,17 @@ appropriate. In the first case the new value is set in struct
 is inappropriate (e.g. the given menu index is not supported by the menu
 control), then this will also result in an ``EINVAL`` error code error.
 
+If ``request_fd`` is set to a valid request file descriptor, then the controls
+will not be applied immediately when calling
+:ref:`VIDIOC_S_EXT_CTRLS <VIDIOC_S_EXT_CTRLS>`, but instead are applied right
+before the driver starts processing a buffer associated to the same request.
+
+All the same, if ``request_fd`` is specified during a call to
+:ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`, then the returned values will
+be the values currently set for the request (or the hardware value if none is
+set) if the request has not yet completed, or the values of the controls at the
+time of request completion if it has already completed.
+
 The driver will only set/get these controls if all control values are
 correct. This prevents the situation where only some of the controls
 were set/get. Only low-level errors (e. g. a failed i2c command) can
@@ -272,8 +283,11 @@ still cause this situation.
 	then you can call :ref:`VIDIOC_TRY_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>` to try to discover the
 	actual control that failed the validation step. Unfortunately,
 	there is no ``TRY`` equivalent for :ref:`VIDIOC_G_EXT_CTRLS <VIDIOC_G_EXT_CTRLS>`.
+    * - __s32
+      - ``request_fd``
+	File descriptor of the request to be used by this operation (0 if none).
     * - __u32
-      - ``reserved``\ [2]
+      - ``reserved``\ [1]
       - Reserved for future extensions.
 
 	Drivers and applications must set the array to zero.
diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
index 9e448a4aa3aa..d7fea37f32e7 100644
--- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
+++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
@@ -98,6 +98,12 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
 :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
 device is closed.
 
+For all buffer types, the ``request_fd`` field can be used when enqueing
+to specify the file descriptor of a request, if requests are in use.
+Setting it means that the buffer will not be passed to the driver until
+the request itself is submitted. Also, the driver will apply any setting
+associated with the request before processing the buffer.
+
 Applications call the ``VIDIOC_DQBUF`` ioctl to dequeue a filled
 (capturing) or displayed (output) buffer from the driver's outgoing
 queue. They just set the ``type``, ``memory`` and ``reserved`` fields of
@@ -115,6 +121,21 @@ queue. When the ``O_NONBLOCK`` flag was given to the
 :ref:`open() <func-open>` function, ``VIDIOC_DQBUF`` returns
 immediately with an ``EAGAIN`` error code when no buffer is available.
 
+If a dequeued buffer was produced as the result of a request, then the
+``request_fd`` field of :c:type:`v4l2_buffer` will be set to the file
+descriptor of the request, regardless of whether the buffer was initially
+queued with a request associated to it or not.
+
+RFC note: a request can be referenced by several FDs, especially if the
+request is shared between different processes. Also a FD can be closed
+after a request is submitted. In that case the FD does not make much sense.
+Maybe it would be better to define request fd as
+
+    union { s32 request_fd; s32 request_id; }
+
+and use request_id when communicating a request to userspace so they can be
+unambiguously referenced?
+
 The struct :c:type:`v4l2_buffer` structure is specified in
 :ref:`buffer`.
 
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 16/17] media: vim2m: add media device

[Alexandre Courbot]

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/platform/vim2m.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..e0eb60310717 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
 	struct v4l2_device	v4l2_dev;
 	struct video_device	vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_device	mdev;
+#endif
 
 	atomic_t		num_inst;
 	struct mutex		dev_mutex;
@@ -1001,6 +1004,13 @@ static int vim2m_probe(struct platform_device *pdev)
 
 	spin_lock_init(&dev->irqlock);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->mdev.dev = &pdev->dev;
+	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+	media_device_init(&dev->mdev);
+	dev->v4l2_dev.mdev = &dev->mdev;
+#endif
+
 	ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
 	if (ret)
 		return ret;
@@ -1034,6 +1044,13 @@ static int vim2m_probe(struct platform_device *pdev)
 		goto err_m2m;
 	}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	/* Register the media device node */
+	ret = media_device_register(&dev->mdev);
+	if (ret)
+		goto err_m2m;
+#endif
+
 	return 0;
 
 err_m2m:
@@ -1050,6 +1067,13 @@ static int vim2m_remove(struct platform_device *pdev)
 	struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
 	v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	if (media_devnode_is_registered(dev->mdev.devnode))
+		media_device_unregister(&dev->mdev);
+	media_device_cleanup(&dev->mdev);
+#endif
+
 	v4l2_m2m_release(dev->m2m_dev);
 	del_timer_sync(&dev->timer);
 	video_unregister_device(&dev->vfd);
-- 
2.16.0.rc1.238.g530d649a79-goog

[RFCv3 17/17] media: vim2m: add request support

[Alexandre Courbot]

Set the necessary ops for supporting requests in vim2m.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/platform/vim2m.c | 55 ++++++++++++++++++++++++++++++++++++++++++
 1 file changed, 55 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index e0eb60310717..96dba60d3c74 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -30,6 +30,9 @@
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-event.h>
 #include <media/videobuf2-vmalloc.h>
+#include <media/media-request.h>
+#include <media/media-request-mgr.h>
+#include <media/v4l2-request.h>
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
@@ -370,6 +373,28 @@ static void job_abort(void *priv)
 	ctx->aborting = 1;
 }
 
+static int apply_request_params(struct media_request *req,
+				 struct vim2m_ctx *ctx)
+{
+	struct media_request_entity_data *_data;
+	struct media_request_v4l2_entity_data *data;
+
+	if (!req)
+		return -EINVAL;
+
+	_data = media_request_get_entity_data(req, &ctx->dev->vfd.entity,
+					      &ctx->fh);
+	if (WARN_ON(!_data))
+		return -EINVAL;
+	data = container_of(_data, struct media_request_v4l2_entity_data, base);
+
+	v4l2_ctrl_request_setup(&data->ctrls);
+
+	data->base.applied = true;
+
+	return 0;
+}
+
 /* device_run() - prepares and starts the device
  *
  * This simulates all the immediate preparations required before starting
@@ -381,12 +406,22 @@ static void device_run(void *priv)
 	struct vim2m_ctx *ctx = priv;
 	struct vim2m_dev *dev = ctx->dev;
 	struct vb2_v4l2_buffer *src_buf, *dst_buf;
+	int ret;
 
 	src_buf = v4l2_m2m_next_src_buf(ctx->fh.m2m_ctx);
 	dst_buf = v4l2_m2m_next_dst_buf(ctx->fh.m2m_ctx);
 
+	WARN_ON(dst_buf->vb2_buf.request != NULL);
+
+	/* Apply request if needed */
+	ret = apply_request_params(src_buf->vb2_buf.request, ctx);
+	WARN_ON(ret);
+
 	device_process(ctx, src_buf, dst_buf);
 
+	/* Inform user about which request produced the destination buffer */
+	dst_buf->request_fd = src_buf->request_fd;
+
 	/* Run a timer, which simulates a hardware irq  */
 	schedule_irq(dev, ctx->transtime);
 }
@@ -841,6 +876,7 @@ static int queue_init(void *priv, struct vb2_queue *src_vq, struct vb2_queue *ds
 	src_vq->mem_ops = &vb2_vmalloc_memops;
 	src_vq->timestamp_flags = V4L2_BUF_FLAG_TIMESTAMP_COPY;
 	src_vq->lock = &ctx->dev->dev_mutex;
+	src_vq->allow_requests = true;
 
 	ret = vb2_queue_init(src_vq);
 	if (ret)
@@ -992,6 +1028,19 @@ static const struct v4l2_m2m_ops m2m_ops = {
 	.job_abort	= job_abort,
 };
 
+struct media_request_entity_data *vim2m_entity_data_alloc(struct media_entity *entity,
+							  void *fh)
+{
+	struct vim2m_ctx *ctx = container_of(fh, struct vim2m_ctx, fh);
+
+	return media_request_v4l2_entity_data_alloc(&ctx->hdl);
+}
+
+static const struct media_entity_request_ops vim2m_entity_req_ops = {
+	.data_alloc	= vim2m_entity_data_alloc,
+	.data_free	= media_request_v4l2_entity_data_free,
+};
+
 static int vim2m_probe(struct platform_device *pdev)
 {
 	struct vim2m_dev *dev;
@@ -1006,6 +1055,9 @@ static int vim2m_probe(struct platform_device *pdev)
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 	dev->mdev.dev = &pdev->dev;
+	dev->mdev.req_mgr = media_request_mgr_alloc(&dev->mdev);
+	if (IS_ERR(dev->mdev.req_mgr))
+		return PTR_ERR(dev->mdev.req_mgr);
 	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
 	media_device_init(&dev->mdev);
 	dev->v4l2_dev.mdev = &dev->mdev;
@@ -1030,6 +1082,9 @@ static int vim2m_probe(struct platform_device *pdev)
 	}
 
 	video_set_drvdata(vfd, dev);
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->vfd.entity.req_ops = &vim2m_entity_req_ops;
+#endif
 	snprintf(vfd->name, sizeof(vfd->name), "%s", vim2m_videodev.name);
 	v4l2_info(&dev->v4l2_dev,
 			"Device registered as /dev/video%d\n", vfd->num);
-- 
2.16.0.rc1.238.g530d649a79-goog

Re: [RFCv3 01/17] media: add request API core and UAPI

[Sakari Ailus]

Hi Alexandre,

On Wed, Feb 07, 2018 at 10:48:05AM +0900, Alexandre Courbot wrote:
> The request API provides a way to group buffers and device parameters
> into units of work to be queued and executed. This patch introduces the
> UAPI and core framework.
> 
> This patch is based on the previous work by Laurent Pinchart. The core
> has changed considerably, but the UAPI is mostly untouched.

Thanks for the rebase.

What's the purpose of the split between media-request.c and
media-request-mgr.c? The use of ops for the request manager suggests it
could be replaced by an alternative implementation but then again the media
request is almost entirely initialised in the media-request-mgr. Either
could make some sense but not both --- I'd simply move the code from the
media-request-mgr to the functions calling them in media-request.c. That
should make this easier to read, too.

A few more comments below.

> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/Makefile               |   3 +-
>  drivers/media/media-device.c         |   7 +
>  drivers/media/media-request-mgr.c    | 105 ++++++++++++
>  drivers/media/media-request.c        | 311 +++++++++++++++++++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
>  include/media/media-device.h         |   3 +
>  include/media/media-entity.h         |   9 +
>  include/media/media-request-mgr.h    |  73 ++++++++
>  include/media/media-request.h        | 186 +++++++++++++++++++++
>  include/uapi/linux/media.h           |  10 ++
>  10 files changed, 707 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/media/media-request-mgr.c
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request-mgr.h
>  create mode 100644 include/media/media-request.h
> 
> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> index 594b462ddf0e..06c43ddb52ea 100644
> --- a/drivers/media/Makefile
> +++ b/drivers/media/Makefile
> @@ -3,7 +3,8 @@
>  # Makefile for the kernel multimedia device drivers.
>  #
>  
> -media-objs	:= media-device.o media-devnode.o media-entity.o
> +media-objs	:= media-device.o media-devnode.o media-entity.o \
> +		   media-request.o media-request-mgr.o
>  
>  #
>  # I2C drivers should come before other drivers, otherwise they'll fail
> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> index e79f72b8b858..024ee81a8334 100644
> --- a/drivers/media/media-device.c
> +++ b/drivers/media/media-device.c
> @@ -32,6 +32,8 @@
>  #include <media/media-device.h>
>  #include <media/media-devnode.h>
>  #include <media/media-entity.h>
> +#include <media/media-request.h>
> +#include <media/media-request-mgr.h>
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  
> @@ -407,6 +409,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
> +	MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
>  };
>  
>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
> @@ -688,6 +691,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
>  
>  void media_device_cleanup(struct media_device *mdev)
>  {
> +	if (mdev->req_mgr) {
> +		media_request_mgr_free(mdev->req_mgr);
> +		mdev->req_mgr = NULL;
> +	}
>  	ida_destroy(&mdev->entity_internal_idx);
>  	mdev->entity_internal_idx_max = 0;
>  	media_graph_walk_cleanup(&mdev->pm_count_walk);
> diff --git a/drivers/media/media-request-mgr.c b/drivers/media/media-request-mgr.c
> new file mode 100644
> index 000000000000..686e877a884b
> --- /dev/null
> +++ b/drivers/media/media-request-mgr.c
> @@ -0,0 +1,105 @@
> +/*
> + * Generic request manager implementation.
> + *
> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/device.h>
> +#include <linux/slab.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-request.h>
> +#include <media/media-request-mgr.h>
> +
> +static struct media_request *
> +media_request_alloc(struct media_request_mgr *mgr)
> +{
> +	struct media_request *req;
> +
> +	req = kzalloc(sizeof(*req), GFP_KERNEL);
> +	if (!req)
> +		return ERR_PTR(-ENOMEM);
> +
> +	req->mgr = mgr;
> +	req->state = MEDIA_REQUEST_STATE_IDLE;
> +	kref_init(&req->kref);
> +	INIT_LIST_HEAD(&req->data);
> +	init_waitqueue_head(&req->complete_wait);
> +	ATOMIC_INIT_NOTIFIER_HEAD(&req->submit_notif);
> +	mutex_init(&req->lock);
> +
> +	mutex_lock(&mgr->mutex);
> +	req->id = ++mgr->req_id;
> +	list_add_tail(&req->list, &mgr->requests);
> +	mutex_unlock(&mgr->mutex);
> +
> +	return req;
> +}
> +
> +static void media_request_free(struct media_request *req)
> +{
> +	struct media_request_mgr *mgr = req->mgr;
> +	struct media_request_entity_data *data, *next;
> +
> +	mutex_lock(&mgr->mutex);
> +	list_del(&req->list);
> +	mutex_unlock(&mgr->mutex);
> +
> +	list_for_each_entry_safe(data, next, &req->data, list) {
> +		list_del(&data->list);
> +		data->entity->req_ops->data_free(data);
> +	}
> +
> +	kfree(req);
> +}
> +
> +void media_request_mgr_free(struct media_request_mgr *mgr)
> +{
> +	struct media_device *mdev = mgr->mdev;
> +
> +	/* Just a sanity check - we should have no remaining requests */
> +	while (!list_empty(&mgr->requests)) {
> +		struct media_request *req;
> +
> +		req = list_first_entry(&mgr->requests, typeof(*req), list);
> +		dev_warn(mdev->dev,
> +			"%s: request %u still referenced, deleting forcibly!\n",
> +			__func__, req->id);
> +		mgr->ops->req_free(req);
> +	}
> +
> +	kfree(mgr);
> +}
> +EXPORT_SYMBOL_GPL(media_request_mgr_free);
> +
> +static const struct media_request_mgr_ops request_mgr_generic_ops = {
> +	.req_alloc = media_request_alloc,
> +	.req_free = media_request_free,
> +};
> +
> +struct media_request_mgr *
> +media_request_mgr_alloc(struct media_device *mdev)
> +{
> +	struct media_request_mgr *mgr;
> +
> +	mgr = kzalloc(sizeof(*mgr), GFP_KERNEL);
> +	if (!mgr)
> +		return ERR_PTR(-ENOMEM);
> +
> +	mgr->mdev = mdev;
> +	mutex_init(&mgr->mutex);
> +	INIT_LIST_HEAD(&mgr->requests);
> +	mgr->ops = &request_mgr_generic_ops;
> +
> +	return mgr;
> +}
> +EXPORT_SYMBOL_GPL(media_request_mgr_alloc);
> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> new file mode 100644
> index 000000000000..30a23235b019
> --- /dev/null
> +++ b/drivers/media/media-request.c
> @@ -0,0 +1,311 @@
> +/*
> + * Request base implementation
> + *
> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/anon_inodes.h>
> +#include <linux/media.h>
> +#include <linux/fs.h>
> +#include <linux/file.h>
> +#include <linux/slab.h>
> +#include <linux/list.h>
> +
> +#include <media/media-device.h>
> +#include <media/media-request.h>
> +#include <media/media-request-mgr.h>
> +
> +const struct file_operations request_fops;
> +
> +struct media_request *media_request_get(struct media_request *req)
> +{
> +	kref_get(&req->kref);
> +	return req;
> +}
> +EXPORT_SYMBOL_GPL(media_request_get);
> +
> +struct media_request *
> +media_request_get_from_fd(int fd)
> +{
> +	struct file *f;
> +	struct media_request *req;
> +
> +	f = fget(fd);
> +	if (!f)
> +		return NULL;
> +
> +	/* Not a request FD? */
> +	if (f->f_op != &request_fops) {
> +		fput(f);
> +		return NULL;
> +	}
> +
> +	req = f->private_data;
> +	media_request_get(req);
> +	fput(f);
> +
> +	return req;
> +}
> +EXPORT_SYMBOL_GPL(media_request_get_from_fd);
> +
> +static void media_request_release(struct kref *kref)
> +{
> +	struct media_request *req =
> +		container_of(kref, typeof(*req), kref);
> +
> +	req->mgr->ops->req_free(req);
> +}
> +
> +void media_request_put(struct media_request *req)
> +{
> +	if (WARN_ON(req == NULL))
> +		return;
> +
> +	kref_put(&req->kref, media_request_release);
> +}
> +EXPORT_SYMBOL_GPL(media_request_put);
> +
> +struct media_request_entity_data *
> +media_request_get_entity_data(struct media_request *req,
> +			      struct media_entity *entity, void *fh)
> +{
> +	struct media_request_entity_data *data;
> +
> +	mutex_lock(&req->lock);
> +
> +	/* First look whether we already have entity data */
> +	list_for_each_entry(data, &req->data, list) {
> +		if (data->entity == entity) {
> +			/*
> +			 * If so, then the fh must match otherwise this means
> +			 * we are trying to set the same entity through
> +			 * different handles
> +			 */
> +			if (data->fh != fh)
> +				data = ERR_PTR(-EINVAL);
> +			goto done;
> +		}
> +	}
> +
> +	/* No entity data found, let's create it */
> +	data = entity->req_ops->data_alloc(entity, fh);
> +	if (IS_ERR(data))
> +		goto done;
> +	data->fh = fh;
> +	data->entity = entity;
> +	list_add_tail(&data->list, &req->data);
> +
> +done:
> +	mutex_unlock(&req->lock);
> +	return data;
> +}
> +EXPORT_SYMBOL_GPL(media_request_get_entity_data);
> +
> +static const char * const media_request_states[] __maybe_unused = {
> +	"IDLE",
> +	"SUBMITTED",
> +	"COMPLETED",
> +	"DELETED",
> +};
> +
> +static const char *media_request_state(enum media_request_state state)
> +{
> +	return state < ARRAY_SIZE(media_request_states) ?
> +		media_request_states[state] : "INVALID";
> +}
> +
> +static int media_device_request_close(struct inode *inode, struct file *filp)
> +{
> +	struct media_request *req = filp->private_data;
> +
> +	if (req == NULL)
> +		return 0;
> +
> +	media_request_put(req);
> +
> +	return 0;
> +}
> +
> +static unsigned int media_request_poll(struct file *file, poll_table *wait)
> +{
> +	struct media_request *req = file->private_data;
> +
> +	poll_wait(file, &req->complete_wait, wait);
> +
> +	if (req->state == MEDIA_REQUEST_STATE_COMPLETED)
> +		return POLLIN | POLLRDNORM;
> +
> +	return 0;
> +}
> +
> +const struct file_operations request_fops = {
> +	.owner = THIS_MODULE,
> +	.poll = media_request_poll,
> +	.release = media_device_request_close,
> +};
> +
> +void media_request_complete(struct media_request *req)
> +{
> +	struct media_request_mgr *mgr = req->mgr;
> +	struct media_device *mdev = mgr->mdev;
> +
> +	mutex_lock(&req->lock);
> +
> +	if (WARN_ON(req->state != MEDIA_REQUEST_STATE_SUBMITTED)) {
> +		dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
> +			__func__, req->id, media_request_state(req->state));
> +		mutex_unlock(&req->lock);
> +		return;
> +	}
> +
> +	req->state = MEDIA_REQUEST_STATE_COMPLETED;
> +
> +	if (mgr->ops->req_complete)
> +		mgr->ops->req_complete(req);
> +
> +	wake_up_interruptible(&req->complete_wait);
> +
> +	mutex_unlock(&req->lock);
> +
> +	/* Release the reference acquired when we submitted the request */
> +	media_request_put(req);
> +}
> +EXPORT_SYMBOL_GPL(media_request_complete);
> +
> +/*
> + * Process the MEDIA_REQ_CMD_ALLOC command
> + */
> +static int media_request_cmd_alloc(struct media_request_mgr *mgr,
> +				   struct media_request_cmd *cmd)
> +{
> +	struct media_request *req;
> +	int fd;
> +
> +	req = mgr->ops->req_alloc(mgr);
> +	if (!req)
> +		return -ENOMEM;
> +
> +	fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
> +	if (fd < 0)
> +		return fd;
> +
> +	cmd->fd = fd;
> +
> +	return 0;
> +}
> +
> +/*
> + * Process the MEDIA_REQ_CMD_SUBMIT command
> + */
> +static int media_request_cmd_submit(struct media_request *req)
> +{
> +	mutex_lock(&req->lock);
> +
> +	if (req->state != MEDIA_REQUEST_STATE_IDLE) {
> +		dev_dbg(req->mgr->mdev->dev,
> +			"%s: unable to submit request in state %s\n",
> +			__func__, media_request_state(req->state));
> +		mutex_unlock(&req->lock);
> +		return -EINVAL;
> +	}
> +
> +	if (atomic_read(&req->buf_cpt) == 0) {
> +		dev_dbg(req->mgr->mdev->dev,
> +			"%s: request has no buffers!\n", __func__);
> +		mutex_unlock(&req->lock);
> +		return -EINVAL;
> +	}
> +
> +	req->state = MEDIA_REQUEST_STATE_SUBMITTED;
> +
> +	/* Hold a reference to the request until it is completed */
> +	media_request_get(req);
> +
> +	mutex_unlock(&req->lock);
> +
> +	atomic_notifier_call_chain(&req->submit_notif, req->state, req);
> +
> +	return 0;
> +}
> +
> +static int media_request_cmd_reinit(struct media_request *req)
> +{
> +	struct media_request_entity_data *data, *next;
> +
> +	mutex_lock(&req->lock);
> +
> +	if (req->state == MEDIA_REQUEST_STATE_SUBMITTED) {
> +		dev_dbg(req->mgr->mdev->dev,
> +			"%s: unable to reinit submitted request\n", __func__);
> +		mutex_unlock(&req->lock);
> +		return -EINVAL;
> +	}
> +
> +	/* delete all entity data */
> +	list_for_each_entry_safe(data, next, &req->data, list) {
> +		list_del(&data->list);
> +		data->entity->req_ops->data_free(data);
> +	}
> +
> +	/* reinitialize request to idle state */
> +	req->state = MEDIA_REQUEST_STATE_IDLE;
> +	atomic_set(&req->buf_cpt, 0);
> +
> +	mutex_unlock(&req->lock);
> +
> +	return 0;
> +}
> +
> +long media_device_request_cmd(struct media_device *mdev,
> +			      struct media_request_cmd *cmd)
> +{
> +	struct media_request *req = NULL;
> +	int ret;
> +
> +	if (!mdev->req_mgr)
> +		return -ENOTTY;
> +
> +	if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
> +		req = media_request_get_from_fd(cmd->fd);
> +		if (IS_ERR(req))
> +			return PTR_ERR(req);
> +
> +		/* requests must belong to this media device's manager */
> +		if (req->mgr != mdev->req_mgr) {
> +			media_request_put(req);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	switch (cmd->cmd) {
> +	case MEDIA_REQ_CMD_ALLOC:
> +		ret = media_request_cmd_alloc(mdev->req_mgr, cmd);
> +		break;
> +
> +	case MEDIA_REQ_CMD_SUBMIT:
> +		ret = media_request_cmd_submit(req);
> +		break;
> +
> +	case MEDIA_REQ_CMD_REINIT:
> +		ret = media_request_cmd_reinit(req);
> +		break;
> +
> +	default:
> +		ret = -EINVAL;
> +		break;
> +	}
> +
> +	if (req)
> +		media_request_put(req);
> +
> +	return ret;
> +}
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 260288ca4f55..e5109e5b8bf5 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -870,7 +870,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
>  	__u32 i;
>  
>  	/* zero the reserved fields */
> -	c->reserved[0] = c->reserved[1] = 0;
> +	c->reserved[0] = 0;
>  	for (i = 0; i < c->count; i++)
>  		c->controls[i].reserved2[0] = 0;
>  
> diff --git a/include/media/media-device.h b/include/media/media-device.h
> index bcc6ec434f1f..f2d471b8a53f 100644
> --- a/include/media/media-device.h
> +++ b/include/media/media-device.h
> @@ -27,6 +27,7 @@
>  
>  struct ida;
>  struct device;
> +struct media_request_mgr;
>  
>  /**
>   * struct media_entity_notify - Media Entity Notify
> @@ -158,6 +159,8 @@ struct media_device {
>  	void (*disable_source)(struct media_entity *entity);
>  
>  	const struct media_device_ops *ops;
> +
> +	struct media_request_mgr *req_mgr;
>  };
>  
>  /* We don't need to include pci.h or usb.h here */
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index a732af1dbba0..39f5e88e2b23 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -172,6 +172,8 @@ struct media_pad {
>  	unsigned long flags;
>  };
>  
> +struct media_request;
> +struct v4l2_ext_controls;
>  /**
>   * struct media_entity_operations - Media entity operations
>   * @get_fwnode_pad:	Return the pad number based on a fwnode endpoint or
> @@ -197,6 +199,11 @@ struct media_entity_operations {
>  	int (*link_validate)(struct media_link *link);
>  };
>  
> +struct media_entity_request_ops {
> +	struct media_request_entity_data *(*data_alloc)(struct media_entity *entity, void *fh);
> +	void (*data_free)(struct media_request_entity_data *data);
> +};
> +
>  /**
>   * enum media_entity_type - Media entity type
>   *
> @@ -281,6 +288,8 @@ struct media_entity {
>  
>  	const struct media_entity_operations *ops;
>  
> +	const struct media_entity_request_ops *req_ops;
> +
>  	int stream_count;
>  	int use_count;
>  
> diff --git a/include/media/media-request-mgr.h b/include/media/media-request-mgr.h
> new file mode 100644
> index 000000000000..a3161fa2add0
> --- /dev/null
> +++ b/include/media/media-request-mgr.h
> @@ -0,0 +1,73 @@
> +/*
> + * Generic request manager.
> + *
> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef _MEDIA_REQUEST_MGR_H
> +#define _MEDIA_REQUEST_MGR_H
> +
> +#include <linux/mutex.h>
> +
> +struct media_request_mgr;
> +
> +/**
> + * struct media_request_mgr_ops - request manager operations
> + *
> + * @req_alloc:	allocate a request
> + * @req_free:	free a previously allocated request
> + * @req_complete: callback invoked when a request has completed
> + *
> + */
> +struct media_request_mgr_ops {
> +	struct media_request *(*req_alloc)(struct media_request_mgr *mgr);
> +	void (*req_free)(struct media_request *req);
> +	void (*req_complete)(struct media_request *req);
> +};
> +
> +/**
> + * struct media_request_mgr - requests manager
> + *
> + * @mdev:	media_device owning this manager
> + * @ops:	implementation of the manager
> + * @mutex:	protects requests, active_request, req_id, and all members of
> + *		struct media_request

The media request has a mutex as well. Is this outdated?

> + * @requests:	list of alive requests produced by this manager
> + * @req_id:	counter used to identify requests for debugging purposes
> + */
> +struct media_request_mgr {
> +	struct media_device *mdev;
> +	const struct media_request_mgr_ops *ops;
> +
> +	struct mutex mutex;
> +	struct list_head requests;
> +	u32 req_id;
> +};
> +
> +/**
> + * media_request_mgr_alloc() - return an instance of the generic manager
> + *
> + * @mdev:	owning media device
> + */
> +struct media_request_mgr *
> +media_request_mgr_alloc(struct media_device *mdev);
> +
> +/**
> + * media_request_mgr_free() - free a media manager
> + *
> + * This should only be called when all requests produced by this manager
> + * has been destroyed. Will warn if that is not the case.
> + *
> + */
> +void media_request_mgr_free(struct media_request_mgr *mgr);
> +
> +#endif
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> new file mode 100644
> index 000000000000..817df13ef6e3
> --- /dev/null
> +++ b/include/media/media-request.h
> @@ -0,0 +1,186 @@
> +/*
> + * Media requests.
> + *
> + * Copyright (C) 2018, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef _MEDIA_REQUEST_H
> +#define _MEDIA_REQUEST_H
> +
> +#include <linux/kref.h>
> +#include <linux/list.h>
> +#include <linux/notifier.h>
> +#include <linux/wait.h>
> +
> +struct media_device;
> +struct media_entity;
> +struct media_request_cmd;
> +struct media_request_entity_data;
> +struct media_request_mgr;
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +
> +enum media_request_state {
> +	MEDIA_REQUEST_STATE_IDLE,
> +	MEDIA_REQUEST_STATE_SUBMITTED,
> +	MEDIA_REQUEST_STATE_COMPLETED,
> +	MEDIA_REQUEST_STATE_DELETED,
> +};
> +
> +/**
> + * struct media_request - Media request base structure
> + * @id:		request id, used internally for debugging
> + * @mgr:	manager this request belongs to
> + * @kref:	reference count
> + * @list:	list entry in the media device requests list
> + * @lock:	protects internal state against concurrent accesses
> + * @state:	current state of the request
> + * @data:	per-entity data list
> + * @complete_wait:	wait queue that signals once the request has completed
> + * @submit_notif:	notification list to call when the request is submitted
> + * @buf_cpt:	counter of queued/completed buffers used to decide when the
> + *		request is completed
> + */
> +struct media_request {
> +	u32 id;
> +	struct media_request_mgr *mgr;
> +	struct kref kref;
> +	struct list_head list;
> +
> +	struct mutex lock;
> +	enum media_request_state state;
> +	struct list_head data;
> +	wait_queue_head_t complete_wait;
> +	struct atomic_notifier_head submit_notif;
> +	atomic_t buf_cpt;
> +};
> +
> +/**
> + * media_request_get() - increment the reference counter of a request
> + *
> + * The calling context must call media_request_put() once it does not need
> + * the reference to the request anymore.
> + *
> + * Returns the request that has been passed as argument.
> + *
> + * @req:	request to acquire a reference of
> + */
> +struct media_request *media_request_get(struct media_request *req);
> +
> +/**
> + * media_request_get_from_fd() - lookup request by fd and acquire a reference.
> + *
> + * Look a request up from its fd, acquire a reference and return a pointer to
> + * the request. As for media_request_get(), media_request_put() must be called
> + * once the reference is not used anymore.
> + *
> + * @req:	request to lookup and acquire.
> + *
> + */
> +struct media_request *media_request_get_from_fd(int fd);
> +
> +/**
> + * media_request_put() - decrement the reference counter of a request
> + *
> + * Mirror function of media_request_get() and media_request_get_from_fd(). Will
> + * free the request if this was the last valid reference.
> + *
> + * @req:	request to release.
> + */
> +void media_request_put(struct media_request *req);
> +
> +/**
> + * media_request_get_entity_data() - get per-entity data for a request
> + * @req:	request to get entity data from
> + * @entity:	entity to get data of
> + * @fh:		cookie identifying the handle from which the entity is accessed
> + *
> + * Search and return the entity data associated associated to the request. If no
> + * such data exists, it is allocated as per the entity operations.
> + *
> + * The fh arguments serves as a cookie to make sure the same entity is not
> + * accessed through different opened file handles. The same handle must be
> + * passed to all calls of this function for the same entity. Failure to do so
> + * will return an error.
> + *
> + * Returns the per-entity data, or an error code if a problem happened. -EINVAL
> + * means that data for the entity already existed, but has been allocated under
> + * a different cookie.
> + */
> +struct media_request_entity_data *
> +media_request_get_entity_data(struct media_request *req,
> +			      struct media_entity *entity, void *fh);
> +
> +
> +/**
> + * media_request_complete() - to be invoked when the request is complete
> + *
> + * @req:	request which has completed
> + */
> +void media_request_complete(struct media_request *req);
> +
> +/**
> + * struct media_request_entity_data - per-entity request data
> + *
> + * Base structure used to store request state data. To be extended by actual
> + * implementation.
> + *
> + * @entity:	entity this data belongs to
> + * @fh:		subsystem-dependent. For V4L2, the v4l2_fh of the opened device
> + * @list:	entry in media_request::data
> + * @applied:	whether this request has already been applied for this entity
> + */
> +struct media_request_entity_data {
> +	struct media_entity *entity;
> +	void *fh;
> +	struct list_head list;
> +	bool applied;
> +};
> +
> +/**
> + * media_device_request_cmd() - perform the REQUEST_CMD ioctl
> + *
> + * @mdev:	media device the ioctl has been called on
> + * @cmd:	user-space request command structure
> + */
> +long media_device_request_cmd(struct media_device *mdev,
> +			      struct media_request_cmd *cmd);
> +
> +#else /* CONFIG_MEDIA_CONTROLLER */
> +
> +static inline media_request_get(struct media_request *req)
> +{
> +}
> +
> +static inline struct media_request *media_request_get_from_fd(int fd)
> +{
> +	return ERR_PTR(-ENOSYS);
> +}
> +
> +static inline void media_request_put(struct media_request *req)
> +{
> +}
> +
> +static inline struct media_request *media_request_get_entity_data(
> +					  struct media_request *req,
> +					  struct media_entity *entity, void *fh)
> +{
> +	return ERR_PTR(-ENOSYS);
> +}
> +
> +static inline void media_request_entity_complete(struct media_request *req)
> +{
> +}
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
> +#endif
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index b9b9446095e9..6e20ac9848b5 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -406,6 +406,15 @@ struct media_v2_topology {
>  	__u64 ptr_links;
>  } __attribute__ ((packed));
>  
> +#define MEDIA_REQ_CMD_ALLOC		0
> +#define MEDIA_REQ_CMD_SUBMIT		1
> +#define MEDIA_REQ_CMD_REINIT		2
> +
> +struct media_request_cmd {
> +	__u32 cmd;
> +	__s32 fd;
> +} __attribute__ ((packed));
> +
>  /* ioctls */
>  
>  #define MEDIA_IOC_DEVICE_INFO		_IOWR('|', 0x00, struct media_device_info)
> @@ -413,5 +422,6 @@ struct media_v2_topology {
>  #define MEDIA_IOC_ENUM_LINKS		_IOWR('|', 0x02, struct media_links_enum)
>  #define MEDIA_IOC_SETUP_LINK		_IOWR('|', 0x03, struct media_link_desc)
>  #define MEDIA_IOC_G_TOPOLOGY		_IOWR('|', 0x04, struct media_v2_topology)
> +#define MEDIA_IOC_REQUEST_CMD		_IOWR('|', 0x05, struct media_request_cmd)
>  
>  #endif /* __LINUX_MEDIA_H */
> -- 
> 2.16.0.rc1.238.g530d649a79-goog
> 

-- 
Sakari Ailus
sakari.ailus@linux.intel.com

[PATCH 1/1] media: request: Add support for tagged request-based objects

[Sakari Ailus]

Allow binding objects to requests that can be later on fetched based on
that tag.

The intent is that the objects are bound at the time data is bound to a
request and later retrieved for validation (and finally applied) when the
request is queued. A tag can be any pointer, as long as it is unique to a
request.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
---
Hi Alexandre,

Here's the patch. It's on top of your current set so if you remove entity
data support there may be some conflicts to resolve. It's only been
compile tested so far, but is rather simple. By providing a tag, the
caller may attach data objects to the request and they can be found later
on, by using the same tag, when the request is queued.

The drivers are still responsible for adding only objects they can support
with requests. The driver must also detach the data from the request, and
at the queue time, make sure that no objects that weren't accounted for
were added.

 drivers/media/media-request.c | 105 ++++++++++++++++++++++++++++++++++++++++++
 include/media/media-request.h |  75 ++++++++++++++++++++++++++++++
 2 files changed, 180 insertions(+)

diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
index 30a2323..c96b747 100644
--- a/drivers/media/media-request.c
+++ b/drivers/media/media-request.c
@@ -57,10 +57,115 @@ media_request_get_from_fd(int fd)
 }
 EXPORT_SYMBOL_GPL(media_request_get_from_fd);
 
+int media_request_data_attach(struct media_request *req, const void *tag,
+			      void *data,
+			      void (*release)(struct media_request *req,
+					      const void *tag, void *data))
+{
+	struct media_request_data *req_data;
+
+	req_data = kzalloc(sizeof(*req_data), GFP_KERNEL);
+	if (!req_data)
+		return -ENOMEM;
+
+	req_data->req = req;
+	req_data->tag = tag;
+	req_data->data = data;
+	req_data->release = release;
+
+	mutex_lock(&req->lock);
+	list_add(&req_data->list, &req->data_list);
+	mutex_unlock(&req->lock);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(media_request_data_attach);
+
+struct media_request_data *__media_request_data_find(
+	struct media_request *req, const void *tag)
+{
+	struct media_request_data *req_data;
+
+	lockdep_assert_held(&req->lock);
+
+	list_for_each_entry(req_data, &req->data_list, list)
+		if (req_data->tag == tag)
+			return req_data;
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__media_request_data_find);
+
+void *media_request_data_find(struct media_request *req, const void *tag)
+{
+	struct media_request_data *req_data;
+	void *data;
+
+	mutex_lock(&req->lock);
+	req_data = __media_request_data_find(req, tag);
+	mutex_lock(&req->lock);
+
+	data = req_data ? req_data->data : NULL;
+
+	return data;
+}
+EXPORT_SYMBOL_GPL(media_request_data_find);
+
+void media_request_data_detach(struct media_request *req, const void *tag)
+{
+	struct media_request_data *req_data;
+
+	mutex_lock(&req->lock);
+
+	req_data = __media_request_data_find(req, tag);
+	if (WARN_ON(!req_data)) {
+		mutex_unlock(&req->lock);
+		return;
+	}
+
+	list_del(&req_data->list);
+
+	mutex_unlock(&req->lock);
+
+	if (req_data->release)
+		req_data->release(req_data->req, req_data->tag, req_data->data);
+
+	kfree(req_data);
+}
+EXPORT_SYMBOL_GPL(media_request_data_detach);
+
+static void __media_request_data_detach(struct media_request_data *req)
+{
+	struct media_request_data *req_data;
+
+	list_del(&req_data->list);
+
+	if (req_data->release)
+		req_data->release(req_data->req, req_data->tag, req_data->data);
+
+	kfree(req_data);
+}
+
+bool media_request_has_data(struct media_request *req)
+{
+	bool ret;
+
+	mutex_lock(&req->lock);
+	ret = list_empty(&req->data_list);
+	mutex_unlock(&req->lock);
+
+	return ret;
+}
+
 static void media_request_release(struct kref *kref)
 {
 	struct media_request *req =
 		container_of(kref, typeof(*req), kref);
+	struct media_request_data *req_data, *req_data_safe;
+
+	/* Last reference; no need to acquire the lock here. */
+	list_for_each_entry_safe(req_data, req_data_safe, &req->data_list, list)
+		__media_request_data_detach(req_data);
 
 	req->mgr->ops->req_free(req);
 }
diff --git a/include/media/media-request.h b/include/media/media-request.h
index 817df13..64b945d 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -36,12 +36,23 @@ enum media_request_state {
 	MEDIA_REQUEST_STATE_DELETED,
 };
 
+struct media_request_data {
+	struct media_request *req;
+	const void *tag;
+	struct list_head list;
+	void *data;
+	void (*release)(struct media_request *req,
+			const void *tag, void *data);
+};
+
 /**
  * struct media_request - Media request base structure
  * @id:		request id, used internally for debugging
  * @mgr:	manager this request belongs to
  * @kref:	reference count
  * @list:	list entry in the media device requests list
+ * @data_list:	list of data entries related to the request;
+ *		struct media_request_data.list
  * @lock:	protects internal state against concurrent accesses
  * @state:	current state of the request
  * @data:	per-entity data list
@@ -55,6 +66,7 @@ struct media_request {
 	struct media_request_mgr *mgr;
 	struct kref kref;
 	struct list_head list;
+	struct list_head data_list;
 
 	struct mutex lock;
 	enum media_request_state state;
@@ -89,6 +101,69 @@ struct media_request *media_request_get(struct media_request *req);
 struct media_request *media_request_get_from_fd(int fd);
 
 /**
+ * media_request_data_attach - Attach data related to a tag in a request
+ *
+ * Attach data to a request. The data can be later on retrieved based on the
+ * tag. The data remains attached until it is detached using
+ * @media_request_data_detach.
+ *
+ * The lifetime of the attachment is the same than that of the request itself.
+ *
+ * @req:	request to which data is to be attached
+ * @tag:	the tag to the data, based on which it can be found later;
+ *		unique to the request
+ * @data:	pointer to the data
+ * @release:	release callback for data, may be NULL
+ */
+int media_request_data_attach(struct media_request *req, const void *tag,
+			      void *data,
+			      void (*release)(struct media_request *req,
+					      const void *tag, void *data));
+
+/**
+ * __media_request_data_find - Find data related to a media request based on a
+ *			       tag
+ *
+ * Retrieve data related to a media request based on a tag.
+ *
+ * Compared to @media_request_data_find, this function returns the underlying
+ * media request data object as well as requires the caller to acquire the
+ * request lock.
+ */
+struct media_request_data *__media_request_data_find(
+	struct media_request *req, const void *tag);
+
+/**
+ * media_request_data_find - Find data related to a media request based on a
+ *			     tag
+ *
+ * Retrieve data related to a media request based on a tag.
+ *
+ * Compared to @__media_request_data_find, this function returns the data only
+ * and performs any locking by itself.
+ */
+void *media_request_data_find(struct media_request *req, const void *tag);
+
+/**
+ * media_request_data_detach - Detach an object from the media request
+ *
+ * Explicitly detach an object from the media request.
+ *
+ * @req:	request to remove the object from
+ * @tag:	tag based on which the object is retrieved for detaching
+ */
+void media_request_data_detach(struct media_request *req, const void *tag);
+
+/**
+ * media_request_has_data - Does the request have object bound to it?
+ *
+ * Return true if the request has objects bound to it, otherwise false.
+ *
+ * @req:	request which is queried for attached objects
+ */
+bool media_request_has_data(struct media_request *req);
+
+/**
  * media_request_put() - decrement the reference counter of a request
  *
  * Mirror function of media_request_get() and media_request_get_from_fd(). Will
-- 
2.7.4

Re: [RFCv3 15/17] v4l2: document the request API interface

[Randy Dunlap]

Hi,

A few comments below...

On 02/06/2018 05:48 PM, Alexandre Courbot wrote:
> Document how the request API can be used along with the existing V4L2
> interface.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  Documentation/media/uapi/v4l/buffer.rst            |  10 +-
>  Documentation/media/uapi/v4l/common.rst            |   1 +
>  Documentation/media/uapi/v4l/request-api.rst       | 236 +++++++++++++++++++++
>  .../media/uapi/v4l/vidioc-g-ext-ctrls.rst          |  16 +-
>  Documentation/media/uapi/v4l/vidioc-qbuf.rst       |  21 ++
>  5 files changed, 280 insertions(+), 4 deletions(-)
>  create mode 100644 Documentation/media/uapi/v4l/request-api.rst


> diff --git a/Documentation/media/uapi/v4l/request-api.rst b/Documentation/media/uapi/v4l/request-api.rst
> new file mode 100644
> index 000000000000..4c61a0dbe3a9
> --- /dev/null
> +++ b/Documentation/media/uapi/v4l/request-api.rst
> @@ -0,0 +1,236 @@
> +.. -*- coding: utf-8; mode: rst -*-
> +
> +.. _media-request-api:
> +
> +Request API
> +===========
> +
> +The Request API has been designed to allow V4L2 to deal with requirements of
> +modern devices (stateless codecs, MIPI cameras, ...) and APIs (Android Codec
> +v2). One such requirement is the ability for devices belonging to the same
> +pipeline to reconfigure and collaborate closely on a per-frame basis. Another is
> +efficient support of stateless codecs, which need per-frame controls to be set
> +asynchronously in order to be efficiently used.

                           to be used efficiently.

> +
> +Supporting these features without the Request API is possible but terribly
> +inefficient: user-space would have to flush all activity on the media pipeline,
> +reconfigure it for the next frame, queue the buffers to be processed with that
> +configuration, and wait until they are all available for dequeing before

"dequeing" should be dequeueing or dequeuing.

> +considering the next frame. This defeats the purpose of having buffer queues
> +since in practice only one buffer would be queued at a time.
> +
> +The Request API allows a specific configuration of the pipeline (media
> +controller topology + controls for each device) to be associated with specific
> +buffers. The parameters are applied by each participating device as buffers
> +associated to a request flow in. This allows user-space to schedule several
> +tasks ("requests") with different parameters in advance, knowing that the
> +parameters will be applied when needed to get the expected result. Controls
> +values at the time of request completion are also available for reading.
> +
> +Usage
> +=====
> +
> +The Request API is used on top of standard media controller and V4L2 calls,
> +which are augmented with an extra ``request_fd`` parameter. All operations on
> +requests themselves are performed using the command parameter of the
> +:c:func:`MEDIA_IOC_REQUEST_CMD` ioctl.
> +
> +Request Allocation
> +------------------
> +
> +User-space allocates requests using the ``MEDIA_REQ_CMD_ALLOC`` command on
> +an opened media device. This returns a file descriptor representing the
> +request. Typically, several such requests will be allocated.
> +
> +Request Preparation
> +-------------------
> +
> +Standard V4L2 ioctls can then receive a request file descriptor to express the
> +fact that the ioctl is part of said request, and is not to be applied
> +immediately. V4L2 ioctls supporting this are :c:func:`VIDIOC_S_EXT_CTRLS` and
> +:c:func:`VIDIOC_QBUF`. Controls set with a request parameter are stored instead
> +of being immediately applied, and queued buffers will block the buffer queue
> +until the request becomes active.
> +
> +RFC Note: currently several buffers can be queued to the same queue with the
> +same request. The request parameters will be only be applied when processing
> +the first buffer. Does it make more sense to allow at most one buffer per
> +request per queue instead?
> +
> +Request Submission
> +------------------
> +
> +Once the parameters and buffers of the request are specified, it can be
> +submitted with the ``MEDIA_REQ_CMD_SUBMIT`` command. This will make the buffers
> +associated to the request available to their driver, which can then apply the
> +saved controls as buffers are processed. A submitted request cannot be modified
> +anymore.
> +
> +If several devices are part of the request, individual drivers may synchronize
> +so the requested pipeline's topology is applied before the buffers are
> +processed. This is at the discretion of the drivers and is not a requirement.
> +
> +Buffers queued without an associated request after a request-bound buffer will
> +be processed using the state of the hardware at the time of the request
> +completion. All the same, controls set without a request are applied
> +immediately, regardless of whether a request is in use or not.
> +
> +User-space can ``poll()`` a request FD in order to wait until the request
> +completes. A request is considered complete once all its associated buffers are
> +available for dequeing. Note that user-space does not need to wait for the

s/dequeing/dequeu[e]ing/

> +request to complete to dequeue its buffers: buffers that are available halfway
> +through a request can be dequeued independently of the request's state.
> +
> +A completed request includes the state of all devices that had queued buffers
> +associated with it at the time of the request completion. User-space can query
> +that state by calling :c:func:`VIDIOC_G_EXT_CTRLS` with the request FD.
> +
> +RFC Note: requests are currently processed in buffer order, meaning that they
> +do not necessarily start being processed in the order of their submission. Also,
> +requests do not need to complete by processing order: a driver can decide to
> +hold the buffer associated to a request in order to process the few next ones,
> +which will result in the first request completing after the others.
> +
> +Recycling and Destruction
> +-------------------------

[snip]

> diff --git a/Documentation/media/uapi/v4l/vidioc-qbuf.rst b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> index 9e448a4aa3aa..d7fea37f32e7 100644
> --- a/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> +++ b/Documentation/media/uapi/v4l/vidioc-qbuf.rst
> @@ -98,6 +98,12 @@ dequeued, until the :ref:`VIDIOC_STREAMOFF <VIDIOC_STREAMON>` or
>  :ref:`VIDIOC_REQBUFS` ioctl is called, or until the
>  device is closed.
>  
> +For all buffer types, the ``request_fd`` field can be used when enqueing

Looks like "enqueing" should be enqueueing or enqueuing.

> +to specify the file descriptor of a request, if requests are in use.
> +Setting it means that the buffer will not be passed to the driver until
> +the request itself is submitted. Also, the driver will apply any setting
> +associated with the request before processing the buffer.
> +
>  Applications call the ``VIDIOC_DQBUF`` ioctl to dequeue a filled
>  (capturing) or displayed (output) buffer from the driver's outgoing
>  queue. They just set the ``type``, ``memory`` and ``reserved`` fields of


-- 
~Randy