[RFC PATCH 0/9] media: base request API support

[RFC PATCH 0/9] media: base request API support

[Alexandre Courbot]

Here is a new attempt at the request API, following the UAPI we agreed on in
Prague. Hopefully this can be used as the basis to move forward.

This series only introduces the very basics of how requests work: allocate a
request, queue buffers to it, queue the request itself, wait for it to complete,
reuse it. It does *not* yet use Hans' work with controls setting. I have
preferred to submit it this way for now as it allows us to concentrate on the
basic request/buffer flow, which was harder to get properly than I initially
thought. I still have a gut feeling that it can be improved, with less back-and-
forth into drivers.

Plugging in controls support should not be too hard a task (basically just apply
the saved controls when the request starts), and I am looking at it now.

The resulting vim2m driver can be successfully used with requests, and my tests
so far have been successful.

There are still some rougher edges:

* locking is currently quite coarse-grained
* too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
  depends on it - I plan to craft the headers so that it becomes unnecessary.
  As it is, some of the code will probably not even compile if
  CONFIG_MEDIA_CONTROLLER is not set

But all in all I think the request flow should be clear and easy to review, and
the possibility of custom queue and entity support implementations should give
us the flexibility we need to support more specific use-cases (I expect the
generic implementations to be sufficient most of the time though).

A very simple test program exercising this API is available here (don't forget
to adapt the /dev/media0 hardcoding):
https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438

Looking forward to your feedback and comments!

Alexandre Courbot (8):
  media: add request API core and UAPI
  media: request: add generic queue
  media: request: add generic entity ops
  media: vb2: add support for requests
  media: vb2: add support for requests in QBUF ioctl
  media: v4l2-mem2mem: add request support
  media: vim2m: add media device
  media: vim2m: add request support

Hans Verkuil (1):
  videodev2.h: Add request field to v4l2_buffer

 drivers/media/Makefile                        |   4 +-
 drivers/media/media-device.c                  |   6 +
 drivers/media/media-request-entity-generic.c  |  56 ++++
 drivers/media/media-request-queue-generic.c   | 150 ++++++++++
 drivers/media/media-request.c                 | 390 ++++++++++++++++++++++++++
 drivers/media/platform/vim2m.c                |  46 +++
 drivers/media/usb/cpia2/cpia2_v4l.c           |   2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   7 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  99 ++++++-
 drivers/media/v4l2-core/v4l2-mem2mem.c        |  34 +++
 drivers/media/v4l2-core/videobuf2-core.c      |  59 +++-
 drivers/media/v4l2-core/videobuf2-v4l2.c      |  32 ++-
 include/media/media-device.h                  |   3 +
 include/media/media-entity.h                  |   6 +
 include/media/media-request.h                 | 282 +++++++++++++++++++
 include/media/v4l2-mem2mem.h                  |  19 ++
 include/media/videobuf2-core.h                |  25 +-
 include/media/videobuf2-v4l2.h                |   2 +
 include/uapi/linux/media.h                    |  11 +
 include/uapi/linux/videodev2.h                |   3 +-
 20 files changed, 1216 insertions(+), 20 deletions(-)
 create mode 100644 drivers/media/media-request-entity-generic.c
 create mode 100644 drivers/media/media-request-queue-generic.c
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request.h

-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 1/9] media: add request API core and UAPI

[Alexandre Courbot]

The request API provides a way to group buffers and device parameters
into units of work to be queued and executed. This patch introduces the
UAPI and core framework.

This patch is based on the previous work by Laurent Pinchart. The core
has changed considerably, but the UAPI is mostly untouched.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/Makefile               |   3 +-
 drivers/media/media-device.c         |   6 +
 drivers/media/media-request.c        | 390 +++++++++++++++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
 include/media/media-device.h         |   3 +
 include/media/media-entity.h         |   6 +
 include/media/media-request.h        | 269 ++++++++++++++++++++++++
 include/uapi/linux/media.h           |  11 +
 8 files changed, 688 insertions(+), 2 deletions(-)
 create mode 100644 drivers/media/media-request.c
 create mode 100644 include/media/media-request.h

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 594b462ddf0e..985d35ec6b29 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -3,7 +3,8 @@
 # Makefile for the kernel multimedia device drivers.
 #
 
-media-objs	:= media-device.o media-devnode.o media-entity.o
+media-objs	:= media-device.o media-devnode.o media-entity.o \
+		   media-request.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
index e79f72b8b858..045cec7d2de9 100644
--- a/drivers/media/media-device.c
+++ b/drivers/media/media-device.c
@@ -32,6 +32,7 @@
 #include <media/media-device.h>
 #include <media/media-devnode.h>
 #include <media/media-entity.h>
+#include <media/media-request.h>
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
@@ -407,6 +408,7 @@ static const struct media_ioctl_info ioctl_info[] = {
 	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
 	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
+	MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
 };
 
 static long media_device_ioctl(struct file *filp, unsigned int cmd,
@@ -688,6 +690,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
 
 void media_device_cleanup(struct media_device *mdev)
 {
+	if (mdev->req_queue) {
+		mdev->req_queue->ops->release(mdev->req_queue);
+		mdev->req_queue = NULL;
+	}
 	ida_destroy(&mdev->entity_internal_idx);
 	mdev->entity_internal_idx_max = 0;
 	media_graph_walk_cleanup(&mdev->pm_count_walk);
diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
new file mode 100644
index 000000000000..15dc65ddfe41
--- /dev/null
+++ b/drivers/media/media-request.c
@@ -0,0 +1,390 @@
+/*
+ * Request and request queue base management
+ *
+ * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/anon_inodes.h>
+#include <linux/media.h>
+#include <linux/fs.h>
+#include <linux/file.h>
+#include <linux/slab.h>
+#include <linux/list.h>
+
+#include <media/media-request.h>
+#include <media/media-device.h>
+
+const struct file_operations request_fops;
+
+void media_request_get(struct media_request *req)
+{
+	kref_get(&req->kref);
+}
+EXPORT_SYMBOL_GPL(media_request_get);
+
+struct media_request *
+media_request_get_from_fd(int fd)
+{
+	struct file *f;
+	struct media_request *req;
+
+	f = fget(fd);
+	if (!f)
+		return NULL;
+
+	/* Not a request FD? */
+	if (f->f_op != &request_fops) {
+		fput(f);
+		return NULL;
+	}
+
+	req = f->private_data;
+	media_request_get(req);
+	fput(f);
+
+	return req;
+}
+EXPORT_SYMBOL_GPL(media_request_get_from_fd);
+
+static void media_request_release(struct kref *kref)
+{
+	struct media_request_entity_data *data, *next;
+	struct media_request *req =
+		container_of(kref, typeof(*req), kref);
+	struct media_device *mdev = req->queue->mdev;
+
+	dev_dbg(mdev->dev, "%s: releasing request %u\n", __func__, req->id);
+
+	list_del(&req->list);
+	list_for_each_entry_safe(data, next, &req->data, list) {
+		list_del(&data->list);
+		data->entity->req_ops->release_data(data);
+	}
+
+	req->queue->ops->req_free(req->queue, req);
+}
+
+void media_request_put(struct media_request *req)
+{
+	kref_put(&req->kref, media_request_release);
+}
+EXPORT_SYMBOL_GPL(media_request_put);
+
+struct media_request_entity_data *
+media_request_get_entity_data(struct media_request *req,
+			      struct media_entity *entity, void *fh)
+{
+	struct media_request_queue *queue = req->queue;
+	struct media_request_entity_data *data;
+
+	media_request_queue_lock(queue);
+
+	/* First look whether we already have entity data */
+	list_for_each_entry(data, &req->data, list) {
+		if (data->entity == entity) {
+			/*
+			 * If so, then the fh must match otherwise this means
+			 * we are trying to set the same entity through
+			 * different handles
+			 */
+			if (data->fh != fh)
+				data = ERR_PTR(-EINVAL);
+			goto done;
+		}
+	}
+
+	/* No entity data found, let's create it */
+	data = entity->req_ops->alloc_data(req, entity);
+	if (IS_ERR(data))
+		goto done;
+	data->fh = fh;
+	data->entity = entity;
+	list_add_tail(&data->list, &req->data);
+
+done:
+	media_request_queue_unlock(queue);
+	return data;
+}
+EXPORT_SYMBOL_GPL(media_request_get_entity_data);
+
+static const char * const media_request_states[] __maybe_unused = {
+	"IDLE",
+	"QUEUED",
+	"COMPLETED",
+	"DELETED",
+};
+
+static const char *media_request_state(enum media_request_state state)
+{
+	return state < ARRAY_SIZE(media_request_states) ?
+		media_request_states[state] : "INVALID";
+}
+
+static int media_device_request_close(struct inode *inode, struct file *filp)
+{
+	struct media_request *req = filp->private_data;
+
+	if (req == NULL)
+		return 0;
+
+	media_request_put(req);
+
+	return 0;
+}
+
+static unsigned int media_request_poll(struct file *file, poll_table *wait)
+{
+	struct media_request *req = file->private_data;
+
+	poll_wait(file, &req->wait_queue, wait);
+
+	if (req->state == MEDIA_REQUEST_STATE_COMPLETE)
+		return POLLIN | POLLRDNORM;
+
+	return 0;
+}
+
+const struct file_operations request_fops = {
+	.owner = THIS_MODULE,
+	.poll = media_request_poll,
+	.release = media_device_request_close,
+};
+
+/*
+ * Called when all the entities taking part in a request have completed their
+ * part.
+ *
+ * The request queue mutex must be held when this function is called.
+ */
+static void media_device_request_complete(struct media_request *req)
+{
+	struct media_request_queue *queue = req->queue;
+	struct media_device *mdev = queue->mdev;
+
+	if (WARN_ON(req->state != MEDIA_REQUEST_STATE_QUEUED)) {
+		dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
+			__func__, req->id, media_request_state(req->state));
+		return;
+	}
+
+	req->state = MEDIA_REQUEST_STATE_COMPLETE;
+
+	wake_up_interruptible(&req->wait_queue);
+
+	queue->ops->req_complete(queue);
+
+	/* Release the reference acquired when we queued the request */
+	media_request_put(req);
+}
+
+void media_request_entity_complete(struct media_request_queue *queue,
+				   struct media_entity *entity)
+{
+	struct media_request_entity_data *data;
+	struct media_request *req;
+	bool completed = true;
+
+	media_request_queue_lock(queue);
+
+	req = queue->active_request;
+
+	list_for_each_entry(data, &req->data, list) {
+		if (data->entity == entity)
+			data->completed = true;
+
+		if (!data->completed)
+			completed = false;
+	}
+
+	/* All entities of the request have completed, request is done */
+	if (completed)
+		media_device_request_complete(req);
+
+	media_request_queue_unlock(queue);
+
+}
+EXPORT_SYMBOL_GPL(media_request_entity_complete);
+
+void media_request_queue_init(struct media_request_queue *queue,
+			      struct media_device *mdev,
+			      const struct media_request_queue_ops *ops)
+{
+	queue->mdev = mdev;
+	mutex_init(&queue->mutex);
+	INIT_LIST_HEAD(&queue->requests);
+	queue->ops = ops;
+}
+EXPORT_SYMBOL_GPL(media_request_queue_init);
+
+void media_request_queue_release(struct media_request_queue *queue)
+{
+	struct media_device *mdev = queue->mdev;
+
+	media_request_queue_lock(queue);
+
+	/* Just a sanity check - we should have no remaining requests */
+	while (!list_empty(&queue->requests)) {
+		struct media_request *req;
+
+		req = list_first_entry(&queue->requests, typeof(*req), list);
+		dev_warn(mdev->dev,
+			"%s: request %u still referenced, deleting forcibly!\n",
+			__func__, req->id);
+		media_request_release(&req->kref);
+	}
+
+	media_request_queue_unlock(queue);
+}
+EXPORT_SYMBOL_GPL(media_request_queue_release);
+
+/*
+ * Process the MEDIA_REQ_CMD_ALLOC command
+ */
+static int media_request_cmd_alloc(struct media_request_queue *queue,
+				   struct media_request_cmd *cmd)
+{
+	struct media_request *req;
+	int fd;
+
+	req = queue->ops->req_alloc(queue);
+	if (!req)
+		return -ENOMEM;
+
+	req->queue = queue;
+	req->state = MEDIA_REQUEST_STATE_IDLE;
+	kref_init(&req->kref);
+	INIT_LIST_HEAD(&req->data);
+	init_waitqueue_head(&req->wait_queue);
+
+	media_request_queue_lock(queue);
+	req->id = ++queue->req_id;
+	list_add_tail(&req->list, &queue->requests);
+	media_request_queue_unlock(queue);
+
+	fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
+	if (fd < 0)
+		return fd;
+
+	cmd->fd = fd;
+
+	return 0;
+}
+
+/*
+ * Process the MEDIA_REQ_CMD_QUEUE command
+ */
+static int media_request_cmd_queue(struct media_request_queue *queue,
+				   struct media_request *req)
+{
+	int ret = 0;
+
+	media_request_queue_lock(queue);
+
+	if (req->state != MEDIA_REQUEST_STATE_IDLE) {
+		dev_dbg(queue->mdev->dev,
+			"%s: unable to queue request %u in state %s\n",
+			__func__, req->id, media_request_state(req->state));
+		media_request_queue_unlock(queue);
+		return -EINVAL;
+	}
+
+	req->state = MEDIA_REQUEST_STATE_QUEUED;
+	ret = queue->ops->req_queue(queue, req);
+	if (ret) {
+		req->state = MEDIA_REQUEST_STATE_IDLE;
+		media_request_queue_unlock(queue);
+		dev_dbg(queue->mdev->dev, "%s: cannot queue %u\n", __func__,
+			req->id);
+		return ret;
+	}
+
+	media_request_queue_unlock(queue);
+
+	/* The queue holds a reference to the request */
+	media_request_get(req);
+
+	return 0;
+}
+
+static int media_request_cmd_reinit(struct media_request_queue *queue,
+				    struct media_request *req)
+{
+	struct media_request_entity_data *data, *next;
+
+	media_request_queue_lock(queue);
+
+	if (req->state == MEDIA_REQUEST_STATE_QUEUED) {
+		dev_dbg(queue->mdev->dev,
+			"%s: unable to reinit queued request %u\n", __func__,
+			req->id);
+		media_request_queue_unlock(queue);
+		return -EINVAL;
+	}
+
+	/* delete all entity data */
+	list_for_each_entry_safe(data, next, &req->data, list) {
+		list_del(&data->list);
+		data->entity->req_ops->release_data(data);
+	}
+
+	/* reinitialize request to idle state */
+	req->state = MEDIA_REQUEST_STATE_IDLE;
+
+	media_request_queue_unlock(queue);
+
+	return 0;
+}
+
+long media_device_request_cmd(struct media_device *mdev,
+			      struct media_request_cmd *cmd)
+{
+	struct media_request *req = NULL;
+	int ret;
+
+	if (!mdev->req_queue)
+		return -ENOTTY;
+
+	if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
+		req = media_request_get_from_fd(cmd->fd);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+
+		/* requests must belong to this media device's queue */
+		if (req->queue != mdev->req_queue) {
+			media_request_put(req);
+			return -EINVAL;
+		}
+	}
+
+	switch (cmd->cmd) {
+	case MEDIA_REQ_CMD_ALLOC:
+		ret = media_request_cmd_alloc(mdev->req_queue, cmd);
+		break;
+
+	case MEDIA_REQ_CMD_QUEUE:
+		ret = media_request_cmd_queue(mdev->req_queue, req);
+		break;
+
+	case MEDIA_REQ_CMD_REINIT:
+		ret = media_request_cmd_reinit(mdev->req_queue, req);
+		break;
+
+	default:
+		ret = -EINVAL;
+		break;
+	}
+
+	if (req)
+		media_request_put(req);
+
+	return ret;
+}
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index b60a6b0841d1..ec4ecd5aa8bf 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -867,7 +867,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
 	__u32 i;
 
 	/* zero the reserved fields */
-	c->reserved[0] = c->reserved[1] = 0;
+	c->reserved[0] = 0;
 	for (i = 0; i < c->count; i++)
 		c->controls[i].reserved2[0] = 0;
 
diff --git a/include/media/media-device.h b/include/media/media-device.h
index bcc6ec434f1f..3feaa02128d1 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -27,6 +27,7 @@
 
 struct ida;
 struct device;
+struct media_request_queue;
 
 /**
  * struct media_entity_notify - Media Entity Notify
@@ -158,6 +159,8 @@ struct media_device {
 	void (*disable_source)(struct media_entity *entity);
 
 	const struct media_device_ops *ops;
+
+	struct media_request_queue *req_queue;
 };
 
 /* We don't need to include pci.h or usb.h here */
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 222d379960b7..bfb7a4565904 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -26,6 +26,9 @@
 #include <linux/list.h>
 #include <linux/media.h>
 
+struct media_request;
+struct media_request_entity_data;
+
 /* Enums used internally at the media controller to represent graphs */
 
 /**
@@ -193,6 +196,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	int (*process_request)(struct media_request *req,
+			       struct media_request_entity_data *data);
 };
 
 /**
@@ -275,6 +280,7 @@ struct media_entity {
 	struct list_head links;
 
 	const struct media_entity_operations *ops;
+	const struct media_request_entity_ops *req_ops;
 
 	int stream_count;
 	int use_count;
diff --git a/include/media/media-request.h b/include/media/media-request.h
new file mode 100644
index 000000000000..ead7fd8898c4
--- /dev/null
+++ b/include/media/media-request.h
@@ -0,0 +1,269 @@
+/*
+ * Generic request queue.
+ *
+ * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#ifndef _MEDIA_REQUEST_H
+#define _MEDIA_REQUEST_H
+
+#include <linux/kref.h>
+#include <linux/wait.h>
+#include <linux/list.h>
+
+struct media_device;
+struct media_request_queue;
+struct media_request_cmd;
+struct media_entity;
+struct media_request_entity_data;
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+
+enum media_request_state {
+	MEDIA_REQUEST_STATE_IDLE,
+	MEDIA_REQUEST_STATE_QUEUED,
+	MEDIA_REQUEST_STATE_COMPLETE,
+	MEDIA_REQUEST_STATE_DELETED,
+};
+
+/**
+ * struct media_request - Media request base structure
+ * @id:		request id, used internally for debugging
+ * @queue:	request queue this request belongs to
+ * @kref: 	reference count
+ * @list: 	list entry in the media device requests list
+ * @state:	current state of the request
+ * @data:	per-entity data list
+ * @wait_queue:	wait queue that signals once the request has completed
+ */
+struct media_request {
+	u32 id;
+	struct media_request_queue *queue;
+	struct kref kref;
+	struct list_head list;
+	enum media_request_state state;
+	struct list_head data;
+	wait_queue_head_t wait_queue;
+};
+
+/**
+ * media_request_get() - increment the reference counter of a request
+ *
+ * The calling context must call media_request_put() once it does not need
+ * the reference to the request anymore.
+ *
+ * @req:	request to acquire a reference of
+ */
+void media_request_get(struct media_request *req);
+
+/**
+ * media_request_get_from_fd() - lookup request by fd and acquire a reference.
+ *
+ * Look a request up from its fd, acquire a reference and return a pointer to
+ * the request. As for media_request_get(), media_request_put() must be called
+ * once the reference is not used anymore.
+ *
+ * @req:	request to lookup and acquire.
+ *
+ */
+struct media_request *media_request_get_from_fd(int fd);
+
+/**
+ * media_request_put() - decrement the reference counter of a request
+ *
+ * Mirror function of media_request_get() and media_request_get_from_fd(). Will
+ * free the request if this was the last valid reference.
+ *
+ * @req:	request to release.
+ */
+void media_request_put(struct media_request *req);
+
+/**
+ * media_request_get_entity_data() - get per-entity data for a request
+ * @req:	request to get entity data from
+ * @entity:	entity to get data of
+ * @fh:		cookie identifying the handle from which the entity is accessed
+ *
+ * Seach and return the entity data associated associated to the request. If no
+ * such data exists, it is allocated as per the entity operations.
+ *
+ * The fh arguments serves as a cookie to make sure the same entity is not
+ * accessed through different opened file handles. The same handle must be
+ * passed to all calls of this function for the same entity. Failure to do so
+ * will return an error.
+ *
+ * Returns the per-entity data, or an error code if a problem happened. -EINVAL
+ * means that data for the entity already existed, but has been allocated under
+ * a different cookie.
+ */
+struct media_request_entity_data *
+media_request_get_entity_data(struct media_request *req,
+			      struct media_entity *entity, void *fh);
+
+
+/**
+ * media_request_entity_complete() - to be invoked when an entity has completed
+ *				     its part of the currently active request
+ *
+ * @queue:	request queue
+ * @entity:	the entity that completed
+ */
+void media_request_entity_complete(struct media_request_queue *queue,
+				   struct media_entity *entity);
+
+/**
+ * struct media_request_queue_ops - request queue operations
+ *
+ * @release:	release all memory associated to the queue
+ * @req_qlloc:	allocate a request for this queue
+ * @req_free:	free a previously allocated request
+ * @req_queue:	queue a request for execution
+ * @req_complete: callback invoked when the active request has completed
+ *
+ */
+struct media_request_queue_ops {
+	void (*release)(struct media_request_queue *queue);
+	struct media_request *(*req_alloc)(struct media_request_queue *queue);
+	void (*req_free)(struct media_request_queue *queue,
+			 struct media_request *req);
+	int (*req_queue)(struct media_request_queue *queue,
+			 struct media_request *req);
+	void (*req_complete)(struct media_request_queue *queue);
+};
+
+/**
+ * struct media_request_queue - queue of requests
+ *
+ * @mdev:	media_device that manages this queue
+ * @ops:	implementation of the queue
+ * @mutex:	protects requests, active_request, req_id, and all members of
+ *		struct media_request
+ * @active_request: request being currently run by this queue
+ * @requests:	list of requests (not in any particular order) that this
+ *		queue owns.
+ * @req_id:	counter used to identify requests for debugging purposes
+ */
+struct media_request_queue {
+	struct media_device *mdev;
+	const struct media_request_queue_ops *ops;
+
+	struct mutex mutex;
+	struct list_head requests;
+	struct media_request *active_request;
+	u32 req_id;
+};
+
+static inline void media_request_queue_lock(struct media_request_queue *queue)
+{
+	mutex_lock(&queue->mutex);
+}
+
+static inline void media_request_queue_unlock(struct media_request_queue *queue)
+{
+	mutex_unlock(&queue->mutex);
+}
+
+/**
+ * media_request_queue_init() - initialize a request queue
+ *
+ * This function is to be called by request queue implementations to initialize
+ * the base structure.
+ *
+ * @queue:	queue to initialize
+ * @mdev:	media device owning this queue
+ * @ops:	ops to be used with this queue
+ */
+void media_request_queue_init(struct media_request_queue *queue,
+			      struct media_device *mdev,
+			      const struct media_request_queue_ops *ops);
+
+/**
+ * media_request_queue_release() - release resources held by a request queue
+ *
+ * The caller is still responsible for freeing the memory held by the queue
+ * structure itself.
+ *
+ * @queue:	queue to be released
+ */
+void media_request_queue_release(struct media_request_queue *queue);
+
+/**
+ * struct media_request_entity_data - per-entity request data
+ *
+ * Base structure used to store request state data. To be extended by actual
+ * implementation.
+ *
+ * @entity:	entity this data belongs to
+ * @fh:		subsystem-dependent. For V4L2, the v4l2_fh of the opened device
+ * @list:	entry in media_request::data
+ * @completed:	whether this entity has completed its part of the request
+ */
+struct media_request_entity_data {
+	struct media_entity *entity;
+	void *fh;
+	struct list_head list;
+	bool completed;
+};
+
+/**
+ * struct media_request_entity_ops - request-related entity operations
+ *
+ * All these operations are mandatory if the request API is to be used with
+ * an entity.
+ *
+ * @alloc_data:		allocate request data for this entity
+ * @release_data:	release previously allocated data
+ * @apply_data:		apply the data to the entity prior to running a request
+ *
+ */
+struct media_request_entity_ops {
+	struct media_request_entity_data *(*alloc_data)
+		(struct media_request *req, struct media_entity *entity);
+	void (*release_data)(struct media_request_entity_data *data);
+	int (*apply_data)(struct media_request_entity_data *data);
+};
+
+/**
+ * media_device_request_cmd() - perform the REQUEST_CMD ioctl
+ *
+ * @mdev:	media device the ioctl has been called on
+ * @cmd:	user-space request command structure
+ */
+long media_device_request_cmd(struct media_device *mdev,
+			      struct media_request_cmd *cmd);
+
+#else /* CONFIG_MEDIA_CONTROLLER */
+
+static inline struct media_request *media_request_get_from_fd(int fd)
+{
+	return ERR_PTR(-ENOTSUPP);
+}
+
+static inline void media_request_put(struct media_request *req)
+{
+}
+
+static inline struct media_request *media_request_get_entity_data(
+					  struct media_request *req,
+					  struct media_entity *entity, void *fh)
+{
+	return ERR_PTR(-ENOTSUPP);
+}
+
+static inline void media_request_entity_complete(struct media_request *req,
+						 struct media_entity *entity)
+{
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
+#endif
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index b9b9446095e9..247823ea0d1a 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -406,6 +406,16 @@ struct media_v2_topology {
 	__u64 ptr_links;
 } __attribute__ ((packed));
 
+#define MEDIA_REQ_CMD_ALLOC		0
+#define MEDIA_REQ_CMD_QUEUE		1
+#define MEDIA_REQ_CMD_REINIT		2
+
+struct media_request_cmd {
+	__u32 cmd;
+	__s32 fd;
+	__u32 flags;
+} __attribute__ ((packed));
+
 /* ioctls */
 
 #define MEDIA_IOC_DEVICE_INFO		_IOWR('|', 0x00, struct media_device_info)
@@ -413,5 +423,6 @@ struct media_v2_topology {
 #define MEDIA_IOC_ENUM_LINKS		_IOWR('|', 0x02, struct media_links_enum)
 #define MEDIA_IOC_SETUP_LINK		_IOWR('|', 0x03, struct media_link_desc)
 #define MEDIA_IOC_G_TOPOLOGY		_IOWR('|', 0x04, struct media_v2_topology)
+#define MEDIA_IOC_REQUEST_CMD		_IOWR('|', 0x05, struct media_request_cmd)
 
 #endif /* __LINUX_MEDIA_H */
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 2/9] media: request: add generic queue

[Alexandre Courbot]

Add a generic request queue that supports most use-case and should be
usable as-is by drivers without special hardware features.

The generic queue stores the requests into a FIFO list and executes them
sequentially.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/Makefile                      |   2 +-
 drivers/media/media-request-queue-generic.c | 150 ++++++++++++++++++++++++++++
 include/media/media-request.h               |   8 ++
 3 files changed, 159 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/media-request-queue-generic.c

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 985d35ec6b29..90117fff1339 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -4,7 +4,7 @@
 #
 
 media-objs	:= media-device.o media-devnode.o media-entity.o \
-		   media-request.o
+		   media-request.o media-request-queue-generic.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-request-queue-generic.c b/drivers/media/media-request-queue-generic.c
new file mode 100644
index 000000000000..780414b6d46a
--- /dev/null
+++ b/drivers/media/media-request-queue-generic.c
@@ -0,0 +1,150 @@
+/*
+ * Generic request queue implementation.
+ *
+ * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/slab.h>
+
+#include <media/media-request.h>
+#include <media/media-entity.h>
+
+/**
+ * struct media_request_generic - request enabled for the generic queue
+ *
+ * @base:	base request member
+ * @queue:	entry in media_request_queue_generic::queued_requests
+ */
+struct media_request_generic {
+	struct media_request base;
+	struct list_head queue;
+};
+#define to_generic_request(r) \
+	container_of(r, struct media_request_generic, base)
+
+/**
+ * struct media_request_queue_generic - generic request queue implementation
+ *
+ * Implements a simple request queue, where the next queued request is executed
+ * as soon as the previous one completes.
+ *
+ * @base:		base request queue member
+ * @mutex:		protects the queue
+ * @queued_requests:	list of requests to be sequentially executed
+ */
+struct media_request_queue_generic {
+	struct media_request_queue base;
+
+	struct list_head queued_requests;
+};
+#define to_generic_queue(q) \
+	container_of(q, struct media_request_queue_generic, base)
+
+static struct media_request *
+media_request_generic_alloc(struct media_request_queue *queue)
+{
+	struct media_request_generic *req;
+
+	req = kzalloc(sizeof(*req), GFP_KERNEL);
+	if (!req)
+		return ERR_PTR(-ENOMEM);
+
+	return &req->base;
+}
+
+static void media_request_generic_free(struct media_request_queue *queue,
+				       struct media_request *_req)
+{
+	struct media_request_generic *req = to_generic_request(_req);
+
+	kfree(req);
+}
+
+static void schedule_next_req(struct media_request_queue_generic *queue)
+{
+	struct media_request_generic *req;
+	struct media_request_entity_data *data;
+
+	req = list_first_entry_or_null(&queue->queued_requests, typeof(*req),
+				       queue);
+	if (!req)
+		return;
+
+	list_del(&req->queue);
+	queue->base.active_request = &req->base;
+
+	list_for_each_entry(data, &req->base.data, list) {
+		int ret;
+
+		ret = data->entity->req_ops->apply_data(data);
+	}
+
+	list_for_each_entry(data, &req->base.data, list) {
+		data->entity->ops->process_request(&req->base, data);
+	}
+}
+
+static void media_request_generic_complete(struct media_request_queue *_queue)
+{
+	struct media_request_queue_generic *queue = to_generic_queue(_queue);
+
+	queue->base.active_request = NULL;
+	schedule_next_req(queue);
+}
+
+static int media_request_generic_queue(struct media_request_queue *_queue,
+			   struct media_request *_req)
+{
+	struct media_request_queue_generic *queue = to_generic_queue(_queue);
+	struct media_request_generic *req = to_generic_request(_req);
+
+	list_add_tail(&req->queue, &queue->queued_requests);
+
+	if (!queue->base.active_request)
+		schedule_next_req(queue);
+
+	return 0;
+}
+
+static void
+media_request_generic_queue_release(struct media_request_queue *_queue)
+{
+	struct media_request_queue_generic *queue = to_generic_queue(_queue);
+
+	media_request_queue_release(&queue->base);
+	kfree(queue);
+}
+
+static const struct media_request_queue_ops request_queue_generic_ops = {
+	.release = media_request_generic_queue_release,
+	.req_alloc = media_request_generic_alloc,
+	.req_free = media_request_generic_free,
+	.req_queue = media_request_generic_queue,
+	.req_complete = media_request_generic_complete,
+};
+
+struct media_request_queue *
+media_request_queue_generic_alloc(struct media_device *mdev)
+{
+	struct media_request_queue_generic *ret;
+
+	ret = kzalloc(sizeof(*ret), GFP_KERNEL);
+	if (!ret)
+		return ERR_PTR(-ENOMEM);
+
+	media_request_queue_init(&ret->base, mdev, &request_queue_generic_ops);
+
+	INIT_LIST_HEAD(&ret->queued_requests);
+
+	return &ret->base;
+}
+EXPORT_SYMBOL(media_request_queue_generic_alloc);
diff --git a/include/media/media-request.h b/include/media/media-request.h
index ead7fd8898c4..583a1116f735 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -196,6 +196,14 @@ void media_request_queue_init(struct media_request_queue *queue,
  */
 void media_request_queue_release(struct media_request_queue *queue);
 
+/**
+ * media_request_queue_generic_alloc() - return an instance of the generic queue
+ *
+ * @mdev:	media device managing this queue
+ */
+struct media_request_queue *
+media_request_queue_generic_alloc(struct media_device *mdev);
+
 /**
  * struct media_request_entity_data - per-entity request data
  *
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 3/9] media: request: add generic entity ops

[Alexandre Courbot]

Add skeleton ops for generic entities. The intent is to provide a
generic mechanism to apply request parameters to entities using regular
media/v4l2 functions.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/Makefile                       |  3 +-
 drivers/media/media-request-entity-generic.c | 56 ++++++++++++++++++++++++++++
 include/media/media-request.h                |  5 +++
 3 files changed, 63 insertions(+), 1 deletion(-)
 create mode 100644 drivers/media/media-request-entity-generic.c

diff --git a/drivers/media/Makefile b/drivers/media/Makefile
index 90117fff1339..dea482f6ab72 100644
--- a/drivers/media/Makefile
+++ b/drivers/media/Makefile
@@ -4,7 +4,8 @@
 #
 
 media-objs	:= media-device.o media-devnode.o media-entity.o \
-		   media-request.o media-request-queue-generic.o
+		   media-request.o media-request-queue-generic.o \
+		   media-request-entity-generic.o
 
 #
 # I2C drivers should come before other drivers, otherwise they'll fail
diff --git a/drivers/media/media-request-entity-generic.c b/drivers/media/media-request-entity-generic.c
new file mode 100644
index 000000000000..18e53f9ce525
--- /dev/null
+++ b/drivers/media/media-request-entity-generic.c
@@ -0,0 +1,56 @@
+/*
+ * Media generic entity ops
+ *
+ * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
+ *
+ * This program is free software; you can redistribute it and/or modify
+ * it under the terms of the GNU General Public License version 2 as
+ * published by the Free Software Foundation.
+ *
+ * This program is distributed in the hope that it will be useful,
+ * but WITHOUT ANY WARRANTY; without even the implied warranty of
+ * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
+ * GNU General Public License for more details.
+ */
+
+#include <linux/slab.h>
+
+#include <media/media-entity.h>
+#include <media/media-request.h>
+
+struct media_request_entity_data_generic {
+	struct media_request_entity_data base;
+};
+
+static struct media_request_entity_data *
+alloc_req_data(struct media_request *req, struct media_entity *entity)
+{
+	struct media_request_entity_data_generic *ret;
+
+	ret = kzalloc(sizeof(*ret), GFP_KERNEL);
+	if (!ret)
+		return ERR_PTR(-ENOMEM);
+
+	return &ret->base;
+}
+
+static void release_req_data(struct media_request_entity_data *_data)
+{
+	struct media_request_entity_data_generic *data;
+
+	data = container_of(_data, typeof(*data), base);
+	kfree(data);
+}
+
+static int apply_req_data(struct media_request_entity_data *_data)
+{
+	return 0;
+}
+
+const struct media_request_entity_ops
+media_entity_request_generic_ops = {
+	.alloc_data = alloc_req_data,
+	.release_data = release_req_data,
+	.apply_data = apply_req_data,
+};
+EXPORT_SYMBOL_GPL(media_entity_request_generic_ops);
diff --git a/include/media/media-request.h b/include/media/media-request.h
index 583a1116f735..de3d6d824ffd 100644
--- a/include/media/media-request.h
+++ b/include/media/media-request.h
@@ -240,6 +240,11 @@ struct media_request_entity_ops {
 	int (*apply_data)(struct media_request_entity_data *data);
 };
 
+/*
+ * Generic entity request support, built on top of standard V4L2 functions
+ */
+extern const struct media_request_entity_ops media_entity_request_generic_ops;
+
 /**
  * media_device_request_cmd() - perform the REQUEST_CMD ioctl
  *
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 4/9] videodev2.h: Add request field to v4l2_buffer

[Alexandre Courbot]

From: Hans Verkuil <hans.verkuil@cisco.com>

When queuing buffers allow for passing the request ID that
should be associated with this buffer.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
[acourbot@chromium.org: make request ID 32-bit]
Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/usb/cpia2/cpia2_v4l.c           | 2 +-
 drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 7 ++++---
 drivers/media/v4l2-core/v4l2-ioctl.c          | 4 ++--
 drivers/media/v4l2-core/videobuf2-v4l2.c      | 3 ++-
 include/media/videobuf2-v4l2.h                | 2 ++
 include/uapi/linux/videodev2.h                | 3 ++-
 6 files changed, 13 insertions(+), 8 deletions(-)

diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c b/drivers/media/usb/cpia2/cpia2_v4l.c
index 3dedd83f0b19..7217dde95a8a 100644
--- a/drivers/media/usb/cpia2/cpia2_v4l.c
+++ b/drivers/media/usb/cpia2/cpia2_v4l.c
@@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
 	buf->sequence = cam->buffers[buf->index].seq;
 	buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
 	buf->length = cam->frame_size;
-	buf->reserved2 = 0;
+	buf->request = 0;
 	buf->reserved = 0;
 	memset(&buf->timecode, 0, sizeof(buf->timecode));
 
diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
index 821f2aa299ae..94f07c3b0b53 100644
--- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
+++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
@@ -370,7 +370,7 @@ struct v4l2_buffer32 {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__u32			request;
 	__u32			reserved;
 };
 
@@ -438,7 +438,8 @@ static int get_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
 		get_user(kp->type, &up->type) ||
 		get_user(kp->flags, &up->flags) ||
 		get_user(kp->memory, &up->memory) ||
-		get_user(kp->length, &up->length))
+		get_user(kp->length, &up->length) ||
+		get_user(kp->request, &up->request))
 			return -EFAULT;
 
 	if (V4L2_TYPE_IS_OUTPUT(kp->type))
@@ -533,7 +534,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
 		put_user(kp->timestamp.tv_usec, &up->timestamp.tv_usec) ||
 		copy_to_user(&up->timecode, &kp->timecode, sizeof(struct v4l2_timecode)) ||
 		put_user(kp->sequence, &up->sequence) ||
-		put_user(kp->reserved2, &up->reserved2) ||
+		put_user(kp->request, &up->request) ||
 		put_user(kp->reserved, &up->reserved) ||
 		put_user(kp->length, &up->length))
 			return -EFAULT;
diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index ec4ecd5aa8bf..8d041247e97f 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool write_only)
 	const struct v4l2_plane *plane;
 	int i;
 
-	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s",
+	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request=%u, flags=0x%08x, field=%s, sequence=%d, memory=%s",
 			p->timestamp.tv_sec / 3600,
 			(int)(p->timestamp.tv_sec / 60) % 60,
 			(int)(p->timestamp.tv_sec % 60),
 			(long)p->timestamp.tv_usec,
 			p->index,
-			prt_names(p->type, v4l2_type_names),
+			prt_names(p->type, v4l2_type_names), p->request,
 			p->flags, prt_names(p->field, v4l2_field_names),
 			p->sequence, prt_names(p->memory, v4l2_memory_names));
 
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
index 0c0669976bdc..bde7b8a3a303 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
 	b->timestamp = ns_to_timeval(vb->timestamp);
 	b->timecode = vbuf->timecode;
 	b->sequence = vbuf->sequence;
-	b->reserved2 = 0;
+	b->request = vbuf->request;
 	b->reserved = 0;
 
 	if (q->is_multiplanar) {
@@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
 	}
 	vb->timestamp = 0;
 	vbuf->sequence = 0;
+	vbuf->request = b->request;
 
 	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
 		if (b->memory == VB2_MEMORY_USERPTR) {
diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
index 036127c54bbf..ef2be0ccff14 100644
--- a/include/media/videobuf2-v4l2.h
+++ b/include/media/videobuf2-v4l2.h
@@ -31,6 +31,7 @@
  * @field:	enum v4l2_field; field order of the image in the buffer
  * @timecode:	frame timecode
  * @sequence:	sequence count of this frame
+ * @request:	request used by the buffer
  *
  * Should contain enough information to be able to cover all the fields
  * of struct v4l2_buffer at videodev2.h
@@ -42,6 +43,7 @@ struct vb2_v4l2_buffer {
 	__u32			field;
 	struct v4l2_timecode	timecode;
 	__u32			sequence;
+	__u32			request;
 };
 
 /*
diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
index 1c095b5a99c5..0650e8d14971 100644
--- a/include/uapi/linux/videodev2.h
+++ b/include/uapi/linux/videodev2.h
@@ -902,6 +902,7 @@ struct v4l2_plane {
  * @length:	size in bytes of the buffer (NOT its payload) for single-plane
  *		buffers (when type != *_MPLANE); number of elements in the
  *		planes array for multi-plane buffers
+ * @request: this buffer should use this request
  *
  * Contains data exchanged by application and driver using one of the Streaming
  * I/O methods.
@@ -925,7 +926,7 @@ struct v4l2_buffer {
 		__s32		fd;
 	} m;
 	__u32			length;
-	__u32			reserved2;
+	__u32			request;
 	__u32			reserved;
 };
 
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 5/9] media: vb2: add support for requests

[Alexandre Courbot]

Add throttling support for buffers when requests are in use on a given
queue. Buffers associated to a request are kept into the vb2 queue until
the request becomes active, at which point all the buffers are passed to
the driver. The queue can also signal that is has processed all of a
request's buffers.

Also add support for the request parameter when handling the QBUF ioctl.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
 drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
 include/media/videobuf2-core.h           | 25 +++++++++++++-
 3 files changed, 104 insertions(+), 9 deletions(-)

diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
index cb115ba6a1d2..c01038b7962a 100644
--- a/drivers/media/v4l2-core/videobuf2-core.c
+++ b/drivers/media/v4l2-core/videobuf2-core.c
@@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		    state != VB2_BUF_STATE_REQUEUEING))
 		state = VB2_BUF_STATE_ERROR;
 
+	WARN_ON(vb->request != q->cur_req);
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Although this is not a callback, it still does have to balance
@@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
 		/* Add the buffer to the done buffers list */
 		list_add_tail(&vb->done_entry, &q->done_list);
 		vb->state = state;
+
+		if (q->cur_req) {
+			WARN_ON(q->req_buf_cnt < 1);
+
+			if (--q->req_buf_cnt == 0)
+				q->cur_req = NULL;
+		}
 	}
 	atomic_dec(&q->owned_by_drv_count);
 	spin_unlock_irqrestore(&q->done_lock, flags);
@@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
 
+static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
+{
+	struct vb2_buffer *vb;
+
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (vb->request == q->cur_req)
+			__enqueue_in_driver(vb);
+	}
+}
+
 /**
  * vb2_start_streaming() - Attempt to start streaming.
  * @q:		videobuf2 queue
@@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	 * If any buffers were queued before streamon,
 	 * we can now pass them to driver for processing.
 	 */
-	list_for_each_entry(vb, &q->queued_list, queued_entry)
-		__enqueue_in_driver(vb);
+	vb2_queue_enqueue_current_buffers(q);
 
 	/* Tell the driver to start streaming */
 	q->start_streaming_called = 1;
@@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
 	return ret;
 }
 
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
+		  struct media_request *req, void *pb)
 {
 	struct vb2_buffer *vb;
 	int ret;
@@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	q->queued_count++;
 	q->waiting_for_buffers = false;
 	vb->state = VB2_BUF_STATE_QUEUED;
+	vb->request = req;
 
 	if (pb)
 		call_void_bufop(q, copy_timestamp, vb, pb);
@@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 	/*
 	 * If already streaming, give the buffer to driver for processing.
 	 * If not, the buffer will be given to driver on next streamon.
+	 *
+	 * If using the request API, the buffer will be given to the driver
+	 * when the request becomes active.
 	 */
-	if (q->start_streaming_called)
+	if (q->start_streaming_called && !req)
 		__enqueue_in_driver(vb);
 
 	/* Fill buffer information for the userspace */
@@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
 }
 EXPORT_SYMBOL_GPL(vb2_core_qbuf);
 
+void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
+{
+	struct vb2_buffer *vb;
+
+	q->req_buf_cnt = 0;
+	list_for_each_entry(vb, &q->queued_list, queued_entry) {
+		if (vb->request == req)
+			++q->req_buf_cnt;
+	}
+
+	/* only consider the request if we actually have buffers for it */
+	if (q->req_buf_cnt == 0)
+		return;
+
+	q->cur_req = req;
+
+	/* If not streaming yet, we will enqueue the buffers later */
+	if (q->start_streaming_called)
+		vb2_queue_enqueue_current_buffers(q);
+}
+EXPORT_SYMBOL_GPL(vb2_queue_start_request);
+
 /**
  * __vb2_wait_for_done_vb() - wait for a buffer to become available
  * for dequeuing
@@ -2242,7 +2287,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
 		 * Queue all buffers.
 		 */
 		for (i = 0; i < q->num_buffers; i++) {
-			ret = vb2_core_qbuf(q, i, NULL);
+			ret = vb2_core_qbuf(q, i, NULL, NULL);
 			if (ret)
 				goto err_reqbufs;
 			fileio->bufs[i].queued = 1;
@@ -2421,7 +2466,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
 
 		if (copy_timestamp)
 			b->timestamp = ktime_get_ns();
-		ret = vb2_core_qbuf(q, index, NULL);
+		ret = vb2_core_qbuf(q, index, NULL, NULL);
 		dprintk(5, "vb2_dbuf result: %d\n", ret);
 		if (ret)
 			return ret;
@@ -2524,7 +2569,7 @@ static int vb2_thread(void *data)
 		if (copy_timestamp)
 			vb->timestamp = ktime_get_ns();;
 		if (!threadio->stop)
-			ret = vb2_core_qbuf(q, vb->index, NULL);
+			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
 		call_void_qop(q, wait_prepare, q);
 		if (ret || threadio->stop)
 			break;
diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
index bde7b8a3a303..55b16b4db9a6 100644
--- a/drivers/media/v4l2-core/videobuf2-v4l2.c
+++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
@@ -28,6 +28,7 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 #include <media/v4l2-common.h>
+#include <media/media-request.h>
 
 #include <media/videobuf2-v4l2.h>
 
@@ -561,6 +562,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
 
 int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 {
+	struct media_request *req = NULL;
 	int ret;
 
 	if (vb2_fileio_is_active(q)) {
@@ -568,8 +570,33 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
 		return -EBUSY;
 	}
 
+	/*
+	 * The caller should have validated that the request is valid,
+	 * so we just need to look it up without further checking
+	 */
+	if (b->request > 0) {
+		req = media_request_get_from_fd(b->request);
+		if (!req)
+			return -EINVAL;
+		media_request_queue_lock(req->queue);
+
+		if (req->state != MEDIA_REQUEST_STATE_IDLE) {
+			ret = -EINVAL;
+			goto done;
+		}
+	}
+
 	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
-	return ret ? ret : vb2_core_qbuf(q, b->index, b);
+	if (!ret)
+		ret = vb2_core_qbuf(q, b->index, req, b);
+
+done:
+	if (req) {
+		media_request_queue_unlock(req->queue);
+		media_request_put(req);
+	}
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(vb2_qbuf);
 
diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
index ef9b64398c8c..7d5e8e53e256 100644
--- a/include/media/videobuf2-core.h
+++ b/include/media/videobuf2-core.h
@@ -44,6 +44,8 @@ enum vb2_memory {
 struct vb2_fileio_data;
 struct vb2_threadio_data;
 
+struct media_request;
+
 /**
  * struct vb2_mem_ops - memory handling/memory allocator operations
  * @alloc:	allocate video memory and, optionally, allocator private data,
@@ -237,6 +239,7 @@ struct vb2_queue;
  *			on an internal driver queue
  * @planes:		private per-plane information; do not change
  * @timestamp:		frame timestamp in ns
+ * @request:		pointer this buffer's request, if any
  */
 struct vb2_buffer {
 	struct vb2_queue	*vb2_queue;
@@ -246,6 +249,7 @@ struct vb2_buffer {
 	unsigned int		num_planes;
 	struct vb2_plane	planes[VB2_MAX_PLANES];
 	u64			timestamp;
+	struct media_request	*request;
 
 	/* private: internal use only
 	 *
@@ -500,6 +504,8 @@ struct vb2_buf_ops {
  *		when a buffer with the V4L2_BUF_FLAG_LAST is dequeued.
  * @fileio:	file io emulator internal data, used only if emulator is active
  * @threadio:	thread io internal data, used only if thread is active
+ * @cur_req:	request currently being processed by this queue
+ * @req_buf_cnt:number of buffers still to process in the current request
  */
 struct vb2_queue {
 	unsigned int			type;
@@ -554,6 +560,9 @@ struct vb2_queue {
 	struct vb2_fileio_data		*fileio;
 	struct vb2_threadio_data	*threadio;
 
+	struct media_request		*cur_req;
+	u32				req_buf_cnt;
+
 #ifdef CONFIG_VIDEO_ADV_DEBUG
 	/*
 	 * Counters for how often these queue-related ops are
@@ -638,6 +647,20 @@ void vb2_discard_done(struct vb2_queue *q);
  */
 int vb2_wait_for_all_buffers(struct vb2_queue *q);
 
+/**
+ * vb2_queue_start_request() - make the queue process the given request
+ *
+ * @q:		videobuf2 queue
+ * @req:	request to make active
+ *
+ * When using the request API, buffers are not passed to drivers until their
+ * request becomes active. Drivers using the request API are thus expected to
+ * call this function whenever a new request becomes active, so all buffers
+ * assigned to it are processed.
+ */
+void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req);
+
+
 /**
  * vb2_core_querybuf() - query video buffer information
  * @q:		videobuf queue
@@ -740,7 +763,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
  * The return values from this function are intended to be directly returned
  * from vidioc_qbuf handler in driver.
  */
-int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
+int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, struct media_request *req, void *pb);
 
 /**
  * vb2_core_dqbuf() - Dequeue a buffer to the userspace
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 6/9] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

Support the request argument of the QBUF ioctl.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-ioctl.c | 93 +++++++++++++++++++++++++++++++++++-
 1 file changed, 92 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 8d041247e97f..28f9c368563e 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -29,6 +29,7 @@
 #include <media/v4l2-device.h>
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mc.h>
+#include <media/media-request.h>
 
 #include <trace/events/v4l2.h>
 
@@ -965,6 +966,81 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
 	return -EINVAL;
 }
 
+/*
+ * Validate that a given request can be used during an ioctl.
+ *
+ * When using the request API, request file descriptors must be matched against
+ * the actual request object. User-space can pass any file descriptor, so we
+ * need to make sure the call is valid before going further.
+ *
+ * This function looks up the request and associated data and performs the
+ * following sanity checks:
+ *
+ * * Make sure that the entity supports requests,
+ * * Make sure that the entity belongs to the media_device managing the passed
+ *   request,
+ * * Make sure that the entity data (if any) is associated to the current file
+ *   handler.
+ *
+ * This function returns a pointer to the valid request, or and error code in
+ * case of failure. When successful, a reference to the request is acquired and
+ * must be properly released.
+ */
+#ifdef CONFIG_MEDIA_CONTROLLER
+static struct media_request *
+check_request(int request, struct file *file, void *fh)
+{
+	struct media_request *req = NULL;
+	struct video_device *vfd = video_devdata(file);
+	struct v4l2_fh *vfh =
+		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
+	struct media_entity *entity = &vfd->entity;
+	const struct media_entity *ent;
+	struct media_request_entity_data *data;
+	bool found = false;
+
+	if (!entity)
+		return ERR_PTR(-EINVAL);
+
+	/* Check that the entity supports requests */
+	if (!entity->req_ops)
+		return ERR_PTR(-ENOTSUPP);
+
+	req = media_request_get_from_fd(request);
+	if (!req)
+		return ERR_PTR(-EINVAL);
+
+	/* Validate that the entity belongs to the media_device managing
+	 * the request queue */
+	media_device_for_each_entity(ent, req->queue->mdev) {
+		if (entity == ent) {
+			found = true;
+			break;
+		}
+	}
+	if (!found) {
+		media_request_put(req);
+		return ERR_PTR(-EINVAL);
+	}
+
+	/* Validate that the entity's data belongs to the correct fh */
+	data = media_request_get_entity_data(req, entity, vfh);
+	if (IS_ERR(data)) {
+		media_request_put(req);
+		return ERR_PTR(PTR_ERR(data));
+	}
+
+	return req;
+}
+#else /* CONFIG_MEDIA_CONTROLLER */
+static struct media_request *
+check_request(int request, struct file *file, void *fh)
+{
+	return ERR_PTR(-ENOTSUPP);
+}
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 static void v4l_sanitize_format(struct v4l2_format *fmt)
 {
 	unsigned int offset;
@@ -1902,10 +1978,25 @@ static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
 static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
 				struct file *file, void *fh, void *arg)
 {
+	struct media_request *req = NULL;
 	struct v4l2_buffer *p = arg;
 	int ret = check_fmt(file, p->type);
 
-	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
+	if (ret)
+		return ret;
+
+	if (p->request > 0) {
+		req = check_request(p->request, file, fh);
+		if (IS_ERR(req))
+			return PTR_ERR(req);
+	}
+
+	ret = ops->vidioc_qbuf(file, fh, p);
+
+	if (req)
+		media_request_put(req);
+
+	return ret;
 }
 
 static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 7/9] media: v4l2-mem2mem: add request support

[Alexandre Courbot]

Support request API in the mem2mem framework. Drivers that specify ops
for the queue and entities can support requests seamlessly.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/v4l2-core/v4l2-mem2mem.c | 34 ++++++++++++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           | 19 +++++++++++++++++++
 2 files changed, 53 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index f62e68aa04c4..eb52bee8e06a 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -22,6 +22,7 @@
 #include <media/v4l2-dev.h>
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
+#include <media/media-request.h>
 
 MODULE_DESCRIPTION("Mem to mem device framework for videobuf");
 MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
@@ -219,6 +220,15 @@ void v4l2_m2m_try_schedule(struct v4l2_m2m_ctx *m2m_ctx)
 	m2m_dev = m2m_ctx->m2m_dev;
 	dprintk("Trying to schedule a job for m2m_ctx: %p\n", m2m_ctx);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	/* request is completed if all queues are done with it */
+	if (m2m_ctx->req_queue && m2m_ctx->req_queue->active_request &&
+	    m2m_ctx->cap_q_ctx.q.cur_req == NULL &&
+	    m2m_ctx->out_q_ctx.q.cur_req == NULL)
+		media_request_entity_complete(m2m_ctx->req_queue,
+					      m2m_ctx->entity);
+#endif
+
 	if (!m2m_ctx->out_q_ctx.q.streaming
 	    || !m2m_ctx->cap_q_ctx.q.streaming) {
 		dprintk("Streaming needs to be on for both queues\n");
@@ -665,6 +675,15 @@ struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev,
 }
 EXPORT_SYMBOL_GPL(v4l2_m2m_ctx_init);
 
+void v4l2_mem_ctx_request_init(struct v4l2_m2m_ctx *ctx,
+			       struct media_request_queue *req_queue,
+			       struct media_entity *entity)
+{
+	ctx->entity = entity;
+	ctx->req_queue = req_queue;
+}
+EXPORT_SYMBOL_GPL(v4l2_mem_ctx_request_init);
+
 void v4l2_m2m_ctx_release(struct v4l2_m2m_ctx *m2m_ctx)
 {
 	/* wait until the current context is dequeued from job_queue */
@@ -810,3 +829,18 @@ unsigned int v4l2_m2m_fop_poll(struct file *file, poll_table *wait)
 }
 EXPORT_SYMBOL_GPL(v4l2_m2m_fop_poll);
 
+int v4l2_m2m_process_request(struct media_request *req,
+			     struct media_request_entity_data *data)
+{
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct v4l2_fh *fh = data->fh;
+	struct v4l2_m2m_ctx *ctx = fh->m2m_ctx;
+
+	vb2_queue_start_request(&ctx->cap_q_ctx.q, req);
+	vb2_queue_start_request(&ctx->out_q_ctx.q, req);
+
+	v4l2_m2m_try_schedule(ctx);
+#endif
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_process_request);
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index e157d5c9b224..1c9925c3d4ce 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -19,6 +19,10 @@
 
 #include <media/videobuf2-v4l2.h>
 
+struct media_entity;
+struct media_request;
+struct media_request_entity_data;
+
 /**
  * struct v4l2_m2m_ops - mem-to-mem device driver callbacks
  * @device_run:	required. Begin the actual job (transaction) inside this
@@ -83,6 +87,7 @@ struct v4l2_m2m_queue_ctx {
  *
  * @q_lock: struct &mutex lock
  * @m2m_dev: opaque pointer to the internal data to handle M2M context
+ * @state: Pointer to state handler for channel support
  * @cap_q_ctx: Capture (output to memory) queue context
  * @out_q_ctx: Output (input from memory) queue context
  * @queue: List of memory to memory contexts
@@ -100,6 +105,8 @@ struct v4l2_m2m_ctx {
 
 	/* internal use only */
 	struct v4l2_m2m_dev		*m2m_dev;
+	struct media_request_queue	*req_queue;
+	struct media_entity		*entity;
 
 	struct v4l2_m2m_queue_ctx	cap_q_ctx;
 
@@ -351,6 +358,16 @@ struct v4l2_m2m_ctx *v4l2_m2m_ctx_init(struct v4l2_m2m_dev *m2m_dev,
 		void *drv_priv,
 		int (*queue_init)(void *priv, struct vb2_queue *src_vq, struct vb2_queue *dst_vq));
 
+/**
+ * v4l2_mem_ctx_request_init() - enable a context to be used with requests
+ *
+ * @ctx: context to potentially use within a channel
+ * @req_queue: request queue that will be used with this context
+ */
+void v4l2_mem_ctx_request_init(struct v4l2_m2m_ctx *ctx,
+			       struct media_request_queue *req_queue,
+			       struct media_entity *entity);
+
 static inline void v4l2_m2m_set_src_buffered(struct v4l2_m2m_ctx *m2m_ctx,
 					     bool buffered)
 {
@@ -602,6 +619,8 @@ int v4l2_m2m_ioctl_streamoff(struct file *file, void *fh,
 				enum v4l2_buf_type type);
 int v4l2_m2m_fop_mmap(struct file *file, struct vm_area_struct *vma);
 unsigned int v4l2_m2m_fop_poll(struct file *file, poll_table *wait);
+int v4l2_m2m_process_request(struct media_request *req,
+			     struct media_request_entity_data *data);
 
 #endif /* _MEDIA_V4L2_MEM2MEM_H */
 
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 8/9] media: vim2m: add media device

[Alexandre Courbot]

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/platform/vim2m.c | 24 ++++++++++++++++++++++++
 1 file changed, 24 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index b01fba020d5f..a32e8a7950eb 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
 	struct v4l2_device	v4l2_dev;
 	struct video_device	vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_device	mdev;
+#endif
 
 	atomic_t		num_inst;
 	struct mutex		dev_mutex;
@@ -1001,6 +1004,13 @@ static int vim2m_probe(struct platform_device *pdev)
 
 	spin_lock_init(&dev->irqlock);
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->mdev.dev = &pdev->dev;
+	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+	media_device_init(&dev->mdev);
+	dev->v4l2_dev.mdev = &dev->mdev;
+#endif
+
 	ret = v4l2_device_register(&pdev->dev, &dev->v4l2_dev);
 	if (ret)
 		return ret;
@@ -1034,6 +1044,13 @@ static int vim2m_probe(struct platform_device *pdev)
 		goto err_m2m;
 	}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	/* Register the media device node */
+	ret = media_device_register(&dev->mdev);
+	if (ret)
+		goto err_m2m;
+#endif
+
 	return 0;
 
 err_m2m:
@@ -1050,6 +1067,13 @@ static int vim2m_remove(struct platform_device *pdev)
 	struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
 	v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	if (media_devnode_is_registered(dev->mdev.devnode))
+		media_device_unregister(&dev->mdev);
+	media_device_cleanup(&dev->mdev);
+#endif
+
 	v4l2_m2m_release(dev->m2m_dev);
 	del_timer_sync(&dev->timer);
 	video_unregister_device(&dev->vfd);
-- 
2.15.1.504.g5279b80103-goog

[RFC PATCH 9/9] media: vim2m: add request support

[Alexandre Courbot]

Set the necessary ops for supporting requests in vim2m.

Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
---
 drivers/media/platform/vim2m.c | 22 ++++++++++++++++++++++
 1 file changed, 22 insertions(+)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index a32e8a7950eb..ffe94ef9214d 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -30,6 +30,7 @@
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-event.h>
 #include <media/videobuf2-vmalloc.h>
+#include <media/media-request.h>
 
 MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
 MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
@@ -142,6 +143,7 @@ struct vim2m_dev {
 	struct video_device	vfd;
 #ifdef CONFIG_MEDIA_CONTROLLER
 	struct media_device	mdev;
+	struct media_request_queue *req_queue;
 #endif
 
 	atomic_t		num_inst;
@@ -937,6 +939,11 @@ static int vim2m_open(struct file *file)
 		goto open_unlock;
 	}
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	v4l2_mem_ctx_request_init(ctx->fh.m2m_ctx, dev->req_queue,
+				  &dev->vfd.entity);
+#endif
+
 	v4l2_fh_add(&ctx->fh);
 	atomic_inc(&dev->num_inst);
 
@@ -992,6 +999,12 @@ static const struct v4l2_m2m_ops m2m_ops = {
 	.job_abort	= job_abort,
 };
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+static const struct media_entity_operations vim2m_entity_ops = {
+	.process_request = v4l2_m2m_process_request,
+};
+#endif
+
 static int vim2m_probe(struct platform_device *pdev)
 {
 	struct vim2m_dev *dev;
@@ -1006,6 +1019,10 @@ static int vim2m_probe(struct platform_device *pdev)
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 	dev->mdev.dev = &pdev->dev;
+	dev->req_queue = media_request_queue_generic_alloc(&dev->mdev);
+	if (IS_ERR(dev->req_queue))
+		return PTR_ERR(dev->req_queue);
+	dev->mdev.req_queue = dev->req_queue;
 	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
 	media_device_init(&dev->mdev);
 	dev->v4l2_dev.mdev = &dev->mdev;
@@ -1023,6 +1040,11 @@ static int vim2m_probe(struct platform_device *pdev)
 	vfd->lock = &dev->dev_mutex;
 	vfd->v4l2_dev = &dev->v4l2_dev;
 
+#ifdef CONFIG_MEDIA_CONTROLLER
+	vfd->entity.ops = &vim2m_entity_ops;
+	vfd->entity.req_ops = &media_entity_request_generic_ops;
+#endif
+
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
 	if (ret) {
 		v4l2_err(&dev->v4l2_dev, "Failed to register video device\n");
-- 
2.15.1.504.g5279b80103-goog

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Philippe Ombredanne]

Alexandre:

On Fri, Dec 15, 2017 at 8:56 AM, Alexandre Courbot
<acourbot@chromium.org> wrote:
> The request API provides a way to group buffers and device parameters
> into units of work to be queued and executed. This patch introduces the
> UAPI and core framework.
>
> This patch is based on the previous work by Laurent Pinchart. The core
> has changed considerably, but the UAPI is mostly untouched.
>
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>

<snip

> --- /dev/null
> +++ b/drivers/media/media-request.c
> @@ -0,0 +1,390 @@
> +/*
> + * Request and request queue base management
> + *
> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */

Have you considered using the new SPDX tags instead of this fine but
long legalese? And if other Chromium contributors could follow suit
and you could spread the word that would be even better!

See Thomas doc patches [1] for details.
Thanks!

[1] https://lkml.org/lkml/2017/12/4/934
-- 
Cordially
Philippe Ombredanne

Re: [RFC PATCH 0/9] media: base request API support

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 4057 bytes --]

Le vendredi 15 décembre 2017 à 16:56 +0900, Alexandre Courbot a écrit :
> Here is a new attempt at the request API, following the UAPI we agreed on in
> Prague. Hopefully this can be used as the basis to move forward.
> 
> This series only introduces the very basics of how requests work: allocate a
> request, queue buffers to it, queue the request itself, wait for it to complete,
> reuse it. It does *not* yet use Hans' work with controls setting. I have
> preferred to submit it this way for now as it allows us to concentrate on the
> basic request/buffer flow, which was harder to get properly than I initially
> thought. I still have a gut feeling that it can be improved, with less back-and-
> forth into drivers.
> 
> Plugging in controls support should not be too hard a task (basically just apply
> the saved controls when the request starts), and I am looking at it now.
> 
> The resulting vim2m driver can be successfully used with requests, and my tests
> so far have been successful.
> 
> There are still some rougher edges:
> 
> * locking is currently quite coarse-grained
> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>   As it is, some of the code will probably not even compile if
>   CONFIG_MEDIA_CONTROLLER is not set
> 
> But all in all I think the request flow should be clear and easy to review, and
> the possibility of custom queue and entity support implementations should give
> us the flexibility we need to support more specific use-cases (I expect the
> generic implementations to be sufficient most of the time though).
> 
> A very simple test program exercising this API is available here (don't forget
> to adapt the /dev/media0 hardcoding):
> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438

It looks like the example uses Hans control work you just mention.
Notably, it uses v4l2_ext_controls ctrls.request.

> 
> Looking forward to your feedback and comments!
> 
> Alexandre Courbot (8):
>   media: add request API core and UAPI
>   media: request: add generic queue
>   media: request: add generic entity ops
>   media: vb2: add support for requests
>   media: vb2: add support for requests in QBUF ioctl
>   media: v4l2-mem2mem: add request support
>   media: vim2m: add media device
>   media: vim2m: add request support
> 
> Hans Verkuil (1):
>   videodev2.h: Add request field to v4l2_buffer
> 
>  drivers/media/Makefile                        |   4 +-
>  drivers/media/media-device.c                  |   6 +
>  drivers/media/media-request-entity-generic.c  |  56 ++++
>  drivers/media/media-request-queue-generic.c   | 150 ++++++++++
>  drivers/media/media-request.c                 | 390 ++++++++++++++++++++++++++
>  drivers/media/platform/vim2m.c                |  46 +++
>  drivers/media/usb/cpia2/cpia2_v4l.c           |   2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   7 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  99 ++++++-
>  drivers/media/v4l2-core/v4l2-mem2mem.c        |  34 +++
>  drivers/media/v4l2-core/videobuf2-core.c      |  59 +++-
>  drivers/media/v4l2-core/videobuf2-v4l2.c      |  32 ++-
>  include/media/media-device.h                  |   3 +
>  include/media/media-entity.h                  |   6 +
>  include/media/media-request.h                 | 282 +++++++++++++++++++
>  include/media/v4l2-mem2mem.h                  |  19 ++
>  include/media/videobuf2-core.h                |  25 +-
>  include/media/videobuf2-v4l2.h                |   2 +
>  include/uapi/linux/media.h                    |  11 +
>  include/uapi/linux/videodev2.h                |   3 +-
>  20 files changed, 1216 insertions(+), 20 deletions(-)
>  create mode 100644 drivers/media/media-request-entity-generic.c
>  create mode 100644 drivers/media/media-request-queue-generic.c
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC PATCH 0/9] media: base request API support

[Nicolas Dufresne]

[-- Attachment #1: Type: text/plain, Size: 4087 bytes --]

Le vendredi 15 décembre 2017 à 16:56 +0900, Alexandre Courbot a écrit :
> Here is a new attempt at the request API, following the UAPI we agreed on in
> Prague. Hopefully this can be used as the basis to move forward.
> 
> This series only introduces the very basics of how requests work: allocate a
> request, queue buffers to it, queue the request itself, wait for it to complete,
> reuse it. It does *not* yet use Hans' work with controls setting. I have
> preferred to submit it this way for now as it allows us to concentrate on the
> basic request/buffer flow, which was harder to get properly than I initially
> thought. I still have a gut feeling that it can be improved, with less back-and-
> forth into drivers.
> 
> Plugging in controls support should not be too hard a task (basically just apply
> the saved controls when the request starts), and I am looking at it now.
> 
> The resulting vim2m driver can be successfully used with requests, and my tests
> so far have been successful.
> 
> There are still some rougher edges:
> 
> * locking is currently quite coarse-grained
> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>   As it is, some of the code will probably not even compile if
>   CONFIG_MEDIA_CONTROLLER is not set

Would it be possible to explain why this relation between request and
the media controller ? Why couldn't request be created from video
devices ?

> 
> But all in all I think the request flow should be clear and easy to review, and
> the possibility of custom queue and entity support implementations should give
> us the flexibility we need to support more specific use-cases (I expect the
> generic implementations to be sufficient most of the time though).
> 
> A very simple test program exercising this API is available here (don't forget
> to adapt the /dev/media0 hardcoding):
> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438
> 
> Looking forward to your feedback and comments!
> 
> Alexandre Courbot (8):
>   media: add request API core and UAPI
>   media: request: add generic queue
>   media: request: add generic entity ops
>   media: vb2: add support for requests
>   media: vb2: add support for requests in QBUF ioctl
>   media: v4l2-mem2mem: add request support
>   media: vim2m: add media device
>   media: vim2m: add request support
> 
> Hans Verkuil (1):
>   videodev2.h: Add request field to v4l2_buffer
> 
>  drivers/media/Makefile                        |   4 +-
>  drivers/media/media-device.c                  |   6 +
>  drivers/media/media-request-entity-generic.c  |  56 ++++
>  drivers/media/media-request-queue-generic.c   | 150 ++++++++++
>  drivers/media/media-request.c                 | 390 ++++++++++++++++++++++++++
>  drivers/media/platform/vim2m.c                |  46 +++
>  drivers/media/usb/cpia2/cpia2_v4l.c           |   2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   7 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  99 ++++++-
>  drivers/media/v4l2-core/v4l2-mem2mem.c        |  34 +++
>  drivers/media/v4l2-core/videobuf2-core.c      |  59 +++-
>  drivers/media/v4l2-core/videobuf2-v4l2.c      |  32 ++-
>  include/media/media-device.h                  |   3 +
>  include/media/media-entity.h                  |   6 +
>  include/media/media-request.h                 | 282 +++++++++++++++++++
>  include/media/v4l2-mem2mem.h                  |  19 ++
>  include/media/videobuf2-core.h                |  25 +-
>  include/media/videobuf2-v4l2.h                |   2 +
>  include/uapi/linux/media.h                    |  11 +
>  include/uapi/linux/videodev2.h                |   3 +-
>  20 files changed, 1216 insertions(+), 20 deletions(-)
>  create mode 100644 drivers/media/media-request-entity-generic.c
>  create mode 100644 drivers/media/media-request-queue-generic.c
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: [RFC PATCH 9/9] media: vim2m: add request support

[Gustavo Padovan]

Hi Alex,

2017-12-15 Alexandre Courbot <acourbot@chromium.org>:

> Set the necessary ops for supporting requests in vim2m.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/platform/vim2m.c | 22 ++++++++++++++++++++++
>  1 file changed, 22 insertions(+)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index a32e8a7950eb..ffe94ef9214d 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -30,6 +30,7 @@
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-event.h>
>  #include <media/videobuf2-vmalloc.h>
> +#include <media/media-request.h>
>  
>  MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
>  MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
> @@ -142,6 +143,7 @@ struct vim2m_dev {
>  	struct video_device	vfd;
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  	struct media_device	mdev;
> +	struct media_request_queue *req_queue;
>  #endif
>  
>  	atomic_t		num_inst;
> @@ -937,6 +939,11 @@ static int vim2m_open(struct file *file)
>  		goto open_unlock;
>  	}
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	v4l2_mem_ctx_request_init(ctx->fh.m2m_ctx, dev->req_queue,
> +				  &dev->vfd.entity);
> +#endif
> +
>  	v4l2_fh_add(&ctx->fh);
>  	atomic_inc(&dev->num_inst);
>  
> @@ -992,6 +999,12 @@ static const struct v4l2_m2m_ops m2m_ops = {
>  	.job_abort	= job_abort,
>  };
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +static const struct media_entity_operations vim2m_entity_ops = {
> +	.process_request = v4l2_m2m_process_request,
> +};
> +#endif
> +
>  static int vim2m_probe(struct platform_device *pdev)
>  {
>  	struct vim2m_dev *dev;
> @@ -1006,6 +1019,10 @@ static int vim2m_probe(struct platform_device *pdev)
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  	dev->mdev.dev = &pdev->dev;
> +	dev->req_queue = media_request_queue_generic_alloc(&dev->mdev);
> +	if (IS_ERR(dev->req_queue))
> +		return PTR_ERR(dev->req_queue);
> +	dev->mdev.req_queue = dev->req_queue;
>  	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>  	media_device_init(&dev->mdev);
>  	dev->v4l2_dev.mdev = &dev->mdev;
> @@ -1023,6 +1040,11 @@ static int vim2m_probe(struct platform_device *pdev)
>  	vfd->lock = &dev->dev_mutex;
>  	vfd->v4l2_dev = &dev->v4l2_dev;
>  
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	vfd->entity.ops = &vim2m_entity_ops;
> +	vfd->entity.req_ops = &media_entity_request_generic_ops;
> +#endif
> +

It seems to me that we can avoid most of this patch and just setup the
request support automatically when the media device node is registered.
The less change we impose to drivers the better I think.

Gustavo

Re: [RFC PATCH 0/9] media: base request API support

[Alexandre Courbot]

On Sat, Dec 16, 2017 at 6:02 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> Le vendredi 15 décembre 2017 à 16:56 +0900, Alexandre Courbot a écrit :
>> Here is a new attempt at the request API, following the UAPI we agreed on in
>> Prague. Hopefully this can be used as the basis to move forward.
>>
>> This series only introduces the very basics of how requests work: allocate a
>> request, queue buffers to it, queue the request itself, wait for it to complete,
>> reuse it. It does *not* yet use Hans' work with controls setting. I have
>> preferred to submit it this way for now as it allows us to concentrate on the
>> basic request/buffer flow, which was harder to get properly than I initially
>> thought. I still have a gut feeling that it can be improved, with less back-and-
>> forth into drivers.
>>
>> Plugging in controls support should not be too hard a task (basically just apply
>> the saved controls when the request starts), and I am looking at it now.
>>
>> The resulting vim2m driver can be successfully used with requests, and my tests
>> so far have been successful.
>>
>> There are still some rougher edges:
>>
>> * locking is currently quite coarse-grained
>> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>>   As it is, some of the code will probably not even compile if
>>   CONFIG_MEDIA_CONTROLLER is not set
>>
>> But all in all I think the request flow should be clear and easy to review, and
>> the possibility of custom queue and entity support implementations should give
>> us the flexibility we need to support more specific use-cases (I expect the
>> generic implementations to be sufficient most of the time though).
>>
>> A very simple test program exercising this API is available here (don't forget
>> to adapt the /dev/media0 hardcoding):
>> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438
>
> It looks like the example uses Hans control work you just mention.
> Notably, it uses v4l2_ext_controls ctrls.request.

Oops, I indeed forgot to remove that bit. You will want to edit that
line out if trying to compile.

Re: [RFC PATCH 0/9] media: base request API support

[Alexandre Courbot]

On Sat, Dec 16, 2017 at 6:04 AM, Nicolas Dufresne <nicolas@ndufresne.ca> wrote:
> Le vendredi 15 décembre 2017 à 16:56 +0900, Alexandre Courbot a écrit :
>> Here is a new attempt at the request API, following the UAPI we agreed on in
>> Prague. Hopefully this can be used as the basis to move forward.
>>
>> This series only introduces the very basics of how requests work: allocate a
>> request, queue buffers to it, queue the request itself, wait for it to complete,
>> reuse it. It does *not* yet use Hans' work with controls setting. I have
>> preferred to submit it this way for now as it allows us to concentrate on the
>> basic request/buffer flow, which was harder to get properly than I initially
>> thought. I still have a gut feeling that it can be improved, with less back-and-
>> forth into drivers.
>>
>> Plugging in controls support should not be too hard a task (basically just apply
>> the saved controls when the request starts), and I am looking at it now.
>>
>> The resulting vim2m driver can be successfully used with requests, and my tests
>> so far have been successful.
>>
>> There are still some rougher edges:
>>
>> * locking is currently quite coarse-grained
>> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>>   As it is, some of the code will probably not even compile if
>>   CONFIG_MEDIA_CONTROLLER is not set
>
> Would it be possible to explain why this relation between request and
> the media controller ? Why couldn't request be created from video
> devices ?

Laurent replied on IRC, but for the record this is because the media
node can act as an orchestrator to the devices it manages. Some
devices may have particular needs in that respect, and we want to be
able to hook somewhere and control that.

Another reason is that in the future the media framework will probably
take more importance, to the point where you could control the whole
chain of devices through it instead of having to open every node
individually. This is a different topic though, and we only touched
the surface of it during the latest mini-summit.

Re: [RFC PATCH 9/9] media: vim2m: add request support

[Alexandre Courbot]

On Tue, Dec 19, 2017 at 5:53 AM, Gustavo Padovan <gustavo@padovan.org> wrote:
> Hi Alex,
>
> 2017-12-15 Alexandre Courbot <acourbot@chromium.org>:
>
>> Set the necessary ops for supporting requests in vim2m.
>>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/platform/vim2m.c | 22 ++++++++++++++++++++++
>>  1 file changed, 22 insertions(+)
>>
>> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
>> index a32e8a7950eb..ffe94ef9214d 100644
>> --- a/drivers/media/platform/vim2m.c
>> +++ b/drivers/media/platform/vim2m.c
>> @@ -30,6 +30,7 @@
>>  #include <media/v4l2-ctrls.h>
>>  #include <media/v4l2-event.h>
>>  #include <media/videobuf2-vmalloc.h>
>> +#include <media/media-request.h>
>>
>>  MODULE_DESCRIPTION("Virtual device for mem2mem framework testing");
>>  MODULE_AUTHOR("Pawel Osciak, <pawel@osciak.com>");
>> @@ -142,6 +143,7 @@ struct vim2m_dev {
>>       struct video_device     vfd;
>>  #ifdef CONFIG_MEDIA_CONTROLLER
>>       struct media_device     mdev;
>> +     struct media_request_queue *req_queue;
>>  #endif
>>
>>       atomic_t                num_inst;
>> @@ -937,6 +939,11 @@ static int vim2m_open(struct file *file)
>>               goto open_unlock;
>>       }
>>
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +     v4l2_mem_ctx_request_init(ctx->fh.m2m_ctx, dev->req_queue,
>> +                               &dev->vfd.entity);
>> +#endif
>> +
>>       v4l2_fh_add(&ctx->fh);
>>       atomic_inc(&dev->num_inst);
>>
>> @@ -992,6 +999,12 @@ static const struct v4l2_m2m_ops m2m_ops = {
>>       .job_abort      = job_abort,
>>  };
>>
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +static const struct media_entity_operations vim2m_entity_ops = {
>> +     .process_request = v4l2_m2m_process_request,
>> +};
>> +#endif
>> +
>>  static int vim2m_probe(struct platform_device *pdev)
>>  {
>>       struct vim2m_dev *dev;
>> @@ -1006,6 +1019,10 @@ static int vim2m_probe(struct platform_device *pdev)
>>
>>  #ifdef CONFIG_MEDIA_CONTROLLER
>>       dev->mdev.dev = &pdev->dev;
>> +     dev->req_queue = media_request_queue_generic_alloc(&dev->mdev);
>> +     if (IS_ERR(dev->req_queue))
>> +             return PTR_ERR(dev->req_queue);
>> +     dev->mdev.req_queue = dev->req_queue;
>>       strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
>>       media_device_init(&dev->mdev);
>>       dev->v4l2_dev.mdev = &dev->mdev;
>> @@ -1023,6 +1040,11 @@ static int vim2m_probe(struct platform_device *pdev)
>>       vfd->lock = &dev->dev_mutex;
>>       vfd->v4l2_dev = &dev->v4l2_dev;
>>
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +     vfd->entity.ops = &vim2m_entity_ops;
>> +     vfd->entity.req_ops = &media_entity_request_generic_ops;
>> +#endif
>> +
>
> It seems to me that we can avoid most of this patch and just setup the
> request support automatically when the media device node is registered.
> The less change we impose to drivers the better I think.

I'd like to move things more towards that direction, if possible. The
main blocker for now is that I want to keep the ability for drivers to
overload the request ops to fit their needs. But maybe we can just
allow some level of specialization if specified, while falling back to
sane defaults otherwise.

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Hans Verkuil]

Hi Alexandre,

A quick review: I'm primarily concentrating on the uAPI as that is the most
critical part to get right at this stage.

On 12/15/17 08:56, Alexandre Courbot wrote:
> The request API provides a way to group buffers and device parameters
> into units of work to be queued and executed. This patch introduces the
> UAPI and core framework.
> 
> This patch is based on the previous work by Laurent Pinchart. The core
> has changed considerably, but the UAPI is mostly untouched.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/Makefile               |   3 +-
>  drivers/media/media-device.c         |   6 +
>  drivers/media/media-request.c        | 390 +++++++++++++++++++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
>  include/media/media-device.h         |   3 +
>  include/media/media-entity.h         |   6 +
>  include/media/media-request.h        | 269 ++++++++++++++++++++++++
>  include/uapi/linux/media.h           |  11 +
>  8 files changed, 688 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 

<snip>

> diff --git a/include/media/media-request.h b/include/media/media-request.h
> new file mode 100644
> index 000000000000..ead7fd8898c4
> --- /dev/null
> +++ b/include/media/media-request.h
> @@ -0,0 +1,269 @@
> +/*
> + * Generic request queue.
> + *
> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef _MEDIA_REQUEST_H
> +#define _MEDIA_REQUEST_H
> +
> +#include <linux/kref.h>
> +#include <linux/wait.h>
> +#include <linux/list.h>
> +
> +struct media_device;
> +struct media_request_queue;
> +struct media_request_cmd;
> +struct media_entity;
> +struct media_request_entity_data;
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +
> +enum media_request_state {
> +	MEDIA_REQUEST_STATE_IDLE,
> +	MEDIA_REQUEST_STATE_QUEUED,
> +	MEDIA_REQUEST_STATE_COMPLETE,

COMPLETE -> COMPLETED

> +	MEDIA_REQUEST_STATE_DELETED,
> +};

Regards,

	Hans

Re: [RFC PATCH 4/9] videodev2.h: Add request field to v4l2_buffer

[Hans Verkuil]

On 12/15/17 08:56, Alexandre Courbot wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> When queuing buffers allow for passing the request ID that
> should be associated with this buffer.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> [acourbot@chromium.org: make request ID 32-bit]
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/usb/cpia2/cpia2_v4l.c           | 2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 7 ++++---
>  drivers/media/v4l2-core/v4l2-ioctl.c          | 4 ++--
>  drivers/media/v4l2-core/videobuf2-v4l2.c      | 3 ++-
>  include/media/videobuf2-v4l2.h                | 2 ++
>  include/uapi/linux/videodev2.h                | 3 ++-
>  6 files changed, 13 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c b/drivers/media/usb/cpia2/cpia2_v4l.c
> index 3dedd83f0b19..7217dde95a8a 100644
> --- a/drivers/media/usb/cpia2/cpia2_v4l.c
> +++ b/drivers/media/usb/cpia2/cpia2_v4l.c
> @@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
>  	buf->sequence = cam->buffers[buf->index].seq;
>  	buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
>  	buf->length = cam->frame_size;
> -	buf->reserved2 = 0;
> +	buf->request = 0;
>  	buf->reserved = 0;
>  	memset(&buf->timecode, 0, sizeof(buf->timecode));
>  
> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> index 821f2aa299ae..94f07c3b0b53 100644
> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
> @@ -370,7 +370,7 @@ struct v4l2_buffer32 {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	__u32			request;
>  	__u32			reserved;
>  };
>  
> @@ -438,7 +438,8 @@ static int get_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
>  		get_user(kp->type, &up->type) ||
>  		get_user(kp->flags, &up->flags) ||
>  		get_user(kp->memory, &up->memory) ||
> -		get_user(kp->length, &up->length))
> +		get_user(kp->length, &up->length) ||
> +		get_user(kp->request, &up->request))
>  			return -EFAULT;
>  
>  	if (V4L2_TYPE_IS_OUTPUT(kp->type))
> @@ -533,7 +534,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
>  		put_user(kp->timestamp.tv_usec, &up->timestamp.tv_usec) ||
>  		copy_to_user(&up->timecode, &kp->timecode, sizeof(struct v4l2_timecode)) ||
>  		put_user(kp->sequence, &up->sequence) ||
> -		put_user(kp->reserved2, &up->reserved2) ||
> +		put_user(kp->request, &up->request) ||
>  		put_user(kp->reserved, &up->reserved) ||
>  		put_user(kp->length, &up->length))
>  			return -EFAULT;
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index ec4ecd5aa8bf..8d041247e97f 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>  	const struct v4l2_plane *plane;
>  	int i;
>  
> -	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s",
> +	pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request=%u, flags=0x%08x, field=%s, sequence=%d, memory=%s",
>  			p->timestamp.tv_sec / 3600,
>  			(int)(p->timestamp.tv_sec / 60) % 60,
>  			(int)(p->timestamp.tv_sec % 60),
>  			(long)p->timestamp.tv_usec,
>  			p->index,
> -			prt_names(p->type, v4l2_type_names),
> +			prt_names(p->type, v4l2_type_names), p->request,
>  			p->flags, prt_names(p->field, v4l2_field_names),
>  			p->sequence, prt_names(p->memory, v4l2_memory_names));
>  
> diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
> index 0c0669976bdc..bde7b8a3a303 100644
> --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
> +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
> @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>  	b->timestamp = ns_to_timeval(vb->timestamp);
>  	b->timecode = vbuf->timecode;
>  	b->sequence = vbuf->sequence;
> -	b->reserved2 = 0;
> +	b->request = vbuf->request;
>  	b->reserved = 0;
>  
>  	if (q->is_multiplanar) {
> @@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
>  	}
>  	vb->timestamp = 0;
>  	vbuf->sequence = 0;
> +	vbuf->request = b->request;
>  
>  	if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>  		if (b->memory == VB2_MEMORY_USERPTR) {
> diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
> index 036127c54bbf..ef2be0ccff14 100644
> --- a/include/media/videobuf2-v4l2.h
> +++ b/include/media/videobuf2-v4l2.h
> @@ -31,6 +31,7 @@
>   * @field:	enum v4l2_field; field order of the image in the buffer
>   * @timecode:	frame timecode
>   * @sequence:	sequence count of this frame
> + * @request:	request used by the buffer
>   *
>   * Should contain enough information to be able to cover all the fields
>   * of struct v4l2_buffer at videodev2.h
> @@ -42,6 +43,7 @@ struct vb2_v4l2_buffer {
>  	__u32			field;
>  	struct v4l2_timecode	timecode;
>  	__u32			sequence;
> +	__u32			request;
>  };
>  
>  /*
> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
> index 1c095b5a99c5..0650e8d14971 100644
> --- a/include/uapi/linux/videodev2.h
> +++ b/include/uapi/linux/videodev2.h
> @@ -902,6 +902,7 @@ struct v4l2_plane {
>   * @length:	size in bytes of the buffer (NOT its payload) for single-plane
>   *		buffers (when type != *_MPLANE); number of elements in the
>   *		planes array for multi-plane buffers
> + * @request: this buffer should use this request
>   *
>   * Contains data exchanged by application and driver using one of the Streaming
>   * I/O methods.
> @@ -925,7 +926,7 @@ struct v4l2_buffer {
>  		__s32		fd;
>  	} m;
>  	__u32			length;
> -	__u32			reserved2;
> +	__u32			request;

Aren't file descriptors signed integers? So this would become __s32.

I also think I would prefer to rename this to request_fd, so it is clear this is a
file descriptor. Same for struct vb2_v4l2_buffer and struct v4l2_buffer32.

Also update the @request descriptions in the comments and v4l_print_buffer.

Regards,

	Hans

>  	__u32			reserved;
>  };
>  
> 

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Hans Verkuil]

On 12/15/17 08:56, Alexandre Courbot wrote:
> Add throttling support for buffers when requests are in use on a given
> queue. Buffers associated to a request are kept into the vb2 queue until
> the request becomes active, at which point all the buffers are passed to
> the driver. The queue can also signal that is has processed all of a
> request's buffers.
> 
> Also add support for the request parameter when handling the QBUF ioctl.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>  3 files changed, 104 insertions(+), 9 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
> index cb115ba6a1d2..c01038b7962a 100644
> --- a/drivers/media/v4l2-core/videobuf2-core.c
> +++ b/drivers/media/v4l2-core/videobuf2-core.c
> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		    state != VB2_BUF_STATE_REQUEUEING))
>  		state = VB2_BUF_STATE_ERROR;
>  
> +	WARN_ON(vb->request != q->cur_req);

What's the reason for this WARN_ON? It's not immediately obvious to me.

> +
>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Although this is not a callback, it still does have to balance
> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>  		/* Add the buffer to the done buffers list */
>  		list_add_tail(&vb->done_entry, &q->done_list);
>  		vb->state = state;
> +
> +		if (q->cur_req) {
> +			WARN_ON(q->req_buf_cnt < 1);
> +
> +			if (--q->req_buf_cnt == 0)
> +				q->cur_req = NULL;
> +		}
>  	}
>  	atomic_dec(&q->owned_by_drv_count);
>  	spin_unlock_irqrestore(&q->done_lock, flags);
> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>  
> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
> +{
> +	struct vb2_buffer *vb;
> +
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->request == q->cur_req)
> +			__enqueue_in_driver(vb);
> +	}
> +}

I think this will clash big time with the v4l2 fence patch series...

> +
>  /**
>   * vb2_start_streaming() - Attempt to start streaming.
>   * @q:		videobuf2 queue
> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	 * If any buffers were queued before streamon,
>  	 * we can now pass them to driver for processing.
>  	 */
> -	list_for_each_entry(vb, &q->queued_list, queued_entry)
> -		__enqueue_in_driver(vb);
> +	vb2_queue_enqueue_current_buffers(q);
>  
>  	/* Tell the driver to start streaming */
>  	q->start_streaming_called = 1;
> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>  	return ret;
>  }
>  
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
> +		  struct media_request *req, void *pb)
>  {
>  	struct vb2_buffer *vb;
>  	int ret;
> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	q->queued_count++;
>  	q->waiting_for_buffers = false;
>  	vb->state = VB2_BUF_STATE_QUEUED;
> +	vb->request = req;
>  
>  	if (pb)
>  		call_void_bufop(q, copy_timestamp, vb, pb);
> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  	/*
>  	 * If already streaming, give the buffer to driver for processing.
>  	 * If not, the buffer will be given to driver on next streamon.
> +	 *
> +	 * If using the request API, the buffer will be given to the driver
> +	 * when the request becomes active.
>  	 */
> -	if (q->start_streaming_called)
> +	if (q->start_streaming_called && !req)
>  		__enqueue_in_driver(vb);
>  
>  	/* Fill buffer information for the userspace */
> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>  }
>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>  
> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
> +{
> +	struct vb2_buffer *vb;
> +
> +	q->req_buf_cnt = 0;
> +	list_for_each_entry(vb, &q->queued_list, queued_entry) {
> +		if (vb->request == req)
> +			++q->req_buf_cnt;
> +	}
> +
> +	/* only consider the request if we actually have buffers for it */
> +	if (q->req_buf_cnt == 0)
> +		return;
> +
> +	q->cur_req = req;
> +
> +	/* If not streaming yet, we will enqueue the buffers later */
> +	if (q->start_streaming_called)
> +		vb2_queue_enqueue_current_buffers(q);

If I understand all this correctly, then you are queuing one request at a
time to the vb2_queue. I.e. all the buffers queued to the driver belong to
the same request (q->cur_req). But that might work for codecs, but not
for camera drivers: you will typically have multiple requests queued up in
the driver.

In any case, I don't think the req_buf_cnt and cur_req fields belong in
vb2_queue.

Another weird thing here is that it appears that you allow for multiple
buffers for the same device in the same request. I'm not sure that is
useful. For one, it will postpone the completion of the request until
all buffers are dequeued.

> +}
> +EXPORT_SYMBOL_GPL(vb2_queue_start_request);
> +
>  /**
>   * __vb2_wait_for_done_vb() - wait for a buffer to become available
>   * for dequeuing
> @@ -2242,7 +2287,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>  		 * Queue all buffers.
>  		 */
>  		for (i = 0; i < q->num_buffers; i++) {
> -			ret = vb2_core_qbuf(q, i, NULL);
> +			ret = vb2_core_qbuf(q, i, NULL, NULL);
>  			if (ret)
>  				goto err_reqbufs;
>  			fileio->bufs[i].queued = 1;
> @@ -2421,7 +2466,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>  
>  		if (copy_timestamp)
>  			b->timestamp = ktime_get_ns();
> -		ret = vb2_core_qbuf(q, index, NULL);
> +		ret = vb2_core_qbuf(q, index, NULL, NULL);
>  		dprintk(5, "vb2_dbuf result: %d\n", ret);
>  		if (ret)
>  			return ret;
> @@ -2524,7 +2569,7 @@ static int vb2_thread(void *data)
>  		if (copy_timestamp)
>  			vb->timestamp = ktime_get_ns();;
>  		if (!threadio->stop)
> -			ret = vb2_core_qbuf(q, vb->index, NULL);
> +			ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>  		call_void_qop(q, wait_prepare, q);
>  		if (ret || threadio->stop)
>  			break;
> diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
> index bde7b8a3a303..55b16b4db9a6 100644
> --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
> +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
> @@ -28,6 +28,7 @@
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-event.h>
>  #include <media/v4l2-common.h>
> +#include <media/media-request.h>
>  
>  #include <media/videobuf2-v4l2.h>
>  
> @@ -561,6 +562,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>  
>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  {
> +	struct media_request *req = NULL;
>  	int ret;
>  
>  	if (vb2_fileio_is_active(q)) {
> @@ -568,8 +570,33 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>  		return -EBUSY;
>  	}
>  
> +	/*
> +	 * The caller should have validated that the request is valid,
> +	 * so we just need to look it up without further checking
> +	 */
> +	if (b->request > 0) {
> +		req = media_request_get_from_fd(b->request);
> +		if (!req)
> +			return -EINVAL;
> +		media_request_queue_lock(req->queue);
> +
> +		if (req->state != MEDIA_REQUEST_STATE_IDLE) {
> +			ret = -EINVAL;
> +			goto done;
> +		}
> +	}
> +
>  	ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
> -	return ret ? ret : vb2_core_qbuf(q, b->index, b);
> +	if (!ret)
> +		ret = vb2_core_qbuf(q, b->index, req, b);
> +
> +done:
> +	if (req) {
> +		media_request_queue_unlock(req->queue);
> +		media_request_put(req);
> +	}
> +
> +	return ret;

I'm trying to remember what we decided w.r.t. mixing buffers associated with
a request and buffers without a request. I can't see anything in the meeting
notes. I have a faint memory that we decided to not allow that (i.e. once
you start using requests, then all buffers should be associated with a
request). Does anyone remember?

>  }
>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>  
> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
> index ef9b64398c8c..7d5e8e53e256 100644
> --- a/include/media/videobuf2-core.h
> +++ b/include/media/videobuf2-core.h
> @@ -44,6 +44,8 @@ enum vb2_memory {
>  struct vb2_fileio_data;
>  struct vb2_threadio_data;
>  
> +struct media_request;
> +
>  /**
>   * struct vb2_mem_ops - memory handling/memory allocator operations
>   * @alloc:	allocate video memory and, optionally, allocator private data,
> @@ -237,6 +239,7 @@ struct vb2_queue;
>   *			on an internal driver queue
>   * @planes:		private per-plane information; do not change
>   * @timestamp:		frame timestamp in ns
> + * @request:		pointer this buffer's request, if any
>   */
>  struct vb2_buffer {
>  	struct vb2_queue	*vb2_queue;
> @@ -246,6 +249,7 @@ struct vb2_buffer {
>  	unsigned int		num_planes;
>  	struct vb2_plane	planes[VB2_MAX_PLANES];
>  	u64			timestamp;
> +	struct media_request	*request;
>  
>  	/* private: internal use only
>  	 *
> @@ -500,6 +504,8 @@ struct vb2_buf_ops {
>   *		when a buffer with the V4L2_BUF_FLAG_LAST is dequeued.
>   * @fileio:	file io emulator internal data, used only if emulator is active
>   * @threadio:	thread io internal data, used only if thread is active
> + * @cur_req:	request currently being processed by this queue
> + * @req_buf_cnt:number of buffers still to process in the current request
>   */
>  struct vb2_queue {
>  	unsigned int			type;
> @@ -554,6 +560,9 @@ struct vb2_queue {
>  	struct vb2_fileio_data		*fileio;
>  	struct vb2_threadio_data	*threadio;
>  
> +	struct media_request		*cur_req;
> +	u32				req_buf_cnt;
> +

As mentioned above, this is very dubious.

>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>  	/*
>  	 * Counters for how often these queue-related ops are
> @@ -638,6 +647,20 @@ void vb2_discard_done(struct vb2_queue *q);
>   */
>  int vb2_wait_for_all_buffers(struct vb2_queue *q);
>  
> +/**
> + * vb2_queue_start_request() - make the queue process the given request
> + *
> + * @q:		videobuf2 queue
> + * @req:	request to make active
> + *
> + * When using the request API, buffers are not passed to drivers until their
> + * request becomes active. Drivers using the request API are thus expected to
> + * call this function whenever a new request becomes active, so all buffers
> + * assigned to it are processed.
> + */
> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req);
> +
> +
>  /**
>   * vb2_core_querybuf() - query video buffer information
>   * @q:		videobuf queue
> @@ -740,7 +763,7 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb);
>   * The return values from this function are intended to be directly returned
>   * from vidioc_qbuf handler in driver.
>   */
> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb);
> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, struct media_request *req, void *pb);
>  
>  /**
>   * vb2_core_dqbuf() - Dequeue a buffer to the userspace
> 

Regards,

	Hans

Re: [RFC PATCH 6/9] media: vb2: add support for requests in QBUF ioctl

[Hans Verkuil]

On 12/15/17 08:56, Alexandre Courbot wrote:
> Support the request argument of the QBUF ioctl.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c | 93 +++++++++++++++++++++++++++++++++++-
>  1 file changed, 92 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 8d041247e97f..28f9c368563e 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -29,6 +29,7 @@
>  #include <media/v4l2-device.h>
>  #include <media/videobuf2-v4l2.h>
>  #include <media/v4l2-mc.h>
> +#include <media/media-request.h>
>  
>  #include <trace/events/v4l2.h>
>  
> @@ -965,6 +966,81 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>  	return -EINVAL;
>  }
>  
> +/*
> + * Validate that a given request can be used during an ioctl.
> + *
> + * When using the request API, request file descriptors must be matched against
> + * the actual request object. User-space can pass any file descriptor, so we
> + * need to make sure the call is valid before going further.
> + *
> + * This function looks up the request and associated data and performs the
> + * following sanity checks:
> + *
> + * * Make sure that the entity supports requests,
> + * * Make sure that the entity belongs to the media_device managing the passed
> + *   request,
> + * * Make sure that the entity data (if any) is associated to the current file
> + *   handler.
> + *
> + * This function returns a pointer to the valid request, or and error code in
> + * case of failure. When successful, a reference to the request is acquired and
> + * must be properly released.
> + */
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +static struct media_request *
> +check_request(int request, struct file *file, void *fh)
> +{
> +	struct media_request *req = NULL;
> +	struct video_device *vfd = video_devdata(file);
> +	struct v4l2_fh *vfh =
> +		test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
> +	struct media_entity *entity = &vfd->entity;
> +	const struct media_entity *ent;
> +	struct media_request_entity_data *data;
> +	bool found = false;
> +
> +	if (!entity)
> +		return ERR_PTR(-EINVAL);
> +
> +	/* Check that the entity supports requests */
> +	if (!entity->req_ops)
> +		return ERR_PTR(-ENOTSUPP);
> +
> +	req = media_request_get_from_fd(request);

You can get the media_device from vfd->v4l2_dev->mdev. So it is much easier
to just pass the media_device as an argument to media_request_get_from_fd()...

> +	if (!req)
> +		return ERR_PTR(-EINVAL);
> +
> +	/* Validate that the entity belongs to the media_device managing
> +	 * the request queue */
> +	media_device_for_each_entity(ent, req->queue->mdev) {
> +		if (entity == ent) {
> +			found = true;
> +			break;
> +		}
> +	}
> +	if (!found) {
> +		media_request_put(req);
> +		return ERR_PTR(-EINVAL);
> +	}

...and then you don't need to do this ^^^ extra validation check.

> +
> +	/* Validate that the entity's data belongs to the correct fh */
> +	data = media_request_get_entity_data(req, entity, vfh);
> +	if (IS_ERR(data)) {
> +		media_request_put(req);
> +		return ERR_PTR(PTR_ERR(data));
> +	}

This assumes that each filehandle has its own state. That's true for codecs,
but not for most (all?) other devices. There the state is per device instance.

I'm not sure if we have a unique identifying mark for such drivers. The closest
is checking if fh->m2m_ctx is non-NULL, but I don't know if all drivers with
per-filehandle state use that field. An alternative might be to check if
fh->ctrl_handler is non-NULL. But again, I'm not sure if that's a 100% valid
check.

> +
> +	return req;
> +}
> +#else /* CONFIG_MEDIA_CONTROLLER */
> +static struct media_request *
> +check_request(int request, struct file *file, void *fh)
> +{
> +	return ERR_PTR(-ENOTSUPP);
> +}
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
>  static void v4l_sanitize_format(struct v4l2_format *fmt)
>  {
>  	unsigned int offset;
> @@ -1902,10 +1978,25 @@ static int v4l_querybuf(const struct v4l2_ioctl_ops *ops,
>  static int v4l_qbuf(const struct v4l2_ioctl_ops *ops,
>  				struct file *file, void *fh, void *arg)
>  {
> +	struct media_request *req = NULL;
>  	struct v4l2_buffer *p = arg;
>  	int ret = check_fmt(file, p->type);
>  
> -	return ret ? ret : ops->vidioc_qbuf(file, fh, p);
> +	if (ret)
> +		return ret;
> +
> +	if (p->request > 0) {
> +		req = check_request(p->request, file, fh);
> +		if (IS_ERR(req))
> +			return PTR_ERR(req);
> +	}
> +
> +	ret = ops->vidioc_qbuf(file, fh, p);
> +
> +	if (req)
> +		media_request_put(req);
> +
> +	return ret;
>  }
>  
>  static int v4l_dqbuf(const struct v4l2_ioctl_ops *ops,
> 

Regards,

	Hans

Re: [RFC PATCH 0/9] media: base request API support

[Hans Verkuil]

Hi Alexandre,

On 12/15/17 08:56, Alexandre Courbot wrote:
> Here is a new attempt at the request API, following the UAPI we agreed on in
> Prague. Hopefully this can be used as the basis to move forward.
> 
> This series only introduces the very basics of how requests work: allocate a
> request, queue buffers to it, queue the request itself, wait for it to complete,
> reuse it. It does *not* yet use Hans' work with controls setting. I have
> preferred to submit it this way for now as it allows us to concentrate on the
> basic request/buffer flow, which was harder to get properly than I initially
> thought. I still have a gut feeling that it can be improved, with less back-and-
> forth into drivers.
> 
> Plugging in controls support should not be too hard a task (basically just apply
> the saved controls when the request starts), and I am looking at it now.
> 
> The resulting vim2m driver can be successfully used with requests, and my tests
> so far have been successful.
> 
> There are still some rougher edges:
> 
> * locking is currently quite coarse-grained
> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>   As it is, some of the code will probably not even compile if
>   CONFIG_MEDIA_CONTROLLER is not set
> 
> But all in all I think the request flow should be clear and easy to review, and
> the possibility of custom queue and entity support implementations should give
> us the flexibility we need to support more specific use-cases (I expect the
> generic implementations to be sufficient most of the time though).
> 
> A very simple test program exercising this API is available here (don't forget
> to adapt the /dev/media0 hardcoding):
> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438
> 
> Looking forward to your feedback and comments!

I think this will become more interesting when the control code is in.

The main thing I've noticed with this patch series is that it is very codec
oriented. Which in some ways is OK (after all, that's the first type of HW
that we want to support), but the vb2 code in particular should be more
generic.

I would also recommend that you start preparing documentation patches: we
can review that and make sure all the corner-cases are correctly documented.

The public API changes are (I think) fairly limited, but the devil is in
the details, so getting that reviewed early on will help you later.

It's a bit unfortunate that the fence patch series is also making vb2 changes,
but I hope that will be merged fairly soon so you can develop on top of that
series.

Regards,

	Hans

> 
> Alexandre Courbot (8):
>   media: add request API core and UAPI
>   media: request: add generic queue
>   media: request: add generic entity ops
>   media: vb2: add support for requests
>   media: vb2: add support for requests in QBUF ioctl
>   media: v4l2-mem2mem: add request support
>   media: vim2m: add media device
>   media: vim2m: add request support
> 
> Hans Verkuil (1):
>   videodev2.h: Add request field to v4l2_buffer
> 
>  drivers/media/Makefile                        |   4 +-
>  drivers/media/media-device.c                  |   6 +
>  drivers/media/media-request-entity-generic.c  |  56 ++++
>  drivers/media/media-request-queue-generic.c   | 150 ++++++++++
>  drivers/media/media-request.c                 | 390 ++++++++++++++++++++++++++
>  drivers/media/platform/vim2m.c                |  46 +++
>  drivers/media/usb/cpia2/cpia2_v4l.c           |   2 +-
>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c |   7 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  99 ++++++-
>  drivers/media/v4l2-core/v4l2-mem2mem.c        |  34 +++
>  drivers/media/v4l2-core/videobuf2-core.c      |  59 +++-
>  drivers/media/v4l2-core/videobuf2-v4l2.c      |  32 ++-
>  include/media/media-device.h                  |   3 +
>  include/media/media-entity.h                  |   6 +
>  include/media/media-request.h                 | 282 +++++++++++++++++++
>  include/media/v4l2-mem2mem.h                  |  19 ++
>  include/media/videobuf2-core.h                |  25 +-
>  include/media/videobuf2-v4l2.h                |   2 +
>  include/uapi/linux/media.h                    |  11 +
>  include/uapi/linux/videodev2.h                |   3 +-
>  20 files changed, 1216 insertions(+), 20 deletions(-)
>  create mode 100644 drivers/media/media-request-entity-generic.c
>  create mode 100644 drivers/media/media-request-queue-generic.c
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 

Re: [RFC PATCH 0/9] media: base request API support

[Alexandre Courbot]

Hi Hans,

On Fri, Jan 12, 2018 at 8:45 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> Hi Alexandre,
>
> On 12/15/17 08:56, Alexandre Courbot wrote:
>> Here is a new attempt at the request API, following the UAPI we agreed on in
>> Prague. Hopefully this can be used as the basis to move forward.
>>
>> This series only introduces the very basics of how requests work: allocate a
>> request, queue buffers to it, queue the request itself, wait for it to complete,
>> reuse it. It does *not* yet use Hans' work with controls setting. I have
>> preferred to submit it this way for now as it allows us to concentrate on the
>> basic request/buffer flow, which was harder to get properly than I initially
>> thought. I still have a gut feeling that it can be improved, with less back-and-
>> forth into drivers.
>>
>> Plugging in controls support should not be too hard a task (basically just apply
>> the saved controls when the request starts), and I am looking at it now.
>>
>> The resulting vim2m driver can be successfully used with requests, and my tests
>> so far have been successful.
>>
>> There are still some rougher edges:
>>
>> * locking is currently quite coarse-grained
>> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>>   As it is, some of the code will probably not even compile if
>>   CONFIG_MEDIA_CONTROLLER is not set
>>
>> But all in all I think the request flow should be clear and easy to review, and
>> the possibility of custom queue and entity support implementations should give
>> us the flexibility we need to support more specific use-cases (I expect the
>> generic implementations to be sufficient most of the time though).
>>
>> A very simple test program exercising this API is available here (don't forget
>> to adapt the /dev/media0 hardcoding):
>> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438
>>
>> Looking forward to your feedback and comments!
>
> I think this will become more interesting when the control code is in.

Definitely.

> The main thing I've noticed with this patch series is that it is very codec
> oriented. Which in some ways is OK (after all, that's the first type of HW
> that we want to support), but the vb2 code in particular should be more
> generic.

I don't want to expand too much into use-cases I do not master ; doing
so would be speculating about how the API will be used. But feel free
to point out where you think my focus on the codec use-case is not
future-proof.

> I would also recommend that you start preparing documentation patches: we
> can review that and make sure all the corner-cases are correctly documented.
>
> The public API changes are (I think) fairly limited, but the devil is in
> the details, so getting that reviewed early on will help you later.

Yeah, I now regret to have submitted this series without
documentation. Won't do that mistake again.

> It's a bit unfortunate that the fence patch series is also making vb2 changes,
> but I hope that will be merged fairly soon so you can develop on top of that
> series.

The fence series may actually make things easier. The vb2 code of this
series is a bit confusing, and fences add a few extra constraints that
should make things more predictable. So I am looking forward to being
able to work on top of it.

Re: [RFC PATCH 4/9] videodev2.h: Add request field to v4l2_buffer

[Alexandre Courbot]

On Fri, Jan 12, 2018 at 7:22 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 12/15/17 08:56, Alexandre Courbot wrote:
>> From: Hans Verkuil <hans.verkuil@cisco.com>
>>
>> When queuing buffers allow for passing the request ID that
>> should be associated with this buffer.
>>
>> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
>> [acourbot@chromium.org: make request ID 32-bit]
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/usb/cpia2/cpia2_v4l.c           | 2 +-
>>  drivers/media/v4l2-core/v4l2-compat-ioctl32.c | 7 ++++---
>>  drivers/media/v4l2-core/v4l2-ioctl.c          | 4 ++--
>>  drivers/media/v4l2-core/videobuf2-v4l2.c      | 3 ++-
>>  include/media/videobuf2-v4l2.h                | 2 ++
>>  include/uapi/linux/videodev2.h                | 3 ++-
>>  6 files changed, 13 insertions(+), 8 deletions(-)
>>
>> diff --git a/drivers/media/usb/cpia2/cpia2_v4l.c b/drivers/media/usb/cpia2/cpia2_v4l.c
>> index 3dedd83f0b19..7217dde95a8a 100644
>> --- a/drivers/media/usb/cpia2/cpia2_v4l.c
>> +++ b/drivers/media/usb/cpia2/cpia2_v4l.c
>> @@ -948,7 +948,7 @@ static int cpia2_dqbuf(struct file *file, void *fh, struct v4l2_buffer *buf)
>>       buf->sequence = cam->buffers[buf->index].seq;
>>       buf->m.offset = cam->buffers[buf->index].data - cam->frame_buffer;
>>       buf->length = cam->frame_size;
>> -     buf->reserved2 = 0;
>> +     buf->request = 0;
>>       buf->reserved = 0;
>>       memset(&buf->timecode, 0, sizeof(buf->timecode));
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> index 821f2aa299ae..94f07c3b0b53 100644
>> --- a/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> +++ b/drivers/media/v4l2-core/v4l2-compat-ioctl32.c
>> @@ -370,7 +370,7 @@ struct v4l2_buffer32 {
>>               __s32           fd;
>>       } m;
>>       __u32                   length;
>> -     __u32                   reserved2;
>> +     __u32                   request;
>>       __u32                   reserved;
>>  };
>>
>> @@ -438,7 +438,8 @@ static int get_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
>>               get_user(kp->type, &up->type) ||
>>               get_user(kp->flags, &up->flags) ||
>>               get_user(kp->memory, &up->memory) ||
>> -             get_user(kp->length, &up->length))
>> +             get_user(kp->length, &up->length) ||
>> +             get_user(kp->request, &up->request))
>>                       return -EFAULT;
>>
>>       if (V4L2_TYPE_IS_OUTPUT(kp->type))
>> @@ -533,7 +534,7 @@ static int put_v4l2_buffer32(struct v4l2_buffer *kp, struct v4l2_buffer32 __user
>>               put_user(kp->timestamp.tv_usec, &up->timestamp.tv_usec) ||
>>               copy_to_user(&up->timecode, &kp->timecode, sizeof(struct v4l2_timecode)) ||
>>               put_user(kp->sequence, &up->sequence) ||
>> -             put_user(kp->reserved2, &up->reserved2) ||
>> +             put_user(kp->request, &up->request) ||
>>               put_user(kp->reserved, &up->reserved) ||
>>               put_user(kp->length, &up->length))
>>                       return -EFAULT;
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index ec4ecd5aa8bf..8d041247e97f 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -437,13 +437,13 @@ static void v4l_print_buffer(const void *arg, bool write_only)
>>       const struct v4l2_plane *plane;
>>       int i;
>>
>> -     pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, flags=0x%08x, field=%s, sequence=%d, memory=%s",
>> +     pr_cont("%02ld:%02d:%02d.%08ld index=%d, type=%s, request=%u, flags=0x%08x, field=%s, sequence=%d, memory=%s",
>>                       p->timestamp.tv_sec / 3600,
>>                       (int)(p->timestamp.tv_sec / 60) % 60,
>>                       (int)(p->timestamp.tv_sec % 60),
>>                       (long)p->timestamp.tv_usec,
>>                       p->index,
>> -                     prt_names(p->type, v4l2_type_names),
>> +                     prt_names(p->type, v4l2_type_names), p->request,
>>                       p->flags, prt_names(p->field, v4l2_field_names),
>>                       p->sequence, prt_names(p->memory, v4l2_memory_names));
>>
>> diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
>> index 0c0669976bdc..bde7b8a3a303 100644
>> --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
>> +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
>> @@ -203,7 +203,7 @@ static void __fill_v4l2_buffer(struct vb2_buffer *vb, void *pb)
>>       b->timestamp = ns_to_timeval(vb->timestamp);
>>       b->timecode = vbuf->timecode;
>>       b->sequence = vbuf->sequence;
>> -     b->reserved2 = 0;
>> +     b->request = vbuf->request;
>>       b->reserved = 0;
>>
>>       if (q->is_multiplanar) {
>> @@ -320,6 +320,7 @@ static int __fill_vb2_buffer(struct vb2_buffer *vb,
>>       }
>>       vb->timestamp = 0;
>>       vbuf->sequence = 0;
>> +     vbuf->request = b->request;
>>
>>       if (V4L2_TYPE_IS_MULTIPLANAR(b->type)) {
>>               if (b->memory == VB2_MEMORY_USERPTR) {
>> diff --git a/include/media/videobuf2-v4l2.h b/include/media/videobuf2-v4l2.h
>> index 036127c54bbf..ef2be0ccff14 100644
>> --- a/include/media/videobuf2-v4l2.h
>> +++ b/include/media/videobuf2-v4l2.h
>> @@ -31,6 +31,7 @@
>>   * @field:   enum v4l2_field; field order of the image in the buffer
>>   * @timecode:        frame timecode
>>   * @sequence:        sequence count of this frame
>> + * @request: request used by the buffer
>>   *
>>   * Should contain enough information to be able to cover all the fields
>>   * of struct v4l2_buffer at videodev2.h
>> @@ -42,6 +43,7 @@ struct vb2_v4l2_buffer {
>>       __u32                   field;
>>       struct v4l2_timecode    timecode;
>>       __u32                   sequence;
>> +     __u32                   request;
>>  };
>>
>>  /*
>> diff --git a/include/uapi/linux/videodev2.h b/include/uapi/linux/videodev2.h
>> index 1c095b5a99c5..0650e8d14971 100644
>> --- a/include/uapi/linux/videodev2.h
>> +++ b/include/uapi/linux/videodev2.h
>> @@ -902,6 +902,7 @@ struct v4l2_plane {
>>   * @length:  size in bytes of the buffer (NOT its payload) for single-plane
>>   *           buffers (when type != *_MPLANE); number of elements in the
>>   *           planes array for multi-plane buffers
>> + * @request: this buffer should use this request
>>   *
>>   * Contains data exchanged by application and driver using one of the Streaming
>>   * I/O methods.
>> @@ -925,7 +926,7 @@ struct v4l2_buffer {
>>               __s32           fd;
>>       } m;
>>       __u32                   length;
>> -     __u32                   reserved2;
>> +     __u32                   request;
>
> Aren't file descriptors signed integers? So this would become __s32.

Indeed, fixed.

>
> I also think I would prefer to rename this to request_fd, so it is clear this is a
> file descriptor. Same for struct vb2_v4l2_buffer and struct v4l2_buffer32.
>
> Also update the @request descriptions in the comments and v4l_print_buffer.

All done, thanks!

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Alexandre Courbot]

On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 12/15/17 08:56, Alexandre Courbot wrote:
>> Add throttling support for buffers when requests are in use on a given
>> queue. Buffers associated to a request are kept into the vb2 queue until
>> the request becomes active, at which point all the buffers are passed to
>> the driver. The queue can also signal that is has processed all of a
>> request's buffers.
>>
>> Also add support for the request parameter when handling the QBUF ioctl.
>>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>> index cb115ba6a1d2..c01038b7962a 100644
>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>                   state != VB2_BUF_STATE_REQUEUEING))
>>               state = VB2_BUF_STATE_ERROR;
>>
>> +     WARN_ON(vb->request != q->cur_req);
>
> What's the reason for this WARN_ON? It's not immediately obvious to me.

This is a safeguard against driver bugs: a buffer should not complete
unless it is part of the request being currently processed.

>
>> +
>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>       /*
>>        * Although this is not a callback, it still does have to balance
>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>               /* Add the buffer to the done buffers list */
>>               list_add_tail(&vb->done_entry, &q->done_list);
>>               vb->state = state;
>> +
>> +             if (q->cur_req) {
>> +                     WARN_ON(q->req_buf_cnt < 1);
>> +
>> +                     if (--q->req_buf_cnt == 0)
>> +                             q->cur_req = NULL;
>> +             }
>>       }
>>       atomic_dec(&q->owned_by_drv_count);
>>       spin_unlock_irqrestore(&q->done_lock, flags);
>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>
>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>> +{
>> +     struct vb2_buffer *vb;
>> +
>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +             if (vb->request == q->cur_req)
>> +                     __enqueue_in_driver(vb);
>> +     }
>> +}
>
> I think this will clash big time with the v4l2 fence patch series...

Indeed, but on the other hand I was not a big fan of going through the
whole list. :) So I welcome the extra throttling introduced by the
fence series.

>
>> +
>>  /**
>>   * vb2_start_streaming() - Attempt to start streaming.
>>   * @q:               videobuf2 queue
>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>        * If any buffers were queued before streamon,
>>        * we can now pass them to driver for processing.
>>        */
>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>> -             __enqueue_in_driver(vb);
>> +     vb2_queue_enqueue_current_buffers(q);
>>
>>       /* Tell the driver to start streaming */
>>       q->start_streaming_called = 1;
>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>       return ret;
>>  }
>>
>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>> +               struct media_request *req, void *pb)
>>  {
>>       struct vb2_buffer *vb;
>>       int ret;
>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>       q->queued_count++;
>>       q->waiting_for_buffers = false;
>>       vb->state = VB2_BUF_STATE_QUEUED;
>> +     vb->request = req;
>>
>>       if (pb)
>>               call_void_bufop(q, copy_timestamp, vb, pb);
>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>       /*
>>        * If already streaming, give the buffer to driver for processing.
>>        * If not, the buffer will be given to driver on next streamon.
>> +      *
>> +      * If using the request API, the buffer will be given to the driver
>> +      * when the request becomes active.
>>        */
>> -     if (q->start_streaming_called)
>> +     if (q->start_streaming_called && !req)
>>               __enqueue_in_driver(vb);
>>
>>       /* Fill buffer information for the userspace */
>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>
>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>> +{
>> +     struct vb2_buffer *vb;
>> +
>> +     q->req_buf_cnt = 0;
>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>> +             if (vb->request == req)
>> +                     ++q->req_buf_cnt;
>> +     }
>> +
>> +     /* only consider the request if we actually have buffers for it */
>> +     if (q->req_buf_cnt == 0)
>> +             return;
>> +
>> +     q->cur_req = req;
>> +
>> +     /* If not streaming yet, we will enqueue the buffers later */
>> +     if (q->start_streaming_called)
>> +             vb2_queue_enqueue_current_buffers(q);
>
> If I understand all this correctly, then you are queuing one request at a
> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
> the same request (q->cur_req).

That is correct.

> But that might work for codecs, but not
> for camera drivers: you will typically have multiple requests queued up in
> the driver.

Aren't requests supposed to be performed sequentially, even in the
camera case? Passing a buffer to the driver means that we allow it to
process it using its current settings ; if another request is
currently active, wouldn't that become an issue?

>
> In any case, I don't think the req_buf_cnt and cur_req fields belong in
> vb2_queue.
>
> Another weird thing here is that it appears that you allow for multiple
> buffers for the same device in the same request. I'm not sure that is
> useful. For one, it will postpone the completion of the request until
> all buffers are dequeued.

s/dequeued/completed. A request is marked as completed as soon as all
its buffers are marked as done, and can be polled before its buffers
are dequeued by user-space.

Do you suggest that we enforce a "one buffer per queue per request"
rule? I cannot thing of any case where this would be a hard limiting
factor (and it would certainly simplify the code), on the other hand
it may slow things down a bit in the case where we want to e.g. take
several shots in fast succession with the same parameters. But I have
no evidence that the extra lag would be noticeable.

>
>> +}
>> +EXPORT_SYMBOL_GPL(vb2_queue_start_request);
>> +
>>  /**
>>   * __vb2_wait_for_done_vb() - wait for a buffer to become available
>>   * for dequeuing
>> @@ -2242,7 +2287,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>>                * Queue all buffers.
>>                */
>>               for (i = 0; i < q->num_buffers; i++) {
>> -                     ret = vb2_core_qbuf(q, i, NULL);
>> +                     ret = vb2_core_qbuf(q, i, NULL, NULL);
>>                       if (ret)
>>                               goto err_reqbufs;
>>                       fileio->bufs[i].queued = 1;
>> @@ -2421,7 +2466,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>>
>>               if (copy_timestamp)
>>                       b->timestamp = ktime_get_ns();
>> -             ret = vb2_core_qbuf(q, index, NULL);
>> +             ret = vb2_core_qbuf(q, index, NULL, NULL);
>>               dprintk(5, "vb2_dbuf result: %d\n", ret);
>>               if (ret)
>>                       return ret;
>> @@ -2524,7 +2569,7 @@ static int vb2_thread(void *data)
>>               if (copy_timestamp)
>>                       vb->timestamp = ktime_get_ns();;
>>               if (!threadio->stop)
>> -                     ret = vb2_core_qbuf(q, vb->index, NULL);
>> +                     ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>>               call_void_qop(q, wait_prepare, q);
>>               if (ret || threadio->stop)
>>                       break;
>> diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
>> index bde7b8a3a303..55b16b4db9a6 100644
>> --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
>> +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
>> @@ -28,6 +28,7 @@
>>  #include <media/v4l2-fh.h>
>>  #include <media/v4l2-event.h>
>>  #include <media/v4l2-common.h>
>> +#include <media/media-request.h>
>>
>>  #include <media/videobuf2-v4l2.h>
>>
>> @@ -561,6 +562,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>>
>>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>>  {
>> +     struct media_request *req = NULL;
>>       int ret;
>>
>>       if (vb2_fileio_is_active(q)) {
>> @@ -568,8 +570,33 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>>               return -EBUSY;
>>       }
>>
>> +     /*
>> +      * The caller should have validated that the request is valid,
>> +      * so we just need to look it up without further checking
>> +      */
>> +     if (b->request > 0) {
>> +             req = media_request_get_from_fd(b->request);
>> +             if (!req)
>> +                     return -EINVAL;
>> +             media_request_queue_lock(req->queue);
>> +
>> +             if (req->state != MEDIA_REQUEST_STATE_IDLE) {
>> +                     ret = -EINVAL;
>> +                     goto done;
>> +             }
>> +     }
>> +
>>       ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
>> -     return ret ? ret : vb2_core_qbuf(q, b->index, b);
>> +     if (!ret)
>> +             ret = vb2_core_qbuf(q, b->index, req, b);
>> +
>> +done:
>> +     if (req) {
>> +             media_request_queue_unlock(req->queue);
>> +             media_request_put(req);
>> +     }
>> +
>> +     return ret;
>
> I'm trying to remember what we decided w.r.t. mixing buffers associated with
> a request and buffers without a request. I can't see anything in the meeting
> notes. I have a faint memory that we decided to not allow that (i.e. once
> you start using requests, then all buffers should be associated with a
> request). Does anyone remember?

My recollection is that we do not support that, on the other hand is
there a case where allowing it would break things? Especially if we
decide to limit each request to one buffer per queue.

>
>>  }
>>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>>
>> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>> index ef9b64398c8c..7d5e8e53e256 100644
>> --- a/include/media/videobuf2-core.h
>> +++ b/include/media/videobuf2-core.h
>> @@ -44,6 +44,8 @@ enum vb2_memory {
>>  struct vb2_fileio_data;
>>  struct vb2_threadio_data;
>>
>> +struct media_request;
>> +
>>  /**
>>   * struct vb2_mem_ops - memory handling/memory allocator operations
>>   * @alloc:   allocate video memory and, optionally, allocator private data,
>> @@ -237,6 +239,7 @@ struct vb2_queue;
>>   *                   on an internal driver queue
>>   * @planes:          private per-plane information; do not change
>>   * @timestamp:               frame timestamp in ns
>> + * @request:         pointer this buffer's request, if any
>>   */
>>  struct vb2_buffer {
>>       struct vb2_queue        *vb2_queue;
>> @@ -246,6 +249,7 @@ struct vb2_buffer {
>>       unsigned int            num_planes;
>>       struct vb2_plane        planes[VB2_MAX_PLANES];
>>       u64                     timestamp;
>> +     struct media_request    *request;
>>
>>       /* private: internal use only
>>        *
>> @@ -500,6 +504,8 @@ struct vb2_buf_ops {
>>   *           when a buffer with the V4L2_BUF_FLAG_LAST is dequeued.
>>   * @fileio:  file io emulator internal data, used only if emulator is active
>>   * @threadio:        thread io internal data, used only if thread is active
>> + * @cur_req: request currently being processed by this queue
>> + * @req_buf_cnt:number of buffers still to process in the current request
>>   */
>>  struct vb2_queue {
>>       unsigned int                    type;
>> @@ -554,6 +560,9 @@ struct vb2_queue {
>>       struct vb2_fileio_data          *fileio;
>>       struct vb2_threadio_data        *threadio;
>>
>> +     struct media_request            *cur_req;
>> +     u32                             req_buf_cnt;
>> +
>
> As mentioned above, this is very dubious.

This wouldn't be needed if I understood your intent properly about
limiting requests to one buffer per queue. If not, where would you
suggest to move this?

Re: [RFC PATCH 6/9] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

On Fri, Jan 12, 2018 at 8:37 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 12/15/17 08:56, Alexandre Courbot wrote:
>> Support the request argument of the QBUF ioctl.
>>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/v4l2-core/v4l2-ioctl.c | 93 +++++++++++++++++++++++++++++++++++-
>>  1 file changed, 92 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 8d041247e97f..28f9c368563e 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -29,6 +29,7 @@
>>  #include <media/v4l2-device.h>
>>  #include <media/videobuf2-v4l2.h>
>>  #include <media/v4l2-mc.h>
>> +#include <media/media-request.h>
>>
>>  #include <trace/events/v4l2.h>
>>
>> @@ -965,6 +966,81 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>       return -EINVAL;
>>  }
>>
>> +/*
>> + * Validate that a given request can be used during an ioctl.
>> + *
>> + * When using the request API, request file descriptors must be matched against
>> + * the actual request object. User-space can pass any file descriptor, so we
>> + * need to make sure the call is valid before going further.
>> + *
>> + * This function looks up the request and associated data and performs the
>> + * following sanity checks:
>> + *
>> + * * Make sure that the entity supports requests,
>> + * * Make sure that the entity belongs to the media_device managing the passed
>> + *   request,
>> + * * Make sure that the entity data (if any) is associated to the current file
>> + *   handler.
>> + *
>> + * This function returns a pointer to the valid request, or and error code in
>> + * case of failure. When successful, a reference to the request is acquired and
>> + * must be properly released.
>> + */
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +static struct media_request *
>> +check_request(int request, struct file *file, void *fh)
>> +{
>> +     struct media_request *req = NULL;
>> +     struct video_device *vfd = video_devdata(file);
>> +     struct v4l2_fh *vfh =
>> +             test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
>> +     struct media_entity *entity = &vfd->entity;
>> +     const struct media_entity *ent;
>> +     struct media_request_entity_data *data;
>> +     bool found = false;
>> +
>> +     if (!entity)
>> +             return ERR_PTR(-EINVAL);
>> +
>> +     /* Check that the entity supports requests */
>> +     if (!entity->req_ops)
>> +             return ERR_PTR(-ENOTSUPP);
>> +
>> +     req = media_request_get_from_fd(request);
>
> You can get the media_device from vfd->v4l2_dev->mdev. So it is much easier
> to just pass the media_device as an argument to media_request_get_from_fd()...
>
>> +     if (!req)
>> +             return ERR_PTR(-EINVAL);
>> +
>> +     /* Validate that the entity belongs to the media_device managing
>> +      * the request queue */
>> +     media_device_for_each_entity(ent, req->queue->mdev) {
>> +             if (entity == ent) {
>> +                     found = true;
>> +                     break;
>> +             }
>> +     }
>> +     if (!found) {
>> +             media_request_put(req);
>> +             return ERR_PTR(-EINVAL);
>> +     }
>
> ...and then you don't need to do this ^^^ extra validation check.

Ah right, all you need to do is check that req->queue->mdev ==
vfd->v4l2_dev->mdev and you can get rid of this whole block. I don't
think we can do that in media_request_get_from_fd() though, since it
is called from other places where (IIUC) we don't have access to the
media_device.

>
>> +
>> +     /* Validate that the entity's data belongs to the correct fh */
>> +     data = media_request_get_entity_data(req, entity, vfh);
>> +     if (IS_ERR(data)) {
>> +             media_request_put(req);
>> +             return ERR_PTR(PTR_ERR(data));
>> +     }
>
> This assumes that each filehandle has its own state. That's true for codecs,
> but not for most (all?) other devices. There the state is per device instance.
>
> I'm not sure if we have a unique identifying mark for such drivers. The closest
> is checking if fh->m2m_ctx is non-NULL, but I don't know if all drivers with
> per-filehandle state use that field. An alternative might be to check if
> fh->ctrl_handler is non-NULL. But again, I'm not sure if that's a 100% valid
> check.

I think the current code already takes that case into account: if the
device does not uses v4l2_fh, then the fh argument passed to
media_request_get_entity_data() will be null, and so will be data->fh.

Re: [RFC PATCH 0/9] media: base request API support

[Hans Verkuil]

On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
> Hi Hans,
> 
> On Fri, Jan 12, 2018 at 8:45 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> Hi Alexandre,
>>
>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>> Here is a new attempt at the request API, following the UAPI we agreed on in
>>> Prague. Hopefully this can be used as the basis to move forward.
>>>
>>> This series only introduces the very basics of how requests work: allocate a
>>> request, queue buffers to it, queue the request itself, wait for it to complete,
>>> reuse it. It does *not* yet use Hans' work with controls setting. I have
>>> preferred to submit it this way for now as it allows us to concentrate on the
>>> basic request/buffer flow, which was harder to get properly than I initially
>>> thought. I still have a gut feeling that it can be improved, with less back-and-
>>> forth into drivers.
>>>
>>> Plugging in controls support should not be too hard a task (basically just apply
>>> the saved controls when the request starts), and I am looking at it now.
>>>
>>> The resulting vim2m driver can be successfully used with requests, and my tests
>>> so far have been successful.
>>>
>>> There are still some rougher edges:
>>>
>>> * locking is currently quite coarse-grained
>>> * too many #ifdef CONFIG_MEDIA_CONTROLLER in the code, as the request API
>>>   depends on it - I plan to craft the headers so that it becomes unnecessary.
>>>   As it is, some of the code will probably not even compile if
>>>   CONFIG_MEDIA_CONTROLLER is not set
>>>
>>> But all in all I think the request flow should be clear and easy to review, and
>>> the possibility of custom queue and entity support implementations should give
>>> us the flexibility we need to support more specific use-cases (I expect the
>>> generic implementations to be sufficient most of the time though).
>>>
>>> A very simple test program exercising this API is available here (don't forget
>>> to adapt the /dev/media0 hardcoding):
>>> https://gist.github.com/Gnurou/dbc3776ed97ea7d4ce6041ea15eb0438
>>>
>>> Looking forward to your feedback and comments!
>>
>> I think this will become more interesting when the control code is in.
> 
> Definitely.
> 
>> The main thing I've noticed with this patch series is that it is very codec
>> oriented. Which in some ways is OK (after all, that's the first type of HW
>> that we want to support), but the vb2 code in particular should be more
>> generic.
> 
> I don't want to expand too much into use-cases I do not master ; doing
> so would be speculating about how the API will be used. But feel free
> to point out where you think my focus on the codec use-case is not
> future-proof.

The vb2 framework is a core component and it should function properly with
the request API for all drivers using it.

Easiest would be to test this using vivid and vimc. The vb2 code is relatively
simple: all it has to do is keep track of requests and buffers and queue them
up to the driver at the right time. It doesn't really need to know if the driver
is a codec or something else, it's all the same to the framework.

Regards,

	Hans

>> I would also recommend that you start preparing documentation patches: we
>> can review that and make sure all the corner-cases are correctly documented.
>>
>> The public API changes are (I think) fairly limited, but the devil is in
>> the details, so getting that reviewed early on will help you later.
> 
> Yeah, I now regret to have submitted this series without
> documentation. Won't do that mistake again.
> 
>> It's a bit unfortunate that the fence patch series is also making vb2 changes,
>> but I hope that will be merged fairly soon so you can develop on top of that
>> series.
> 
> The fence series may actually make things easier. The vb2 code of this
> series is a bit confusing, and fences add a few extra constraints that
> should make things more predictable. So I am looking forward to being
> able to work on top of it.
> 

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Hans Verkuil]

On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
> On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>> Add throttling support for buffers when requests are in use on a given
>>> queue. Buffers associated to a request are kept into the vb2 queue until
>>> the request becomes active, at which point all the buffers are passed to
>>> the driver. The queue can also signal that is has processed all of a
>>> request's buffers.
>>>
>>> Also add support for the request parameter when handling the QBUF ioctl.
>>>
>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>> ---
>>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>>
>>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>>> index cb115ba6a1d2..c01038b7962a 100644
>>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>                   state != VB2_BUF_STATE_REQUEUEING))
>>>               state = VB2_BUF_STATE_ERROR;
>>>
>>> +     WARN_ON(vb->request != q->cur_req);
>>
>> What's the reason for this WARN_ON? It's not immediately obvious to me.
> 
> This is a safeguard against driver bugs: a buffer should not complete
> unless it is part of the request being currently processed.
> 
>>
>>> +
>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>       /*
>>>        * Although this is not a callback, it still does have to balance
>>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>               /* Add the buffer to the done buffers list */
>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>               vb->state = state;
>>> +
>>> +             if (q->cur_req) {
>>> +                     WARN_ON(q->req_buf_cnt < 1);
>>> +
>>> +                     if (--q->req_buf_cnt == 0)
>>> +                             q->cur_req = NULL;
>>> +             }
>>>       }
>>>       atomic_dec(&q->owned_by_drv_count);
>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>>
>>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>>> +{
>>> +     struct vb2_buffer *vb;
>>> +
>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>> +             if (vb->request == q->cur_req)
>>> +                     __enqueue_in_driver(vb);
>>> +     }
>>> +}
>>
>> I think this will clash big time with the v4l2 fence patch series...
> 
> Indeed, but on the other hand I was not a big fan of going through the
> whole list. :) So I welcome the extra throttling introduced by the
> fence series.

There is only throttling if fences are used by userspace. Otherwise there
is no change.

> 
>>
>>> +
>>>  /**
>>>   * vb2_start_streaming() - Attempt to start streaming.
>>>   * @q:               videobuf2 queue
>>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>        * If any buffers were queued before streamon,
>>>        * we can now pass them to driver for processing.
>>>        */
>>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>>> -             __enqueue_in_driver(vb);
>>> +     vb2_queue_enqueue_current_buffers(q);
>>>
>>>       /* Tell the driver to start streaming */
>>>       q->start_streaming_called = 1;
>>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>       return ret;
>>>  }
>>>
>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>>> +               struct media_request *req, void *pb)
>>>  {
>>>       struct vb2_buffer *vb;
>>>       int ret;
>>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>       q->queued_count++;
>>>       q->waiting_for_buffers = false;
>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>> +     vb->request = req;
>>>
>>>       if (pb)
>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>       /*
>>>        * If already streaming, give the buffer to driver for processing.
>>>        * If not, the buffer will be given to driver on next streamon.
>>> +      *
>>> +      * If using the request API, the buffer will be given to the driver
>>> +      * when the request becomes active.
>>>        */
>>> -     if (q->start_streaming_called)
>>> +     if (q->start_streaming_called && !req)
>>>               __enqueue_in_driver(vb);
>>>
>>>       /* Fill buffer information for the userspace */
>>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>
>>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>>> +{
>>> +     struct vb2_buffer *vb;
>>> +
>>> +     q->req_buf_cnt = 0;
>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>> +             if (vb->request == req)
>>> +                     ++q->req_buf_cnt;
>>> +     }
>>> +
>>> +     /* only consider the request if we actually have buffers for it */
>>> +     if (q->req_buf_cnt == 0)
>>> +             return;
>>> +
>>> +     q->cur_req = req;
>>> +
>>> +     /* If not streaming yet, we will enqueue the buffers later */
>>> +     if (q->start_streaming_called)
>>> +             vb2_queue_enqueue_current_buffers(q);
>>
>> If I understand all this correctly, then you are queuing one request at a
>> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
>> the same request (q->cur_req).
> 
> That is correct.
> 
>> But that might work for codecs, but not
>> for camera drivers: you will typically have multiple requests queued up in
>> the driver.
> 
> Aren't requests supposed to be performed sequentially, even in the
> camera case? Passing a buffer to the driver means that we allow it to
> process it using its current settings ; if another request is
> currently active, wouldn't that become an issue?

Drivers often need multiple buffers queued before they can start the DMA
engine (usually at least two buffers have to be queued, just do a
git grep min_buffers_needed drivers/media).

In addition, sensors often have to be programmed one or two frames earlier
for a new setting to take effect for the required frame.

In other words: drivers need to be able to look ahead and vb2 should just
queue buffers/requests as soon as they are ready.

> 
>>
>> In any case, I don't think the req_buf_cnt and cur_req fields belong in
>> vb2_queue.
>>
>> Another weird thing here is that it appears that you allow for multiple
>> buffers for the same device in the same request. I'm not sure that is
>> useful. For one, it will postpone the completion of the request until
>> all buffers are dequeued.
> 
> s/dequeued/completed. A request is marked as completed as soon as all
> its buffers are marked as done, and can be polled before its buffers
> are dequeued by user-space.
> 
> Do you suggest that we enforce a "one buffer per queue per request"
> rule? I cannot thing of any case where this would be a hard limiting
> factor (and it would certainly simplify the code), on the other hand
> it may slow things down a bit in the case where we want to e.g. take
> several shots in fast succession with the same parameters. But I have
> no evidence that the extra lag would be noticeable.

This is something that should go into an RFC for discussion. My opinion
at this time is that you should stick with one buffer per queue per request.
This can always be relaxed in the future. For now it is better to keep
it simple.

> 
>>
>>> +}
>>> +EXPORT_SYMBOL_GPL(vb2_queue_start_request);
>>> +
>>>  /**
>>>   * __vb2_wait_for_done_vb() - wait for a buffer to become available
>>>   * for dequeuing
>>> @@ -2242,7 +2287,7 @@ static int __vb2_init_fileio(struct vb2_queue *q, int read)
>>>                * Queue all buffers.
>>>                */
>>>               for (i = 0; i < q->num_buffers; i++) {
>>> -                     ret = vb2_core_qbuf(q, i, NULL);
>>> +                     ret = vb2_core_qbuf(q, i, NULL, NULL);
>>>                       if (ret)
>>>                               goto err_reqbufs;
>>>                       fileio->bufs[i].queued = 1;
>>> @@ -2421,7 +2466,7 @@ static size_t __vb2_perform_fileio(struct vb2_queue *q, char __user *data, size_
>>>
>>>               if (copy_timestamp)
>>>                       b->timestamp = ktime_get_ns();
>>> -             ret = vb2_core_qbuf(q, index, NULL);
>>> +             ret = vb2_core_qbuf(q, index, NULL, NULL);
>>>               dprintk(5, "vb2_dbuf result: %d\n", ret);
>>>               if (ret)
>>>                       return ret;
>>> @@ -2524,7 +2569,7 @@ static int vb2_thread(void *data)
>>>               if (copy_timestamp)
>>>                       vb->timestamp = ktime_get_ns();;
>>>               if (!threadio->stop)
>>> -                     ret = vb2_core_qbuf(q, vb->index, NULL);
>>> +                     ret = vb2_core_qbuf(q, vb->index, NULL, NULL);
>>>               call_void_qop(q, wait_prepare, q);
>>>               if (ret || threadio->stop)
>>>                       break;
>>> diff --git a/drivers/media/v4l2-core/videobuf2-v4l2.c b/drivers/media/v4l2-core/videobuf2-v4l2.c
>>> index bde7b8a3a303..55b16b4db9a6 100644
>>> --- a/drivers/media/v4l2-core/videobuf2-v4l2.c
>>> +++ b/drivers/media/v4l2-core/videobuf2-v4l2.c
>>> @@ -28,6 +28,7 @@
>>>  #include <media/v4l2-fh.h>
>>>  #include <media/v4l2-event.h>
>>>  #include <media/v4l2-common.h>
>>> +#include <media/media-request.h>
>>>
>>>  #include <media/videobuf2-v4l2.h>
>>>
>>> @@ -561,6 +562,7 @@ EXPORT_SYMBOL_GPL(vb2_create_bufs);
>>>
>>>  int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>>>  {
>>> +     struct media_request *req = NULL;
>>>       int ret;
>>>
>>>       if (vb2_fileio_is_active(q)) {
>>> @@ -568,8 +570,33 @@ int vb2_qbuf(struct vb2_queue *q, struct v4l2_buffer *b)
>>>               return -EBUSY;
>>>       }
>>>
>>> +     /*
>>> +      * The caller should have validated that the request is valid,
>>> +      * so we just need to look it up without further checking
>>> +      */
>>> +     if (b->request > 0) {
>>> +             req = media_request_get_from_fd(b->request);
>>> +             if (!req)
>>> +                     return -EINVAL;
>>> +             media_request_queue_lock(req->queue);
>>> +
>>> +             if (req->state != MEDIA_REQUEST_STATE_IDLE) {
>>> +                     ret = -EINVAL;
>>> +                     goto done;
>>> +             }
>>> +     }
>>> +
>>>       ret = vb2_queue_or_prepare_buf(q, b, "qbuf");
>>> -     return ret ? ret : vb2_core_qbuf(q, b->index, b);
>>> +     if (!ret)
>>> +             ret = vb2_core_qbuf(q, b->index, req, b);
>>> +
>>> +done:
>>> +     if (req) {
>>> +             media_request_queue_unlock(req->queue);
>>> +             media_request_put(req);
>>> +     }
>>> +
>>> +     return ret;
>>
>> I'm trying to remember what we decided w.r.t. mixing buffers associated with
>> a request and buffers without a request. I can't see anything in the meeting
>> notes. I have a faint memory that we decided to not allow that (i.e. once
>> you start using requests, then all buffers should be associated with a
>> request). Does anyone remember?
> 
> My recollection is that we do not support that, on the other hand is
> there a case where allowing it would break things? Especially if we
> decide to limit each request to one buffer per queue.

Also something for an RFC.

> 
>>
>>>  }
>>>  EXPORT_SYMBOL_GPL(vb2_qbuf);
>>>
>>> diff --git a/include/media/videobuf2-core.h b/include/media/videobuf2-core.h
>>> index ef9b64398c8c..7d5e8e53e256 100644
>>> --- a/include/media/videobuf2-core.h
>>> +++ b/include/media/videobuf2-core.h
>>> @@ -44,6 +44,8 @@ enum vb2_memory {
>>>  struct vb2_fileio_data;
>>>  struct vb2_threadio_data;
>>>
>>> +struct media_request;
>>> +
>>>  /**
>>>   * struct vb2_mem_ops - memory handling/memory allocator operations
>>>   * @alloc:   allocate video memory and, optionally, allocator private data,
>>> @@ -237,6 +239,7 @@ struct vb2_queue;
>>>   *                   on an internal driver queue
>>>   * @planes:          private per-plane information; do not change
>>>   * @timestamp:               frame timestamp in ns
>>> + * @request:         pointer this buffer's request, if any
>>>   */
>>>  struct vb2_buffer {
>>>       struct vb2_queue        *vb2_queue;
>>> @@ -246,6 +249,7 @@ struct vb2_buffer {
>>>       unsigned int            num_planes;
>>>       struct vb2_plane        planes[VB2_MAX_PLANES];
>>>       u64                     timestamp;
>>> +     struct media_request    *request;
>>>
>>>       /* private: internal use only
>>>        *
>>> @@ -500,6 +504,8 @@ struct vb2_buf_ops {
>>>   *           when a buffer with the V4L2_BUF_FLAG_LAST is dequeued.
>>>   * @fileio:  file io emulator internal data, used only if emulator is active
>>>   * @threadio:        thread io internal data, used only if thread is active
>>> + * @cur_req: request currently being processed by this queue
>>> + * @req_buf_cnt:number of buffers still to process in the current request
>>>   */
>>>  struct vb2_queue {
>>>       unsigned int                    type;
>>> @@ -554,6 +560,9 @@ struct vb2_queue {
>>>       struct vb2_fileio_data          *fileio;
>>>       struct vb2_threadio_data        *threadio;
>>>
>>> +     struct media_request            *cur_req;
>>> +     u32                             req_buf_cnt;
>>> +
>>
>> As mentioned above, this is very dubious.
> 
> This wouldn't be needed if I understood your intent properly about
> limiting requests to one buffer per queue. If not, where would you
> suggest to move this?
> 

You understood correctly. I think this just can be dropped.

Regards,

	Hans

Re: [RFC PATCH 6/9] media: vb2: add support for requests in QBUF ioctl

[Hans Verkuil]

On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
> On Fri, Jan 12, 2018 at 8:37 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>> Support the request argument of the QBUF ioctl.
>>>
>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>> ---
>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 93 +++++++++++++++++++++++++++++++++++-
>>>  1 file changed, 92 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> index 8d041247e97f..28f9c368563e 100644
>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>> @@ -29,6 +29,7 @@
>>>  #include <media/v4l2-device.h>
>>>  #include <media/videobuf2-v4l2.h>
>>>  #include <media/v4l2-mc.h>
>>> +#include <media/media-request.h>
>>>
>>>  #include <trace/events/v4l2.h>
>>>
>>> @@ -965,6 +966,81 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>>       return -EINVAL;
>>>  }
>>>
>>> +/*
>>> + * Validate that a given request can be used during an ioctl.
>>> + *
>>> + * When using the request API, request file descriptors must be matched against
>>> + * the actual request object. User-space can pass any file descriptor, so we
>>> + * need to make sure the call is valid before going further.
>>> + *
>>> + * This function looks up the request and associated data and performs the
>>> + * following sanity checks:
>>> + *
>>> + * * Make sure that the entity supports requests,
>>> + * * Make sure that the entity belongs to the media_device managing the passed
>>> + *   request,
>>> + * * Make sure that the entity data (if any) is associated to the current file
>>> + *   handler.
>>> + *
>>> + * This function returns a pointer to the valid request, or and error code in
>>> + * case of failure. When successful, a reference to the request is acquired and
>>> + * must be properly released.
>>> + */
>>> +#ifdef CONFIG_MEDIA_CONTROLLER
>>> +static struct media_request *
>>> +check_request(int request, struct file *file, void *fh)
>>> +{
>>> +     struct media_request *req = NULL;
>>> +     struct video_device *vfd = video_devdata(file);
>>> +     struct v4l2_fh *vfh =
>>> +             test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
>>> +     struct media_entity *entity = &vfd->entity;
>>> +     const struct media_entity *ent;
>>> +     struct media_request_entity_data *data;
>>> +     bool found = false;
>>> +
>>> +     if (!entity)
>>> +             return ERR_PTR(-EINVAL);
>>> +
>>> +     /* Check that the entity supports requests */
>>> +     if (!entity->req_ops)
>>> +             return ERR_PTR(-ENOTSUPP);
>>> +
>>> +     req = media_request_get_from_fd(request);
>>
>> You can get the media_device from vfd->v4l2_dev->mdev. So it is much easier
>> to just pass the media_device as an argument to media_request_get_from_fd()...
>>
>>> +     if (!req)
>>> +             return ERR_PTR(-EINVAL);
>>> +
>>> +     /* Validate that the entity belongs to the media_device managing
>>> +      * the request queue */
>>> +     media_device_for_each_entity(ent, req->queue->mdev) {
>>> +             if (entity == ent) {
>>> +                     found = true;
>>> +                     break;
>>> +             }
>>> +     }
>>> +     if (!found) {
>>> +             media_request_put(req);
>>> +             return ERR_PTR(-EINVAL);
>>> +     }
>>
>> ...and then you don't need to do this ^^^ extra validation check.
> 
> Ah right, all you need to do is check that req->queue->mdev ==
> vfd->v4l2_dev->mdev and you can get rid of this whole block.

Correct.

 I don't
> think we can do that in media_request_get_from_fd() though, since it
> is called from other places where (IIUC) we don't have access to the
> media_device.

Yes, sorry about that. Ignore my comment about media_request_get_from_fd().
I misunderstood what that function did.

> 
>>
>>> +
>>> +     /* Validate that the entity's data belongs to the correct fh */
>>> +     data = media_request_get_entity_data(req, entity, vfh);
>>> +     if (IS_ERR(data)) {
>>> +             media_request_put(req);
>>> +             return ERR_PTR(PTR_ERR(data));
>>> +     }
>>
>> This assumes that each filehandle has its own state. That's true for codecs,
>> but not for most (all?) other devices. There the state is per device instance.
>>
>> I'm not sure if we have a unique identifying mark for such drivers. The closest
>> is checking if fh->m2m_ctx is non-NULL, but I don't know if all drivers with
>> per-filehandle state use that field. An alternative might be to check if
>> fh->ctrl_handler is non-NULL. But again, I'm not sure if that's a 100% valid
>> check.
> 
> I think the current code already takes that case into account: if the
> device does not uses v4l2_fh, then the fh argument passed to
> media_request_get_entity_data() will be null, and so will be data->fh.
> 

Using v4l2_fh doesn't mean it has per-filehandle state. Most drivers only have global
state and they still use v4l2_fh.

This is why you should also look at vivid and vimc for the request API otherwise you
miss what non-codec drivers do. And those are the vast bulk of the drivers.

Regards,

	Hans

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Alexandre Courbot]

On Mon, Jan 15, 2018 at 6:07 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
>> On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>>> Add throttling support for buffers when requests are in use on a given
>>>> queue. Buffers associated to a request are kept into the vb2 queue until
>>>> the request becomes active, at which point all the buffers are passed to
>>>> the driver. The queue can also signal that is has processed all of a
>>>> request's buffers.
>>>>
>>>> Also add support for the request parameter when handling the QBUF ioctl.
>>>>
>>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>>> ---
>>>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>>>> index cb115ba6a1d2..c01038b7962a 100644
>>>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>>>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>>>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>                   state != VB2_BUF_STATE_REQUEUEING))
>>>>               state = VB2_BUF_STATE_ERROR;
>>>>
>>>> +     WARN_ON(vb->request != q->cur_req);
>>>
>>> What's the reason for this WARN_ON? It's not immediately obvious to me.
>>
>> This is a safeguard against driver bugs: a buffer should not complete
>> unless it is part of the request being currently processed.
>>
>>>
>>>> +
>>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>>       /*
>>>>        * Although this is not a callback, it still does have to balance
>>>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>               /* Add the buffer to the done buffers list */
>>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>>               vb->state = state;
>>>> +
>>>> +             if (q->cur_req) {
>>>> +                     WARN_ON(q->req_buf_cnt < 1);
>>>> +
>>>> +                     if (--q->req_buf_cnt == 0)
>>>> +                             q->cur_req = NULL;
>>>> +             }
>>>>       }
>>>>       atomic_dec(&q->owned_by_drv_count);
>>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>>>
>>>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>>>> +{
>>>> +     struct vb2_buffer *vb;
>>>> +
>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>> +             if (vb->request == q->cur_req)
>>>> +                     __enqueue_in_driver(vb);
>>>> +     }
>>>> +}
>>>
>>> I think this will clash big time with the v4l2 fence patch series...
>>
>> Indeed, but on the other hand I was not a big fan of going through the
>> whole list. :) So I welcome the extra throttling introduced by the
>> fence series.
>
> There is only throttling if fences are used by userspace. Otherwise there
> is no change.
>
>>
>>>
>>>> +
>>>>  /**
>>>>   * vb2_start_streaming() - Attempt to start streaming.
>>>>   * @q:               videobuf2 queue
>>>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>        * If any buffers were queued before streamon,
>>>>        * we can now pass them to driver for processing.
>>>>        */
>>>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>>>> -             __enqueue_in_driver(vb);
>>>> +     vb2_queue_enqueue_current_buffers(q);
>>>>
>>>>       /* Tell the driver to start streaming */
>>>>       q->start_streaming_called = 1;
>>>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>       return ret;
>>>>  }
>>>>
>>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>>>> +               struct media_request *req, void *pb)
>>>>  {
>>>>       struct vb2_buffer *vb;
>>>>       int ret;
>>>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>       q->queued_count++;
>>>>       q->waiting_for_buffers = false;
>>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>>> +     vb->request = req;
>>>>
>>>>       if (pb)
>>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>       /*
>>>>        * If already streaming, give the buffer to driver for processing.
>>>>        * If not, the buffer will be given to driver on next streamon.
>>>> +      *
>>>> +      * If using the request API, the buffer will be given to the driver
>>>> +      * when the request becomes active.
>>>>        */
>>>> -     if (q->start_streaming_called)
>>>> +     if (q->start_streaming_called && !req)
>>>>               __enqueue_in_driver(vb);
>>>>
>>>>       /* Fill buffer information for the userspace */
>>>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>  }
>>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>>
>>>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>>>> +{
>>>> +     struct vb2_buffer *vb;
>>>> +
>>>> +     q->req_buf_cnt = 0;
>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>> +             if (vb->request == req)
>>>> +                     ++q->req_buf_cnt;
>>>> +     }
>>>> +
>>>> +     /* only consider the request if we actually have buffers for it */
>>>> +     if (q->req_buf_cnt == 0)
>>>> +             return;
>>>> +
>>>> +     q->cur_req = req;
>>>> +
>>>> +     /* If not streaming yet, we will enqueue the buffers later */
>>>> +     if (q->start_streaming_called)
>>>> +             vb2_queue_enqueue_current_buffers(q);
>>>
>>> If I understand all this correctly, then you are queuing one request at a
>>> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
>>> the same request (q->cur_req).
>>
>> That is correct.
>>
>>> But that might work for codecs, but not
>>> for camera drivers: you will typically have multiple requests queued up in
>>> the driver.
>>
>> Aren't requests supposed to be performed sequentially, even in the
>> camera case? Passing a buffer to the driver means that we allow it to
>> process it using its current settings ; if another request is
>> currently active, wouldn't that become an issue?
>
> Drivers often need multiple buffers queued before they can start the DMA
> engine (usually at least two buffers have to be queued, just do a
> git grep min_buffers_needed drivers/media).
>
> In addition, sensors often have to be programmed one or two frames earlier
> for a new setting to take effect for the required frame.
>
> In other words: drivers need to be able to look ahead and vb2 should just
> queue buffers/requests as soon as they are ready.

Cannot drivers simply peek into their vb2_queue if they need to look
ahead? My main concern here is that I would like to avoid having to
make individual drivers aware of requests as much as possible. With
the current design, drivers just need to care about unconditionally
processing all the buffers that are passed to them by vb2, and not
keeping it that way would complicate things.

The issue of programming sensors ahead-of-time is more complex. As
always, it would probably be preferable to manage as much of this at
the framework level. I am not sure that a bulletproof solution exists
here, especially given the extra synchronization introduced by fences:
you may want to program a sensor two frames ahead, but what if the
frame in between has a non-yet signaled input fence? You could
potentially block and miss the timing. The solution here would be to
delay the processing of frame F-2 until the fence is signaled, or
introduce an empty "latency frame" once the fence signals to get the
timing right. Unless I am missing something, the ability to look into
the vb2_queue should be enough to achieve this.

>
>>
>>>
>>> In any case, I don't think the req_buf_cnt and cur_req fields belong in
>>> vb2_queue.
>>>
>>> Another weird thing here is that it appears that you allow for multiple
>>> buffers for the same device in the same request. I'm not sure that is
>>> useful. For one, it will postpone the completion of the request until
>>> all buffers are dequeued.
>>
>> s/dequeued/completed. A request is marked as completed as soon as all
>> its buffers are marked as done, and can be polled before its buffers
>> are dequeued by user-space.
>>
>> Do you suggest that we enforce a "one buffer per queue per request"
>> rule? I cannot thing of any case where this would be a hard limiting
>> factor (and it would certainly simplify the code), on the other hand
>> it may slow things down a bit in the case where we want to e.g. take
>> several shots in fast succession with the same parameters. But I have
>> no evidence that the extra lag would be noticeable.
>
> This is something that should go into an RFC for discussion. My opinion
> at this time is that you should stick with one buffer per queue per request.
> This can always be relaxed in the future. For now it is better to keep
> it simple.

The more I think about this idea, the more it looks like a good one.
It also answers the question "can we queue buffers not associated to a
request after we started using requests?". The answer would be yes:
such buffers would be processed immediately after the preceding
request completes, almost emulating the behavior of having several
buffers per request (with the exceptions that the request completes
earlier and with a different ending state, but this can be fixed by
associated another request without any control set to the last frame).

Re: [RFC PATCH 6/9] media: vb2: add support for requests in QBUF ioctl

[Alexandre Courbot]

On Mon, Jan 15, 2018 at 6:19 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
>> On Fri, Jan 12, 2018 at 8:37 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>>> Support the request argument of the QBUF ioctl.
>>>>
>>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>>> ---
>>>>  drivers/media/v4l2-core/v4l2-ioctl.c | 93 +++++++++++++++++++++++++++++++++++-
>>>>  1 file changed, 92 insertions(+), 1 deletion(-)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> index 8d041247e97f..28f9c368563e 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>>>> @@ -29,6 +29,7 @@
>>>>  #include <media/v4l2-device.h>
>>>>  #include <media/videobuf2-v4l2.h>
>>>>  #include <media/v4l2-mc.h>
>>>> +#include <media/media-request.h>
>>>>
>>>>  #include <trace/events/v4l2.h>
>>>>
>>>> @@ -965,6 +966,81 @@ static int check_fmt(struct file *file, enum v4l2_buf_type type)
>>>>       return -EINVAL;
>>>>  }
>>>>
>>>> +/*
>>>> + * Validate that a given request can be used during an ioctl.
>>>> + *
>>>> + * When using the request API, request file descriptors must be matched against
>>>> + * the actual request object. User-space can pass any file descriptor, so we
>>>> + * need to make sure the call is valid before going further.
>>>> + *
>>>> + * This function looks up the request and associated data and performs the
>>>> + * following sanity checks:
>>>> + *
>>>> + * * Make sure that the entity supports requests,
>>>> + * * Make sure that the entity belongs to the media_device managing the passed
>>>> + *   request,
>>>> + * * Make sure that the entity data (if any) is associated to the current file
>>>> + *   handler.
>>>> + *
>>>> + * This function returns a pointer to the valid request, or and error code in
>>>> + * case of failure. When successful, a reference to the request is acquired and
>>>> + * must be properly released.
>>>> + */
>>>> +#ifdef CONFIG_MEDIA_CONTROLLER
>>>> +static struct media_request *
>>>> +check_request(int request, struct file *file, void *fh)
>>>> +{
>>>> +     struct media_request *req = NULL;
>>>> +     struct video_device *vfd = video_devdata(file);
>>>> +     struct v4l2_fh *vfh =
>>>> +             test_bit(V4L2_FL_USES_V4L2_FH, &vfd->flags) ? fh : NULL;
>>>> +     struct media_entity *entity = &vfd->entity;
>>>> +     const struct media_entity *ent;
>>>> +     struct media_request_entity_data *data;
>>>> +     bool found = false;
>>>> +
>>>> +     if (!entity)
>>>> +             return ERR_PTR(-EINVAL);
>>>> +
>>>> +     /* Check that the entity supports requests */
>>>> +     if (!entity->req_ops)
>>>> +             return ERR_PTR(-ENOTSUPP);
>>>> +
>>>> +     req = media_request_get_from_fd(request);
>>>
>>> You can get the media_device from vfd->v4l2_dev->mdev. So it is much easier
>>> to just pass the media_device as an argument to media_request_get_from_fd()...
>>>
>>>> +     if (!req)
>>>> +             return ERR_PTR(-EINVAL);
>>>> +
>>>> +     /* Validate that the entity belongs to the media_device managing
>>>> +      * the request queue */
>>>> +     media_device_for_each_entity(ent, req->queue->mdev) {
>>>> +             if (entity == ent) {
>>>> +                     found = true;
>>>> +                     break;
>>>> +             }
>>>> +     }
>>>> +     if (!found) {
>>>> +             media_request_put(req);
>>>> +             return ERR_PTR(-EINVAL);
>>>> +     }
>>>
>>> ...and then you don't need to do this ^^^ extra validation check.
>>
>> Ah right, all you need to do is check that req->queue->mdev ==
>> vfd->v4l2_dev->mdev and you can get rid of this whole block.
>
> Correct.
>
>  I don't
>> think we can do that in media_request_get_from_fd() though, since it
>> is called from other places where (IIUC) we don't have access to the
>> media_device.
>
> Yes, sorry about that. Ignore my comment about media_request_get_from_fd().
> I misunderstood what that function did.
>
>>
>>>
>>>> +
>>>> +     /* Validate that the entity's data belongs to the correct fh */
>>>> +     data = media_request_get_entity_data(req, entity, vfh);
>>>> +     if (IS_ERR(data)) {
>>>> +             media_request_put(req);
>>>> +             return ERR_PTR(PTR_ERR(data));
>>>> +     }
>>>
>>> This assumes that each filehandle has its own state. That's true for codecs,
>>> but not for most (all?) other devices. There the state is per device instance.
>>>
>>> I'm not sure if we have a unique identifying mark for such drivers. The closest
>>> is checking if fh->m2m_ctx is non-NULL, but I don't know if all drivers with
>>> per-filehandle state use that field. An alternative might be to check if
>>> fh->ctrl_handler is non-NULL. But again, I'm not sure if that's a 100% valid
>>> check.
>>
>> I think the current code already takes that case into account: if the
>> device does not uses v4l2_fh, then the fh argument passed to
>> media_request_get_entity_data() will be null, and so will be data->fh.
>>
>
> Using v4l2_fh doesn't mean it has per-filehandle state. Most drivers only have global
> state and they still use v4l2_fh.
>
> This is why you should also look at vivid and vimc for the request API otherwise you
> miss what non-codec drivers do. And those are the vast bulk of the drivers.

Thanks for spotting this, I clearly misunderstood the purpose of
v4l2_fh here. I will take a look as you suggested.

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Hans Verkuil]

On 01/16/2018 10:39 AM, Alexandre Courbot wrote:
> On Mon, Jan 15, 2018 at 6:07 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
>>> On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>>>> Add throttling support for buffers when requests are in use on a given
>>>>> queue. Buffers associated to a request are kept into the vb2 queue until
>>>>> the request becomes active, at which point all the buffers are passed to
>>>>> the driver. The queue can also signal that is has processed all of a
>>>>> request's buffers.
>>>>>
>>>>> Also add support for the request parameter when handling the QBUF ioctl.
>>>>>
>>>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>>>> ---
>>>>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>>>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>>>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>>>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>>>>
>>>>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>>>>> index cb115ba6a1d2..c01038b7962a 100644
>>>>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>>>>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>>>>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>                   state != VB2_BUF_STATE_REQUEUEING))
>>>>>               state = VB2_BUF_STATE_ERROR;
>>>>>
>>>>> +     WARN_ON(vb->request != q->cur_req);
>>>>
>>>> What's the reason for this WARN_ON? It's not immediately obvious to me.
>>>
>>> This is a safeguard against driver bugs: a buffer should not complete
>>> unless it is part of the request being currently processed.
>>>
>>>>
>>>>> +
>>>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>>>       /*
>>>>>        * Although this is not a callback, it still does have to balance
>>>>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>               /* Add the buffer to the done buffers list */
>>>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>>>               vb->state = state;
>>>>> +
>>>>> +             if (q->cur_req) {
>>>>> +                     WARN_ON(q->req_buf_cnt < 1);
>>>>> +
>>>>> +                     if (--q->req_buf_cnt == 0)
>>>>> +                             q->cur_req = NULL;
>>>>> +             }
>>>>>       }
>>>>>       atomic_dec(&q->owned_by_drv_count);
>>>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>>>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>  }
>>>>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>>>>
>>>>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>>>>> +{
>>>>> +     struct vb2_buffer *vb;
>>>>> +
>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>> +             if (vb->request == q->cur_req)
>>>>> +                     __enqueue_in_driver(vb);
>>>>> +     }
>>>>> +}
>>>>
>>>> I think this will clash big time with the v4l2 fence patch series...
>>>
>>> Indeed, but on the other hand I was not a big fan of going through the
>>> whole list. :) So I welcome the extra throttling introduced by the
>>> fence series.
>>
>> There is only throttling if fences are used by userspace. Otherwise there
>> is no change.
>>
>>>
>>>>
>>>>> +
>>>>>  /**
>>>>>   * vb2_start_streaming() - Attempt to start streaming.
>>>>>   * @q:               videobuf2 queue
>>>>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>        * If any buffers were queued before streamon,
>>>>>        * we can now pass them to driver for processing.
>>>>>        */
>>>>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>>>>> -             __enqueue_in_driver(vb);
>>>>> +     vb2_queue_enqueue_current_buffers(q);
>>>>>
>>>>>       /* Tell the driver to start streaming */
>>>>>       q->start_streaming_called = 1;
>>>>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>       return ret;
>>>>>  }
>>>>>
>>>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>>>>> +               struct media_request *req, void *pb)
>>>>>  {
>>>>>       struct vb2_buffer *vb;
>>>>>       int ret;
>>>>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>       q->queued_count++;
>>>>>       q->waiting_for_buffers = false;
>>>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>>>> +     vb->request = req;
>>>>>
>>>>>       if (pb)
>>>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>>>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>       /*
>>>>>        * If already streaming, give the buffer to driver for processing.
>>>>>        * If not, the buffer will be given to driver on next streamon.
>>>>> +      *
>>>>> +      * If using the request API, the buffer will be given to the driver
>>>>> +      * when the request becomes active.
>>>>>        */
>>>>> -     if (q->start_streaming_called)
>>>>> +     if (q->start_streaming_called && !req)
>>>>>               __enqueue_in_driver(vb);
>>>>>
>>>>>       /* Fill buffer information for the userspace */
>>>>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>  }
>>>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>>>
>>>>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>>>>> +{
>>>>> +     struct vb2_buffer *vb;
>>>>> +
>>>>> +     q->req_buf_cnt = 0;
>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>> +             if (vb->request == req)
>>>>> +                     ++q->req_buf_cnt;
>>>>> +     }
>>>>> +
>>>>> +     /* only consider the request if we actually have buffers for it */
>>>>> +     if (q->req_buf_cnt == 0)
>>>>> +             return;
>>>>> +
>>>>> +     q->cur_req = req;
>>>>> +
>>>>> +     /* If not streaming yet, we will enqueue the buffers later */
>>>>> +     if (q->start_streaming_called)
>>>>> +             vb2_queue_enqueue_current_buffers(q);
>>>>
>>>> If I understand all this correctly, then you are queuing one request at a
>>>> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
>>>> the same request (q->cur_req).
>>>
>>> That is correct.
>>>
>>>> But that might work for codecs, but not
>>>> for camera drivers: you will typically have multiple requests queued up in
>>>> the driver.
>>>
>>> Aren't requests supposed to be performed sequentially, even in the
>>> camera case? Passing a buffer to the driver means that we allow it to
>>> process it using its current settings ; if another request is
>>> currently active, wouldn't that become an issue?
>>
>> Drivers often need multiple buffers queued before they can start the DMA
>> engine (usually at least two buffers have to be queued, just do a
>> git grep min_buffers_needed drivers/media).
>>
>> In addition, sensors often have to be programmed one or two frames earlier
>> for a new setting to take effect for the required frame.
>>
>> In other words: drivers need to be able to look ahead and vb2 should just
>> queue buffers/requests as soon as they are ready.
> 
> Cannot drivers simply peek into their vb2_queue if they need to look
> ahead? My main concern here is that I would like to avoid having to
> make individual drivers aware of requests as much as possible. With
> the current design, drivers just need to care about unconditionally
> processing all the buffers that are passed to them by vb2, and not
> keeping it that way would complicate things.

I'm not sure what the problem is. Once buffers are ready (i.e. not waiting
for fences or unfinished requests) then they should be queued to the driver.
At that time the driver can look at whatever associated request data the
newly queued buffer has and program the hardware.

Once a buffer is ready, just queue it up to the driver. Drivers should never
look into buffers not explicitly queued to them. They don't need to either,
since those buffers are not ready anyway.

It's also a matter of who owns a buffer: that can be either userspace, vb2
or the driver. Each 'owner' can only touch the buffers it owns and should
never access buffers from other owners.

> The issue of programming sensors ahead-of-time is more complex. As
> always, it would probably be preferable to manage as much of this at
> the framework level. I am not sure that a bulletproof solution exists
> here, especially given the extra synchronization introduced by fences:
> you may want to program a sensor two frames ahead, but what if the
> frame in between has a non-yet signaled input fence? You could
> potentially block and miss the timing. The solution here would be to
> delay the processing of frame F-2 until the fence is signaled, or
> introduce an empty "latency frame" once the fence signals to get the
> timing right. Unless I am missing something, the ability to look into
> the vb2_queue should be enough to achieve this.

It is the responsibility of the application to make sure buffers aren't
held up waiting for fences. That's no different to what happens today
if the application cannot process buffers fast enough and the DMA engine
is starved of buffers.

Handling such complex scenarios is out-of-scope to the request API. The
request framework should just hand over the buffers + request data to
the driver as soon as it can, and that's it.

We will probably want to provide helper functions for drivers to use
to simplify such complex scenarios, but that's something for the future.

Right now we just need to get core functionality right.

> 
>>
>>>
>>>>
>>>> In any case, I don't think the req_buf_cnt and cur_req fields belong in
>>>> vb2_queue.
>>>>
>>>> Another weird thing here is that it appears that you allow for multiple
>>>> buffers for the same device in the same request. I'm not sure that is
>>>> useful. For one, it will postpone the completion of the request until
>>>> all buffers are dequeued.
>>>
>>> s/dequeued/completed. A request is marked as completed as soon as all
>>> its buffers are marked as done, and can be polled before its buffers
>>> are dequeued by user-space.
>>>
>>> Do you suggest that we enforce a "one buffer per queue per request"
>>> rule? I cannot thing of any case where this would be a hard limiting
>>> factor (and it would certainly simplify the code), on the other hand
>>> it may slow things down a bit in the case where we want to e.g. take
>>> several shots in fast succession with the same parameters. But I have
>>> no evidence that the extra lag would be noticeable.
>>
>> This is something that should go into an RFC for discussion. My opinion
>> at this time is that you should stick with one buffer per queue per request.
>> This can always be relaxed in the future. For now it is better to keep
>> it simple.
> 
> The more I think about this idea, the more it looks like a good one.
> It also answers the question "can we queue buffers not associated to a
> request after we started using requests?". The answer would be yes:
> such buffers would be processed immediately after the preceding
> request completes, almost emulating the behavior of having several
> buffers per request (with the exceptions that the request completes
> earlier and with a different ending state, but this can be fixed by
> associated another request without any control set to the last frame).
> 

It sounds reasonable, but, as always, the devil is in the details.

Regards,

	Hans

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Alexandre Courbot]

On Tue, Jan 16, 2018 at 7:37 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
> On 01/16/2018 10:39 AM, Alexandre Courbot wrote:
>> On Mon, Jan 15, 2018 at 6:07 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>> On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
>>>> On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>>>>> Add throttling support for buffers when requests are in use on a given
>>>>>> queue. Buffers associated to a request are kept into the vb2 queue until
>>>>>> the request becomes active, at which point all the buffers are passed to
>>>>>> the driver. The queue can also signal that is has processed all of a
>>>>>> request's buffers.
>>>>>>
>>>>>> Also add support for the request parameter when handling the QBUF ioctl.
>>>>>>
>>>>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>>>>> ---
>>>>>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>>>>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>>>>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>>>>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>>>>>
>>>>>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>>>>>> index cb115ba6a1d2..c01038b7962a 100644
>>>>>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>>>>>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>>>>>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>>                   state != VB2_BUF_STATE_REQUEUEING))
>>>>>>               state = VB2_BUF_STATE_ERROR;
>>>>>>
>>>>>> +     WARN_ON(vb->request != q->cur_req);
>>>>>
>>>>> What's the reason for this WARN_ON? It's not immediately obvious to me.
>>>>
>>>> This is a safeguard against driver bugs: a buffer should not complete
>>>> unless it is part of the request being currently processed.
>>>>
>>>>>
>>>>>> +
>>>>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>>>>       /*
>>>>>>        * Although this is not a callback, it still does have to balance
>>>>>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>>               /* Add the buffer to the done buffers list */
>>>>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>>>>               vb->state = state;
>>>>>> +
>>>>>> +             if (q->cur_req) {
>>>>>> +                     WARN_ON(q->req_buf_cnt < 1);
>>>>>> +
>>>>>> +                     if (--q->req_buf_cnt == 0)
>>>>>> +                             q->cur_req = NULL;
>>>>>> +             }
>>>>>>       }
>>>>>>       atomic_dec(&q->owned_by_drv_count);
>>>>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>>>>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>  }
>>>>>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>>>>>
>>>>>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>>>>>> +{
>>>>>> +     struct vb2_buffer *vb;
>>>>>> +
>>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>>> +             if (vb->request == q->cur_req)
>>>>>> +                     __enqueue_in_driver(vb);
>>>>>> +     }
>>>>>> +}
>>>>>
>>>>> I think this will clash big time with the v4l2 fence patch series...
>>>>
>>>> Indeed, but on the other hand I was not a big fan of going through the
>>>> whole list. :) So I welcome the extra throttling introduced by the
>>>> fence series.
>>>
>>> There is only throttling if fences are used by userspace. Otherwise there
>>> is no change.
>>>
>>>>
>>>>>
>>>>>> +
>>>>>>  /**
>>>>>>   * vb2_start_streaming() - Attempt to start streaming.
>>>>>>   * @q:               videobuf2 queue
>>>>>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>>        * If any buffers were queued before streamon,
>>>>>>        * we can now pass them to driver for processing.
>>>>>>        */
>>>>>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>>>>>> -             __enqueue_in_driver(vb);
>>>>>> +     vb2_queue_enqueue_current_buffers(q);
>>>>>>
>>>>>>       /* Tell the driver to start streaming */
>>>>>>       q->start_streaming_called = 1;
>>>>>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>>       return ret;
>>>>>>  }
>>>>>>
>>>>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>>>>>> +               struct media_request *req, void *pb)
>>>>>>  {
>>>>>>       struct vb2_buffer *vb;
>>>>>>       int ret;
>>>>>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>       q->queued_count++;
>>>>>>       q->waiting_for_buffers = false;
>>>>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>>>>> +     vb->request = req;
>>>>>>
>>>>>>       if (pb)
>>>>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>>>>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>       /*
>>>>>>        * If already streaming, give the buffer to driver for processing.
>>>>>>        * If not, the buffer will be given to driver on next streamon.
>>>>>> +      *
>>>>>> +      * If using the request API, the buffer will be given to the driver
>>>>>> +      * when the request becomes active.
>>>>>>        */
>>>>>> -     if (q->start_streaming_called)
>>>>>> +     if (q->start_streaming_called && !req)
>>>>>>               __enqueue_in_driver(vb);
>>>>>>
>>>>>>       /* Fill buffer information for the userspace */
>>>>>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>  }
>>>>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>>>>
>>>>>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>>>>>> +{
>>>>>> +     struct vb2_buffer *vb;
>>>>>> +
>>>>>> +     q->req_buf_cnt = 0;
>>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>>> +             if (vb->request == req)
>>>>>> +                     ++q->req_buf_cnt;
>>>>>> +     }
>>>>>> +
>>>>>> +     /* only consider the request if we actually have buffers for it */
>>>>>> +     if (q->req_buf_cnt == 0)
>>>>>> +             return;
>>>>>> +
>>>>>> +     q->cur_req = req;
>>>>>> +
>>>>>> +     /* If not streaming yet, we will enqueue the buffers later */
>>>>>> +     if (q->start_streaming_called)
>>>>>> +             vb2_queue_enqueue_current_buffers(q);
>>>>>
>>>>> If I understand all this correctly, then you are queuing one request at a
>>>>> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
>>>>> the same request (q->cur_req).
>>>>
>>>> That is correct.
>>>>
>>>>> But that might work for codecs, but not
>>>>> for camera drivers: you will typically have multiple requests queued up in
>>>>> the driver.
>>>>
>>>> Aren't requests supposed to be performed sequentially, even in the
>>>> camera case? Passing a buffer to the driver means that we allow it to
>>>> process it using its current settings ; if another request is
>>>> currently active, wouldn't that become an issue?
>>>
>>> Drivers often need multiple buffers queued before they can start the DMA
>>> engine (usually at least two buffers have to be queued, just do a
>>> git grep min_buffers_needed drivers/media).
>>>
>>> In addition, sensors often have to be programmed one or two frames earlier
>>> for a new setting to take effect for the required frame.
>>>
>>> In other words: drivers need to be able to look ahead and vb2 should just
>>> queue buffers/requests as soon as they are ready.
>>
>> Cannot drivers simply peek into their vb2_queue if they need to look
>> ahead? My main concern here is that I would like to avoid having to
>> make individual drivers aware of requests as much as possible. With
>> the current design, drivers just need to care about unconditionally
>> processing all the buffers that are passed to them by vb2, and not
>> keeping it that way would complicate things.
>
> I'm not sure what the problem is. Once buffers are ready (i.e. not waiting
> for fences or unfinished requests) then they should be queued to the driver.
> At that time the driver can look at whatever associated request data the
> newly queued buffer has and program the hardware.

I see what you mean and indeed if we want to maintain proper buffer
ownership that would be the way to go.

What I wanted to avoid (but maybe this is unavoidable?) is having to
make each individual driver aware of requests. I.e. the upper
framework would take care of scheduling the buffers and of setting the
controls when the time to process the request has come. While this
would work well for simple hardware that can only process buffers one
at a time, it would be sub-optimal for e.g. more sophisticated codecs
with their own buffer queues and shadow registers.

However, it is not clear to me how we can make the different IPs of a
complex pipeline, all controlled by their own driver, cooperate and
synchronize in order to properly set the pipeline topography and
process the buffers only once that topography is set. Wouldn't that
mean extra back-and-forth between the drivers and the request
framework?

Re: [RFC PATCH 5/9] media: vb2: add support for requests

[Hans Verkuil]

On 01/17/18 09:01, Alexandre Courbot wrote:
> On Tue, Jan 16, 2018 at 7:37 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>> On 01/16/2018 10:39 AM, Alexandre Courbot wrote:
>>> On Mon, Jan 15, 2018 at 6:07 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>> On 01/15/2018 09:24 AM, Alexandre Courbot wrote:
>>>>> On Fri, Jan 12, 2018 at 7:49 PM, Hans Verkuil <hverkuil@xs4all.nl> wrote:
>>>>>> On 12/15/17 08:56, Alexandre Courbot wrote:
>>>>>>> Add throttling support for buffers when requests are in use on a given
>>>>>>> queue. Buffers associated to a request are kept into the vb2 queue until
>>>>>>> the request becomes active, at which point all the buffers are passed to
>>>>>>> the driver. The queue can also signal that is has processed all of a
>>>>>>> request's buffers.
>>>>>>>
>>>>>>> Also add support for the request parameter when handling the QBUF ioctl.
>>>>>>>
>>>>>>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>>>>>>> ---
>>>>>>>  drivers/media/v4l2-core/videobuf2-core.c | 59 ++++++++++++++++++++++++++++----
>>>>>>>  drivers/media/v4l2-core/videobuf2-v4l2.c | 29 +++++++++++++++-
>>>>>>>  include/media/videobuf2-core.h           | 25 +++++++++++++-
>>>>>>>  3 files changed, 104 insertions(+), 9 deletions(-)
>>>>>>>
>>>>>>> diff --git a/drivers/media/v4l2-core/videobuf2-core.c b/drivers/media/v4l2-core/videobuf2-core.c
>>>>>>> index cb115ba6a1d2..c01038b7962a 100644
>>>>>>> --- a/drivers/media/v4l2-core/videobuf2-core.c
>>>>>>> +++ b/drivers/media/v4l2-core/videobuf2-core.c
>>>>>>> @@ -898,6 +898,8 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>>>                   state != VB2_BUF_STATE_REQUEUEING))
>>>>>>>               state = VB2_BUF_STATE_ERROR;
>>>>>>>
>>>>>>> +     WARN_ON(vb->request != q->cur_req);
>>>>>>
>>>>>> What's the reason for this WARN_ON? It's not immediately obvious to me.
>>>>>
>>>>> This is a safeguard against driver bugs: a buffer should not complete
>>>>> unless it is part of the request being currently processed.
>>>>>
>>>>>>
>>>>>>> +
>>>>>>>  #ifdef CONFIG_VIDEO_ADV_DEBUG
>>>>>>>       /*
>>>>>>>        * Although this is not a callback, it still does have to balance
>>>>>>> @@ -920,6 +922,13 @@ void vb2_buffer_done(struct vb2_buffer *vb, enum vb2_buffer_state state)
>>>>>>>               /* Add the buffer to the done buffers list */
>>>>>>>               list_add_tail(&vb->done_entry, &q->done_list);
>>>>>>>               vb->state = state;
>>>>>>> +
>>>>>>> +             if (q->cur_req) {
>>>>>>> +                     WARN_ON(q->req_buf_cnt < 1);
>>>>>>> +
>>>>>>> +                     if (--q->req_buf_cnt == 0)
>>>>>>> +                             q->cur_req = NULL;
>>>>>>> +             }
>>>>>>>       }
>>>>>>>       atomic_dec(&q->owned_by_drv_count);
>>>>>>>       spin_unlock_irqrestore(&q->done_lock, flags);
>>>>>>> @@ -1298,6 +1307,16 @@ int vb2_core_prepare_buf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>>  }
>>>>>>>  EXPORT_SYMBOL_GPL(vb2_core_prepare_buf);
>>>>>>>
>>>>>>> +static void vb2_queue_enqueue_current_buffers(struct vb2_queue *q)
>>>>>>> +{
>>>>>>> +     struct vb2_buffer *vb;
>>>>>>> +
>>>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>>>> +             if (vb->request == q->cur_req)
>>>>>>> +                     __enqueue_in_driver(vb);
>>>>>>> +     }
>>>>>>> +}
>>>>>>
>>>>>> I think this will clash big time with the v4l2 fence patch series...
>>>>>
>>>>> Indeed, but on the other hand I was not a big fan of going through the
>>>>> whole list. :) So I welcome the extra throttling introduced by the
>>>>> fence series.
>>>>
>>>> There is only throttling if fences are used by userspace. Otherwise there
>>>> is no change.
>>>>
>>>>>
>>>>>>
>>>>>>> +
>>>>>>>  /**
>>>>>>>   * vb2_start_streaming() - Attempt to start streaming.
>>>>>>>   * @q:               videobuf2 queue
>>>>>>> @@ -1318,8 +1337,7 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>>>        * If any buffers were queued before streamon,
>>>>>>>        * we can now pass them to driver for processing.
>>>>>>>        */
>>>>>>> -     list_for_each_entry(vb, &q->queued_list, queued_entry)
>>>>>>> -             __enqueue_in_driver(vb);
>>>>>>> +     vb2_queue_enqueue_current_buffers(q);
>>>>>>>
>>>>>>>       /* Tell the driver to start streaming */
>>>>>>>       q->start_streaming_called = 1;
>>>>>>> @@ -1361,7 +1379,8 @@ static int vb2_start_streaming(struct vb2_queue *q)
>>>>>>>       return ret;
>>>>>>>  }
>>>>>>>
>>>>>>> -int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>> +int vb2_core_qbuf(struct vb2_queue *q, unsigned int index,
>>>>>>> +               struct media_request *req, void *pb)
>>>>>>>  {
>>>>>>>       struct vb2_buffer *vb;
>>>>>>>       int ret;
>>>>>>> @@ -1392,6 +1411,7 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>>       q->queued_count++;
>>>>>>>       q->waiting_for_buffers = false;
>>>>>>>       vb->state = VB2_BUF_STATE_QUEUED;
>>>>>>> +     vb->request = req;
>>>>>>>
>>>>>>>       if (pb)
>>>>>>>               call_void_bufop(q, copy_timestamp, vb, pb);
>>>>>>> @@ -1401,8 +1421,11 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>>       /*
>>>>>>>        * If already streaming, give the buffer to driver for processing.
>>>>>>>        * If not, the buffer will be given to driver on next streamon.
>>>>>>> +      *
>>>>>>> +      * If using the request API, the buffer will be given to the driver
>>>>>>> +      * when the request becomes active.
>>>>>>>        */
>>>>>>> -     if (q->start_streaming_called)
>>>>>>> +     if (q->start_streaming_called && !req)
>>>>>>>               __enqueue_in_driver(vb);
>>>>>>>
>>>>>>>       /* Fill buffer information for the userspace */
>>>>>>> @@ -1427,6 +1450,28 @@ int vb2_core_qbuf(struct vb2_queue *q, unsigned int index, void *pb)
>>>>>>>  }
>>>>>>>  EXPORT_SYMBOL_GPL(vb2_core_qbuf);
>>>>>>>
>>>>>>> +void vb2_queue_start_request(struct vb2_queue *q, struct media_request *req)
>>>>>>> +{
>>>>>>> +     struct vb2_buffer *vb;
>>>>>>> +
>>>>>>> +     q->req_buf_cnt = 0;
>>>>>>> +     list_for_each_entry(vb, &q->queued_list, queued_entry) {
>>>>>>> +             if (vb->request == req)
>>>>>>> +                     ++q->req_buf_cnt;
>>>>>>> +     }
>>>>>>> +
>>>>>>> +     /* only consider the request if we actually have buffers for it */
>>>>>>> +     if (q->req_buf_cnt == 0)
>>>>>>> +             return;
>>>>>>> +
>>>>>>> +     q->cur_req = req;
>>>>>>> +
>>>>>>> +     /* If not streaming yet, we will enqueue the buffers later */
>>>>>>> +     if (q->start_streaming_called)
>>>>>>> +             vb2_queue_enqueue_current_buffers(q);
>>>>>>
>>>>>> If I understand all this correctly, then you are queuing one request at a
>>>>>> time to the vb2_queue. I.e. all the buffers queued to the driver belong to
>>>>>> the same request (q->cur_req).
>>>>>
>>>>> That is correct.
>>>>>
>>>>>> But that might work for codecs, but not
>>>>>> for camera drivers: you will typically have multiple requests queued up in
>>>>>> the driver.
>>>>>
>>>>> Aren't requests supposed to be performed sequentially, even in the
>>>>> camera case? Passing a buffer to the driver means that we allow it to
>>>>> process it using its current settings ; if another request is
>>>>> currently active, wouldn't that become an issue?
>>>>
>>>> Drivers often need multiple buffers queued before they can start the DMA
>>>> engine (usually at least two buffers have to be queued, just do a
>>>> git grep min_buffers_needed drivers/media).
>>>>
>>>> In addition, sensors often have to be programmed one or two frames earlier
>>>> for a new setting to take effect for the required frame.
>>>>
>>>> In other words: drivers need to be able to look ahead and vb2 should just
>>>> queue buffers/requests as soon as they are ready.
>>>
>>> Cannot drivers simply peek into their vb2_queue if they need to look
>>> ahead? My main concern here is that I would like to avoid having to
>>> make individual drivers aware of requests as much as possible. With
>>> the current design, drivers just need to care about unconditionally
>>> processing all the buffers that are passed to them by vb2, and not
>>> keeping it that way would complicate things.
>>
>> I'm not sure what the problem is. Once buffers are ready (i.e. not waiting
>> for fences or unfinished requests) then they should be queued to the driver.
>> At that time the driver can look at whatever associated request data the
>> newly queued buffer has and program the hardware.
> 
> I see what you mean and indeed if we want to maintain proper buffer
> ownership that would be the way to go.
> 
> What I wanted to avoid (but maybe this is unavoidable?) is having to
> make each individual driver aware of requests. I.e. the upper
> framework would take care of scheduling the buffers and of setting the
> controls when the time to process the request has come. While this
> would work well for simple hardware that can only process buffers one
> at a time, it would be sub-optimal for e.g. more sophisticated codecs
> with their own buffer queues and shadow registers.

You really can't do this. Only for codecs would this be feasible, not
other drivers.

This also means that we need a way to signal to userspace that a device
supports the request API since drivers indeed need to be modified for
this to work. But most capture drivers don't need the request API. The
current way of working is perfectly fine for them. It's just codecs and
complex camera drivers that need this.

> However, it is not clear to me how we can make the different IPs of a
> complex pipeline, all controlled by their own driver, cooperate and
> synchronize in order to properly set the pipeline topography and
> process the buffers only once that topography is set. Wouldn't that
> mean extra back-and-forth between the drivers and the request
> framework?

That would have to be coordinated by the top-level driver (usually
the driver that also creates the media device). But in any case, this
is out-of-scope of the request API since this is likely to be very
hardware dependent.

Regards,

	Hans

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Sakari Ailus]

Hi Alexandre,

I remember it was discussed that the work after the V4L2 jobs API would
continue from the existing request API patches. I see that at least the
rather important support for events is missing in this version. Why was it
left out?

I also see that variable size IOCTL argument support is no longer included.

On Fri, Dec 15, 2017 at 04:56:17PM +0900, Alexandre Courbot wrote:
> The request API provides a way to group buffers and device parameters
> into units of work to be queued and executed. This patch introduces the
> UAPI and core framework.
> 
> This patch is based on the previous work by Laurent Pinchart. The core
> has changed considerably, but the UAPI is mostly untouched.
> 
> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> ---
>  drivers/media/Makefile               |   3 +-
>  drivers/media/media-device.c         |   6 +
>  drivers/media/media-request.c        | 390 +++++++++++++++++++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
>  include/media/media-device.h         |   3 +
>  include/media/media-entity.h         |   6 +
>  include/media/media-request.h        | 269 ++++++++++++++++++++++++
>  include/uapi/linux/media.h           |  11 +
>  8 files changed, 688 insertions(+), 2 deletions(-)
>  create mode 100644 drivers/media/media-request.c
>  create mode 100644 include/media/media-request.h
> 
> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> index 594b462ddf0e..985d35ec6b29 100644
> --- a/drivers/media/Makefile
> +++ b/drivers/media/Makefile
> @@ -3,7 +3,8 @@
>  # Makefile for the kernel multimedia device drivers.
>  #
>  
> -media-objs	:= media-device.o media-devnode.o media-entity.o
> +media-objs	:= media-device.o media-devnode.o media-entity.o \
> +		   media-request.o
>  
>  #
>  # I2C drivers should come before other drivers, otherwise they'll fail
> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> index e79f72b8b858..045cec7d2de9 100644
> --- a/drivers/media/media-device.c
> +++ b/drivers/media/media-device.c
> @@ -32,6 +32,7 @@
>  #include <media/media-device.h>
>  #include <media/media-devnode.h>
>  #include <media/media-entity.h>
> +#include <media/media-request.h>
>  
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  
> @@ -407,6 +408,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>  	MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>  	MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
> +	MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
>  };
>  
>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
> @@ -688,6 +690,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
>  
>  void media_device_cleanup(struct media_device *mdev)
>  {
> +	if (mdev->req_queue) {
> +		mdev->req_queue->ops->release(mdev->req_queue);
> +		mdev->req_queue = NULL;
> +	}
>  	ida_destroy(&mdev->entity_internal_idx);
>  	mdev->entity_internal_idx_max = 0;
>  	media_graph_walk_cleanup(&mdev->pm_count_walk);
> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> new file mode 100644
> index 000000000000..15dc65ddfe41
> --- /dev/null
> +++ b/drivers/media/media-request.c
> @@ -0,0 +1,390 @@
> +/*
> + * Request and request queue base management
> + *
> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#include <linux/anon_inodes.h>
> +#include <linux/media.h>
> +#include <linux/fs.h>
> +#include <linux/file.h>
> +#include <linux/slab.h>
> +#include <linux/list.h>
> +
> +#include <media/media-request.h>
> +#include <media/media-device.h>

Alphabetic order, please.

> +
> +const struct file_operations request_fops;
> +
> +void media_request_get(struct media_request *req)

Please return the media request, too.

> +{
> +	kref_get(&req->kref);
> +}
> +EXPORT_SYMBOL_GPL(media_request_get);
> +
> +struct media_request *
> +media_request_get_from_fd(int fd)
> +{
> +	struct file *f;
> +	struct media_request *req;
> +
> +	f = fget(fd);
> +	if (!f)
> +		return NULL;
> +
> +	/* Not a request FD? */
> +	if (f->f_op != &request_fops) {
> +		fput(f);
> +		return NULL;
> +	}
> +
> +	req = f->private_data;
> +	media_request_get(req);
> +	fput(f);
> +
> +	return req;
> +}
> +EXPORT_SYMBOL_GPL(media_request_get_from_fd);
> +
> +static void media_request_release(struct kref *kref)
> +{
> +	struct media_request_entity_data *data, *next;
> +	struct media_request *req =
> +		container_of(kref, typeof(*req), kref);
> +	struct media_device *mdev = req->queue->mdev;
> +
> +	dev_dbg(mdev->dev, "%s: releasing request %u\n", __func__, req->id);

I'd leave out __func__. It doesn't provide any additional information here.

> +
> +	list_del(&req->list);

You appear to have added a list of requests in the queue. You'll need to
serialise access to that list unless you can otherwise prove you're doing
this safely --- and that's not the case here.

> +	list_for_each_entry_safe(data, next, &req->data, list) {
> +		list_del(&data->list);
> +		data->entity->req_ops->release_data(data);
> +	}
> +
> +	req->queue->ops->req_free(req->queue, req);
> +}
> +
> +void media_request_put(struct media_request *req)
> +{

How about checking req is non-NULL here?

> +	kref_put(&req->kref, media_request_release);
> +}
> +EXPORT_SYMBOL_GPL(media_request_put);
> +
> +struct media_request_entity_data *
> +media_request_get_entity_data(struct media_request *req,
> +			      struct media_entity *entity, void *fh)

This makes the assumption that request data is bound to entities. How does
this work with links?

I wonder if it should rather be bound to graph objects, or certain graph
objects. Having a standard way to bind request specific information e.g. to
entities is definitely worth having, though.

V4L2 framework specific information would be needed across the media graph
and it'd be good to store it in a non-driver specific way. What I think
you'd need is an interface that allows storing information based on two
keys --- the request and e.g. a pointer provided by the caller. The V4L2
framework would have one key, e.g. a pointer to an empty struct defined
somewhere in the V4L2 framework could be used for the purpose.

Going forward, the entire media graph state will be subject to changing
through requests. This includes link state, media bus and pixel formats,
cropping and scaling configurations, everything. Let's not try to go there
yet in this patchset, but what I'm asking is to keep the goal in mind when
implementating the request API.

> +{
> +	struct media_request_queue *queue = req->queue;
> +	struct media_request_entity_data *data;
> +
> +	media_request_queue_lock(queue);
> +
> +	/* First look whether we already have entity data */
> +	list_for_each_entry(data, &req->data, list) {
> +		if (data->entity == entity) {

This is fine for now but something more efficient will be needed in the
long run.

> +			/*
> +			 * If so, then the fh must match otherwise this means
> +			 * we are trying to set the same entity through
> +			 * different handles
> +			 */

What's the purpose of this? The request is identified by the file handle
already.

> +			if (data->fh != fh)
> +				data = ERR_PTR(-EINVAL);
> +			goto done;
> +		}
> +	}
> +
> +	/* No entity data found, let's create it */
> +	data = entity->req_ops->alloc_data(req, entity);
> +	if (IS_ERR(data))
> +		goto done;
> +	data->fh = fh;
> +	data->entity = entity;
> +	list_add_tail(&data->list, &req->data);
> +
> +done:
> +	media_request_queue_unlock(queue);
> +	return data;
> +}
> +EXPORT_SYMBOL_GPL(media_request_get_entity_data);
> +
> +static const char * const media_request_states[] __maybe_unused = {
> +	"IDLE",
> +	"QUEUED",
> +	"COMPLETED",
> +	"DELETED",
> +};
> +
> +static const char *media_request_state(enum media_request_state state)
> +{
> +	return state < ARRAY_SIZE(media_request_states) ?
> +		media_request_states[state] : "INVALID";
> +}
> +
> +static int media_device_request_close(struct inode *inode, struct file *filp)
> +{
> +	struct media_request *req = filp->private_data;
> +
> +	if (req == NULL)
> +		return 0;
> +
> +	media_request_put(req);
> +
> +	return 0;
> +}
> +
> +static unsigned int media_request_poll(struct file *file, poll_table *wait)
> +{
> +	struct media_request *req = file->private_data;
> +
> +	poll_wait(file, &req->wait_queue, wait);
> +
> +	if (req->state == MEDIA_REQUEST_STATE_COMPLETE)
> +		return POLLIN | POLLRDNORM;
> +
> +	return 0;
> +}
> +
> +const struct file_operations request_fops = {
> +	.owner = THIS_MODULE,
> +	.poll = media_request_poll,
> +	.release = media_device_request_close,
> +};
> +
> +/*
> + * Called when all the entities taking part in a request have completed their
> + * part.
> + *
> + * The request queue mutex must be held when this function is called.
> + */
> +static void media_device_request_complete(struct media_request *req)
> +{
> +	struct media_request_queue *queue = req->queue;
> +	struct media_device *mdev = queue->mdev;
> +
> +	if (WARN_ON(req->state != MEDIA_REQUEST_STATE_QUEUED)) {
> +		dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
> +			__func__, req->id, media_request_state(req->state));
> +		return;
> +	}
> +
> +	req->state = MEDIA_REQUEST_STATE_COMPLETE;
> +
> +	wake_up_interruptible(&req->wait_queue);
> +
> +	queue->ops->req_complete(queue);
> +
> +	/* Release the reference acquired when we queued the request */
> +	media_request_put(req);
> +}
> +
> +void media_request_entity_complete(struct media_request_queue *queue,
> +				   struct media_entity *entity)
> +{
> +	struct media_request_entity_data *data;
> +	struct media_request *req;
> +	bool completed = true;
> +
> +	media_request_queue_lock(queue);
> +
> +	req = queue->active_request;
> +
> +	list_for_each_entry(data, &req->data, list) {
> +		if (data->entity == entity)
> +			data->completed = true;
> +
> +		if (!data->completed)
> +			completed = false;
> +	}
> +
> +	/* All entities of the request have completed, request is done */
> +	if (completed)
> +		media_device_request_complete(req);
> +
> +	media_request_queue_unlock(queue);
> +
> +}
> +EXPORT_SYMBOL_GPL(media_request_entity_complete);
> +
> +void media_request_queue_init(struct media_request_queue *queue,
> +			      struct media_device *mdev,
> +			      const struct media_request_queue_ops *ops)
> +{
> +	queue->mdev = mdev;
> +	mutex_init(&queue->mutex);
> +	INIT_LIST_HEAD(&queue->requests);
> +	queue->ops = ops;
> +}
> +EXPORT_SYMBOL_GPL(media_request_queue_init);
> +
> +void media_request_queue_release(struct media_request_queue *queue)
> +{
> +	struct media_device *mdev = queue->mdev;
> +
> +	media_request_queue_lock(queue);
> +
> +	/* Just a sanity check - we should have no remaining requests */
> +	while (!list_empty(&queue->requests)) {
> +		struct media_request *req;
> +
> +		req = list_first_entry(&queue->requests, typeof(*req), list);
> +		dev_warn(mdev->dev,
> +			"%s: request %u still referenced, deleting forcibly!\n",
> +			__func__, req->id);
> +		media_request_release(&req->kref);
> +	}
> +
> +	media_request_queue_unlock(queue);
> +}
> +EXPORT_SYMBOL_GPL(media_request_queue_release);
> +
> +/*
> + * Process the MEDIA_REQ_CMD_ALLOC command
> + */
> +static int media_request_cmd_alloc(struct media_request_queue *queue,
> +				   struct media_request_cmd *cmd)
> +{
> +	struct media_request *req;
> +	int fd;
> +
> +	req = queue->ops->req_alloc(queue);
> +	if (!req)
> +		return -ENOMEM;
> +
> +	req->queue = queue;
> +	req->state = MEDIA_REQUEST_STATE_IDLE;
> +	kref_init(&req->kref);
> +	INIT_LIST_HEAD(&req->data);
> +	init_waitqueue_head(&req->wait_queue);
> +
> +	media_request_queue_lock(queue);
> +	req->id = ++queue->req_id;
> +	list_add_tail(&req->list, &queue->requests);
> +	media_request_queue_unlock(queue);
> +
> +	fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
> +	if (fd < 0)
> +		return fd;
> +
> +	cmd->fd = fd;
> +
> +	return 0;
> +}
> +
> +/*
> + * Process the MEDIA_REQ_CMD_QUEUE command
> + */
> +static int media_request_cmd_queue(struct media_request_queue *queue,
> +				   struct media_request *req)
> +{
> +	int ret = 0;
> +
> +	media_request_queue_lock(queue);
> +
> +	if (req->state != MEDIA_REQUEST_STATE_IDLE) {
> +		dev_dbg(queue->mdev->dev,
> +			"%s: unable to queue request %u in state %s\n",
> +			__func__, req->id, media_request_state(req->state));
> +		media_request_queue_unlock(queue);
> +		return -EINVAL;
> +	}
> +
> +	req->state = MEDIA_REQUEST_STATE_QUEUED;
> +	ret = queue->ops->req_queue(queue, req);

Queueing a request will be a complex operation. Therefore I'd allow the
driver to manage locking by itself.

> +	if (ret) {
> +		req->state = MEDIA_REQUEST_STATE_IDLE;
> +		media_request_queue_unlock(queue);
> +		dev_dbg(queue->mdev->dev, "%s: cannot queue %u\n", __func__,
> +			req->id);
> +		return ret;
> +	}
> +
> +	media_request_queue_unlock(queue);
> +
> +	/* The queue holds a reference to the request */

I'd put this differently. How about:

	/*
	 * Hold a reference to queued requests, to be put at request complete
	 * time.
	 */

Or the comment omit altogether.

> +	media_request_get(req);

You have to do this before you release the lock above.

> +
> +	return 0;
> +}
> +
> +static int media_request_cmd_reinit(struct media_request_queue *queue,
> +				    struct media_request *req)

I'd refrain from trying to implement recycling media requestsfor now . It
can always be implemeted later on.

> +{
> +	struct media_request_entity_data *data, *next;
> +
> +	media_request_queue_lock(queue);
> +
> +	if (req->state == MEDIA_REQUEST_STATE_QUEUED) {
> +		dev_dbg(queue->mdev->dev,
> +			"%s: unable to reinit queued request %u\n", __func__,
> +			req->id);
> +		media_request_queue_unlock(queue);
> +		return -EINVAL;
> +	}
> +
> +	/* delete all entity data */
> +	list_for_each_entry_safe(data, next, &req->data, list) {
> +		list_del(&data->list);
> +		data->entity->req_ops->release_data(data);
> +	}
> +
> +	/* reinitialize request to idle state */
> +	req->state = MEDIA_REQUEST_STATE_IDLE;
> +
> +	media_request_queue_unlock(queue);
> +
> +	return 0;
> +}
> +
> +long media_device_request_cmd(struct media_device *mdev,
> +			      struct media_request_cmd *cmd)
> +{
> +	struct media_request *req = NULL;
> +	int ret;
> +
> +	if (!mdev->req_queue)
> +		return -ENOTTY;
> +
> +	if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
> +		req = media_request_get_from_fd(cmd->fd);
> +		if (IS_ERR(req))
> +			return PTR_ERR(req);
> +
> +		/* requests must belong to this media device's queue */
> +		if (req->queue != mdev->req_queue) {
> +			media_request_put(req);
> +			return -EINVAL;
> +		}
> +	}
> +
> +	switch (cmd->cmd) {
> +	case MEDIA_REQ_CMD_ALLOC:
> +		ret = media_request_cmd_alloc(mdev->req_queue, cmd);
> +		break;
> +
> +	case MEDIA_REQ_CMD_QUEUE:
> +		ret = media_request_cmd_queue(mdev->req_queue, req);
> +		break;
> +
> +	case MEDIA_REQ_CMD_REINIT:
> +		ret = media_request_cmd_reinit(mdev->req_queue, req);
> +		break;
> +
> +	default:
> +		ret = -EINVAL;
> +		break;
> +	}
> +
> +	if (req)
> +		media_request_put(req);
> +
> +	return ret;
> +}
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index b60a6b0841d1..ec4ecd5aa8bf 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -867,7 +867,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
>  	__u32 i;
>  
>  	/* zero the reserved fields */
> -	c->reserved[0] = c->reserved[1] = 0;
> +	c->reserved[0] = 0;
>  	for (i = 0; i < c->count; i++)
>  		c->controls[i].reserved2[0] = 0;
>  
> diff --git a/include/media/media-device.h b/include/media/media-device.h
> index bcc6ec434f1f..3feaa02128d1 100644
> --- a/include/media/media-device.h
> +++ b/include/media/media-device.h
> @@ -27,6 +27,7 @@
>  
>  struct ida;
>  struct device;
> +struct media_request_queue;
>  
>  /**
>   * struct media_entity_notify - Media Entity Notify
> @@ -158,6 +159,8 @@ struct media_device {
>  	void (*disable_source)(struct media_entity *entity);
>  
>  	const struct media_device_ops *ops;
> +
> +	struct media_request_queue *req_queue;
>  };
>  
>  /* We don't need to include pci.h or usb.h here */
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 222d379960b7..bfb7a4565904 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -26,6 +26,9 @@
>  #include <linux/list.h>
>  #include <linux/media.h>
>  
> +struct media_request;
> +struct media_request_entity_data;
> +
>  /* Enums used internally at the media controller to represent graphs */
>  
>  /**
> @@ -193,6 +196,8 @@ struct media_entity_operations {
>  			  const struct media_pad *local,
>  			  const struct media_pad *remote, u32 flags);
>  	int (*link_validate)(struct media_link *link);
> +	int (*process_request)(struct media_request *req,
> +			       struct media_request_entity_data *data);

process_request remains unused here.

>  };
>  
>  /**
> @@ -275,6 +280,7 @@ struct media_entity {
>  	struct list_head links;
>  
>  	const struct media_entity_operations *ops;
> +	const struct media_request_entity_ops *req_ops;
>  
>  	int stream_count;
>  	int use_count;
> diff --git a/include/media/media-request.h b/include/media/media-request.h
> new file mode 100644
> index 000000000000..ead7fd8898c4
> --- /dev/null
> +++ b/include/media/media-request.h
> @@ -0,0 +1,269 @@
> +/*
> + * Generic request queue.
> + *
> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> + *
> + * This program is free software; you can redistribute it and/or modify
> + * it under the terms of the GNU General Public License version 2 as
> + * published by the Free Software Foundation.
> + *
> + * This program is distributed in the hope that it will be useful,
> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> + * GNU General Public License for more details.
> + */
> +
> +#ifndef _MEDIA_REQUEST_H
> +#define _MEDIA_REQUEST_H
> +
> +#include <linux/kref.h>
> +#include <linux/wait.h>
> +#include <linux/list.h>

Alphabetical order?

> +
> +struct media_device;
> +struct media_request_queue;
> +struct media_request_cmd;
> +struct media_entity;
> +struct media_request_entity_data;

Here, too?

> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +
> +enum media_request_state {
> +	MEDIA_REQUEST_STATE_IDLE,
> +	MEDIA_REQUEST_STATE_QUEUED,
> +	MEDIA_REQUEST_STATE_COMPLETE,
> +	MEDIA_REQUEST_STATE_DELETED,
> +};
> +
> +/**
> + * struct media_request - Media request base structure
> + * @id:		request id, used internally for debugging
> + * @queue:	request queue this request belongs to
> + * @kref: 	reference count
> + * @list: 	list entry in the media device requests list
> + * @state:	current state of the request
> + * @data:	per-entity data list
> + * @wait_queue:	wait queue that signals once the request has completed
> + */
> +struct media_request {
> +	u32 id;
> +	struct media_request_queue *queue;
> +	struct kref kref;
> +	struct list_head list;
> +	enum media_request_state state;
> +	struct list_head data;
> +	wait_queue_head_t wait_queue;
> +};
> +
> +/**
> + * media_request_get() - increment the reference counter of a request
> + *
> + * The calling context must call media_request_put() once it does not need
> + * the reference to the request anymore.
> + *
> + * @req:	request to acquire a reference of
> + */
> +void media_request_get(struct media_request *req);
> +
> +/**
> + * media_request_get_from_fd() - lookup request by fd and acquire a reference.
> + *
> + * Look a request up from its fd, acquire a reference and return a pointer to
> + * the request. As for media_request_get(), media_request_put() must be called
> + * once the reference is not used anymore.
> + *
> + * @req:	request to lookup and acquire.
> + *
> + */
> +struct media_request *media_request_get_from_fd(int fd);
> +
> +/**
> + * media_request_put() - decrement the reference counter of a request
> + *
> + * Mirror function of media_request_get() and media_request_get_from_fd(). Will
> + * free the request if this was the last valid reference.
> + *
> + * @req:	request to release.
> + */
> +void media_request_put(struct media_request *req);
> +
> +/**
> + * media_request_get_entity_data() - get per-entity data for a request
> + * @req:	request to get entity data from
> + * @entity:	entity to get data of
> + * @fh:		cookie identifying the handle from which the entity is accessed
> + *
> + * Seach and return the entity data associated associated to the request. If no
> + * such data exists, it is allocated as per the entity operations.
> + *
> + * The fh arguments serves as a cookie to make sure the same entity is not
> + * accessed through different opened file handles. The same handle must be
> + * passed to all calls of this function for the same entity. Failure to do so
> + * will return an error.
> + *
> + * Returns the per-entity data, or an error code if a problem happened. -EINVAL
> + * means that data for the entity already existed, but has been allocated under
> + * a different cookie.
> + */
> +struct media_request_entity_data *
> +media_request_get_entity_data(struct media_request *req,
> +			      struct media_entity *entity, void *fh);
> +
> +
> +/**
> + * media_request_entity_complete() - to be invoked when an entity has completed
> + *				     its part of the currently active request
> + *
> + * @queue:	request queue
> + * @entity:	the entity that completed
> + */
> +void media_request_entity_complete(struct media_request_queue *queue,
> +				   struct media_entity *entity);
> +
> +/**
> + * struct media_request_queue_ops - request queue operations
> + *
> + * @release:	release all memory associated to the queue
> + * @req_qlloc:	allocate a request for this queue
> + * @req_free:	free a previously allocated request
> + * @req_queue:	queue a request for execution
> + * @req_complete: callback invoked when the active request has completed
> + *
> + */
> +struct media_request_queue_ops {
> +	void (*release)(struct media_request_queue *queue);
> +	struct media_request *(*req_alloc)(struct media_request_queue *queue);
> +	void (*req_free)(struct media_request_queue *queue,
> +			 struct media_request *req);
> +	int (*req_queue)(struct media_request_queue *queue,
> +			 struct media_request *req);
> +	void (*req_complete)(struct media_request_queue *queue);
> +};
> +
> +/**
> + * struct media_request_queue - queue of requests
> + *
> + * @mdev:	media_device that manages this queue
> + * @ops:	implementation of the queue
> + * @mutex:	protects requests, active_request, req_id, and all members of
> + *		struct media_request
> + * @active_request: request being currently run by this queue
> + * @requests:	list of requests (not in any particular order) that this
> + *		queue owns.
> + * @req_id:	counter used to identify requests for debugging purposes
> + */
> +struct media_request_queue {
> +	struct media_device *mdev;
> +	const struct media_request_queue_ops *ops;
> +
> +	struct mutex mutex;

Any particular reason for using a mutex? The request queue lock will need
to be acquired from interrupts, too, so this should be changed to a
spinlock.

A media device, presumably, has a single request queue. I might just put
the fields to struct media_device.

> +	struct list_head requests;
> +	struct media_request *active_request;
> +	u32 req_id;
> +};
> +
> +static inline void media_request_queue_lock(struct media_request_queue *queue)
> +{
> +	mutex_lock(&queue->mutex);
> +}
> +
> +static inline void media_request_queue_unlock(struct media_request_queue *queue)
> +{
> +	mutex_unlock(&queue->mutex);
> +}
> +
> +/**
> + * media_request_queue_init() - initialize a request queue
> + *
> + * This function is to be called by request queue implementations to initialize
> + * the base structure.
> + *
> + * @queue:	queue to initialize
> + * @mdev:	media device owning this queue
> + * @ops:	ops to be used with this queue
> + */
> +void media_request_queue_init(struct media_request_queue *queue,
> +			      struct media_device *mdev,
> +			      const struct media_request_queue_ops *ops);
> +
> +/**
> + * media_request_queue_release() - release resources held by a request queue
> + *
> + * The caller is still responsible for freeing the memory held by the queue
> + * structure itself.
> + *
> + * @queue:	queue to be released
> + */
> +void media_request_queue_release(struct media_request_queue *queue);
> +
> +/**
> + * struct media_request_entity_data - per-entity request data
> + *
> + * Base structure used to store request state data. To be extended by actual
> + * implementation.
> + *
> + * @entity:	entity this data belongs to
> + * @fh:		subsystem-dependent. For V4L2, the v4l2_fh of the opened device
> + * @list:	entry in media_request::data
> + * @completed:	whether this entity has completed its part of the request
> + */
> +struct media_request_entity_data {
> +	struct media_entity *entity;
> +	void *fh;
> +	struct list_head list;
> +	bool completed;
> +};
> +
> +/**
> + * struct media_request_entity_ops - request-related entity operations
> + *
> + * All these operations are mandatory if the request API is to be used with
> + * an entity.
> + *
> + * @alloc_data:		allocate request data for this entity
> + * @release_data:	release previously allocated data
> + * @apply_data:		apply the data to the entity prior to running a request
> + *
> + */
> +struct media_request_entity_ops {
> +	struct media_request_entity_data *(*alloc_data)
> +		(struct media_request *req, struct media_entity *entity);
> +	void (*release_data)(struct media_request_entity_data *data);
> +	int (*apply_data)(struct media_request_entity_data *data);

The media device already has an ops struct. How about putting these into
that?

> +};
> +
> +/**
> + * media_device_request_cmd() - perform the REQUEST_CMD ioctl
> + *
> + * @mdev:	media device the ioctl has been called on
> + * @cmd:	user-space request command structure
> + */
> +long media_device_request_cmd(struct media_device *mdev,
> +			      struct media_request_cmd *cmd);
> +
> +#else /* CONFIG_MEDIA_CONTROLLER */
> +
> +static inline struct media_request *media_request_get_from_fd(int fd)
> +{
> +	return ERR_PTR(-ENOTSUPP);
> +}
> +
> +static inline void media_request_put(struct media_request *req)
> +{
> +}
> +
> +static inline struct media_request *media_request_get_entity_data(
> +					  struct media_request *req,
> +					  struct media_entity *entity, void *fh)
> +{
> +	return ERR_PTR(-ENOTSUPP);
> +}
> +
> +static inline void media_request_entity_complete(struct media_request *req,
> +						 struct media_entity *entity)
> +{
> +}
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
> +#endif
> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index b9b9446095e9..247823ea0d1a 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -406,6 +406,16 @@ struct media_v2_topology {
>  	__u64 ptr_links;
>  } __attribute__ ((packed));
>  
> +#define MEDIA_REQ_CMD_ALLOC		0
> +#define MEDIA_REQ_CMD_QUEUE		1
> +#define MEDIA_REQ_CMD_REINIT		2
> +
> +struct media_request_cmd {
> +	__u32 cmd;
> +	__s32 fd;
> +	__u32 flags;
> +} __attribute__ ((packed));
> +
>  /* ioctls */
>  
>  #define MEDIA_IOC_DEVICE_INFO		_IOWR('|', 0x00, struct media_device_info)
> @@ -413,5 +423,6 @@ struct media_v2_topology {
>  #define MEDIA_IOC_ENUM_LINKS		_IOWR('|', 0x02, struct media_links_enum)
>  #define MEDIA_IOC_SETUP_LINK		_IOWR('|', 0x03, struct media_link_desc)
>  #define MEDIA_IOC_G_TOPOLOGY		_IOWR('|', 0x04, struct media_v2_topology) 
> +#define MEDIA_IOC_REQUEST_CMD		_IOWR('|', 0x05, struct media_request_cmd)
>  
>  #endif /* __LINUX_MEDIA_H */

-- 
Kind regards,

Sakari Ailus
e-mail: sakari.ailus@iki.fi

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Alexandre Courbot]

Hi Sakari, thanks for the review!

The version you reviewed is not the latest one, but I suppose most of
your comments still apply.

On Fri, Jan 26, 2018 at 5:39 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> Hi Alexandre,
>
> I remember it was discussed that the work after the V4L2 jobs API would
> continue from the existing request API patches. I see that at least the
> rather important support for events is missing in this version. Why was it
> left out?

Request completion is signaled by polling on the request FD, so we
don't need to rely on V4L2 events to signal this anymore. If we want
to signal different kinds of events on requests we could implement a
more sophisticated event system on top of that, but for our current
needs polling is sufficient.

What other kind of event besides completion could we want to deliver
to user-space from a request?

>
> I also see that variable size IOCTL argument support is no longer included.

Do we need this for the request API?

>
> On Fri, Dec 15, 2017 at 04:56:17PM +0900, Alexandre Courbot wrote:
>> The request API provides a way to group buffers and device parameters
>> into units of work to be queued and executed. This patch introduces the
>> UAPI and core framework.
>>
>> This patch is based on the previous work by Laurent Pinchart. The core
>> has changed considerably, but the UAPI is mostly untouched.
>>
>> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
>> ---
>>  drivers/media/Makefile               |   3 +-
>>  drivers/media/media-device.c         |   6 +
>>  drivers/media/media-request.c        | 390 +++++++++++++++++++++++++++++++++++
>>  drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
>>  include/media/media-device.h         |   3 +
>>  include/media/media-entity.h         |   6 +
>>  include/media/media-request.h        | 269 ++++++++++++++++++++++++
>>  include/uapi/linux/media.h           |  11 +
>>  8 files changed, 688 insertions(+), 2 deletions(-)
>>  create mode 100644 drivers/media/media-request.c
>>  create mode 100644 include/media/media-request.h
>>
>> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
>> index 594b462ddf0e..985d35ec6b29 100644
>> --- a/drivers/media/Makefile
>> +++ b/drivers/media/Makefile
>> @@ -3,7 +3,8 @@
>>  # Makefile for the kernel multimedia device drivers.
>>  #
>>
>> -media-objs   := media-device.o media-devnode.o media-entity.o
>> +media-objs   := media-device.o media-devnode.o media-entity.o \
>> +                media-request.o
>>
>>  #
>>  # I2C drivers should come before other drivers, otherwise they'll fail
>> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
>> index e79f72b8b858..045cec7d2de9 100644
>> --- a/drivers/media/media-device.c
>> +++ b/drivers/media/media-device.c
>> @@ -32,6 +32,7 @@
>>  #include <media/media-device.h>
>>  #include <media/media-devnode.h>
>>  #include <media/media-entity.h>
>> +#include <media/media-request.h>
>>
>>  #ifdef CONFIG_MEDIA_CONTROLLER
>>
>> @@ -407,6 +408,7 @@ static const struct media_ioctl_info ioctl_info[] = {
>>       MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
>>       MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
>>       MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
>> +     MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
>>  };
>>
>>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
>> @@ -688,6 +690,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
>>
>>  void media_device_cleanup(struct media_device *mdev)
>>  {
>> +     if (mdev->req_queue) {
>> +             mdev->req_queue->ops->release(mdev->req_queue);
>> +             mdev->req_queue = NULL;
>> +     }
>>       ida_destroy(&mdev->entity_internal_idx);
>>       mdev->entity_internal_idx_max = 0;
>>       media_graph_walk_cleanup(&mdev->pm_count_walk);
>> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
>> new file mode 100644
>> index 000000000000..15dc65ddfe41
>> --- /dev/null
>> +++ b/drivers/media/media-request.c
>> @@ -0,0 +1,390 @@
>> +/*
>> + * Request and request queue base management
>> + *
>> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#include <linux/anon_inodes.h>
>> +#include <linux/media.h>
>> +#include <linux/fs.h>
>> +#include <linux/file.h>
>> +#include <linux/slab.h>
>> +#include <linux/list.h>
>> +
>> +#include <media/media-request.h>
>> +#include <media/media-device.h>
>
> Alphabetic order, please.

Fixed.

>
>> +
>> +const struct file_operations request_fops;
>> +
>> +void media_request_get(struct media_request *req)
>
> Please return the media request, too.

Fixed.

>
>> +{
>> +     kref_get(&req->kref);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get);
>> +
>> +struct media_request *
>> +media_request_get_from_fd(int fd)
>> +{
>> +     struct file *f;
>> +     struct media_request *req;
>> +
>> +     f = fget(fd);
>> +     if (!f)
>> +             return NULL;
>> +
>> +     /* Not a request FD? */
>> +     if (f->f_op != &request_fops) {
>> +             fput(f);
>> +             return NULL;
>> +     }
>> +
>> +     req = f->private_data;
>> +     media_request_get(req);
>> +     fput(f);
>> +
>> +     return req;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get_from_fd);
>> +
>> +static void media_request_release(struct kref *kref)
>> +{
>> +     struct media_request_entity_data *data, *next;
>> +     struct media_request *req =
>> +             container_of(kref, typeof(*req), kref);
>> +     struct media_device *mdev = req->queue->mdev;
>> +
>> +     dev_dbg(mdev->dev, "%s: releasing request %u\n", __func__, req->id);
>
> I'd leave out __func__. It doesn't provide any additional information here.

This should be removed in the new version.

>
>> +
>> +     list_del(&req->list);
>
> You appear to have added a list of requests in the queue. You'll need to
> serialise access to that list unless you can otherwise prove you're doing
> this safely --- and that's not the case here.

I believe this is fixed in the newest version.

>
>> +     list_for_each_entry_safe(data, next, &req->data, list) {
>> +             list_del(&data->list);
>> +             data->entity->req_ops->release_data(data);
>> +     }
>> +
>> +     req->queue->ops->req_free(req->queue, req);
>> +}
>> +
>> +void media_request_put(struct media_request *req)
>> +{
>
> How about checking req is non-NULL here?

It would not hurt to add a WARN_ON indeed.

>
>> +     kref_put(&req->kref, media_request_release);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_put);
>> +
>> +struct media_request_entity_data *
>> +media_request_get_entity_data(struct media_request *req,
>> +                           struct media_entity *entity, void *fh)
>
> This makes the assumption that request data is bound to entities. How does
> this work with links?
>
> I wonder if it should rather be bound to graph objects, or certain graph
> objects. Having a standard way to bind request specific information e.g. to
> entities is definitely worth having, though.
>
> V4L2 framework specific information would be needed across the media graph
> and it'd be good to store it in a non-driver specific way. What I think
> you'd need is an interface that allows storing information based on two
> keys --- the request and e.g. a pointer provided by the caller. The V4L2
> framework would have one key, e.g. a pointer to an empty struct defined
> somewhere in the V4L2 framework could be used for the purpose.
>
> Going forward, the entire media graph state will be subject to changing
> through requests. This includes link state, media bus and pixel formats,
> cropping and scaling configurations, everything. Let's not try to go there
> yet in this patchset, but what I'm asking is to keep the goal in mind when
> implementating the request API.

Yeah, I think a similar idea is brought up in the cover letter of the
next revision (although for different reasons). Entities are probably
not a one-fit for all use-cases.

For the case of links though, I believe that the "entity" that would
control them would be the media controller itself, since it is the one
that takes the MEDIA_IOC_SETUP_LINK ioctl. But even for this case, we
cannot use an entity to look up the media_device, so something more
generic like an opaque key would probably be needed.

>
>> +{
>> +     struct media_request_queue *queue = req->queue;
>> +     struct media_request_entity_data *data;
>> +
>> +     media_request_queue_lock(queue);
>> +
>> +     /* First look whether we already have entity data */
>> +     list_for_each_entry(data, &req->data, list) {
>> +             if (data->entity == entity) {
>
> This is fine for now but something more efficient will be needed in the
> long run.

Agreed, although I am not sure we will even have that many entities per request.

>
>> +                     /*
>> +                      * If so, then the fh must match otherwise this means
>> +                      * we are trying to set the same entity through
>> +                      * different handles
>> +                      */
>
> What's the purpose of this? The request is identified by the file handle
> already.

That's the file handle of the device we are looking up (e.g. the V4L2
device taking the ioctl operation that triggered this lookup), not the
FH of the request itself.

This is needed to make sure we are not using two different FDs coming
from the same device in the same request. I.e. you cannot open a video
device twice, and then use the same request on the two different FHs
you have obtained, since the device can only be configured for one of
these FHs at a given time.

>
>> +                     if (data->fh != fh)
>> +                             data = ERR_PTR(-EINVAL);
>> +                     goto done;
>> +             }
>> +     }
>> +
>> +     /* No entity data found, let's create it */
>> +     data = entity->req_ops->alloc_data(req, entity);
>> +     if (IS_ERR(data))
>> +             goto done;
>> +     data->fh = fh;
>> +     data->entity = entity;
>> +     list_add_tail(&data->list, &req->data);
>> +
>> +done:
>> +     media_request_queue_unlock(queue);
>> +     return data;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_get_entity_data);
>> +
>> +static const char * const media_request_states[] __maybe_unused = {
>> +     "IDLE",
>> +     "QUEUED",
>> +     "COMPLETED",
>> +     "DELETED",
>> +};
>> +
>> +static const char *media_request_state(enum media_request_state state)
>> +{
>> +     return state < ARRAY_SIZE(media_request_states) ?
>> +             media_request_states[state] : "INVALID";
>> +}
>> +
>> +static int media_device_request_close(struct inode *inode, struct file *filp)
>> +{
>> +     struct media_request *req = filp->private_data;
>> +
>> +     if (req == NULL)
>> +             return 0;
>> +
>> +     media_request_put(req);
>> +
>> +     return 0;
>> +}
>> +
>> +static unsigned int media_request_poll(struct file *file, poll_table *wait)
>> +{
>> +     struct media_request *req = file->private_data;
>> +
>> +     poll_wait(file, &req->wait_queue, wait);
>> +
>> +     if (req->state == MEDIA_REQUEST_STATE_COMPLETE)
>> +             return POLLIN | POLLRDNORM;
>> +
>> +     return 0;
>> +}
>> +
>> +const struct file_operations request_fops = {
>> +     .owner = THIS_MODULE,
>> +     .poll = media_request_poll,
>> +     .release = media_device_request_close,
>> +};
>> +
>> +/*
>> + * Called when all the entities taking part in a request have completed their
>> + * part.
>> + *
>> + * The request queue mutex must be held when this function is called.
>> + */
>> +static void media_device_request_complete(struct media_request *req)
>> +{
>> +     struct media_request_queue *queue = req->queue;
>> +     struct media_device *mdev = queue->mdev;
>> +
>> +     if (WARN_ON(req->state != MEDIA_REQUEST_STATE_QUEUED)) {
>> +             dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
>> +                     __func__, req->id, media_request_state(req->state));
>> +             return;
>> +     }
>> +
>> +     req->state = MEDIA_REQUEST_STATE_COMPLETE;
>> +
>> +     wake_up_interruptible(&req->wait_queue);
>> +
>> +     queue->ops->req_complete(queue);
>> +
>> +     /* Release the reference acquired when we queued the request */
>> +     media_request_put(req);
>> +}
>> +
>> +void media_request_entity_complete(struct media_request_queue *queue,
>> +                                struct media_entity *entity)
>> +{
>> +     struct media_request_entity_data *data;
>> +     struct media_request *req;
>> +     bool completed = true;
>> +
>> +     media_request_queue_lock(queue);
>> +
>> +     req = queue->active_request;
>> +
>> +     list_for_each_entry(data, &req->data, list) {
>> +             if (data->entity == entity)
>> +                     data->completed = true;
>> +
>> +             if (!data->completed)
>> +                     completed = false;
>> +     }
>> +
>> +     /* All entities of the request have completed, request is done */
>> +     if (completed)
>> +             media_device_request_complete(req);
>> +
>> +     media_request_queue_unlock(queue);
>> +
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_entity_complete);
>> +
>> +void media_request_queue_init(struct media_request_queue *queue,
>> +                           struct media_device *mdev,
>> +                           const struct media_request_queue_ops *ops)
>> +{
>> +     queue->mdev = mdev;
>> +     mutex_init(&queue->mutex);
>> +     INIT_LIST_HEAD(&queue->requests);
>> +     queue->ops = ops;
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_queue_init);
>> +
>> +void media_request_queue_release(struct media_request_queue *queue)
>> +{
>> +     struct media_device *mdev = queue->mdev;
>> +
>> +     media_request_queue_lock(queue);
>> +
>> +     /* Just a sanity check - we should have no remaining requests */
>> +     while (!list_empty(&queue->requests)) {
>> +             struct media_request *req;
>> +
>> +             req = list_first_entry(&queue->requests, typeof(*req), list);
>> +             dev_warn(mdev->dev,
>> +                     "%s: request %u still referenced, deleting forcibly!\n",
>> +                     __func__, req->id);
>> +             media_request_release(&req->kref);
>> +     }
>> +
>> +     media_request_queue_unlock(queue);
>> +}
>> +EXPORT_SYMBOL_GPL(media_request_queue_release);
>> +
>> +/*
>> + * Process the MEDIA_REQ_CMD_ALLOC command
>> + */
>> +static int media_request_cmd_alloc(struct media_request_queue *queue,
>> +                                struct media_request_cmd *cmd)
>> +{
>> +     struct media_request *req;
>> +     int fd;
>> +
>> +     req = queue->ops->req_alloc(queue);
>> +     if (!req)
>> +             return -ENOMEM;
>> +
>> +     req->queue = queue;
>> +     req->state = MEDIA_REQUEST_STATE_IDLE;
>> +     kref_init(&req->kref);
>> +     INIT_LIST_HEAD(&req->data);
>> +     init_waitqueue_head(&req->wait_queue);
>> +
>> +     media_request_queue_lock(queue);
>> +     req->id = ++queue->req_id;
>> +     list_add_tail(&req->list, &queue->requests);
>> +     media_request_queue_unlock(queue);
>> +
>> +     fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
>> +     if (fd < 0)
>> +             return fd;
>> +
>> +     cmd->fd = fd;
>> +
>> +     return 0;
>> +}
>> +
>> +/*
>> + * Process the MEDIA_REQ_CMD_QUEUE command
>> + */
>> +static int media_request_cmd_queue(struct media_request_queue *queue,
>> +                                struct media_request *req)
>> +{
>> +     int ret = 0;
>> +
>> +     media_request_queue_lock(queue);
>> +
>> +     if (req->state != MEDIA_REQUEST_STATE_IDLE) {
>> +             dev_dbg(queue->mdev->dev,
>> +                     "%s: unable to queue request %u in state %s\n",
>> +                     __func__, req->id, media_request_state(req->state));
>> +             media_request_queue_unlock(queue);
>> +             return -EINVAL;
>> +     }
>> +
>> +     req->state = MEDIA_REQUEST_STATE_QUEUED;
>> +     ret = queue->ops->req_queue(queue, req);
>
> Queueing a request will be a complex operation. Therefore I'd allow the
> driver to manage locking by itself.
>
>> +     if (ret) {
>> +             req->state = MEDIA_REQUEST_STATE_IDLE;
>> +             media_request_queue_unlock(queue);
>> +             dev_dbg(queue->mdev->dev, "%s: cannot queue %u\n", __func__,
>> +                     req->id);
>> +             return ret;
>> +     }
>> +
>> +     media_request_queue_unlock(queue);
>> +
>> +     /* The queue holds a reference to the request */
>
> I'd put this differently. How about:
>
>         /*
>          * Hold a reference to queued requests, to be put at request complete
>          * time.
>          */
>
> Or the comment omit altogether.

The next revision says "Hold a reference to the request until it is
completed", which I think is more succint.

>
>> +     media_request_get(req);
>
> You have to do this before you release the lock above.

Fixed.

>
>> +
>> +     return 0;
>> +}
>> +
>> +static int media_request_cmd_reinit(struct media_request_queue *queue,
>> +                                 struct media_request *req)
>
> I'd refrain from trying to implement recycling media requestsfor now . It
> can always be implemeted later on.

Why not? It is not particularly difficult to do, and way faster than
reallocating a request from start every time.

>
>> +{
>> +     struct media_request_entity_data *data, *next;
>> +
>> +     media_request_queue_lock(queue);
>> +
>> +     if (req->state == MEDIA_REQUEST_STATE_QUEUED) {
>> +             dev_dbg(queue->mdev->dev,
>> +                     "%s: unable to reinit queued request %u\n", __func__,
>> +                     req->id);
>> +             media_request_queue_unlock(queue);
>> +             return -EINVAL;
>> +     }
>> +
>> +     /* delete all entity data */
>> +     list_for_each_entry_safe(data, next, &req->data, list) {
>> +             list_del(&data->list);
>> +             data->entity->req_ops->release_data(data);
>> +     }
>> +
>> +     /* reinitialize request to idle state */
>> +     req->state = MEDIA_REQUEST_STATE_IDLE;
>> +
>> +     media_request_queue_unlock(queue);
>> +
>> +     return 0;
>> +}
>> +
>> +long media_device_request_cmd(struct media_device *mdev,
>> +                           struct media_request_cmd *cmd)
>> +{
>> +     struct media_request *req = NULL;
>> +     int ret;
>> +
>> +     if (!mdev->req_queue)
>> +             return -ENOTTY;
>> +
>> +     if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
>> +             req = media_request_get_from_fd(cmd->fd);
>> +             if (IS_ERR(req))
>> +                     return PTR_ERR(req);
>> +
>> +             /* requests must belong to this media device's queue */
>> +             if (req->queue != mdev->req_queue) {
>> +                     media_request_put(req);
>> +                     return -EINVAL;
>> +             }
>> +     }
>> +
>> +     switch (cmd->cmd) {
>> +     case MEDIA_REQ_CMD_ALLOC:
>> +             ret = media_request_cmd_alloc(mdev->req_queue, cmd);
>> +             break;
>> +
>> +     case MEDIA_REQ_CMD_QUEUE:
>> +             ret = media_request_cmd_queue(mdev->req_queue, req);
>> +             break;
>> +
>> +     case MEDIA_REQ_CMD_REINIT:
>> +             ret = media_request_cmd_reinit(mdev->req_queue, req);
>> +             break;
>> +
>> +     default:
>> +             ret = -EINVAL;
>> +             break;
>> +     }
>> +
>> +     if (req)
>> +             media_request_put(req);
>> +
>> +     return ret;
>> +}
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index b60a6b0841d1..ec4ecd5aa8bf 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -867,7 +867,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
>>       __u32 i;
>>
>>       /* zero the reserved fields */
>> -     c->reserved[0] = c->reserved[1] = 0;
>> +     c->reserved[0] = 0;
>>       for (i = 0; i < c->count; i++)
>>               c->controls[i].reserved2[0] = 0;
>>
>> diff --git a/include/media/media-device.h b/include/media/media-device.h
>> index bcc6ec434f1f..3feaa02128d1 100644
>> --- a/include/media/media-device.h
>> +++ b/include/media/media-device.h
>> @@ -27,6 +27,7 @@
>>
>>  struct ida;
>>  struct device;
>> +struct media_request_queue;
>>
>>  /**
>>   * struct media_entity_notify - Media Entity Notify
>> @@ -158,6 +159,8 @@ struct media_device {
>>       void (*disable_source)(struct media_entity *entity);
>>
>>       const struct media_device_ops *ops;
>> +
>> +     struct media_request_queue *req_queue;
>>  };
>>
>>  /* We don't need to include pci.h or usb.h here */
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index 222d379960b7..bfb7a4565904 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -26,6 +26,9 @@
>>  #include <linux/list.h>
>>  #include <linux/media.h>
>>
>> +struct media_request;
>> +struct media_request_entity_data;
>> +
>>  /* Enums used internally at the media controller to represent graphs */
>>
>>  /**
>> @@ -193,6 +196,8 @@ struct media_entity_operations {
>>                         const struct media_pad *local,
>>                         const struct media_pad *remote, u32 flags);
>>       int (*link_validate)(struct media_link *link);
>> +     int (*process_request)(struct media_request *req,
>> +                            struct media_request_entity_data *data);
>
> process_request remains unused here.

This has been removed in the new revision.

>
>>  };
>>
>>  /**
>> @@ -275,6 +280,7 @@ struct media_entity {
>>       struct list_head links;
>>
>>       const struct media_entity_operations *ops;
>> +     const struct media_request_entity_ops *req_ops;
>>
>>       int stream_count;
>>       int use_count;
>> diff --git a/include/media/media-request.h b/include/media/media-request.h
>> new file mode 100644
>> index 000000000000..ead7fd8898c4
>> --- /dev/null
>> +++ b/include/media/media-request.h
>> @@ -0,0 +1,269 @@
>> +/*
>> + * Generic request queue.
>> + *
>> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
>> + *
>> + * This program is free software; you can redistribute it and/or modify
>> + * it under the terms of the GNU General Public License version 2 as
>> + * published by the Free Software Foundation.
>> + *
>> + * This program is distributed in the hope that it will be useful,
>> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
>> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
>> + * GNU General Public License for more details.
>> + */
>> +
>> +#ifndef _MEDIA_REQUEST_H
>> +#define _MEDIA_REQUEST_H
>> +
>> +#include <linux/kref.h>
>> +#include <linux/wait.h>
>> +#include <linux/list.h>
>
> Alphabetical order?

Fixed.

>
>> +
>> +struct media_device;
>> +struct media_request_queue;
>> +struct media_request_cmd;
>> +struct media_entity;
>> +struct media_request_entity_data;
>
> Here, too?

Fixed.

>
>> +
>> +#ifdef CONFIG_MEDIA_CONTROLLER
>> +
>> +enum media_request_state {
>> +     MEDIA_REQUEST_STATE_IDLE,
>> +     MEDIA_REQUEST_STATE_QUEUED,
>> +     MEDIA_REQUEST_STATE_COMPLETE,
>> +     MEDIA_REQUEST_STATE_DELETED,
>> +};
>> +
>> +/**
>> + * struct media_request - Media request base structure
>> + * @id:              request id, used internally for debugging
>> + * @queue:   request queue this request belongs to
>> + * @kref:    reference count
>> + * @list:    list entry in the media device requests list
>> + * @state:   current state of the request
>> + * @data:    per-entity data list
>> + * @wait_queue:      wait queue that signals once the request has completed
>> + */
>> +struct media_request {
>> +     u32 id;
>> +     struct media_request_queue *queue;
>> +     struct kref kref;
>> +     struct list_head list;
>> +     enum media_request_state state;
>> +     struct list_head data;
>> +     wait_queue_head_t wait_queue;
>> +};
>> +
>> +/**
>> + * media_request_get() - increment the reference counter of a request
>> + *
>> + * The calling context must call media_request_put() once it does not need
>> + * the reference to the request anymore.
>> + *
>> + * @req:     request to acquire a reference of
>> + */
>> +void media_request_get(struct media_request *req);
>> +
>> +/**
>> + * media_request_get_from_fd() - lookup request by fd and acquire a reference.
>> + *
>> + * Look a request up from its fd, acquire a reference and return a pointer to
>> + * the request. As for media_request_get(), media_request_put() must be called
>> + * once the reference is not used anymore.
>> + *
>> + * @req:     request to lookup and acquire.
>> + *
>> + */
>> +struct media_request *media_request_get_from_fd(int fd);
>> +
>> +/**
>> + * media_request_put() - decrement the reference counter of a request
>> + *
>> + * Mirror function of media_request_get() and media_request_get_from_fd(). Will
>> + * free the request if this was the last valid reference.
>> + *
>> + * @req:     request to release.
>> + */
>> +void media_request_put(struct media_request *req);
>> +
>> +/**
>> + * media_request_get_entity_data() - get per-entity data for a request
>> + * @req:     request to get entity data from
>> + * @entity:  entity to get data of
>> + * @fh:              cookie identifying the handle from which the entity is accessed
>> + *
>> + * Seach and return the entity data associated associated to the request. If no
>> + * such data exists, it is allocated as per the entity operations.
>> + *
>> + * The fh arguments serves as a cookie to make sure the same entity is not
>> + * accessed through different opened file handles. The same handle must be
>> + * passed to all calls of this function for the same entity. Failure to do so
>> + * will return an error.
>> + *
>> + * Returns the per-entity data, or an error code if a problem happened. -EINVAL
>> + * means that data for the entity already existed, but has been allocated under
>> + * a different cookie.
>> + */
>> +struct media_request_entity_data *
>> +media_request_get_entity_data(struct media_request *req,
>> +                           struct media_entity *entity, void *fh);
>> +
>> +
>> +/**
>> + * media_request_entity_complete() - to be invoked when an entity has completed
>> + *                                its part of the currently active request
>> + *
>> + * @queue:   request queue
>> + * @entity:  the entity that completed
>> + */
>> +void media_request_entity_complete(struct media_request_queue *queue,
>> +                                struct media_entity *entity);
>> +
>> +/**
>> + * struct media_request_queue_ops - request queue operations
>> + *
>> + * @release: release all memory associated to the queue
>> + * @req_qlloc:       allocate a request for this queue
>> + * @req_free:        free a previously allocated request
>> + * @req_queue:       queue a request for execution
>> + * @req_complete: callback invoked when the active request has completed
>> + *
>> + */
>> +struct media_request_queue_ops {
>> +     void (*release)(struct media_request_queue *queue);
>> +     struct media_request *(*req_alloc)(struct media_request_queue *queue);
>> +     void (*req_free)(struct media_request_queue *queue,
>> +                      struct media_request *req);
>> +     int (*req_queue)(struct media_request_queue *queue,
>> +                      struct media_request *req);
>> +     void (*req_complete)(struct media_request_queue *queue);
>> +};
>> +
>> +/**
>> + * struct media_request_queue - queue of requests
>> + *
>> + * @mdev:    media_device that manages this queue
>> + * @ops:     implementation of the queue
>> + * @mutex:   protects requests, active_request, req_id, and all members of
>> + *           struct media_request
>> + * @active_request: request being currently run by this queue
>> + * @requests:        list of requests (not in any particular order) that this
>> + *           queue owns.
>> + * @req_id:  counter used to identify requests for debugging purposes
>> + */
>> +struct media_request_queue {
>> +     struct media_device *mdev;
>> +     const struct media_request_queue_ops *ops;
>> +
>> +     struct mutex mutex;
>
> Any particular reason for using a mutex? The request queue lock will need
> to be acquired from interrupts, too, so this should be changed to a
> spinlock.

Will it be acquired from interrupts? In any case it should be possible
to change this to a spinlock.

>
> A media device, presumably, has a single request queue. I might just put
> the fields to struct media_device.

I find it easier to understand when different aspects are separated
into different data structures.

>
>> +     struct list_head requests;
>> +     struct media_request *active_request;
>> +     u32 req_id;
>> +};
>> +
>> +static inline void media_request_queue_lock(struct media_request_queue *queue)
>> +{
>> +     mutex_lock(&queue->mutex);
>> +}
>> +
>> +static inline void media_request_queue_unlock(struct media_request_queue *queue)
>> +{
>> +     mutex_unlock(&queue->mutex);
>> +}
>> +
>> +/**
>> + * media_request_queue_init() - initialize a request queue
>> + *
>> + * This function is to be called by request queue implementations to initialize
>> + * the base structure.
>> + *
>> + * @queue:   queue to initialize
>> + * @mdev:    media device owning this queue
>> + * @ops:     ops to be used with this queue
>> + */
>> +void media_request_queue_init(struct media_request_queue *queue,
>> +                           struct media_device *mdev,
>> +                           const struct media_request_queue_ops *ops);
>> +
>> +/**
>> + * media_request_queue_release() - release resources held by a request queue
>> + *
>> + * The caller is still responsible for freeing the memory held by the queue
>> + * structure itself.
>> + *
>> + * @queue:   queue to be released
>> + */
>> +void media_request_queue_release(struct media_request_queue *queue);
>> +
>> +/**
>> + * struct media_request_entity_data - per-entity request data
>> + *
>> + * Base structure used to store request state data. To be extended by actual
>> + * implementation.
>> + *
>> + * @entity:  entity this data belongs to
>> + * @fh:              subsystem-dependent. For V4L2, the v4l2_fh of the opened device
>> + * @list:    entry in media_request::data
>> + * @completed:       whether this entity has completed its part of the request
>> + */
>> +struct media_request_entity_data {
>> +     struct media_entity *entity;
>> +     void *fh;
>> +     struct list_head list;
>> +     bool completed;
>> +};
>> +
>> +/**
>> + * struct media_request_entity_ops - request-related entity operations
>> + *
>> + * All these operations are mandatory if the request API is to be used with
>> + * an entity.
>> + *
>> + * @alloc_data:              allocate request data for this entity
>> + * @release_data:    release previously allocated data
>> + * @apply_data:              apply the data to the entity prior to running a request
>> + *
>> + */
>> +struct media_request_entity_ops {
>> +     struct media_request_entity_data *(*alloc_data)
>> +             (struct media_request *req, struct media_entity *entity);
>> +     void (*release_data)(struct media_request_entity_data *data);
>> +     int (*apply_data)(struct media_request_entity_data *data);
>
> The media device already has an ops struct. How about putting these into
> that?

Again, separation of aspects into dedicated structures, especially
since we are not completely sure that this should end into
media_entity.

>
>> +};
>> +
>> +/**
>> + * media_device_request_cmd() - perform the REQUEST_CMD ioctl
>> + *
>> + * @mdev:    media device the ioctl has been called on
>> + * @cmd:     user-space request command structure
>> + */
>> +long media_device_request_cmd(struct media_device *mdev,
>> +                           struct media_request_cmd *cmd);
>> +
>> +#else /* CONFIG_MEDIA_CONTROLLER */
>> +
>> +static inline struct media_request *media_request_get_from_fd(int fd)
>> +{
>> +     return ERR_PTR(-ENOTSUPP);
>> +}
>> +
>> +static inline void media_request_put(struct media_request *req)
>> +{
>> +}
>> +
>> +static inline struct media_request *media_request_get_entity_data(
>> +                                       struct media_request *req,
>> +                                       struct media_entity *entity, void *fh)
>> +{
>> +     return ERR_PTR(-ENOTSUPP);
>> +}
>> +
>> +static inline void media_request_entity_complete(struct media_request *req,
>> +                                              struct media_entity *entity)
>> +{
>> +}
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> +
>> +#endif
>> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
>> index b9b9446095e9..247823ea0d1a 100644
>> --- a/include/uapi/linux/media.h
>> +++ b/include/uapi/linux/media.h
>> @@ -406,6 +406,16 @@ struct media_v2_topology {
>>       __u64 ptr_links;
>>  } __attribute__ ((packed));
>>
>> +#define MEDIA_REQ_CMD_ALLOC          0
>> +#define MEDIA_REQ_CMD_QUEUE          1
>> +#define MEDIA_REQ_CMD_REINIT         2
>> +
>> +struct media_request_cmd {
>> +     __u32 cmd;
>> +     __s32 fd;
>> +     __u32 flags;
>> +} __attribute__ ((packed));
>> +
>>  /* ioctls */
>>
>>  #define MEDIA_IOC_DEVICE_INFO                _IOWR('|', 0x00, struct media_device_info)
>> @@ -413,5 +423,6 @@ struct media_v2_topology {
>>  #define MEDIA_IOC_ENUM_LINKS         _IOWR('|', 0x02, struct media_links_enum)
>>  #define MEDIA_IOC_SETUP_LINK         _IOWR('|', 0x03, struct media_link_desc)
>>  #define MEDIA_IOC_G_TOPOLOGY         _IOWR('|', 0x04, struct media_v2_topology)
>> +#define MEDIA_IOC_REQUEST_CMD                _IOWR('|', 0x05, struct media_request_cmd)
>>
>>  #endif /* __LINUX_MEDIA_H */
>
> --
> Kind regards,
>
> Sakari Ailus
> e-mail: sakari.ailus@iki.fi

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Sakari Ailus]

Hi Alexandre,

On Tue, Jan 30, 2018 at 01:23:05PM +0900, Alexandre Courbot wrote:
> Hi Sakari, thanks for the review!
> 
> The version you reviewed is not the latest one, but I suppose most of
> your comments still apply.
> 
> On Fri, Jan 26, 2018 at 5:39 PM, Sakari Ailus <sakari.ailus@iki.fi> wrote:
> > Hi Alexandre,
> >
> > I remember it was discussed that the work after the V4L2 jobs API would
> > continue from the existing request API patches. I see that at least the
> > rather important support for events is missing in this version. Why was it
> > left out?
> 
> Request completion is signaled by polling on the request FD, so we
> don't need to rely on V4L2 events to signal this anymore. If we want
> to signal different kinds of events on requests we could implement a
> more sophisticated event system on top of that, but for our current
> needs polling is sufficient.

Right. This works for now indeed. We will need to revisit this when
requests are moved to the media device in the future.

> 
> What other kind of event besides completion could we want to deliver
> to user-space from a request?
> 
> >
> > I also see that variable size IOCTL argument support is no longer included.
> 
> Do we need this for the request API?

Technically there's no strict need for that now. However when the requests
are moved to the media device (i.e. other device nodes are not needed
anymore), then this is a must.

It was proposed and AFAIR agreed on as well that new media device
IOCTLs would not use reserved fields any longer but rely on variable size
IOCTL arguments instead. This is in line with your request argument struct
having no reserved fields and I don't think we should add them there.

> 
> >
> > On Fri, Dec 15, 2017 at 04:56:17PM +0900, Alexandre Courbot wrote:
> >> The request API provides a way to group buffers and device parameters
> >> into units of work to be queued and executed. This patch introduces the
> >> UAPI and core framework.
> >>
> >> This patch is based on the previous work by Laurent Pinchart. The core
> >> has changed considerably, but the UAPI is mostly untouched.
> >>
> >> Signed-off-by: Alexandre Courbot <acourbot@chromium.org>
> >> ---
> >>  drivers/media/Makefile               |   3 +-
> >>  drivers/media/media-device.c         |   6 +
> >>  drivers/media/media-request.c        | 390 +++++++++++++++++++++++++++++++++++
> >>  drivers/media/v4l2-core/v4l2-ioctl.c |   2 +-
> >>  include/media/media-device.h         |   3 +
> >>  include/media/media-entity.h         |   6 +
> >>  include/media/media-request.h        | 269 ++++++++++++++++++++++++
> >>  include/uapi/linux/media.h           |  11 +
> >>  8 files changed, 688 insertions(+), 2 deletions(-)
> >>  create mode 100644 drivers/media/media-request.c
> >>  create mode 100644 include/media/media-request.h
> >>
> >> diff --git a/drivers/media/Makefile b/drivers/media/Makefile
> >> index 594b462ddf0e..985d35ec6b29 100644
> >> --- a/drivers/media/Makefile
> >> +++ b/drivers/media/Makefile
> >> @@ -3,7 +3,8 @@
> >>  # Makefile for the kernel multimedia device drivers.
> >>  #
> >>
> >> -media-objs   := media-device.o media-devnode.o media-entity.o
> >> +media-objs   := media-device.o media-devnode.o media-entity.o \
> >> +                media-request.o
> >>
> >>  #
> >>  # I2C drivers should come before other drivers, otherwise they'll fail
> >> diff --git a/drivers/media/media-device.c b/drivers/media/media-device.c
> >> index e79f72b8b858..045cec7d2de9 100644
> >> --- a/drivers/media/media-device.c
> >> +++ b/drivers/media/media-device.c
> >> @@ -32,6 +32,7 @@
> >>  #include <media/media-device.h>
> >>  #include <media/media-devnode.h>
> >>  #include <media/media-entity.h>
> >> +#include <media/media-request.h>
> >>
> >>  #ifdef CONFIG_MEDIA_CONTROLLER
> >>
> >> @@ -407,6 +408,7 @@ static const struct media_ioctl_info ioctl_info[] = {
> >>       MEDIA_IOC(ENUM_LINKS, media_device_enum_links, MEDIA_IOC_FL_GRAPH_MUTEX),
> >>       MEDIA_IOC(SETUP_LINK, media_device_setup_link, MEDIA_IOC_FL_GRAPH_MUTEX),
> >>       MEDIA_IOC(G_TOPOLOGY, media_device_get_topology, MEDIA_IOC_FL_GRAPH_MUTEX),
> >> +     MEDIA_IOC(REQUEST_CMD, media_device_request_cmd, 0),
> >>  };
> >>
> >>  static long media_device_ioctl(struct file *filp, unsigned int cmd,
> >> @@ -688,6 +690,10 @@ EXPORT_SYMBOL_GPL(media_device_init);
> >>
> >>  void media_device_cleanup(struct media_device *mdev)
> >>  {
> >> +     if (mdev->req_queue) {
> >> +             mdev->req_queue->ops->release(mdev->req_queue);
> >> +             mdev->req_queue = NULL;
> >> +     }
> >>       ida_destroy(&mdev->entity_internal_idx);
> >>       mdev->entity_internal_idx_max = 0;
> >>       media_graph_walk_cleanup(&mdev->pm_count_walk);
> >> diff --git a/drivers/media/media-request.c b/drivers/media/media-request.c
> >> new file mode 100644
> >> index 000000000000..15dc65ddfe41
> >> --- /dev/null
> >> +++ b/drivers/media/media-request.c
> >> @@ -0,0 +1,390 @@
> >> +/*
> >> + * Request and request queue base management
> >> + *
> >> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License version 2 as
> >> + * published by the Free Software Foundation.
> >> + *
> >> + * This program is distributed in the hope that it will be useful,
> >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >> + * GNU General Public License for more details.
> >> + */
> >> +
> >> +#include <linux/anon_inodes.h>
> >> +#include <linux/media.h>
> >> +#include <linux/fs.h>
> >> +#include <linux/file.h>
> >> +#include <linux/slab.h>
> >> +#include <linux/list.h>
> >> +
> >> +#include <media/media-request.h>
> >> +#include <media/media-device.h>
> >
> > Alphabetic order, please.
> 
> Fixed.
> 
> >
> >> +
> >> +const struct file_operations request_fops;

Can this be static?

> >> +
> >> +void media_request_get(struct media_request *req)
> >
> > Please return the media request, too.
> 
> Fixed.
> 
> >
> >> +{
> >> +     kref_get(&req->kref);
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_get);
> >> +
> >> +struct media_request *
> >> +media_request_get_from_fd(int fd)
> >> +{
> >> +     struct file *f;
> >> +     struct media_request *req;
> >> +
> >> +     f = fget(fd);
> >> +     if (!f)
> >> +             return NULL;
> >> +
> >> +     /* Not a request FD? */
> >> +     if (f->f_op != &request_fops) {
> >> +             fput(f);
> >> +             return NULL;
> >> +     }
> >> +
> >> +     req = f->private_data;
> >> +     media_request_get(req);
> >> +     fput(f);
> >> +
> >> +     return req;
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_get_from_fd);
> >> +
> >> +static void media_request_release(struct kref *kref)
> >> +{
> >> +     struct media_request_entity_data *data, *next;
> >> +     struct media_request *req =
> >> +             container_of(kref, typeof(*req), kref);
> >> +     struct media_device *mdev = req->queue->mdev;
> >> +
> >> +     dev_dbg(mdev->dev, "%s: releasing request %u\n", __func__, req->id);
> >
> > I'd leave out __func__. It doesn't provide any additional information here.
> 
> This should be removed in the new version.
> 
> >
> >> +
> >> +     list_del(&req->list);
> >
> > You appear to have added a list of requests in the queue. You'll need to
> > serialise access to that list unless you can otherwise prove you're doing
> > this safely --- and that's not the case here.
> 
> I believe this is fixed in the newest version.
> 
> >
> >> +     list_for_each_entry_safe(data, next, &req->data, list) {
> >> +             list_del(&data->list);
> >> +             data->entity->req_ops->release_data(data);
> >> +     }
> >> +
> >> +     req->queue->ops->req_free(req->queue, req);
> >> +}
> >> +
> >> +void media_request_put(struct media_request *req)
> >> +{
> >
> > How about checking req is non-NULL here?
> 
> It would not hurt to add a WARN_ON indeed.

Why? In many cases you can simply omit that check in the caller.

> 
> >
> >> +     kref_put(&req->kref, media_request_release);
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_put);
> >> +
> >> +struct media_request_entity_data *
> >> +media_request_get_entity_data(struct media_request *req,
> >> +                           struct media_entity *entity, void *fh)
> >
> > This makes the assumption that request data is bound to entities. How does
> > this work with links?
> >
> > I wonder if it should rather be bound to graph objects, or certain graph
> > objects. Having a standard way to bind request specific information e.g. to
> > entities is definitely worth having, though.
> >
> > V4L2 framework specific information would be needed across the media graph
> > and it'd be good to store it in a non-driver specific way. What I think
> > you'd need is an interface that allows storing information based on two
> > keys --- the request and e.g. a pointer provided by the caller. The V4L2
> > framework would have one key, e.g. a pointer to an empty struct defined
> > somewhere in the V4L2 framework could be used for the purpose.
> >
> > Going forward, the entire media graph state will be subject to changing
> > through requests. This includes link state, media bus and pixel formats,
> > cropping and scaling configurations, everything. Let's not try to go there
> > yet in this patchset, but what I'm asking is to keep the goal in mind when
> > implementating the request API.
> 
> Yeah, I think a similar idea is brought up in the cover letter of the
> next revision (although for different reasons). Entities are probably
> not a one-fit for all use-cases.
> 
> For the case of links though, I believe that the "entity" that would
> control them would be the media controller itself, since it is the one
> that takes the MEDIA_IOC_SETUP_LINK ioctl. But even for this case, we
> cannot use an entity to look up the media_device, so something more
> generic like an opaque key would probably be needed.

Perhaps in the near future we still need a little less than that. Changing
something that has a state in V4L2 will be troublesome and will require
managing state of what is now stream centric.

I still think that the framework would need to do the job of managing the
video buffers related to a request as well as controls without necessarily
trying to generalise that right now. But how to store these in a meaningful
way? Putting them to the request itself would be one option: you'll need to
dig the request up anyway when things are associated to it, and the driver
needs it when it is queued.

I wonder what Hans and Laurent think.

> 
> >
> >> +{
> >> +     struct media_request_queue *queue = req->queue;
> >> +     struct media_request_entity_data *data;
> >> +
> >> +     media_request_queue_lock(queue);
> >> +
> >> +     /* First look whether we already have entity data */
> >> +     list_for_each_entry(data, &req->data, list) {
> >> +             if (data->entity == entity) {
> >
> > This is fine for now but something more efficient will be needed in the
> > long run.
> 
> Agreed, although I am not sure we will even have that many entities per request.

Perhaps not for video codecs, but this isn't limited to video codecs.

> 
> >
> >> +                     /*
> >> +                      * If so, then the fh must match otherwise this means
> >> +                      * we are trying to set the same entity through
> >> +                      * different handles
> >> +                      */
> >
> > What's the purpose of this? The request is identified by the file handle
> > already.
> 
> That's the file handle of the device we are looking up (e.g. the V4L2
> device taking the ioctl operation that triggered this lookup), not the
> FH of the request itself.
> 
> This is needed to make sure we are not using two different FDs coming
> from the same device in the same request. I.e. you cannot open a video
> device twice, and then use the same request on the two different FHs
> you have obtained, since the device can only be configured for one of
> these FHs at a given time.

Hmm, why not? I can't see how the file handle of e.g. a video device is
related to this. The request file handle already identifies a request, no
other file handles are needed for that.

> 
> >
> >> +                     if (data->fh != fh)
> >> +                             data = ERR_PTR(-EINVAL);
> >> +                     goto done;
> >> +             }
> >> +     }
> >> +
> >> +     /* No entity data found, let's create it */
> >> +     data = entity->req_ops->alloc_data(req, entity);
> >> +     if (IS_ERR(data))
> >> +             goto done;
> >> +     data->fh = fh;
> >> +     data->entity = entity;
> >> +     list_add_tail(&data->list, &req->data);
> >> +
> >> +done:
> >> +     media_request_queue_unlock(queue);
> >> +     return data;
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_get_entity_data);
> >> +
> >> +static const char * const media_request_states[] __maybe_unused = {
> >> +     "IDLE",
> >> +     "QUEUED",
> >> +     "COMPLETED",
> >> +     "DELETED",
> >> +};
> >> +
> >> +static const char *media_request_state(enum media_request_state state)
> >> +{
> >> +     return state < ARRAY_SIZE(media_request_states) ?
> >> +             media_request_states[state] : "INVALID";
> >> +}
> >> +
> >> +static int media_device_request_close(struct inode *inode, struct file *filp)
> >> +{
> >> +     struct media_request *req = filp->private_data;
> >> +
> >> +     if (req == NULL)
> >> +             return 0;
> >> +
> >> +     media_request_put(req);
> >> +
> >> +     return 0;
> >> +}
> >> +
> >> +static unsigned int media_request_poll(struct file *file, poll_table *wait)
> >> +{
> >> +     struct media_request *req = file->private_data;
> >> +
> >> +     poll_wait(file, &req->wait_queue, wait);
> >> +
> >> +     if (req->state == MEDIA_REQUEST_STATE_COMPLETE)
> >> +             return POLLIN | POLLRDNORM;
> >> +
> >> +     return 0;
> >> +}
> >> +
> >> +const struct file_operations request_fops = {
> >> +     .owner = THIS_MODULE,
> >> +     .poll = media_request_poll,
> >> +     .release = media_device_request_close,
> >> +};
> >> +
> >> +/*
> >> + * Called when all the entities taking part in a request have completed their
> >> + * part.
> >> + *
> >> + * The request queue mutex must be held when this function is called.
> >> + */
> >> +static void media_device_request_complete(struct media_request *req)
> >> +{
> >> +     struct media_request_queue *queue = req->queue;
> >> +     struct media_device *mdev = queue->mdev;
> >> +
> >> +     if (WARN_ON(req->state != MEDIA_REQUEST_STATE_QUEUED)) {
> >> +             dev_dbg(mdev->dev, "%s: can't complete %u, state %s\n",
> >> +                     __func__, req->id, media_request_state(req->state));
> >> +             return;
> >> +     }
> >> +
> >> +     req->state = MEDIA_REQUEST_STATE_COMPLETE;
> >> +
> >> +     wake_up_interruptible(&req->wait_queue);
> >> +
> >> +     queue->ops->req_complete(queue);
> >> +
> >> +     /* Release the reference acquired when we queued the request */
> >> +     media_request_put(req);
> >> +}
> >> +
> >> +void media_request_entity_complete(struct media_request_queue *queue,
> >> +                                struct media_entity *entity)
> >> +{
> >> +     struct media_request_entity_data *data;
> >> +     struct media_request *req;
> >> +     bool completed = true;
> >> +
> >> +     media_request_queue_lock(queue);
> >> +
> >> +     req = queue->active_request;
> >> +
> >> +     list_for_each_entry(data, &req->data, list) {
> >> +             if (data->entity == entity)
> >> +                     data->completed = true;
> >> +
> >> +             if (!data->completed)
> >> +                     completed = false;
> >> +     }
> >> +
> >> +     /* All entities of the request have completed, request is done */
> >> +     if (completed)
> >> +             media_device_request_complete(req);
> >> +
> >> +     media_request_queue_unlock(queue);
> >> +
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_entity_complete);
> >> +
> >> +void media_request_queue_init(struct media_request_queue *queue,
> >> +                           struct media_device *mdev,
> >> +                           const struct media_request_queue_ops *ops)
> >> +{
> >> +     queue->mdev = mdev;
> >> +     mutex_init(&queue->mutex);
> >> +     INIT_LIST_HEAD(&queue->requests);
> >> +     queue->ops = ops;
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_queue_init);
> >> +
> >> +void media_request_queue_release(struct media_request_queue *queue)
> >> +{
> >> +     struct media_device *mdev = queue->mdev;
> >> +
> >> +     media_request_queue_lock(queue);
> >> +
> >> +     /* Just a sanity check - we should have no remaining requests */
> >> +     while (!list_empty(&queue->requests)) {
> >> +             struct media_request *req;
> >> +
> >> +             req = list_first_entry(&queue->requests, typeof(*req), list);
> >> +             dev_warn(mdev->dev,
> >> +                     "%s: request %u still referenced, deleting forcibly!\n",
> >> +                     __func__, req->id);
> >> +             media_request_release(&req->kref);
> >> +     }
> >> +
> >> +     media_request_queue_unlock(queue);
> >> +}
> >> +EXPORT_SYMBOL_GPL(media_request_queue_release);
> >> +
> >> +/*
> >> + * Process the MEDIA_REQ_CMD_ALLOC command
> >> + */
> >> +static int media_request_cmd_alloc(struct media_request_queue *queue,
> >> +                                struct media_request_cmd *cmd)
> >> +{
> >> +     struct media_request *req;
> >> +     int fd;
> >> +
> >> +     req = queue->ops->req_alloc(queue);
> >> +     if (!req)
> >> +             return -ENOMEM;
> >> +
> >> +     req->queue = queue;
> >> +     req->state = MEDIA_REQUEST_STATE_IDLE;
> >> +     kref_init(&req->kref);
> >> +     INIT_LIST_HEAD(&req->data);
> >> +     init_waitqueue_head(&req->wait_queue);
> >> +
> >> +     media_request_queue_lock(queue);
> >> +     req->id = ++queue->req_id;
> >> +     list_add_tail(&req->list, &queue->requests);
> >> +     media_request_queue_unlock(queue);
> >> +
> >> +     fd = anon_inode_getfd("media_request", &request_fops, req, O_CLOEXEC);
> >> +     if (fd < 0)
> >> +             return fd;
> >> +
> >> +     cmd->fd = fd;
> >> +
> >> +     return 0;
> >> +}
> >> +
> >> +/*
> >> + * Process the MEDIA_REQ_CMD_QUEUE command
> >> + */
> >> +static int media_request_cmd_queue(struct media_request_queue *queue,
> >> +                                struct media_request *req)
> >> +{
> >> +     int ret = 0;
> >> +
> >> +     media_request_queue_lock(queue);
> >> +
> >> +     if (req->state != MEDIA_REQUEST_STATE_IDLE) {
> >> +             dev_dbg(queue->mdev->dev,
> >> +                     "%s: unable to queue request %u in state %s\n",
> >> +                     __func__, req->id, media_request_state(req->state));
> >> +             media_request_queue_unlock(queue);
> >> +             return -EINVAL;
> >> +     }
> >> +
> >> +     req->state = MEDIA_REQUEST_STATE_QUEUED;
> >> +     ret = queue->ops->req_queue(queue, req);
> >
> > Queueing a request will be a complex operation. Therefore I'd allow the
> > driver to manage locking by itself.

Could you comment this one, please?

> >
> >> +     if (ret) {
> >> +             req->state = MEDIA_REQUEST_STATE_IDLE;
> >> +             media_request_queue_unlock(queue);
> >> +             dev_dbg(queue->mdev->dev, "%s: cannot queue %u\n", __func__,
> >> +                     req->id);
> >> +             return ret;
> >> +     }
> >> +
> >> +     media_request_queue_unlock(queue);
> >> +
> >> +     /* The queue holds a reference to the request */
> >
> > I'd put this differently. How about:
> >
> >         /*
> >          * Hold a reference to queued requests, to be put at request complete
> >          * time.
> >          */
> >
> > Or the comment omit altogether.
> 
> The next revision says "Hold a reference to the request until it is
> completed", which I think is more succint.

Sounds good.

> 
> >
> >> +     media_request_get(req);
> >
> > You have to do this before you release the lock above.
> 
> Fixed.
> 
> >
> >> +
> >> +     return 0;
> >> +}
> >> +
> >> +static int media_request_cmd_reinit(struct media_request_queue *queue,
> >> +                                 struct media_request *req)
> >
> > I'd refrain from trying to implement recycling media requestsfor now . It
> > can always be implemeted later on.
> 
> Why not? It is not particularly difficult to do, and way faster than
> reallocating a request from start every time.

If you think you can, go for it. I just think it introduces extra hassle
that we don't exactly need now.

> 
> >
> >> +{
> >> +     struct media_request_entity_data *data, *next;
> >> +
> >> +     media_request_queue_lock(queue);
> >> +
> >> +     if (req->state == MEDIA_REQUEST_STATE_QUEUED) {
> >> +             dev_dbg(queue->mdev->dev,
> >> +                     "%s: unable to reinit queued request %u\n", __func__,
> >> +                     req->id);
> >> +             media_request_queue_unlock(queue);
> >> +             return -EINVAL;
> >> +     }
> >> +
> >> +     /* delete all entity data */
> >> +     list_for_each_entry_safe(data, next, &req->data, list) {
> >> +             list_del(&data->list);
> >> +             data->entity->req_ops->release_data(data);
> >> +     }
> >> +
> >> +     /* reinitialize request to idle state */
> >> +     req->state = MEDIA_REQUEST_STATE_IDLE;
> >> +
> >> +     media_request_queue_unlock(queue);
> >> +
> >> +     return 0;
> >> +}
> >> +
> >> +long media_device_request_cmd(struct media_device *mdev,
> >> +                           struct media_request_cmd *cmd)
> >> +{
> >> +     struct media_request *req = NULL;
> >> +     int ret;
> >> +
> >> +     if (!mdev->req_queue)
> >> +             return -ENOTTY;
> >> +
> >> +     if (cmd->cmd != MEDIA_REQ_CMD_ALLOC) {
> >> +             req = media_request_get_from_fd(cmd->fd);
> >> +             if (IS_ERR(req))
> >> +                     return PTR_ERR(req);
> >> +
> >> +             /* requests must belong to this media device's queue */
> >> +             if (req->queue != mdev->req_queue) {
> >> +                     media_request_put(req);
> >> +                     return -EINVAL;
> >> +             }
> >> +     }
> >> +
> >> +     switch (cmd->cmd) {
> >> +     case MEDIA_REQ_CMD_ALLOC:
> >> +             ret = media_request_cmd_alloc(mdev->req_queue, cmd);
> >> +             break;
> >> +
> >> +     case MEDIA_REQ_CMD_QUEUE:
> >> +             ret = media_request_cmd_queue(mdev->req_queue, req);
> >> +             break;
> >> +
> >> +     case MEDIA_REQ_CMD_REINIT:
> >> +             ret = media_request_cmd_reinit(mdev->req_queue, req);
> >> +             break;
> >> +
> >> +     default:
> >> +             ret = -EINVAL;
> >> +             break;
> >> +     }
> >> +
> >> +     if (req)
> >> +             media_request_put(req);
> >> +
> >> +     return ret;
> >> +}
> >> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> >> index b60a6b0841d1..ec4ecd5aa8bf 100644
> >> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> >> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> >> @@ -867,7 +867,7 @@ static int check_ext_ctrls(struct v4l2_ext_controls *c, int allow_priv)
> >>       __u32 i;
> >>
> >>       /* zero the reserved fields */
> >> -     c->reserved[0] = c->reserved[1] = 0;
> >> +     c->reserved[0] = 0;
> >>       for (i = 0; i < c->count; i++)
> >>               c->controls[i].reserved2[0] = 0;
> >>
> >> diff --git a/include/media/media-device.h b/include/media/media-device.h
> >> index bcc6ec434f1f..3feaa02128d1 100644
> >> --- a/include/media/media-device.h
> >> +++ b/include/media/media-device.h
> >> @@ -27,6 +27,7 @@
> >>
> >>  struct ida;
> >>  struct device;
> >> +struct media_request_queue;
> >>
> >>  /**
> >>   * struct media_entity_notify - Media Entity Notify
> >> @@ -158,6 +159,8 @@ struct media_device {
> >>       void (*disable_source)(struct media_entity *entity);
> >>
> >>       const struct media_device_ops *ops;
> >> +
> >> +     struct media_request_queue *req_queue;
> >>  };
> >>
> >>  /* We don't need to include pci.h or usb.h here */
> >> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> >> index 222d379960b7..bfb7a4565904 100644
> >> --- a/include/media/media-entity.h
> >> +++ b/include/media/media-entity.h
> >> @@ -26,6 +26,9 @@
> >>  #include <linux/list.h>
> >>  #include <linux/media.h>
> >>
> >> +struct media_request;
> >> +struct media_request_entity_data;
> >> +
> >>  /* Enums used internally at the media controller to represent graphs */
> >>
> >>  /**
> >> @@ -193,6 +196,8 @@ struct media_entity_operations {
> >>                         const struct media_pad *local,
> >>                         const struct media_pad *remote, u32 flags);
> >>       int (*link_validate)(struct media_link *link);
> >> +     int (*process_request)(struct media_request *req,
> >> +                            struct media_request_entity_data *data);
> >
> > process_request remains unused here.
> 
> This has been removed in the new revision.
> 
> >
> >>  };
> >>
> >>  /**
> >> @@ -275,6 +280,7 @@ struct media_entity {
> >>       struct list_head links;
> >>
> >>       const struct media_entity_operations *ops;
> >> +     const struct media_request_entity_ops *req_ops;
> >>
> >>       int stream_count;
> >>       int use_count;
> >> diff --git a/include/media/media-request.h b/include/media/media-request.h
> >> new file mode 100644
> >> index 000000000000..ead7fd8898c4
> >> --- /dev/null
> >> +++ b/include/media/media-request.h
> >> @@ -0,0 +1,269 @@
> >> +/*
> >> + * Generic request queue.
> >> + *
> >> + * Copyright (C) 2017, The Chromium OS Authors.  All rights reserved.
> >> + *
> >> + * This program is free software; you can redistribute it and/or modify
> >> + * it under the terms of the GNU General Public License version 2 as
> >> + * published by the Free Software Foundation.
> >> + *
> >> + * This program is distributed in the hope that it will be useful,
> >> + * but WITHOUT ANY WARRANTY; without even the implied warranty of
> >> + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
> >> + * GNU General Public License for more details.
> >> + */
> >> +
> >> +#ifndef _MEDIA_REQUEST_H
> >> +#define _MEDIA_REQUEST_H
> >> +
> >> +#include <linux/kref.h>
> >> +#include <linux/wait.h>
> >> +#include <linux/list.h>
> >
> > Alphabetical order?
> 
> Fixed.
> 
> >
> >> +
> >> +struct media_device;
> >> +struct media_request_queue;
> >> +struct media_request_cmd;
> >> +struct media_entity;
> >> +struct media_request_entity_data;
> >
> > Here, too?
> 
> Fixed.
> 
> >
> >> +
> >> +#ifdef CONFIG_MEDIA_CONTROLLER
> >> +
> >> +enum media_request_state {
> >> +     MEDIA_REQUEST_STATE_IDLE,
> >> +     MEDIA_REQUEST_STATE_QUEUED,
> >> +     MEDIA_REQUEST_STATE_COMPLETE,
> >> +     MEDIA_REQUEST_STATE_DELETED,
> >> +};
> >> +
> >> +/**
> >> + * struct media_request - Media request base structure
> >> + * @id:              request id, used internally for debugging
> >> + * @queue:   request queue this request belongs to
> >> + * @kref:    reference count
> >> + * @list:    list entry in the media device requests list
> >> + * @state:   current state of the request
> >> + * @data:    per-entity data list
> >> + * @wait_queue:      wait queue that signals once the request has completed
> >> + */
> >> +struct media_request {
> >> +     u32 id;
> >> +     struct media_request_queue *queue;
> >> +     struct kref kref;
> >> +     struct list_head list;
> >> +     enum media_request_state state;
> >> +     struct list_head data;
> >> +     wait_queue_head_t wait_queue;
> >> +};
> >> +
> >> +/**
> >> + * media_request_get() - increment the reference counter of a request
> >> + *
> >> + * The calling context must call media_request_put() once it does not need
> >> + * the reference to the request anymore.
> >> + *
> >> + * @req:     request to acquire a reference of
> >> + */
> >> +void media_request_get(struct media_request *req);
> >> +
> >> +/**
> >> + * media_request_get_from_fd() - lookup request by fd and acquire a reference.
> >> + *
> >> + * Look a request up from its fd, acquire a reference and return a pointer to
> >> + * the request. As for media_request_get(), media_request_put() must be called
> >> + * once the reference is not used anymore.
> >> + *
> >> + * @req:     request to lookup and acquire.
> >> + *
> >> + */
> >> +struct media_request *media_request_get_from_fd(int fd);
> >> +
> >> +/**
> >> + * media_request_put() - decrement the reference counter of a request
> >> + *
> >> + * Mirror function of media_request_get() and media_request_get_from_fd(). Will
> >> + * free the request if this was the last valid reference.
> >> + *
> >> + * @req:     request to release.
> >> + */
> >> +void media_request_put(struct media_request *req);
> >> +
> >> +/**
> >> + * media_request_get_entity_data() - get per-entity data for a request
> >> + * @req:     request to get entity data from
> >> + * @entity:  entity to get data of
> >> + * @fh:              cookie identifying the handle from which the entity is accessed
> >> + *
> >> + * Seach and return the entity data associated associated to the request. If no
> >> + * such data exists, it is allocated as per the entity operations.
> >> + *
> >> + * The fh arguments serves as a cookie to make sure the same entity is not
> >> + * accessed through different opened file handles. The same handle must be
> >> + * passed to all calls of this function for the same entity. Failure to do so
> >> + * will return an error.
> >> + *
> >> + * Returns the per-entity data, or an error code if a problem happened. -EINVAL
> >> + * means that data for the entity already existed, but has been allocated under
> >> + * a different cookie.
> >> + */
> >> +struct media_request_entity_data *
> >> +media_request_get_entity_data(struct media_request *req,
> >> +                           struct media_entity *entity, void *fh);
> >> +
> >> +
> >> +/**
> >> + * media_request_entity_complete() - to be invoked when an entity has completed
> >> + *                                its part of the currently active request
> >> + *
> >> + * @queue:   request queue
> >> + * @entity:  the entity that completed
> >> + */
> >> +void media_request_entity_complete(struct media_request_queue *queue,
> >> +                                struct media_entity *entity);
> >> +
> >> +/**
> >> + * struct media_request_queue_ops - request queue operations
> >> + *
> >> + * @release: release all memory associated to the queue
> >> + * @req_qlloc:       allocate a request for this queue
> >> + * @req_free:        free a previously allocated request
> >> + * @req_queue:       queue a request for execution
> >> + * @req_complete: callback invoked when the active request has completed
> >> + *
> >> + */
> >> +struct media_request_queue_ops {
> >> +     void (*release)(struct media_request_queue *queue);
> >> +     struct media_request *(*req_alloc)(struct media_request_queue *queue);
> >> +     void (*req_free)(struct media_request_queue *queue,
> >> +                      struct media_request *req);
> >> +     int (*req_queue)(struct media_request_queue *queue,
> >> +                      struct media_request *req);
> >> +     void (*req_complete)(struct media_request_queue *queue);
> >> +};
> >> +
> >> +/**
> >> + * struct media_request_queue - queue of requests
> >> + *
> >> + * @mdev:    media_device that manages this queue
> >> + * @ops:     implementation of the queue
> >> + * @mutex:   protects requests, active_request, req_id, and all members of
> >> + *           struct media_request
> >> + * @active_request: request being currently run by this queue
> >> + * @requests:        list of requests (not in any particular order) that this
> >> + *           queue owns.
> >> + * @req_id:  counter used to identify requests for debugging purposes
> >> + */
> >> +struct media_request_queue {
> >> +     struct media_device *mdev;
> >> +     const struct media_request_queue_ops *ops;
> >> +
> >> +     struct mutex mutex;
> >
> > Any particular reason for using a mutex? The request queue lock will need
> > to be acquired from interrupts, too, so this should be changed to a
> > spinlock.
> 
> Will it be acquired from interrupts? In any case it should be possible
> to change this to a spinlock.

Using mutexes will effectively make this impossible, and I don't think we
can safely say there's not going to be a need for that. So spinlocks,
please.

> 
> >
> > A media device, presumably, has a single request queue. I might just put
> > the fields to struct media_device.
> 
> I find it easier to understand when different aspects are separated
> into different data structures.

You may add a comment saying the fields below are related to requests. For
now this just makes the implementation a bit confusing.

> 
> >
> >> +     struct list_head requests;
> >> +     struct media_request *active_request;
> >> +     u32 req_id;
> >> +};
> >> +
> >> +static inline void media_request_queue_lock(struct media_request_queue *queue)
> >> +{
> >> +     mutex_lock(&queue->mutex);
> >> +}
> >> +
> >> +static inline void media_request_queue_unlock(struct media_request_queue *queue)
> >> +{
> >> +     mutex_unlock(&queue->mutex);
> >> +}
> >> +
> >> +/**
> >> + * media_request_queue_init() - initialize a request queue
> >> + *
> >> + * This function is to be called by request queue implementations to initialize
> >> + * the base structure.
> >> + *
> >> + * @queue:   queue to initialize
> >> + * @mdev:    media device owning this queue
> >> + * @ops:     ops to be used with this queue
> >> + */
> >> +void media_request_queue_init(struct media_request_queue *queue,
> >> +                           struct media_device *mdev,
> >> +                           const struct media_request_queue_ops *ops);
> >> +
> >> +/**
> >> + * media_request_queue_release() - release resources held by a request queue
> >> + *
> >> + * The caller is still responsible for freeing the memory held by the queue
> >> + * structure itself.
> >> + *
> >> + * @queue:   queue to be released
> >> + */
> >> +void media_request_queue_release(struct media_request_queue *queue);
> >> +
> >> +/**
> >> + * struct media_request_entity_data - per-entity request data
> >> + *
> >> + * Base structure used to store request state data. To be extended by actual
> >> + * implementation.
> >> + *
> >> + * @entity:  entity this data belongs to
> >> + * @fh:              subsystem-dependent. For V4L2, the v4l2_fh of the opened device
> >> + * @list:    entry in media_request::data
> >> + * @completed:       whether this entity has completed its part of the request
> >> + */
> >> +struct media_request_entity_data {
> >> +     struct media_entity *entity;
> >> +     void *fh;
> >> +     struct list_head list;
> >> +     bool completed;
> >> +};
> >> +
> >> +/**
> >> + * struct media_request_entity_ops - request-related entity operations
> >> + *
> >> + * All these operations are mandatory if the request API is to be used with
> >> + * an entity.
> >> + *
> >> + * @alloc_data:              allocate request data for this entity
> >> + * @release_data:    release previously allocated data
> >> + * @apply_data:              apply the data to the entity prior to running a request
> >> + *
> >> + */
> >> +struct media_request_entity_ops {
> >> +     struct media_request_entity_data *(*alloc_data)
> >> +             (struct media_request *req, struct media_entity *entity);
> >> +     void (*release_data)(struct media_request_entity_data *data);
> >> +     int (*apply_data)(struct media_request_entity_data *data);
> >
> > The media device already has an ops struct. How about putting these into
> > that?
> 
> Again, separation of aspects into dedicated structures, especially
> since we are not completely sure that this should end into
> media_entity.

In that case I'd just keep it internal to the driver until we see how at
least a few drivers end up doing this job.

> 
> >
> >> +};
> >> +
> >> +/**
> >> + * media_device_request_cmd() - perform the REQUEST_CMD ioctl
> >> + *
> >> + * @mdev:    media device the ioctl has been called on
> >> + * @cmd:     user-space request command structure
> >> + */
> >> +long media_device_request_cmd(struct media_device *mdev,
> >> +                           struct media_request_cmd *cmd);
> >> +
> >> +#else /* CONFIG_MEDIA_CONTROLLER */
> >> +
> >> +static inline struct media_request *media_request_get_from_fd(int fd)
> >> +{
> >> +     return ERR_PTR(-ENOTSUPP);
> >> +}
> >> +
> >> +static inline void media_request_put(struct media_request *req)
> >> +{
> >> +}
> >> +
> >> +static inline struct media_request *media_request_get_entity_data(
> >> +                                       struct media_request *req,
> >> +                                       struct media_entity *entity, void *fh)
> >> +{
> >> +     return ERR_PTR(-ENOTSUPP);
> >> +}
> >> +
> >> +static inline void media_request_entity_complete(struct media_request *req,
> >> +                                              struct media_entity *entity)
> >> +{
> >> +}
> >> +
> >> +#endif /* CONFIG_MEDIA_CONTROLLER */
> >> +
> >> +#endif
> >> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> >> index b9b9446095e9..247823ea0d1a 100644
> >> --- a/include/uapi/linux/media.h
> >> +++ b/include/uapi/linux/media.h
> >> @@ -406,6 +406,16 @@ struct media_v2_topology {
> >>       __u64 ptr_links;
> >>  } __attribute__ ((packed));
> >>
> >> +#define MEDIA_REQ_CMD_ALLOC          0
> >> +#define MEDIA_REQ_CMD_QUEUE          1
> >> +#define MEDIA_REQ_CMD_REINIT         2
> >> +
> >> +struct media_request_cmd {
> >> +     __u32 cmd;
> >> +     __s32 fd;
> >> +     __u32 flags;
> >> +} __attribute__ ((packed));
> >> +
> >>  /* ioctls */
> >>
> >>  #define MEDIA_IOC_DEVICE_INFO                _IOWR('|', 0x00, struct media_device_info)
> >> @@ -413,5 +423,6 @@ struct media_v2_topology {
> >>  #define MEDIA_IOC_ENUM_LINKS         _IOWR('|', 0x02, struct media_links_enum)
> >>  #define MEDIA_IOC_SETUP_LINK         _IOWR('|', 0x03, struct media_link_desc)
> >>  #define MEDIA_IOC_G_TOPOLOGY         _IOWR('|', 0x04, struct media_v2_topology)
> >> +#define MEDIA_IOC_REQUEST_CMD                _IOWR('|', 0x05, struct media_request_cmd)
> >>
> >>  #endif /* __LINUX_MEDIA_H */
> >

-- 
Regards,

Sakari Ailus
sakari.ailus@linux.intel.com

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Tomasz Figa]

On Fri, Feb 2, 2018 at 4:33 PM, Sakari Ailus
<sakari.ailus@linux.intel.com> wrote:
>> >> +/**
>> >> + * struct media_request_queue - queue of requests
>> >> + *
>> >> + * @mdev:    media_device that manages this queue
>> >> + * @ops:     implementation of the queue
>> >> + * @mutex:   protects requests, active_request, req_id, and all members of
>> >> + *           struct media_request
>> >> + * @active_request: request being currently run by this queue
>> >> + * @requests:        list of requests (not in any particular order) that this
>> >> + *           queue owns.
>> >> + * @req_id:  counter used to identify requests for debugging purposes
>> >> + */
>> >> +struct media_request_queue {
>> >> +     struct media_device *mdev;
>> >> +     const struct media_request_queue_ops *ops;
>> >> +
>> >> +     struct mutex mutex;
>> >
>> > Any particular reason for using a mutex? The request queue lock will need
>> > to be acquired from interrupts, too, so this should be changed to a
>> > spinlock.
>>
>> Will it be acquired from interrupts? In any case it should be possible
>> to change this to a spinlock.
>
> Using mutexes will effectively make this impossible, and I don't think we
> can safely say there's not going to be a need for that. So spinlocks,
> please.
>

IMHO whether a mutex or spinlock is the right thing depends on what
kind of critical section it is used for. If it only protects data (and
according to the comment, this one seems to do so), spinlock might
actually have better properties, e.g. not introducing the need to
reschedule, if another CPU is accessing the data at the moment. It
might also depend on how heavy the data accesses are, though. We
shouldn't need to spin for too long time.

Best regards,
Tomasz

Re: [RFC PATCH 1/9] media: add request API core and UAPI

[Hans Verkuil]

On 02/02/2018 08:33 AM, Sakari Ailus wrote:

<snip>

>>>> +struct media_request_entity_data *
>>>> +media_request_get_entity_data(struct media_request *req,
>>>> +                           struct media_entity *entity, void *fh)
>>>
>>> This makes the assumption that request data is bound to entities. How does
>>> this work with links?
>>>
>>> I wonder if it should rather be bound to graph objects, or certain graph
>>> objects. Having a standard way to bind request specific information e.g. to
>>> entities is definitely worth having, though.
>>>
>>> V4L2 framework specific information would be needed across the media graph
>>> and it'd be good to store it in a non-driver specific way. What I think
>>> you'd need is an interface that allows storing information based on two
>>> keys --- the request and e.g. a pointer provided by the caller. The V4L2
>>> framework would have one key, e.g. a pointer to an empty struct defined
>>> somewhere in the V4L2 framework could be used for the purpose.
>>>
>>> Going forward, the entire media graph state will be subject to changing
>>> through requests. This includes link state, media bus and pixel formats,
>>> cropping and scaling configurations, everything. Let's not try to go there
>>> yet in this patchset, but what I'm asking is to keep the goal in mind when
>>> implementating the request API.
>>
>> Yeah, I think a similar idea is brought up in the cover letter of the
>> next revision (although for different reasons). Entities are probably
>> not a one-fit for all use-cases.
>>
>> For the case of links though, I believe that the "entity" that would
>> control them would be the media controller itself, since it is the one
>> that takes the MEDIA_IOC_SETUP_LINK ioctl. But even for this case, we
>> cannot use an entity to look up the media_device, so something more
>> generic like an opaque key would probably be needed.
> 
> Perhaps in the near future we still need a little less than that. Changing
> something that has a state in V4L2 will be troublesome and will require
> managing state of what is now stream centric.
> 
> I still think that the framework would need to do the job of managing the
> video buffers related to a request as well as controls without necessarily
> trying to generalise that right now. But how to store these in a meaningful
> way? Putting them to the request itself would be one option: you'll need to
> dig the request up anyway when things are associated to it, and the driver
> needs it when it is queued.
> 
> I wonder what Hans and Laurent think.

I think this is something for the future. I want to avoid delaying the Request
API for endless internal design discussions. The public API should be solid,
but the internal framework will undoubtedly need to change in the future.

That's OK. The reality is that there is a lot of demand for the Request API for
stateless codecs, and that is what we should concentrate on.

Should the framework manage the buffers? I don't even know what is meant with
that exactly, let alone that I can give an answer.

Let's stay focused: 1) solid uAPI, 2) stateless codec support. The internal
framework shouldn't of course make it harder than it needs to to later extend
the support to camera pipelines, but neither should we spend much time on it
or we will never get this in.

Regards,

	Hans