[PATCH v2 0/2] Memory-to-memory media controller topology

[PATCH v2 0/2] Memory-to-memory media controller topology

[Ezequiel Garcia]

As discussed on IRC, memory-to-memory need to be modeled
properly in order to be supported by the media controller
framework, and thus to support the Request API.

First commit introduces a register/unregister API,
that creates/destroys all the entities and pads needed,
and links them.

The second commit uses this API to support the vim2m driver.

Topology (media-ctl -p output)
==============================

Device topology
- entity 1: source (1 pad, 1 link)
            type Node subtype V4L flags 0
	pad0: Source
		-> "proc":1 [ENABLED,IMMUTABLE]

- entity 3: proc (2 pads, 2 links)
            type Node subtype Unknown flags 0
	pad0: Source
		-> "sink":0 [ENABLED,IMMUTABLE]
	pad1: Sink
		<- "source":0 [ENABLED,IMMUTABLE]

- entity 6: sink (1 pad, 1 link)
            type Node subtype V4L flags 0
	pad0: Sink

Compliance output
=================

Compliance test for device /dev/media0:

Media Driver Info:
	Driver name      : vim2m
	Model            : vim2m
	Serial           : 
	Bus info         : 
	Media version    : 4.17.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 4.17.0

Required ioctls:
	test MEDIA_IOC_DEVICE_INFO: OK

Allow for multiple opens:
	test second /dev/media0 open: OK
	test MEDIA_IOC_DEVICE_INFO: OK
	test for unlimited opens: OK

Media Controller ioctls:
	test MEDIA_IOC_G_TOPOLOGY: OK
	Entities: 3 Interfaces: 1 Pads: 4 Links: 4
		fail: v4l2-test-media.cpp(333): found_source
	test MEDIA_IOC_ENUM_ENTITIES/LINKS: FAIL
	test MEDIA_IOC_SETUP_LINK: OK

--------------------------------------------------------------------------------
Compliance test for device /dev/video2:

Driver Info:
	Driver name      : vim2m
	Card type        : vim2m
	Bus info         : platform:vim2m
	Driver version   : 4.17.0
	Capabilities     : 0x84208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
		Device Capabilities
	Device Caps      : 0x04208000
		Video Memory-to-Memory
		Streaming
		Extended Pix Format
Media Driver Info:
	Driver name      : vim2m
	Model            : vim2m
	Serial           : 
	Bus info         : 
	Media version    : 4.17.0
	Hardware revision: 0x00000000 (0)
	Driver version   : 4.17.0
Interface Info:
	ID               : 0x0300000c
	Type             : V4L Video
	Major            : 81
	Minor            : 7
Entity Info:
	ID               : 0x00000001 (1)
	Name             : source
	Function         : V4L2 I/O
	Pad 0x01000002   : Source
	  Link 0x0200000a: from remote pad 0x1000004 of entity 'proc': Data, Enabled, Immutable

Required ioctls:
	test MC information (see 'Media Driver Info' above): OK
	test VIDIOC_QUERYCAP: OK

Allow for multiple opens:
	test second /dev/video2 open: OK
	test VIDIOC_QUERYCAP: OK
	test VIDIOC_G/S_PRIORITY: OK
	test for unlimited opens: OK

Debug ioctls:
	test VIDIOC_DBG_G/S_REGISTER: OK
	test VIDIOC_LOG_STATUS: OK (Not Supported)

Input ioctls:
	test VIDIOC_G/S_TUNER/ENUM_FREQ_BANDS: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_S_HW_FREQ_SEEK: OK (Not Supported)
	test VIDIOC_ENUMAUDIO: OK (Not Supported)
	test VIDIOC_G/S/ENUMINPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDIO: OK (Not Supported)
	Inputs: 0 Audio Inputs: 0 Tuners: 0

Output ioctls:
	test VIDIOC_G/S_MODULATOR: OK (Not Supported)
	test VIDIOC_G/S_FREQUENCY: OK (Not Supported)
	test VIDIOC_ENUMAUDOUT: OK (Not Supported)
	test VIDIOC_G/S/ENUMOUTPUT: OK (Not Supported)
	test VIDIOC_G/S_AUDOUT: OK (Not Supported)
	Outputs: 0 Audio Outputs: 0 Modulators: 0

Input/Output configuration ioctls:
	test VIDIOC_ENUM/G/S/QUERY_STD: OK (Not Supported)
	test VIDIOC_ENUM/G/S/QUERY_DV_TIMINGS: OK (Not Supported)
	test VIDIOC_DV_TIMINGS_CAP: OK (Not Supported)
	test VIDIOC_G/S_EDID: OK (Not Supported)

Control ioctls:
	test VIDIOC_QUERY_EXT_CTRL/QUERYMENU: OK
	test VIDIOC_QUERYCTRL: OK
	test VIDIOC_G/S_CTRL: OK
	test VIDIOC_G/S/TRY_EXT_CTRLS: OK
	test VIDIOC_(UN)SUBSCRIBE_EVENT/DQEVENT: OK
	test VIDIOC_G/S_JPEGCOMP: OK (Not Supported)
	Standard Controls: 3 Private Controls: 2

Format ioctls:
	test VIDIOC_ENUM_FMT/FRAMESIZES/FRAMEINTERVALS: OK
	test VIDIOC_G/S_PARM: OK (Not Supported)
	test VIDIOC_G_FBUF: OK (Not Supported)
	test VIDIOC_G_FMT: OK
	test VIDIOC_TRY_FMT: OK
	test VIDIOC_S_FMT: OK
	test VIDIOC_G_SLICED_VBI_CAP: OK (Not Supported)
	test Cropping: OK (Not Supported)
	test Composing: OK (Not Supported)
	test Scaling: OK

Codec ioctls:
	test VIDIOC_(TRY_)ENCODER_CMD: OK (Not Supported)
	test VIDIOC_G_ENC_INDEX: OK (Not Supported)
	test VIDIOC_(TRY_)DECODER_CMD: OK (Not Supported)

Buffer ioctls:
	test VIDIOC_REQBUFS/CREATE_BUFS/QUERYBUF: OK
	test VIDIOC_EXPBUF: OK

Total: 51, Succeeded: 50, Failed: 1, Warnings: 0

I am not sure if the compliance failure makes sense,
as a 'proc' entity connecting the two pads seems legal.
Commenting the failing test in v4l2-test-media.cpp
makes the tool pass with no errors.

v2:
  * Fix compile error when MEDIA_CONTROLLER was not enabled.
  * Fix the 'proc' entity link, which was wrongly connecting
    source to source and sink to sink :-P

Ezequiel Garcia (1):
  media: add helpers for memory-to-memory media controller

Hans Verkuil (1):
  vim2m: add media device

 drivers/media/platform/vim2m.c         |  41 +++++-
 drivers/media/v4l2-core/v4l2-dev.c     |  13 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c | 174 +++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           |  19 +++
 include/uapi/linux/media.h             |   3 +
 5 files changed, 241 insertions(+), 9 deletions(-)

-- 
2.17.1

[PATCH v2 1/2] media: add helpers for memory-to-memory media controller

[Ezequiel Garcia]

A memory-to-memory pipeline device consists in three
entities: two DMA engine and one video processing entities.
The DMA engine entities are linked to a V4L interface.

This commit add a new v4l2_m2m_{un}register_media_controller
API to register this topology.

For instance, a typical mem2mem device topology would
look like this:

Device topology
- entity 1: source (1 pad, 1 link)
            type Node subtype V4L flags 0
	pad0: Source
		-> "proc":1 [ENABLED,IMMUTABLE]

- entity 3: proc (2 pads, 2 links)
            type Node subtype Unknown flags 0
	pad0: Source
		-> "sink":0 [ENABLED,IMMUTABLE]
	pad1: Sink
		<- "source":0 [ENABLED,IMMUTABLE]

- entity 6: sink (1 pad, 1 link)
            type Node subtype V4L flags 0
	pad0: Sink
		<- "proc":0 [ENABLED,IMMUTABLE]

Suggested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Suggested-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/v4l2-core/v4l2-dev.c     |  13 +-
 drivers/media/v4l2-core/v4l2-mem2mem.c | 174 +++++++++++++++++++++++++
 include/media/v4l2-mem2mem.h           |  19 +++
 include/uapi/linux/media.h             |   3 +
 4 files changed, 204 insertions(+), 5 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 4ffd7d60a901..c1996d73ca74 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -202,7 +202,7 @@ static void v4l2_device_release(struct device *cd)
 	mutex_unlock(&videodev_lock);
 
 #if defined(CONFIG_MEDIA_CONTROLLER)
-	if (v4l2_dev->mdev) {
+	if (v4l2_dev->mdev && vdev->vfl_dir != VFL_DIR_M2M) {
 		/* Remove interfaces and interface links */
 		media_devnode_remove(vdev->intf_devnode);
 		if (vdev->entity.function != MEDIA_ENT_F_UNKNOWN)
@@ -733,19 +733,22 @@ static void determine_valid_ioctls(struct video_device *vdev)
 			BASE_VIDIOC_PRIVATE);
 }
 
-static int video_register_media_controller(struct video_device *vdev, int type)
+static int video_register_media_controller(struct video_device *vdev)
 {
 #if defined(CONFIG_MEDIA_CONTROLLER)
 	u32 intf_type;
 	int ret;
 
-	if (!vdev->v4l2_dev->mdev)
+	/* Memory-to-memory devices are more complex and use
+	 * their own function to register its mc entities.
+	 */
+	if (!vdev->v4l2_dev->mdev || vdev->vfl_dir == VFL_DIR_M2M)
 		return 0;
 
 	vdev->entity.obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
 	vdev->entity.function = MEDIA_ENT_F_UNKNOWN;
 
-	switch (type) {
+	switch (vdev->vfl_type) {
 	case VFL_TYPE_GRABBER:
 		intf_type = MEDIA_INTF_T_V4L_VIDEO;
 		vdev->entity.function = MEDIA_ENT_F_IO_V4L;
@@ -993,7 +996,7 @@ int __video_register_device(struct video_device *vdev,
 	v4l2_device_get(vdev->v4l2_dev);
 
 	/* Part 5: Register the entity. */
-	ret = video_register_media_controller(vdev, type);
+	ret = video_register_media_controller(vdev);
 
 	/* Part 6: Activate this minor. The char device can now be used. */
 	set_bit(V4L2_FL_REGISTERED, &vdev->flags);
diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
index c4f963d96a79..e0e7262b7e75 100644
--- a/drivers/media/v4l2-core/v4l2-mem2mem.c
+++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
@@ -17,9 +17,11 @@
 #include <linux/sched.h>
 #include <linux/slab.h>
 
+#include <media/media-device.h>
 #include <media/videobuf2-v4l2.h>
 #include <media/v4l2-mem2mem.h>
 #include <media/v4l2-dev.h>
+#include <media/v4l2-device.h>
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 
@@ -50,6 +52,19 @@ module_param(debug, bool, 0644);
  * offsets but for different queues */
 #define DST_QUEUE_OFF_BASE	(1 << 30)
 
+enum v4l2_m2m_entity_type {
+	MEM2MEM_ENT_TYPE_SOURCE,
+	MEM2MEM_ENT_TYPE_SINK,
+	MEM2MEM_ENT_TYPE_PROC,
+	MEM2MEM_ENT_TYPE_MAX
+};
+
+static const char * const m2m_entity_name[] = {
+	"source",
+	"sink",
+	"proc"
+};
+
 
 /**
  * struct v4l2_m2m_dev - per-device context
@@ -60,6 +75,15 @@ module_param(debug, bool, 0644);
  */
 struct v4l2_m2m_dev {
 	struct v4l2_m2m_ctx	*curr_ctx;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_entity	*source;
+	struct media_pad	source_pad;
+	struct media_entity	sink;
+	struct media_pad	sink_pad;
+	struct media_entity	proc;
+	struct media_pad	proc_pads[2];
+	struct media_intf_devnode *intf_devnode;
+#endif
 
 	struct list_head	job_queue;
 	spinlock_t		job_spinlock;
@@ -595,6 +619,156 @@ int v4l2_m2m_mmap(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
 }
 EXPORT_SYMBOL(v4l2_m2m_mmap);
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+void v4l2_m2m_unregister_media_controller(struct v4l2_m2m_dev *m2m_dev)
+{
+	media_remove_intf_links(&m2m_dev->intf_devnode->intf);
+	media_devnode_remove(m2m_dev->intf_devnode);
+
+	media_entity_remove_links(m2m_dev->source);
+	media_entity_remove_links(&m2m_dev->sink);
+	media_entity_remove_links(&m2m_dev->proc);
+	media_device_unregister_entity(m2m_dev->source);
+	media_device_unregister_entity(&m2m_dev->sink);
+	media_device_unregister_entity(&m2m_dev->proc);
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_unregister_media_controller);
+
+static int v4l2_m2m_register_entity(struct media_device *mdev,
+	struct v4l2_m2m_dev *m2m_dev, enum v4l2_m2m_entity_type type,
+	int function)
+{
+	struct media_entity *entity;
+	struct media_pad *pads;
+	int num_pads;
+	int ret;
+
+	switch (type) {
+	case MEM2MEM_ENT_TYPE_SOURCE:
+		entity = m2m_dev->source;
+		pads = &m2m_dev->source_pad;
+		entity->name = m2m_entity_name[type];
+		pads[0].flags = MEDIA_PAD_FL_SOURCE;
+		num_pads = 1;
+		break;
+	case MEM2MEM_ENT_TYPE_SINK:
+		entity = &m2m_dev->sink;
+		pads = &m2m_dev->sink_pad;
+		pads[0].flags = MEDIA_PAD_FL_SINK;
+		num_pads = 1;
+		break;
+	case MEM2MEM_ENT_TYPE_PROC:
+		entity = &m2m_dev->proc;
+		pads = m2m_dev->proc_pads;
+		pads[0].flags = MEDIA_PAD_FL_SOURCE;
+		pads[1].flags = MEDIA_PAD_FL_SINK;
+		num_pads = 2;
+		break;
+	default:
+		return -EINVAL;
+	}
+
+	entity->obj_type = MEDIA_ENTITY_TYPE_BASE;
+	entity->name = m2m_entity_name[type];
+	entity->function = function;
+
+	ret = media_entity_pads_init(entity, num_pads, pads);
+	if (ret)
+		return ret;
+	ret = media_device_register_entity(mdev, entity);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+int v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
+		struct video_device *vdev, int function)
+{
+	struct media_device *mdev = vdev->v4l2_dev->mdev;
+	struct media_link *link;
+	int ret;
+
+	if (!mdev)
+		return 0;
+
+	/* A memory-to-memory device consists in two
+	 * DMA engine and one video processing entities.
+	 * The DMA engine entities are linked to a V4L interface
+	 */
+
+	/* Create the three entities with their pads */
+	m2m_dev->source = &vdev->entity;
+	ret = v4l2_m2m_register_entity(mdev, m2m_dev,
+			MEM2MEM_ENT_TYPE_SOURCE, MEDIA_ENT_F_IO_V4L);
+	if (ret)
+		return ret;
+	ret = v4l2_m2m_register_entity(mdev, m2m_dev,
+			MEM2MEM_ENT_TYPE_PROC, function);
+	if (ret)
+		goto err_rel_entity0;
+	ret = v4l2_m2m_register_entity(mdev, m2m_dev,
+			MEM2MEM_ENT_TYPE_SINK, MEDIA_ENT_F_IO_V4L);
+	if (ret)
+		goto err_rel_entity1;
+
+	/* Connect the three entities */
+	ret = media_create_pad_link(m2m_dev->source, 0, &m2m_dev->proc, 1,
+			MEDIA_LNK_FL_IMMUTABLE | MEDIA_LNK_FL_ENABLED);
+	if (ret)
+		goto err_rel_entity2;
+
+	ret = media_create_pad_link(&m2m_dev->proc, 0, &m2m_dev->sink, 0,
+			MEDIA_LNK_FL_IMMUTABLE | MEDIA_LNK_FL_ENABLED);
+	if (ret)
+		goto err_rm_links0;
+
+	/* Create video interface */
+	m2m_dev->intf_devnode = media_devnode_create(mdev,
+			MEDIA_INTF_T_V4L_VIDEO, 0,
+			VIDEO_MAJOR, vdev->minor);
+	if (!m2m_dev->intf_devnode) {
+		ret = -ENOMEM;
+		goto err_rm_links1;
+	}
+
+	/* Connect the two DMA engines to the interface */
+	link = media_create_intf_link(m2m_dev->source,
+			&m2m_dev->intf_devnode->intf, MEDIA_LNK_FL_ENABLED);
+	if (!link) {
+		ret = -ENOMEM;
+		goto err_rm_devnode;
+	}
+
+	link = media_create_intf_link(&m2m_dev->sink,
+			&m2m_dev->intf_devnode->intf, MEDIA_LNK_FL_ENABLED);
+	if (!link) {
+		ret = -ENOMEM;
+		goto err_rm_intf_link;
+	}
+	return 0;
+
+err_rm_intf_link:
+	media_remove_intf_links(&m2m_dev->intf_devnode->intf);
+err_rm_devnode:
+	media_devnode_remove(m2m_dev->intf_devnode);
+err_rm_links1:
+	media_entity_remove_links(&m2m_dev->sink);
+err_rm_links0:
+	media_entity_remove_links(&m2m_dev->proc);
+	media_entity_remove_links(m2m_dev->source);
+err_rel_entity2:
+	media_device_unregister_entity(&m2m_dev->proc);
+err_rel_entity1:
+	media_device_unregister_entity(&m2m_dev->sink);
+err_rel_entity0:
+	media_device_unregister_entity(m2m_dev->source);
+	return ret;
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_m2m_register_media_controller);
+#endif
+
 struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops)
 {
 	struct v4l2_m2m_dev *m2m_dev;
diff --git a/include/media/v4l2-mem2mem.h b/include/media/v4l2-mem2mem.h
index 3d07ba3a8262..24fcc99653de 100644
--- a/include/media/v4l2-mem2mem.h
+++ b/include/media/v4l2-mem2mem.h
@@ -53,6 +53,7 @@ struct v4l2_m2m_ops {
 	void (*unlock)(void *priv);
 };
 
+struct video_device;
 struct v4l2_m2m_dev;
 
 /**
@@ -328,6 +329,24 @@ int v4l2_m2m_mmap(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
  */
 struct v4l2_m2m_dev *v4l2_m2m_init(const struct v4l2_m2m_ops *m2m_ops);
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+void v4l2_m2m_unregister_media_controller(struct v4l2_m2m_dev *m2m_dev);
+int v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
+			struct video_device *vdev, int function);
+#else
+static inline void
+v4l2_m2m_unregister_media_controller(struct v4l2_m2m_dev *m2m_dev)
+{
+}
+
+static inline int
+v4l2_m2m_register_media_controller(struct v4l2_m2m_dev *m2m_dev,
+		struct video_device *vdev, int function)
+{
+	return 0;
+}
+#endif
+
 /**
  * v4l2_m2m_release() - cleans up and frees a m2m_dev structure
  *
diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
index c7e9a5cba24e..5f58c7ac04c0 100644
--- a/include/uapi/linux/media.h
+++ b/include/uapi/linux/media.h
@@ -132,6 +132,9 @@ struct media_device_info {
 #define MEDIA_ENT_F_PROC_VIDEO_LUT		(MEDIA_ENT_F_BASE + 0x4004)
 #define MEDIA_ENT_F_PROC_VIDEO_SCALER		(MEDIA_ENT_F_BASE + 0x4005)
 #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
+#define MEDIA_ENT_F_PROC_VIDEO_DECODER		(MEDIA_ENT_F_BASE + 0x4007)
+#define MEDIA_ENT_F_PROC_VIDEO_ENCODER		(MEDIA_ENT_F_BASE + 0x4008)
+#define MEDIA_ENT_F_PROC_VIDEO_DEINTERLACER	(MEDIA_ENT_F_BASE + 0x4009)
 
 /*
  * Switch and bridge entity functions
-- 
2.17.1

[PATCH v2 2/2] vim2m: add media device

[Ezequiel Garcia]

From: Hans Verkuil <hans.verkuil@cisco.com>

Request API requires a media node. Add one to the vim2m driver so we can
use requests with it.

Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
---
 drivers/media/platform/vim2m.c | 41 ++++++++++++++++++++++++++++++----
 1 file changed, 37 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
index 065483e62db4..da13a8927f3f 100644
--- a/drivers/media/platform/vim2m.c
+++ b/drivers/media/platform/vim2m.c
@@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
 struct vim2m_dev {
 	struct v4l2_device	v4l2_dev;
 	struct video_device	vfd;
+#ifdef CONFIG_MEDIA_CONTROLLER
+	struct media_device	mdev;
+#endif
 
 	atomic_t		num_inst;
 	struct mutex		dev_mutex;
@@ -1016,7 +1019,7 @@ static int vim2m_probe(struct platform_device *pdev)
 	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
 	if (ret) {
 		v4l2_err(&dev->v4l2_dev, "Failed to register video device\n");
-		goto unreg_dev;
+		goto unreg_v4l2;
 	}
 
 	video_set_drvdata(vfd, dev);
@@ -1031,15 +1034,39 @@ static int vim2m_probe(struct platform_device *pdev)
 	if (IS_ERR(dev->m2m_dev)) {
 		v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem device\n");
 		ret = PTR_ERR(dev->m2m_dev);
-		goto err_m2m;
+		goto unreg_dev;
+	}
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	dev->mdev.dev = &pdev->dev;
+	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
+	media_device_init(&dev->mdev);
+	dev->v4l2_dev.mdev = &dev->mdev;
+
+	ret = v4l2_m2m_register_media_controller(dev->m2m_dev,
+			vfd, MEDIA_ENT_F_PROC_VIDEO_SCALER);
+	if (ret) {
+		v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem media controller\n");
+		goto unreg_m2m;
 	}
 
+	ret = media_device_register(&dev->mdev);
+	if (ret) {
+		v4l2_err(&dev->v4l2_dev, "Failed to register mem2mem media device\n");
+		goto unreg_m2m_mc;
+	}
+#endif
 	return 0;
 
-err_m2m:
+#ifdef CONFIG_MEDIA_CONTROLLER
+unreg_m2m_mc:
+	v4l2_m2m_unregister_media_controller(dev->m2m_dev);
+unreg_m2m:
 	v4l2_m2m_release(dev->m2m_dev);
-	video_unregister_device(&dev->vfd);
+#endif
 unreg_dev:
+	video_unregister_device(&dev->vfd);
+unreg_v4l2:
 	v4l2_device_unregister(&dev->v4l2_dev);
 
 	return ret;
@@ -1050,6 +1077,12 @@ static int vim2m_remove(struct platform_device *pdev)
 	struct vim2m_dev *dev = platform_get_drvdata(pdev);
 
 	v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
+
+#ifdef CONFIG_MEDIA_CONTROLLER
+	media_device_unregister(&dev->mdev);
+	v4l2_m2m_unregister_media_controller(dev->m2m_dev);
+	media_device_cleanup(&dev->mdev);
+#endif
 	v4l2_m2m_release(dev->m2m_dev);
 	del_timer_sync(&dev->timer);
 	video_unregister_device(&dev->vfd);
-- 
2.17.1

Re: [PATCH v2 1/2] media: add helpers for memory-to-memory media controller

[Hans Verkuil]

On 06/21/2018 10:38 PM, Ezequiel Garcia wrote:
> A memory-to-memory pipeline device consists in three
> entities: two DMA engine and one video processing entities.
> The DMA engine entities are linked to a V4L interface.
> 
> This commit add a new v4l2_m2m_{un}register_media_controller
> API to register this topology.
> 
> For instance, a typical mem2mem device topology would
> look like this:
> 
> Device topology
> - entity 1: source (1 pad, 1 link)
>             type Node subtype V4L flags 0
> 	pad0: Source
> 		-> "proc":1 [ENABLED,IMMUTABLE]
> 
> - entity 3: proc (2 pads, 2 links)
>             type Node subtype Unknown flags 0
> 	pad0: Source
> 		-> "sink":0 [ENABLED,IMMUTABLE]
> 	pad1: Sink
> 		<- "source":0 [ENABLED,IMMUTABLE]
> 
> - entity 6: sink (1 pad, 1 link)
>             type Node subtype V4L flags 0
> 	pad0: Sink
> 		<- "proc":0 [ENABLED,IMMUTABLE]
> 
> Suggested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Suggested-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-dev.c     |  13 +-
>  drivers/media/v4l2-core/v4l2-mem2mem.c | 174 +++++++++++++++++++++++++
>  include/media/v4l2-mem2mem.h           |  19 +++
>  include/uapi/linux/media.h             |   3 +
>  4 files changed, 204 insertions(+), 5 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index 4ffd7d60a901..c1996d73ca74 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -202,7 +202,7 @@ static void v4l2_device_release(struct device *cd)
>  	mutex_unlock(&videodev_lock);
>  
>  #if defined(CONFIG_MEDIA_CONTROLLER)
> -	if (v4l2_dev->mdev) {
> +	if (v4l2_dev->mdev && vdev->vfl_dir != VFL_DIR_M2M) {
>  		/* Remove interfaces and interface links */
>  		media_devnode_remove(vdev->intf_devnode);
>  		if (vdev->entity.function != MEDIA_ENT_F_UNKNOWN)
> @@ -733,19 +733,22 @@ static void determine_valid_ioctls(struct video_device *vdev)
>  			BASE_VIDIOC_PRIVATE);
>  }
>  
> -static int video_register_media_controller(struct video_device *vdev, int type)
> +static int video_register_media_controller(struct video_device *vdev)
>  {
>  #if defined(CONFIG_MEDIA_CONTROLLER)
>  	u32 intf_type;
>  	int ret;
>  
> -	if (!vdev->v4l2_dev->mdev)
> +	/* Memory-to-memory devices are more complex and use
> +	 * their own function to register its mc entities.
> +	 */
> +	if (!vdev->v4l2_dev->mdev || vdev->vfl_dir == VFL_DIR_M2M)
>  		return 0;
>  
>  	vdev->entity.obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
>  	vdev->entity.function = MEDIA_ENT_F_UNKNOWN;
>  
> -	switch (type) {
> +	switch (vdev->vfl_type) {
>  	case VFL_TYPE_GRABBER:
>  		intf_type = MEDIA_INTF_T_V4L_VIDEO;
>  		vdev->entity.function = MEDIA_ENT_F_IO_V4L;
> @@ -993,7 +996,7 @@ int __video_register_device(struct video_device *vdev,
>  	v4l2_device_get(vdev->v4l2_dev);
>  
>  	/* Part 5: Register the entity. */
> -	ret = video_register_media_controller(vdev, type);
> +	ret = video_register_media_controller(vdev);
>  
>  	/* Part 6: Activate this minor. The char device can now be used. */
>  	set_bit(V4L2_FL_REGISTERED, &vdev->flags);
> diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c b/drivers/media/v4l2-core/v4l2-mem2mem.c
> index c4f963d96a79..e0e7262b7e75 100644
> --- a/drivers/media/v4l2-core/v4l2-mem2mem.c
> +++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
> @@ -17,9 +17,11 @@
>  #include <linux/sched.h>
>  #include <linux/slab.h>
>  
> +#include <media/media-device.h>
>  #include <media/videobuf2-v4l2.h>
>  #include <media/v4l2-mem2mem.h>
>  #include <media/v4l2-dev.h>
> +#include <media/v4l2-device.h>
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-event.h>
>  
> @@ -50,6 +52,19 @@ module_param(debug, bool, 0644);
>   * offsets but for different queues */
>  #define DST_QUEUE_OFF_BASE	(1 << 30)
>  
> +enum v4l2_m2m_entity_type {
> +	MEM2MEM_ENT_TYPE_SOURCE,
> +	MEM2MEM_ENT_TYPE_SINK,
> +	MEM2MEM_ENT_TYPE_PROC,
> +	MEM2MEM_ENT_TYPE_MAX
> +};
> +
> +static const char * const m2m_entity_name[] = {
> +	"source",
> +	"sink",
> +	"proc"
> +};
> +
>  
>  /**
>   * struct v4l2_m2m_dev - per-device context
> @@ -60,6 +75,15 @@ module_param(debug, bool, 0644);
>   */
>  struct v4l2_m2m_dev {
>  	struct v4l2_m2m_ctx	*curr_ctx;
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	struct media_entity	*source;
> +	struct media_pad	source_pad;
> +	struct media_entity	sink;
> +	struct media_pad	sink_pad;
> +	struct media_entity	proc;
> +	struct media_pad	proc_pads[2];
> +	struct media_intf_devnode *intf_devnode;
> +#endif
>  
>  	struct list_head	job_queue;
>  	spinlock_t		job_spinlock;
> @@ -595,6 +619,156 @@ int v4l2_m2m_mmap(struct file *file, struct v4l2_m2m_ctx *m2m_ctx,
>  }
>  EXPORT_SYMBOL(v4l2_m2m_mmap);
>  
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +void v4l2_m2m_unregister_media_controller(struct v4l2_m2m_dev *m2m_dev)
> +{
> +	media_remove_intf_links(&m2m_dev->intf_devnode->intf);
> +	media_devnode_remove(m2m_dev->intf_devnode);
> +
> +	media_entity_remove_links(m2m_dev->source);
> +	media_entity_remove_links(&m2m_dev->sink);
> +	media_entity_remove_links(&m2m_dev->proc);
> +	media_device_unregister_entity(m2m_dev->source);
> +	media_device_unregister_entity(&m2m_dev->sink);
> +	media_device_unregister_entity(&m2m_dev->proc);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_m2m_unregister_media_controller);
> +
> +static int v4l2_m2m_register_entity(struct media_device *mdev,
> +	struct v4l2_m2m_dev *m2m_dev, enum v4l2_m2m_entity_type type,
> +	int function)
> +{
> +	struct media_entity *entity;
> +	struct media_pad *pads;
> +	int num_pads;
> +	int ret;
> +
> +	switch (type) {
> +	case MEM2MEM_ENT_TYPE_SOURCE:
> +		entity = m2m_dev->source;
> +		pads = &m2m_dev->source_pad;
> +		entity->name = m2m_entity_name[type];
> +		pads[0].flags = MEDIA_PAD_FL_SOURCE;
> +		num_pads = 1;
> +		break;
> +	case MEM2MEM_ENT_TYPE_SINK:
> +		entity = &m2m_dev->sink;
> +		pads = &m2m_dev->sink_pad;
> +		pads[0].flags = MEDIA_PAD_FL_SINK;
> +		num_pads = 1;
> +		break;
> +	case MEM2MEM_ENT_TYPE_PROC:
> +		entity = &m2m_dev->proc;
> +		pads = m2m_dev->proc_pads;
> +		pads[0].flags = MEDIA_PAD_FL_SOURCE;
> +		pads[1].flags = MEDIA_PAD_FL_SINK;

Can you swap this? The v4l2-compliance test gave a "fail: v4l2-test-media.cpp(333): found_source"
message which is because (I think) it expects sink pads before source pads.

I'm not actually sure that this is a requirement, at least I can't find this in the spec,
but for some reason I have this memory that this actually is the right order. It might just
be a custom rather than a requirement.

Anyway, it doesn't matter for this code, so it is easiest to swap it and run v4l2-compliance -m
again.

I am interested in the v4l2-compliance output of the topology.

Regards,

	Hans

Re: [PATCH v2 1/2] media: add helpers for memory-to-memory media controller

[Ezequiel Garcia]

On Fri, 2018-06-22 at 08:58 +0200, Hans Verkuil wrote:
> On 06/21/2018 10:38 PM, Ezequiel Garcia wrote:
> > A memory-to-memory pipeline device consists in three
> > entities: two DMA engine and one video processing entities.
> > The DMA engine entities are linked to a V4L interface.
> > 
> > This commit add a new v4l2_m2m_{un}register_media_controller
> > API to register this topology.
> > 
> > For instance, a typical mem2mem device topology would
> > look like this:
> > 
> > Device topology
> > - entity 1: source (1 pad, 1 link)
> >             type Node subtype V4L flags 0
> > 	pad0: Source
> > 		-> "proc":1 [ENABLED,IMMUTABLE]
> > 
> > - entity 3: proc (2 pads, 2 links)
> >             type Node subtype Unknown flags 0
> > 	pad0: Source
> > 		-> "sink":0 [ENABLED,IMMUTABLE]
> > 	pad1: Sink
> > 		<- "source":0 [ENABLED,IMMUTABLE]
> > 
> > - entity 6: sink (1 pad, 1 link)
> >             type Node subtype V4L flags 0
> > 	pad0: Sink
> > 		<- "proc":0 [ENABLED,IMMUTABLE]
> > 
> > Suggested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Suggested-by: Hans Verkuil <hans.verkuil@cisco.com>
> > Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-dev.c     |  13 +-
> >  drivers/media/v4l2-core/v4l2-mem2mem.c | 174
> > +++++++++++++++++++++++++
> >  include/media/v4l2-mem2mem.h           |  19 +++
> >  include/uapi/linux/media.h             |   3 +
> >  4 files changed, 204 insertions(+), 5 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-dev.c
> > b/drivers/media/v4l2-core/v4l2-dev.c
> > index 4ffd7d60a901..c1996d73ca74 100644
> > --- a/drivers/media/v4l2-core/v4l2-dev.c
> > +++ b/drivers/media/v4l2-core/v4l2-dev.c
> > @@ -202,7 +202,7 @@ static void v4l2_device_release(struct device
> > *cd)
> >  	mutex_unlock(&videodev_lock);
> >  
> >  #if defined(CONFIG_MEDIA_CONTROLLER)
> > -	if (v4l2_dev->mdev) {
> > +	if (v4l2_dev->mdev && vdev->vfl_dir != VFL_DIR_M2M) {
> >  		/* Remove interfaces and interface links */
> >  		media_devnode_remove(vdev->intf_devnode);
> >  		if (vdev->entity.function != MEDIA_ENT_F_UNKNOWN)
> > @@ -733,19 +733,22 @@ static void determine_valid_ioctls(struct
> > video_device *vdev)
> >  			BASE_VIDIOC_PRIVATE);
> >  }
> >  
> > -static int video_register_media_controller(struct video_device
> > *vdev, int type)
> > +static int video_register_media_controller(struct video_device
> > *vdev)
> >  {
> >  #if defined(CONFIG_MEDIA_CONTROLLER)
> >  	u32 intf_type;
> >  	int ret;
> >  
> > -	if (!vdev->v4l2_dev->mdev)
> > +	/* Memory-to-memory devices are more complex and use
> > +	 * their own function to register its mc entities.
> > +	 */
> > +	if (!vdev->v4l2_dev->mdev || vdev->vfl_dir == VFL_DIR_M2M)
> >  		return 0;
> >  
> >  	vdev->entity.obj_type = MEDIA_ENTITY_TYPE_VIDEO_DEVICE;
> >  	vdev->entity.function = MEDIA_ENT_F_UNKNOWN;
> >  
> > -	switch (type) {
> > +	switch (vdev->vfl_type) {
> >  	case VFL_TYPE_GRABBER:
> >  		intf_type = MEDIA_INTF_T_V4L_VIDEO;
> >  		vdev->entity.function = MEDIA_ENT_F_IO_V4L;
> > @@ -993,7 +996,7 @@ int __video_register_device(struct video_device
> > *vdev,
> >  	v4l2_device_get(vdev->v4l2_dev);
> >  
> >  	/* Part 5: Register the entity. */
> > -	ret = video_register_media_controller(vdev, type);
> > +	ret = video_register_media_controller(vdev);
> >  
> >  	/* Part 6: Activate this minor. The char device can now be
> > used. */
> >  	set_bit(V4L2_FL_REGISTERED, &vdev->flags);
> > diff --git a/drivers/media/v4l2-core/v4l2-mem2mem.c
> > b/drivers/media/v4l2-core/v4l2-mem2mem.c
> > index c4f963d96a79..e0e7262b7e75 100644
> > --- a/drivers/media/v4l2-core/v4l2-mem2mem.c
> > +++ b/drivers/media/v4l2-core/v4l2-mem2mem.c
> > @@ -17,9 +17,11 @@
> >  #include <linux/sched.h>
> >  #include <linux/slab.h>
> >  
> > +#include <media/media-device.h>
> >  #include <media/videobuf2-v4l2.h>
> >  #include <media/v4l2-mem2mem.h>
> >  #include <media/v4l2-dev.h>
> > +#include <media/v4l2-device.h>
> >  #include <media/v4l2-fh.h>
> >  #include <media/v4l2-event.h>
> >  
> > @@ -50,6 +52,19 @@ module_param(debug, bool, 0644);
> >   * offsets but for different queues */
> >  #define DST_QUEUE_OFF_BASE	(1 << 30)
> >  
> > +enum v4l2_m2m_entity_type {
> > +	MEM2MEM_ENT_TYPE_SOURCE,
> > +	MEM2MEM_ENT_TYPE_SINK,
> > +	MEM2MEM_ENT_TYPE_PROC,
> > +	MEM2MEM_ENT_TYPE_MAX
> > +};
> > +
> > +static const char * const m2m_entity_name[] = {
> > +	"source",
> > +	"sink",
> > +	"proc"
> > +};
> > +
> >  
> >  /**
> >   * struct v4l2_m2m_dev - per-device context
> > @@ -60,6 +75,15 @@ module_param(debug, bool, 0644);
> >   */
> >  struct v4l2_m2m_dev {
> >  	struct v4l2_m2m_ctx	*curr_ctx;
> > +#ifdef CONFIG_MEDIA_CONTROLLER
> > +	struct media_entity	*source;
> > +	struct media_pad	source_pad;
> > +	struct media_entity	sink;
> > +	struct media_pad	sink_pad;
> > +	struct media_entity	proc;
> > +	struct media_pad	proc_pads[2];
> > +	struct media_intf_devnode *intf_devnode;
> > +#endif
> >  
> >  	struct list_head	job_queue;
> >  	spinlock_t		job_spinlock;
> > @@ -595,6 +619,156 @@ int v4l2_m2m_mmap(struct file *file, struct
> > v4l2_m2m_ctx *m2m_ctx,
> >  }
> >  EXPORT_SYMBOL(v4l2_m2m_mmap);
> >  
> > +#if defined(CONFIG_MEDIA_CONTROLLER)
> > +void v4l2_m2m_unregister_media_controller(struct v4l2_m2m_dev
> > *m2m_dev)
> > +{
> > +	media_remove_intf_links(&m2m_dev->intf_devnode->intf);
> > +	media_devnode_remove(m2m_dev->intf_devnode);
> > +
> > +	media_entity_remove_links(m2m_dev->source);
> > +	media_entity_remove_links(&m2m_dev->sink);
> > +	media_entity_remove_links(&m2m_dev->proc);
> > +	media_device_unregister_entity(m2m_dev->source);
> > +	media_device_unregister_entity(&m2m_dev->sink);
> > +	media_device_unregister_entity(&m2m_dev->proc);
> > +}
> > +EXPORT_SYMBOL_GPL(v4l2_m2m_unregister_media_controller);
> > +
> > +static int v4l2_m2m_register_entity(struct media_device *mdev,
> > +	struct v4l2_m2m_dev *m2m_dev, enum v4l2_m2m_entity_type
> > type,
> > +	int function)
> > +{
> > +	struct media_entity *entity;
> > +	struct media_pad *pads;
> > +	int num_pads;
> > +	int ret;
> > +
> > +	switch (type) {
> > +	case MEM2MEM_ENT_TYPE_SOURCE:
> > +		entity = m2m_dev->source;
> > +		pads = &m2m_dev->source_pad;
> > +		entity->name = m2m_entity_name[type];
> > +		pads[0].flags = MEDIA_PAD_FL_SOURCE;
> > +		num_pads = 1;
> > +		break;
> > +	case MEM2MEM_ENT_TYPE_SINK:
> > +		entity = &m2m_dev->sink;
> > +		pads = &m2m_dev->sink_pad;
> > +		pads[0].flags = MEDIA_PAD_FL_SINK;
> > +		num_pads = 1;
> > +		break;
> > +	case MEM2MEM_ENT_TYPE_PROC:
> > +		entity = &m2m_dev->proc;
> > +		pads = m2m_dev->proc_pads;
> > +		pads[0].flags = MEDIA_PAD_FL_SOURCE;
> > +		pads[1].flags = MEDIA_PAD_FL_SINK;
> 
> Can you swap this? The v4l2-compliance test gave a "fail: v4l2-test-
> media.cpp(333): found_source"
> message which is because (I think) it expects sink pads before source
> pads.
> 

Yes, that did it.

> I'm not actually sure that this is a requirement, at least I can't
> find this in the spec,
> but for some reason I have this memory that this actually is the
> right order. It might just
> be a custom rather than a requirement.
> 
> Anyway, it doesn't matter for this code, so it is easiest to swap it
> and run v4l2-compliance -m
> again.
> 
> I am interested in the v4l2-compliance output of the topology.
> 

Here it goes:

Media Controller ioctls:
		Entity: 0x00000001 (Name: 'source', Function:
0x00010001)
		Entity: 0x00000003 (Name: 'proc', Function: 0x00004005)
		Entity: 0x00000006 (Name: 'sink', Function: 0x00010001)
		Interface: 0x0300000c (Type: 0x00000200)
		Pad: 0x01000002
		Pad: 0x01000004
		Pad: 0x01000005
		Pad: 0x01000007
		Link: 0x02000008
		Link: 0x0200000a
		Link: 0x0200000d
		Link: 0x0200000e
	test MEDIA_IOC_G_TOPOLOGY: OK
	Entities: 3 Interfaces: 1 Pads: 4 Links: 4
		Entity: 0x00000001 (Name: 'source', Type: 0x00010001)
		Entity: 0x00000003 (Name: 'proc', Type: 0x0001ffff)
		Entity: 0x00000006 (Name: 'sink', Type: 0x00010001)
		Entity Links: 0x00000001 (Name: 'source')
		Entity Links: 0x00000003 (Name: 'proc')
		Entity Links: 0x00000006 (Name: 'sink')
	test MEDIA_IOC_ENUM_ENTITIES/LINKS: OK
	test MEDIA_IOC_SETUP_LINK: OK

I will submit a v3 shortly with this change, and a couple other
nitpicks.

Regards,
Eze

Re: [PATCH v2 1/2] media: add helpers for memory-to-memory media controller

[Hans Verkuil]

On 06/21/2018 10:38 PM, Ezequiel Garcia wrote:
> A memory-to-memory pipeline device consists in three
> entities: two DMA engine and one video processing entities.
> The DMA engine entities are linked to a V4L interface.
> 
> This commit add a new v4l2_m2m_{un}register_media_controller
> API to register this topology.
> 
> For instance, a typical mem2mem device topology would
> look like this:
> 
> Device topology
> - entity 1: source (1 pad, 1 link)
>             type Node subtype V4L flags 0
> 	pad0: Source
> 		-> "proc":1 [ENABLED,IMMUTABLE]
> 
> - entity 3: proc (2 pads, 2 links)
>             type Node subtype Unknown flags 0
> 	pad0: Source
> 		-> "sink":0 [ENABLED,IMMUTABLE]
> 	pad1: Sink
> 		<- "source":0 [ENABLED,IMMUTABLE]
> 
> - entity 6: sink (1 pad, 1 link)
>             type Node subtype V4L flags 0
> 	pad0: Sink
> 		<- "proc":0 [ENABLED,IMMUTABLE]
> 
> Suggested-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Suggested-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/v4l2-core/v4l2-dev.c     |  13 +-
>  drivers/media/v4l2-core/v4l2-mem2mem.c | 174 +++++++++++++++++++++++++
>  include/media/v4l2-mem2mem.h           |  19 +++
>  include/uapi/linux/media.h             |   3 +
>  4 files changed, 204 insertions(+), 5 deletions(-)
> 

<snip>

> diff --git a/include/uapi/linux/media.h b/include/uapi/linux/media.h
> index c7e9a5cba24e..5f58c7ac04c0 100644
> --- a/include/uapi/linux/media.h
> +++ b/include/uapi/linux/media.h
> @@ -132,6 +132,9 @@ struct media_device_info {
>  #define MEDIA_ENT_F_PROC_VIDEO_LUT		(MEDIA_ENT_F_BASE + 0x4004)
>  #define MEDIA_ENT_F_PROC_VIDEO_SCALER		(MEDIA_ENT_F_BASE + 0x4005)
>  #define MEDIA_ENT_F_PROC_VIDEO_STATISTICS	(MEDIA_ENT_F_BASE + 0x4006)
> +#define MEDIA_ENT_F_PROC_VIDEO_DECODER		(MEDIA_ENT_F_BASE + 0x4007)
> +#define MEDIA_ENT_F_PROC_VIDEO_ENCODER		(MEDIA_ENT_F_BASE + 0x4008)
> +#define MEDIA_ENT_F_PROC_VIDEO_DEINTERLACER	(MEDIA_ENT_F_BASE + 0x4009)
>  
>  /*
>   * Switch and bridge entity functions
> 

This last chunk is unrelated to this patch series.

May I suggest that you post this as a separate patch, together with an update to
the documentation (media-types.rst), that sits on top of:

https://git.linuxtv.org/hverkuil/media_tree.git/log/?h=media-missing2

That branch reformats the tables in media-types.rst, making it easier to add new
entries. In addition it already has one patch adding a new function ("media.h: add
MEDIA_ENT_F_DTV_ENCODER"). I'm happy to add your patch to this series and to
include it in the eventual pull request. The patches in this branch were also
posted to the mailinglist:

https://www.mail-archive.com/linux-media@vger.kernel.org/msg132942.html

Otherwise this patch looks great (after swapping the source/sink pad order).

Regards,

	Hans

Re: [PATCH v2 2/2] vim2m: add media device

[Hans Verkuil]

On 06/21/2018 10:38 PM, Ezequiel Garcia wrote:
> From: Hans Verkuil <hans.verkuil@cisco.com>
> 
> Request API requires a media node. Add one to the vim2m driver so we can
> use requests with it.
> 
> Signed-off-by: Hans Verkuil <hans.verkuil@cisco.com>
> Signed-off-by: Ezequiel Garcia <ezequiel@collabora.com>
> ---
>  drivers/media/platform/vim2m.c | 41 ++++++++++++++++++++++++++++++----
>  1 file changed, 37 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/platform/vim2m.c b/drivers/media/platform/vim2m.c
> index 065483e62db4..da13a8927f3f 100644
> --- a/drivers/media/platform/vim2m.c
> +++ b/drivers/media/platform/vim2m.c
> @@ -140,6 +140,9 @@ static struct vim2m_fmt *find_format(struct v4l2_format *f)
>  struct vim2m_dev {
>  	struct v4l2_device	v4l2_dev;
>  	struct video_device	vfd;
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	struct media_device	mdev;
> +#endif
>  
>  	atomic_t		num_inst;
>  	struct mutex		dev_mutex;
> @@ -1016,7 +1019,7 @@ static int vim2m_probe(struct platform_device *pdev)
>  	ret = video_register_device(vfd, VFL_TYPE_GRABBER, 0);
>  	if (ret) {
>  		v4l2_err(&dev->v4l2_dev, "Failed to register video device\n");
> -		goto unreg_dev;
> +		goto unreg_v4l2;
>  	}
>  
>  	video_set_drvdata(vfd, dev);
> @@ -1031,15 +1034,39 @@ static int vim2m_probe(struct platform_device *pdev)
>  	if (IS_ERR(dev->m2m_dev)) {
>  		v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem device\n");
>  		ret = PTR_ERR(dev->m2m_dev);
> -		goto err_m2m;
> +		goto unreg_dev;
> +	}
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	dev->mdev.dev = &pdev->dev;
> +	strlcpy(dev->mdev.model, "vim2m", sizeof(dev->mdev.model));
> +	media_device_init(&dev->mdev);
> +	dev->v4l2_dev.mdev = &dev->mdev;
> +
> +	ret = v4l2_m2m_register_media_controller(dev->m2m_dev,
> +			vfd, MEDIA_ENT_F_PROC_VIDEO_SCALER);
> +	if (ret) {
> +		v4l2_err(&dev->v4l2_dev, "Failed to init mem2mem media controller\n");
> +		goto unreg_m2m;
>  	}
>  
> +	ret = media_device_register(&dev->mdev);
> +	if (ret) {
> +		v4l2_err(&dev->v4l2_dev, "Failed to register mem2mem media device\n");
> +		goto unreg_m2m_mc;
> +	}
> +#endif
>  	return 0;
>  
> -err_m2m:
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +unreg_m2m_mc:
> +	v4l2_m2m_unregister_media_controller(dev->m2m_dev);
> +unreg_m2m:
>  	v4l2_m2m_release(dev->m2m_dev);
> -	video_unregister_device(&dev->vfd);
> +#endif
>  unreg_dev:
> +	video_unregister_device(&dev->vfd);
> +unreg_v4l2:
>  	v4l2_device_unregister(&dev->v4l2_dev);
>  
>  	return ret;
> @@ -1050,6 +1077,12 @@ static int vim2m_remove(struct platform_device *pdev)
>  	struct vim2m_dev *dev = platform_get_drvdata(pdev);
>  
>  	v4l2_info(&dev->v4l2_dev, "Removing " MEM2MEM_NAME);
> +
> +#ifdef CONFIG_MEDIA_CONTROLLER
> +	media_device_unregister(&dev->mdev);
> +	v4l2_m2m_unregister_media_controller(dev->m2m_dev);
> +	media_device_cleanup(&dev->mdev);
> +#endif
>  	v4l2_m2m_release(dev->m2m_dev);
>  	del_timer_sync(&dev->timer);
>  	video_unregister_device(&dev->vfd);
> 

Looks good!

Regards,

	Hans