[PATCH v14 00/34]

[PATCH v14 00/34]

[Tomi Valkeinen]

Hi,

This is v14 of the streams series. The v13 can be found from:

https://lore.kernel.org/all/20220810121122.3149086-1-tomi.valkeinen@ideasonboard.com/

My work branch with additional drivers can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git streams/work-v14

And there's also the v4l-utils series v2 to add support to v4l2-ctl and
media-ctl:

https://lore.kernel.org/all/20220714132116.132498-1-tomi.valkeinen@ideasonboard.com/

Changes in v14:

- Rebased on top of v6.0-rc3
- Include minmax.h instead of kernel.h for min()
- Simplify new_pipe handling in media_pipeline_alloc_start()
- Add missing kdocs
- Fix doc references to media entities where media pad should now be
  used.
- Change media_entity_pipeline() to return the first pipeline found in
  the entity's pads (instead of returning the pipe from the first pad).
- Mark media_entity_pipeline() as deprecated. It is only used in two
  drivers after this series.

 Tomi

Jacopo Mondi (3):
  media: mc: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (9):
  media: mc: entity: Merge media_entity_enum_init and
    __media_entity_enum_init
  media: mc: entity: Move media_entity_get_fwnode_pad() out of graph
    walk section
  media: mc: entity: Add media_entity_pipeline() to access the media
    pipeline
  media: mc: entity: Rewrite media_pipeline_start()
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations
  media: subdev: add v4l2_subdev_routing_validate() helper
  media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper
  media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations
  media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

Sakari Ailus (1):
  media: Add stream to frame descriptor

Tomi Valkeinen (21):
  media: Documentation: mc: add definitions for stream and pipeline
  media: media-entity.h: add include for min()
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  media: mc: entity: Rename streaming_count -> start_count
  media: v4l2-dev: Add videodev wrappers for media pipelines
  media: drivers: use video device pipeline start/stop
  media: drivers: use video_device_pipeline()
  media: mc: entity: add alloc variant of pipeline_start
  media: drivers: use video_device_pipeline_alloc_start()
  media: mc: entity: Add has_pad_interdep entity operation
  media: mc: convert pipeline funcs to take media_pad
  media: add V4L2_SUBDEV_FL_STREAMS
  media: add V4L2_SUBDEV_CAP_STREAMS
  media: subdev: add v4l2_subdev_has_pad_interdep()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add streams to v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

 .clang-format                                 |    1 +
 Documentation/driver-api/media/mc-core.rst    |   19 +-
 .../driver-api/media/v4l2-subdev.rst          |    8 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  175 +++
 .../userspace-api/media/v4l/user-func.rst     |    1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |    5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |    5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |    5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |    5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |    5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |    5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |  150 +++
 .../media/v4l/vidioc-subdev-g-selection.rst   |    5 +-
 drivers/media/mc/mc-device.c                  |   13 +-
 drivers/media/mc/mc-entity.c                  |  648 ++++++++--
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |    6 +-
 .../media/platform/qcom/camss/camss-video.c   |    6 +-
 .../platform/renesas/rcar-vin/rcar-core.c     |    5 +-
 .../platform/renesas/rcar-vin/rcar-dma.c      |   18 +-
 .../media/platform/renesas/vsp1/vsp1_video.c  |    6 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   14 +-
 .../samsung/exynos4-is/fimc-capture.c         |    9 +-
 .../samsung/exynos4-is/fimc-isp-video.c       |    9 +-
 .../platform/samsung/exynos4-is/fimc-lite.c   |    9 +-
 .../samsung/s3c-camif/camif-capture.c         |    6 +-
 drivers/media/platform/st/stm32/stm32-dcmi.c  |    6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |    6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |    6 +-
 drivers/media/platform/ti/cal/cal-video.c     |    6 +-
 drivers/media/platform/ti/cal/cal.h           |    1 -
 drivers/media/platform/ti/omap3isp/isp.c      |    4 +-
 drivers/media/platform/ti/omap3isp/ispvideo.c |    9 +-
 drivers/media/platform/ti/omap3isp/ispvideo.h |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |    9 +-
 .../media/test-drivers/vimc/vimc-capture.c    |    7 +-
 drivers/media/usb/au0828/au0828-core.c        |    8 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   72 ++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 1038 ++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |    8 +-
 drivers/staging/media/imx/imx7-media-csi.c    |    6 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |    6 +-
 drivers/staging/media/omap4iss/iss.c          |    4 +-
 drivers/staging/media/omap4iss/iss_video.c    |    9 +-
 drivers/staging/media/omap4iss/iss_video.h    |   11 +-
 drivers/staging/media/tegra-video/tegra210.c  |    6 +-
 include/media/media-device.h                  |   15 -
 include/media/media-entity.h                  |  169 ++-
 include/media/v4l2-dev.h                      |  102 ++
 include/media/v4l2-subdev.h                   |  382 +++++-
 include/uapi/linux/v4l2-subdev.h              |   83 +-
 52 files changed, 2825 insertions(+), 333 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.34.1

[PATCH v14 01/34] media: Documentation: mc: add definitions for stream and pipeline

[Tomi Valkeinen]

The doc talks about streams and pipelines, but doesn't really define
them. This is an attempt to define them according to my understanding.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 84aa7cdb5341..4bb062d5c2e7 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -214,6 +214,18 @@ Link properties can be modified at runtime by calling
 Pipelines and media streams
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+A media stream is a stream of pixels or metadata originating from one or more
+source devices (such as a sensors) and flowing through media entity pads
+towards the final sinks. The stream can be modified on the route by the
+devices (e.g. scaling or pixel format conversions), or it can be split into
+multiple branches, or multiple branches can be merged.
+
+A media pipeline is a set of media streams which are interdependent. This
+interdependency can be caused by the hardware (e.g. configuration of a second
+stream cannot be changed if the first stream has been enabled) or by the driver
+due to the software design. Most commonly a media pipeline consists of a single
+stream which does not branch.
+
 When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
-- 
2.34.1

[PATCH v14 02/34] media: media-entity.h: add include for min()

[Tomi Valkeinen]

./include/media/media-entity.h:595:34: error: implicit declaration of function ‘min’

Include minmax.h to get the definition for min().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 include/media/media-entity.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index f16ffe70f7a6..4a67b1dfdc69 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -17,6 +17,7 @@
 #include <linux/fwnode.h>
 #include <linux/list.h>
 #include <linux/media.h>
+#include <linux/minmax.h>
 #include <linux/types.h>
 
 /* Enums used internally at the media controller to represent graphs */
-- 
2.34.1

[PATCH v14 03/34] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 9689f38a0af1..3797b99bb408 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -358,7 +358,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.34.1

[PATCH v14 04/34] media: mc: entity: Rename streaming_count -> start_count

[Tomi Valkeinen]

'streaming_count' is a bit misleading name, as the count is increased
with media_pipeline_start(). Let's rename it to 'start_count' instead.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                            | 8 ++++----
 drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c | 4 ++--
 include/media/media-entity.h                            | 4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index afd1bd7ff7b6..67d009b617ce 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -415,8 +415,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	struct media_link *link;
 	int ret;
 
-	if (pipe->streaming_count) {
-		pipe->streaming_count++;
+	if (pipe->start_count) {
+		pipe->start_count++;
 		return 0;
 	}
 
@@ -499,7 +499,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		}
 	}
 
-	pipe->streaming_count++;
+	pipe->start_count++;
 
 	return 0;
 
@@ -552,7 +552,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	if (--pipe->streaming_count)
+	if (--pipe->start_count)
 		return;
 
 	media_graph_walk_start(graph, entity);
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index d5904c96ff3f..bf0d4789374c 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -926,7 +926,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes shouldn't
 	 * be disabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count < 2)
+	if (rkisp1->pipe.start_count < 2)
 		v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, false);
 
 	v4l2_subdev_call(&rkisp1->resizer_devs[cap->id].sd, video, s_stream,
@@ -956,7 +956,7 @@ static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes are already
 	 * enabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count > 1)
+	if (rkisp1->pipe.start_count > 1)
 		return 0;
 
 	ret = v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, true);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 4a67b1dfdc69..198ea1416ddd 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -100,11 +100,11 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
- * @streaming_count:	Streaming start count - streaming stop count
+ * @start_count:	Media pipeline start - stop count
  * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
-	int streaming_count;
+	int start_count;
 	struct media_graph graph;
 };
 
-- 
2.34.1

[PATCH v14 05/34] media: mc: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index b8176a3b76d3..25020d58eb06 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -610,7 +610,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -639,9 +639,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 67d009b617ce..682f424a15ca 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -193,7 +193,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -204,12 +205,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					  &iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 198ea1416ddd..a5a50350e954 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -316,6 +316,18 @@ struct media_entity {
 	} info;
 };
 
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
+
 /**
  * struct media_interface - A media interface graph object.
  *
-- 
2.34.1

[PATCH v14 06/34] media: mc: entity: Merge media_entity_enum_init and __media_entity_enum_init

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_enum_init() function is a wrapper around
__media_entity_enum_init() that turns a media_device pointer argument
into the maximum entity ID in the corresponding media graph.
__media_entity_enum_init() is never used outside of
media_entity_enum_init(), so the two functions can be merged together.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 10 ++++++----
 include/media/media-device.h | 15 ---------------
 include/media/media-entity.h | 10 +++++-----
 3 files changed, 11 insertions(+), 24 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 682f424a15ca..48d8cc98ae04 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -59,10 +59,12 @@ static inline const char *link_type_name(struct media_link *link)
 	}
 }
 
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max)
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev)
 {
-	idx_max = ALIGN(idx_max, BITS_PER_LONG);
+	int idx_max;
+
+	idx_max = ALIGN(mdev->entity_internal_idx_max + 1, BITS_PER_LONG);
 	ent_enum->bmap = bitmap_zalloc(idx_max, GFP_KERNEL);
 	if (!ent_enum->bmap)
 		return -ENOMEM;
@@ -71,7 +73,7 @@ __must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(__media_entity_enum_init);
+EXPORT_SYMBOL_GPL(media_entity_enum_init);
 
 void media_entity_enum_cleanup(struct media_entity_enum *ent_enum)
 {
diff --git a/include/media/media-device.h b/include/media/media-device.h
index a10b30507524..86716ee7cc6c 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -191,21 +191,6 @@ struct usb_device;
 #define MEDIA_DEV_NOTIFY_PRE_LINK_CH	0
 #define MEDIA_DEV_NOTIFY_POST_LINK_CH	1
 
-/**
- * media_entity_enum_init - Initialise an entity enumeration
- *
- * @ent_enum: Entity enumeration to be initialised
- * @mdev: The related media device
- *
- * Return: zero on success or a negative error code.
- */
-static inline __must_check int media_entity_enum_init(
-	struct media_entity_enum *ent_enum, struct media_device *mdev)
-{
-	return __media_entity_enum_init(ent_enum,
-					mdev->entity_internal_idx_max + 1);
-}
-
 /**
  * media_device_init() - Initializes a media device element
  *
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a5a50350e954..1030e45e8ee6 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -439,15 +439,15 @@ static inline bool is_media_entity_v4l2_subdev(struct media_entity *entity)
 }
 
 /**
- * __media_entity_enum_init - Initialise an entity enumeration
+ * media_entity_enum_init - Initialise an entity enumeration
  *
  * @ent_enum: Entity enumeration to be initialised
- * @idx_max: Maximum number of entities in the enumeration
+ * @mdev: The related media device
  *
- * Return: Returns zero on success or a negative error code.
+ * Return: zero on success or a negative error code.
  */
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max);
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev);
 
 /**
  * media_entity_enum_cleanup - Release resources of an entity enumeration
-- 
2.34.1

[PATCH v14 07/34] media: mc: entity: Move media_entity_get_fwnode_pad() out of graph walk section

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_get_fwnode_pad() function is unrelated to the graph
traversal code that it is currently bundled with. Move it with the
media_entity_remote_pad() function.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 70 ++++++++++++++++++------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 48d8cc98ae04..c5c66befed0f 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -370,41 +370,6 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
-int media_entity_get_fwnode_pad(struct media_entity *entity,
-				struct fwnode_handle *fwnode,
-				unsigned long direction_flags)
-{
-	struct fwnode_endpoint endpoint;
-	unsigned int i;
-	int ret;
-
-	if (!entity->ops || !entity->ops->get_fwnode_pad) {
-		for (i = 0; i < entity->num_pads; i++) {
-			if (entity->pads[i].flags & direction_flags)
-				return i;
-		}
-
-		return -ENXIO;
-	}
-
-	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
-	if (ret)
-		return ret;
-
-	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
-	if (ret < 0)
-		return ret;
-
-	if (ret >= entity->num_pads)
-		return -ENXIO;
-
-	if (!(entity->pads[ret].flags & direction_flags))
-		return -ENXIO;
-
-	return ret;
-}
-EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
-
 /* -----------------------------------------------------------------------------
  * Pipeline management
  */
@@ -994,6 +959,41 @@ struct media_pad *media_pad_remote_pad_unique(const struct media_pad *pad)
 }
 EXPORT_SYMBOL_GPL(media_pad_remote_pad_unique);
 
+int media_entity_get_fwnode_pad(struct media_entity *entity,
+				struct fwnode_handle *fwnode,
+				unsigned long direction_flags)
+{
+	struct fwnode_endpoint endpoint;
+	unsigned int i;
+	int ret;
+
+	if (!entity->ops || !entity->ops->get_fwnode_pad) {
+		for (i = 0; i < entity->num_pads; i++) {
+			if (entity->pads[i].flags & direction_flags)
+				return i;
+		}
+
+		return -ENXIO;
+	}
+
+	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
+	if (ret)
+		return ret;
+
+	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
+	if (ret < 0)
+		return ret;
+
+	if (ret >= entity->num_pads)
+		return -ENXIO;
+
+	if (!(entity->pads[ret].flags & direction_flags))
+		return -ENXIO;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
-- 
2.34.1

[PATCH v14 08/34] media: mc: entity: Add media_entity_pipeline() to access the media pipeline

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Replace direct access to the pipe field in drivers with a new helper
function. This will allow easier refactoring of media pipeline handling
in the MC core behind the scenes without affecting drivers.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                   |  6 ++++++
 .../platform/renesas/rcar-vin/rcar-core.c      |  5 ++---
 .../media/platform/renesas/rcar-vin/rcar-dma.c |  2 +-
 drivers/media/platform/ti/omap3isp/isp.c       |  4 +---
 drivers/media/platform/ti/omap3isp/ispvideo.c  |  3 +--
 drivers/media/platform/ti/omap3isp/ispvideo.h  | 11 +++++++++--
 drivers/media/platform/xilinx/xilinx-dma.c     |  3 +--
 drivers/media/platform/xilinx/xilinx-dma.h     |  7 ++++++-
 drivers/staging/media/imx/imx-media-utils.c    |  2 +-
 drivers/staging/media/omap4iss/iss.c           |  4 +---
 drivers/staging/media/omap4iss/iss_video.c     |  3 +--
 drivers/staging/media/omap4iss/iss_video.h     | 11 +++++++++--
 include/media/media-entity.h                   | 18 ++++++++++++++++++
 13 files changed, 57 insertions(+), 22 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index c5c66befed0f..7fb97c6dc897 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -994,6 +994,12 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
+{
+	return entity->pipe;
+}
+EXPORT_SYMBOL_GPL(media_entity_pipeline);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-core.c b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
index 968a74234e92..2f7daa853ed8 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
@@ -786,9 +786,8 @@ static int rvin_csi2_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
 	media_device_for_each_entity(entity, &group->mdev)
 		if (media_entity_is_streaming(entity))
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 8d37fbdc266a..e72bc6fa049f 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1281,7 +1281,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/isp.c b/drivers/media/platform/ti/omap3isp/isp.c
index d251736eb420..84f8ed0edaba 100644
--- a/drivers/media/platform/ti/omap3isp/isp.c
+++ b/drivers/media/platform/ti/omap3isp/isp.c
@@ -937,10 +937,8 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_isp_pipeline(me);
-	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_pad_remote_pad_first(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
index d7059180e80e..5746cac4e046 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.c
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
@@ -1093,8 +1093,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
-	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
+	pipe = to_isp_pipeline(&video->video.entity) ? : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.h b/drivers/media/platform/ti/omap3isp/ispvideo.h
index a0908670c0cf..1d23df576e6b 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.h
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.h
@@ -99,8 +99,15 @@ struct isp_pipeline {
 	unsigned int external_width;
 };
 
-#define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+static inline struct isp_pipeline *to_isp_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct isp_pipeline, pipe);
+}
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 2d1ef7a25c33..3a4d62be0f27 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,8 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
-	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
+	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
 	if (ret < 0)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..3ea10f6b0bb9 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,12 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	struct media_pipeline *pipe = media_entity_pipeline(e);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 294c808b2ebe..e9a3c6d2c66f 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -871,7 +871,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (media_entity_pipeline(entity))
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 28aacda0f5a7..fa2a36d829d3 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,10 +548,8 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_iss_pipeline(me);
-	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_pad_remote_pad_first(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 9512cd3314f2..abadaf13c0e1 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -870,8 +870,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = to_iss_pipeline(&video->video.entity) ? : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..ca2d5edb6261 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -90,8 +90,15 @@ struct iss_pipeline {
 	int external_bpp;
 };
 
-#define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+static inline struct iss_pipeline *to_iss_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct iss_pipeline, pipe);
+}
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 1030e45e8ee6..aaf276f765cf 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -948,6 +948,24 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
 	return entity->pipe;
 }
 
+/**
+ * media_entity_pipeline - Get the media pipeline an entity is part of
+ * @entity: The entity
+ *
+ * This function returns the media pipeline that an entity has been associated
+ * with when constructing the pipeline with media_pipeline_start(). The pointer
+ * remains valid until media_pipeline_stop() is called.
+ *
+ * In general, entities can be part of multiple pipelines, when carrying
+ * multiple streams (either on different pads, or on the same pad using
+ * multiplexed streams). This function is to be used only for entities that
+ * do not support multiple pipelines.
+ *
+ * Return: The media_pipeline the entity is part of, or NULL if the entity is
+ * not part of any pipeline.
+ */
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
+
 /**
  * media_entity_get_fwnode_pad - Get pad number from fwnode
  *
-- 
2.34.1

[PATCH v14 09/34] media: v4l2-dev: Add videodev wrappers for media pipelines

[Tomi Valkeinen]

With the upcoming stream related improvements to the pipelines, the
pipelines are moved from media entities to media pads. As the drivers
currently use the pipelines with the entity based model, moving the
pipelines to pads will cause changes to the drivers.

However, most of the uses of media pipelines are related to a video
device (a DMA engine) with a single pad, and thus there's never a need
to support multiple pads in these use cases. We can avoid pushing the
complexities of the pad based model to the drivers by adding video
device wrappers for the pipeline related functions.

This patch adds a number of wrappers to media_pipeline functions, all of
which take a video_device as a parameter (instead of a media_entity),
and verify that there's just one pad.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-dev.c | 61 +++++++++++++++++++++
 include/media/v4l2-dev.h           | 88 ++++++++++++++++++++++++++++++
 2 files changed, 149 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index d00237ee4cae..7f933ff89fd4 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1095,6 +1095,67 @@ void video_unregister_device(struct video_device *vdev)
 }
 EXPORT_SYMBOL(video_unregister_device);
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+__must_check int video_device_pipeline_start(struct video_device *vdev,
+					     struct media_pipeline *pipe)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_start(entity, pipe);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_start);
+
+__must_check int __video_device_pipeline_start(struct video_device *vdev,
+					       struct media_pipeline *pipe)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return __media_pipeline_start(entity, pipe);
+}
+EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
+
+void video_device_pipeline_stop(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return;
+
+	return media_pipeline_stop(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
+
+void __video_device_pipeline_stop(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return;
+
+	return __media_pipeline_stop(entity);
+}
+EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
+
+struct media_pipeline *video_device_pipeline(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return NULL;
+
+	return media_entity_pipeline(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 /*
  *	Initialise video for linux
  */
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 5cf1edefb822..494685872254 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -539,4 +539,92 @@ static inline int video_is_registered(struct video_device *vdev)
 	return test_bit(V4L2_FL_REGISTERED, &vdev->flags);
 }
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+/**
+ * video_device_pipeline_start - Mark a pipeline as streaming
+ * @vdev: Starting video device
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * Mark all entities connected to a given video device through enabled links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every entity in the pipeline and stored in the media_entity pipe
+ * field.
+ *
+ * Calls to this function can be nested, in which case the same number of
+ * video_device_pipeline_stop() calls will be required to stop streaming. The
+ * pipeline pointer must be identical for all nested calls to
+ * video_device_pipeline_start().
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper around media_pipeline_start().
+ */
+__must_check int video_device_pipeline_start(struct video_device *vdev,
+					     struct media_pipeline *pipe);
+
+/**
+ * __video_device_pipeline_start - Mark a pipeline as streaming
+ * @vdev: Starting video device
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * ..note:: This is the non-locking version of video_device_pipeline_start()
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper around __media_pipeline_start().
+ */
+__must_check int __video_device_pipeline_start(struct video_device *vdev,
+					       struct media_pipeline *pipe);
+
+/**
+ * video_device_pipeline_stop - Mark a pipeline as not streaming
+ * @vdev: Starting video device
+ *
+ * Mark all entities connected to a given video device through enabled links,
+ * either directly or indirectly, as not streaming. The media_entity pipe field
+ * is reset to %NULL.
+ *
+ * If multiple calls to media_pipeline_start() have been made, the same
+ * number of calls to this function are required to mark the pipeline as not
+ * streaming.
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper around media_pipeline_stop().
+ */
+void video_device_pipeline_stop(struct video_device *vdev);
+
+/**
+ * __video_device_pipeline_stop - Mark a pipeline as not streaming
+ * @vdev: Starting video device
+ *
+ * .. note:: This is the non-locking version of media_pipeline_stop()
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper around __media_pipeline_stop().
+ */
+void __video_device_pipeline_stop(struct video_device *vdev);
+
+/**
+ * video_device_pipeline - Get the media pipeline a video device is part of
+ * @vdev: The video device
+ *
+ * This function returns the media pipeline that a video device has been
+ * associated with when constructing the pipeline with
+ * video_device_pipeline_start(). The pointer remains valid until
+ * video_device_pipeline_stop() is called.
+ *
+ * Return: The media_pipeline the video device is part of, or NULL if the video
+ * device is not part of any pipeline.
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper around media_entity_pipeline().
+ */
+struct media_pipeline *video_device_pipeline(struct video_device *vdev);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 #endif /* _V4L2_DEV_H */
-- 
2.34.1

[PATCH v14 10/34] media: drivers: use video device pipeline start/stop

[Tomi Valkeinen]

Convert the media drivers to use video device based pipeline start/stop
where possible.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c          |  6 +++---
 drivers/media/platform/qcom/camss/camss-video.c        |  6 +++---
 drivers/media/platform/renesas/rcar-vin/rcar-dma.c     |  6 +++---
 drivers/media/platform/renesas/vsp1/vsp1_video.c       |  6 +++---
 .../media/platform/rockchip/rkisp1/rkisp1-capture.c    | 10 +++++-----
 .../media/platform/samsung/exynos4-is/fimc-capture.c   |  9 ++++-----
 .../media/platform/samsung/exynos4-is/fimc-isp-video.c |  9 ++++-----
 drivers/media/platform/samsung/exynos4-is/fimc-lite.c  |  9 ++++-----
 drivers/media/platform/st/stm32/stm32-dcmi.c           |  6 +++---
 drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c     |  6 +++---
 drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c   |  6 +++---
 drivers/media/platform/ti/cal/cal-video.c              |  6 +++---
 drivers/media/platform/ti/omap3isp/ispvideo.c          |  6 +++---
 drivers/media/platform/xilinx/xilinx-dma.c             |  6 +++---
 drivers/media/test-drivers/vimc/vimc-capture.c         |  7 +++----
 drivers/staging/media/imx/imx7-media-csi.c             |  6 +++---
 drivers/staging/media/ipu3/ipu3-v4l2.c                 |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c             |  6 +++---
 drivers/staging/media/tegra-video/tegra210.c           |  6 +++---
 19 files changed, 62 insertions(+), 66 deletions(-)

diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index a3fe547b7fce..390bd5ea3472 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -989,7 +989,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = video_device_pipeline_start(&q->vdev, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1009,7 +1009,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	video_device_pipeline_stop(&q->vdev);
 fail_pipeline:
 	dev_dbg(dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1030,7 +1030,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	video_device_pipeline_stop(&q->vdev);
 	pm_runtime_put(dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index 290df04c4d02..81fb3a5bc1d5 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -493,7 +493,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = video_device_pipeline_start(vdev, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -522,7 +522,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	video_device_pipeline_stop(vdev);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -553,7 +553,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	video_device_pipeline_stop(vdev);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index e72bc6fa049f..879dd02bbb55 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1265,7 +1265,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		video_device_pipeline_stop(&vin->vdev);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1282,7 +1282,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __video_device_pipeline_start(&vin->vdev, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1291,7 +1291,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		video_device_pipeline_stop(&vin->vdev);
 
 	return ret;
 }
diff --git a/drivers/media/platform/renesas/vsp1/vsp1_video.c b/drivers/media/platform/renesas/vsp1/vsp1_video.c
index e8e0ee5f2277..f8e4ee1c60ab 100644
--- a/drivers/media/platform/renesas/vsp1/vsp1_video.c
+++ b/drivers/media/platform/renesas/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1046,7 +1046,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1070,7 +1070,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index bf0d4789374c..677b65e273d0 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -913,7 +913,7 @@ static void rkisp1_cap_stream_disable(struct rkisp1_capture *cap)
  *
  * Call s_stream(false) in the reverse order from
  * rkisp1_pipeline_stream_enable() and disable the DMA engine.
- * Should be called before media_pipeline_stop()
+ * Should be called before video_device_pipeline_stop()
  */
 static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
 	__must_hold(&cap->rkisp1->stream_lock)
@@ -937,7 +937,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
  * rkisp1_pipeline_stream_enable - enable nodes in the pipeline
  *
  * Enable the DMA Engine and call s_stream(true) through the pipeline.
- * Should be called after media_pipeline_start()
+ * Should be called after video_device_pipeline_start()
  */
 static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
 	__must_hold(&cap->rkisp1->stream_lock)
@@ -994,7 +994,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -1008,7 +1008,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = video_device_pipeline_start(&cap->vnode.vdev, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1044,7 +1044,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&cap->vnode.vdev);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
index 03638c8f772d..e3b95a2b7e04 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		video_device_pipeline_stop(&vc->ve.vdev);
 		vc->streaming = false;
 	}
 
@@ -1176,7 +1176,6 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_vid_cap *vc = &fimc->vid_cap;
-	struct media_entity *entity = &vc->ve.vdev.entity;
 	struct fimc_source_info *si = NULL;
 	struct v4l2_subdev *sd;
 	int ret;
@@ -1184,7 +1183,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = video_device_pipeline_start(&vc->ve.vdev, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1217,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&vc->ve.vdev);
 	return ret;
 }
 
@@ -1234,7 +1233,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		video_device_pipeline_stop(&vc->ve.vdev);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
index 8f12240b0eb7..f6a302fa8d37 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
@@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
 	is_singular_file = v4l2_fh_is_singular_file(file);
 
 	if (is_singular_file && ivc->streaming) {
-		media_pipeline_stop(entity);
+		video_device_pipeline_stop(&ivc->ve.vdev);
 		ivc->streaming = 0;
 	}
 
@@ -490,10 +490,9 @@ static int isp_video_streamon(struct file *file, void *priv,
 {
 	struct fimc_isp *isp = video_drvdata(file);
 	struct exynos_video_entity *ve = &isp->video_capture.ve;
-	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = video_device_pipeline_start(&ve->vdev, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -508,7 +507,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	video_device_pipeline_stop(&ve->vdev);
 	return ret;
 }
 
@@ -523,7 +522,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	video_device_pipeline_stop(&video->ve.vdev);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
index 41b0a4a5929a..e185a40305a8 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			video_device_pipeline_stop(&fimc->ve.vdev);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -812,13 +812,12 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 			      enum v4l2_buf_type type)
 {
 	struct fimc_lite *fimc = video_drvdata(file);
-	struct media_entity *entity = &fimc->ve.vdev.entity;
 	int ret;
 
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = video_device_pipeline_start(&fimc->ve.vdev, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -835,7 +834,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&fimc->ve.vdev);
 	return 0;
 }
 
@@ -849,7 +848,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	video_device_pipeline_stop(&fimc->ve.vdev);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/st/stm32/stm32-dcmi.c b/drivers/media/platform/st/stm32/stm32-dcmi.c
index 2ca95ab2b0fe..37458d4d9564 100644
--- a/drivers/media/platform/st/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/st/stm32/stm32-dcmi.c
@@ -751,7 +751,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_unlocked;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = video_device_pipeline_start(dcmi->vdev, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -865,7 +865,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	video_device_pipeline_stop(dcmi->vdev);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -892,7 +892,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	video_device_pipeline_stop(dcmi->vdev);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 0912a1b6d525..17ad9a3caaa5 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	video_device_pipeline_stop(&csi->vdev);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	video_device_pipeline_stop(&csi->vdev);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 1d46e113d01d..21a6b92494ba 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	video_device_pipeline_stop(&video->vdev);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	video_device_pipeline_stop(&video->vdev);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
index 776da0cfcdbe..6a57efb0ed91 100644
--- a/drivers/media/platform/ti/cal/cal-video.c
+++ b/drivers/media/platform/ti/cal/cal-video.c
@@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -760,7 +760,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	cal_ctx_unprepare(ctx);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	video_device_pipeline_stop(&ctx->vdev);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -781,7 +781,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	video_device_pipeline_stop(&ctx->vdev);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
index 5746cac4e046..9b352660cbdc 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.c
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
@@ -1103,7 +1103,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1160,7 +1160,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1227,7 +1227,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 3a4d62be0f27..9a177337e934 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -404,7 +404,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 */
 	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -430,7 +430,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	video_device_pipeline_stop(&dma->video);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -458,7 +458,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	video_device_pipeline_stop(&dma->video);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index 6c437802f91f..aa944270e716 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -241,13 +241,12 @@ static void vimc_capture_return_all_buffers(struct vimc_capture_device *vcapture
 static int vimc_capture_start_streaming(struct vb2_queue *vq, unsigned int count)
 {
 	struct vimc_capture_device *vcapture = vb2_get_drv_priv(vq);
-	struct media_entity *entity = &vcapture->vdev.entity;
 	int ret;
 
 	vcapture->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcapture->stream.pipe);
+	ret = video_device_pipeline_start(&vcapture->vdev, &vcapture->stream.pipe);
 	if (ret) {
 		vimc_capture_return_all_buffers(vcapture, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -255,7 +254,7 @@ static int vimc_capture_start_streaming(struct vb2_queue *vq, unsigned int count
 
 	ret = vimc_streamer_s_stream(&vcapture->stream, &vcapture->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		video_device_pipeline_stop(&vcapture->vdev);
 		vimc_capture_return_all_buffers(vcapture, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -274,7 +273,7 @@ static void vimc_capture_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcapture->stream, &vcapture->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcapture->vdev.entity);
+	video_device_pipeline_stop(&vcapture->vdev);
 
 	/* Release all active buffers */
 	vimc_capture_return_all_buffers(vcapture, VB2_BUF_STATE_ERROR);
diff --git a/drivers/staging/media/imx/imx7-media-csi.c b/drivers/staging/media/imx/imx7-media-csi.c
index a0553c24cce4..45f27a855255 100644
--- a/drivers/staging/media/imx/imx7-media-csi.c
+++ b/drivers/staging/media/imx/imx7-media-csi.c
@@ -1360,7 +1360,7 @@ static int imx7_csi_video_start_streaming(struct vb2_queue *vq,
 
 	mutex_lock(&csi->mdev.graph_mutex);
 
-	ret = __media_pipeline_start(&csi->sd.entity, &csi->pipe);
+	ret = __video_device_pipeline_start(csi->vdev, &csi->pipe);
 	if (ret)
 		goto err_unlock;
 
@@ -1373,7 +1373,7 @@ static int imx7_csi_video_start_streaming(struct vb2_queue *vq,
 	return 0;
 
 err_stop:
-	__media_pipeline_stop(&csi->sd.entity);
+	__video_device_pipeline_stop(csi->vdev);
 err_unlock:
 	mutex_unlock(&csi->mdev.graph_mutex);
 	dev_err(csi->dev, "pipeline start failed with %d\n", ret);
@@ -1396,7 +1396,7 @@ static void imx7_csi_video_stop_streaming(struct vb2_queue *vq)
 
 	mutex_lock(&csi->mdev.graph_mutex);
 	v4l2_subdev_call(&csi->sd, video, s_stream, 0);
-	__media_pipeline_stop(&csi->sd.entity);
+	__video_device_pipeline_stop(csi->vdev);
 	mutex_unlock(&csi->mdev.graph_mutex);
 
 	/* release all active buffers */
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index d1c539cefba8..e06d2012a4ef 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -486,7 +486,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
 	atomic_set(&node->sequence, 0);
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = video_device_pipeline_start(&node->vdev, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -511,7 +511,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index abadaf13c0e1..a0a43dbda16c 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -886,7 +886,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -977,7 +977,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1031,7 +1031,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d58370a84737 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = video_device_pipeline_start(&chan->video, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	video_device_pipeline_stop(&chan->video);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	video_device_pipeline_stop(&chan->video);
 }
 
 /*
-- 
2.34.1

[PATCH v14 11/34] media: drivers: use video_device_pipeline()

[Tomi Valkeinen]

Use video_device_pipeline() in the drivers instead of
media_entity_pipeline().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 4 ++--
 drivers/media/platform/xilinx/xilinx-dma.h | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 9a177337e934..0a7fd8642a65 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
+	pipe = to_xvip_pipeline(&dma->video) ? : &dma->pipe;
 
 	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
 	if (ret < 0)
@@ -447,7 +447,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 {
 	struct xvip_dma *dma = vb2_get_drv_priv(vq);
-	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video.entity);
+	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video);
 	struct xvip_dma_buffer *buf, *nbuf;
 
 	/* Stop the pipeline. */
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 3ea10f6b0bb9..9c6d4c18d1a9 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -45,9 +45,9 @@ struct xvip_pipeline {
 	struct xvip_dma *output;
 };
 
-static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
+static inline struct xvip_pipeline *to_xvip_pipeline(struct video_device *vdev)
 {
-	struct media_pipeline *pipe = media_entity_pipeline(e);
+	struct media_pipeline *pipe = video_device_pipeline(vdev);
 
 	if (!pipe)
 		return NULL;
-- 
2.34.1

[PATCH v14 12/34] media: mc: entity: add alloc variant of pipeline_start

[Tomi Valkeinen]

Add new variant of media_pipeline_start(), media_pipeline_alloc_start().

media_pipeline_alloc_start() can be used by drivers that do not need to
extend the media_pipeline. The function will either use the pipeline
already associated with the entity, if such exists, or allocate a new
pipeline.

When media_pipeline_stop() is called and the pipeline's use count drops
to zero, the pipeline is automatically freed.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c       | 38 ++++++++++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-dev.c | 11 +++++++++
 include/media/media-entity.h       | 15 ++++++++++++
 include/media/v4l2-dev.h           | 14 +++++++++++
 4 files changed, 78 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 7fb97c6dc897..ad153a426a36 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -530,6 +530,8 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_cleanup(graph);
 
+	if (pipe->allocated)
+		kfree(pipe);
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
@@ -543,6 +545,42 @@ void media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
+__must_check int media_pipeline_alloc_start(struct media_entity *entity)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pipeline *new_pipe = NULL;
+	struct media_pipeline *pipe;
+	int ret;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	/*
+	 * Is the entity already part of a pipeline? If not, we need to allocate
+	 * a pipe.
+	 */
+	pipe = media_entity_pipeline(entity);
+	if (!pipe) {
+		new_pipe = kzalloc(sizeof(*new_pipe), GFP_KERNEL);
+		if (!new_pipe) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		pipe = new_pipe;
+		pipe->allocated = true;
+	}
+
+	ret = __media_pipeline_start(entity, pipe);
+	if (ret)
+		kfree(new_pipe);
+
+out:
+	mutex_unlock(&mdev->graph_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
+
 /* -----------------------------------------------------------------------------
  * Links management
  */
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 7f933ff89fd4..945bb867a4c1 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1143,6 +1143,17 @@ void __video_device_pipeline_stop(struct video_device *vdev)
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
 
+__must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_alloc_start(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
+
 struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 {
 	struct media_entity *entity = &vdev->entity;
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index aaf276f765cf..a77933afaa48 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -100,10 +100,12 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
+ * @allocated:		Media pipeline allocated and freed by the framework
  * @start_count:	Media pipeline start - stop count
  * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
+	bool allocated;
 	int start_count;
 	struct media_graph graph;
 };
@@ -1092,6 +1094,19 @@ void media_pipeline_stop(struct media_entity *entity);
  */
 void __media_pipeline_stop(struct media_entity *entity);
 
+/**
+ * media_pipeline_alloc_start - Mark a pipeline as streaming
+ * @entity: Starting entity
+ *
+ * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
+ * of working on a given pipeline the function will use an existing pipeline if
+ * the entity is already part of a pipeline, or allocate a new pipeline.
+ *
+ * Calls to media_pipeline_alloc_start() must be matched with
+ * media_pipeline_stop().
+ */
+__must_check int media_pipeline_alloc_start(struct media_entity *entity);
+
 /**
  * media_devnode_create() - creates and initializes a device node interface
  *
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 494685872254..643da0740ab0 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -607,6 +607,20 @@ void video_device_pipeline_stop(struct video_device *vdev);
  */
 void __video_device_pipeline_stop(struct video_device *vdev);
 
+/**
+ * video_device_pipeline_alloc_start - Mark a pipeline as streaming
+ * @vdev: Starting video device
+ *
+ * video_device_pipeline_alloc_start() is similar to video_device_pipeline_start()
+ * but instead of working on a given pipeline the function will use an
+ * existing pipeline if the video device is already part of a pipeline, or
+ * allocate a new pipeline.
+ *
+ * Calls to video_device_pipeline_alloc_start() must be matched with
+ * video_device_pipeline_stop().
+ */
+__must_check int video_device_pipeline_alloc_start(struct video_device *vdev);
+
 /**
  * video_device_pipeline - Get the media pipeline a video device is part of
  * @vdev: The video device
-- 
2.34.1

[PATCH v14 13/34] media: drivers: use video_device_pipeline_alloc_start()

[Tomi Valkeinen]

Use video_device_pipeline_alloc_start() instead of manually
allocating/managing the media pipeline storage.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-vin/rcar-dma.c | 14 +-------------
 drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c |  2 +-
 .../media/platform/sunxi/sun6i-csi/sun6i_video.c   |  2 +-
 drivers/media/platform/ti/cal/cal-video.c          |  2 +-
 drivers/media/platform/ti/cal/cal.h                |  1 -
 5 files changed, 4 insertions(+), 17 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 879dd02bbb55..3aea96d85165 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1244,8 +1244,6 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd,
 
 static int rvin_set_stream(struct rvin_dev *vin, int on)
 {
-	struct media_pipeline *pipe;
-	struct media_device *mdev;
 	struct v4l2_subdev *sd;
 	struct media_pad *pad;
 	int ret;
@@ -1273,17 +1271,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret)
 		return ret;
 
-	/*
-	 * The graph lock needs to be taken to protect concurrent
-	 * starts of multiple VIN instances as they might share
-	 * a common subdevice down the line and then should use
-	 * the same pipe.
-	 */
-	mdev = vin->vdev.entity.graph_obj.mdev;
-	mutex_lock(&mdev->graph_mutex);
-	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
-	ret = __video_device_pipeline_start(&vin->vdev, pipe);
-	mutex_unlock(&mdev->graph_mutex);
+	ret = video_device_pipeline_alloc_start(&vin->vdev);
 	if (ret)
 		return ret;
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 17ad9a3caaa5..a3e826a755fc 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
+	ret = video_device_pipeline_alloc_start(&csi->vdev);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 21a6b92494ba..7d39cbbf6a88 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
+	ret = video_device_pipeline_alloc_start(&video->vdev);
 	if (ret < 0)
 		goto clear_dma_queue;
 
diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
index 6a57efb0ed91..5813345656af 100644
--- a/drivers/media/platform/ti/cal/cal-video.c
+++ b/drivers/media/platform/ti/cal/cal-video.c
@@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
+	ret = video_device_pipeline_alloc_start(&ctx->vdev);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
diff --git a/drivers/media/platform/ti/cal/cal.h b/drivers/media/platform/ti/cal/cal.h
index 61409ddced98..648cec43c3fc 100644
--- a/drivers/media/platform/ti/cal/cal.h
+++ b/drivers/media/platform/ti/cal/cal.h
@@ -174,7 +174,6 @@ struct cal_camerarx {
 	struct device_node	*source_ep_node;
 	struct device_node	*source_node;
 	struct v4l2_subdev	*source;
-	struct media_pipeline	pipe;
 
 	struct v4l2_subdev	subdev;
 	struct media_pad	pads[CAL_CAMERARX_NUM_PADS];
-- 
2.34.1

[PATCH v14 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

[Note: the code is mostly from Laurent but the patch description is from Tomi]

The media_pipeline_start() and media_pipeline_stop() functions use the
media graph walk API to traverse the graph and validate the pipeline.
The graph walk traverses the media graph following links between the
entities.

Also, while the pipeline can't change between the start and stop calls,
the graph is walked again from scratch at stop time, or any time a
driver needs to inspect the pipeline.

With the upcoming multiplexed streams support we will need a bit more
intelligent pipeline construction, as e.g. two independent streams may
be passing through a single entity via separate pads in which case
those pads should not be part of the same pipeline.

This patch essentially rewrites the media_pipeline_start/stop so that
a pipeline is defined as a set of pads instead of entities and the media
graph traversal considers the pad interdependencies when choosing which
links to follow.

Currently all the entity's pads are considered as interdependent. This
means that the behavior with all the current drivers stays the same, but
in the future we can define a more fine-grained pipeline construction.

Additionally the media pipeline's pads are cached at
media_pipeline_start() time, and re-used at media_pipeline_stop() which
avoid the need to re-walk the whole graph as the previous implementation
did.

Also, caching pads in the pipeline can serve in the future as the
foundation to provide a better API than the media graph walk to drivers
to iterate over pads and entities in the pipeline.

Note that the old media_pipeline_start/stop used the media graph walk
API. The new version does not use the media graph walk API, but instead
a new implementation.

There are two reason for not changing the graph walk: it proved to be
rather difficult to change the graph walk to have the features
implemented in this patch, and second, this keeps the backward
compatibility of the graph walk as there are users of the graph walk API

The long term plan is that all the existing code would be converted to
use the new cached pipeline, thus allowing us to remove the graph walk.

Signed-off-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst |   7 +-
 drivers/media/mc/mc-entity.c               | 517 ++++++++++++++++++---
 include/media/media-entity.h               |  71 ++-
 3 files changed, 509 insertions(+), 86 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 4bb062d5c2e7..400b8ca29367 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -230,14 +230,13 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads which are part of the pipeline as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
+pipeline through the struct media_pad
 pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index ad153a426a36..0a5c92b8bbce 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -226,6 +226,27 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+/*
+ * This function checks the interdependency inside the entity between @pad0
+ * and @pad1. If two pads are interdependent they are part of the same pipeline
+ * and enabling one of the pads means that the other pad will become "locked"
+ * and doesn't allow configuration changes.
+ *
+ * For the time being all pads are considered interdependent.
+ */
+static bool media_entity_has_pad_interdep(struct media_entity *entity,
+					  unsigned int pad0, unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (entity->pads[pad0].flags & entity->pads[pad1].flags &
+	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
+		return false;
+
+	return true;
+}
+
 static struct media_entity *
 media_entity_other(struct media_entity *entity, struct media_link *link)
 {
@@ -374,97 +395,436 @@ EXPORT_SYMBOL_GPL(media_graph_walk_next);
  * Pipeline management
  */
 
+/*
+ * The pipeline traversal stack stores pads that are reached during graph
+ * traversal, with a list of links to be visited to continue the traversal.
+ * When a new pad is reached, an entry is pushed on the top of the stack and
+ * points to the incoming pad and the first link of the entity.
+ *
+ * To find further pads in the pipeline, the traversal algorithm follows
+ * internal pad dependencies in the entity, and then links in the graph. It
+ * does so by iterating over all links of the entity, and following enabled
+ * links that originate from a pad that is internally connected to the incoming
+ * pad, as reported by the media_entity_has_pad_interdep() function.
+ */
+
+/**
+ * struct media_pipeline_walk_entry - Entry in the pipeline traversal stack
+ *
+ * @pad: The media pad being visited
+ * @links: Links left to be visited
+ */
+struct media_pipeline_walk_entry {
+	struct media_pad *pad;
+	struct list_head *links;
+};
+
+/**
+ * struct media_pipeline_walk - State used by the media pipeline traversal
+ *				algorithm
+ *
+ * @mdev: The media device
+ * @stack: Depth-first search stack
+ * @stack.size: Number of allocated entries in @stack.entries
+ * @stack.top: Index of the top stack entry (-1 if the stack is empty)
+ * @stack.entries: Stack entries
+ */
+struct media_pipeline_walk {
+	struct media_device *mdev;
+
+	struct {
+		unsigned int size;
+		int top;
+		struct media_pipeline_walk_entry *entries;
+	} stack;
+};
+
+#define MEDIA_PIPELINE_STACK_GROW_STEP		16
+
+static struct media_pipeline_walk_entry *
+media_pipeline_walk_top(struct media_pipeline_walk *walk)
+{
+	return &walk->stack.entries[walk->stack.top];
+}
+
+static bool media_pipeline_walk_empty(struct media_pipeline_walk *walk)
+{
+	return walk->stack.top == -1;
+}
+
+/* Increase the stack size by MEDIA_PIPELINE_STACK_GROW_STEP elements. */
+static int media_pipeline_walk_resize(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entries;
+	unsigned int new_size;
+
+	/* Safety check, to avoid stack overflows in case of bugs. */
+	if (walk->stack.size >= 256)
+		return -E2BIG;
+
+	new_size = walk->stack.size + MEDIA_PIPELINE_STACK_GROW_STEP;
+
+	entries = krealloc(walk->stack.entries,
+			   new_size * sizeof(*walk->stack.entries),
+			   GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	walk->stack.entries = entries;
+	walk->stack.size = new_size;
+
+	return 0;
+}
+
+/* Push a new entry on the stack. */
+static int media_pipeline_walk_push(struct media_pipeline_walk *walk,
+				    struct media_pad *pad)
+{
+	struct media_pipeline_walk_entry *entry;
+	int ret;
+
+	if (walk->stack.top + 1 >= walk->stack.size) {
+		ret = media_pipeline_walk_resize(walk);
+		if (ret)
+			return ret;
+	}
+
+	walk->stack.top++;
+	entry = media_pipeline_walk_top(walk);
+	entry->pad = pad;
+	entry->links = pad->entity->links.next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: pushed entry %u: '%s':%u\n",
+		walk->stack.top, pad->entity->name, pad->index);
+
+	return 0;
+}
+
+/*
+ * Move the top entry link cursor to the next link. If all links of the entry
+ * have been visited, pop the entry itself.
+ */
+static void media_pipeline_walk_pop(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry;
+
+	if (WARN_ON(walk->stack.top < 0))
+		return;
+
+	entry = media_pipeline_walk_top(walk);
+
+	if (entry->links->next == &entry->pad->entity->links) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: entry %u has no more links, popping\n",
+			walk->stack.top);
+
+		walk->stack.top--;
+		return;
+	}
+
+	entry->links = entry->links->next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: moved entry %u to next link\n",
+		walk->stack.top);
+}
+
+/* Free all memory allocated while walking the pipeline. */
+static void media_pipeline_walk_destroy(struct media_pipeline_walk *walk)
+{
+	kfree(walk->stack.entries);
+}
+
+/* Add a pad to the pipeline and push it to the stack. */
+static int media_pipeline_add_pad(struct media_pipeline *pipe,
+				  struct media_pipeline_walk *walk,
+				  struct media_pad *pad)
+{
+	struct media_pipeline_pad *ppad;
+
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		if (ppad->pad == pad) {
+			dev_dbg(pad->graph_obj.mdev->dev,
+				"media pipeline: already contains pad '%s':%u\n",
+				pad->entity->name, pad->index);
+			return 0;
+		}
+	}
+
+	ppad = kzalloc(sizeof(*ppad), GFP_KERNEL);
+	if (!ppad)
+		return -ENOMEM;
+
+	ppad->pipe = pipe;
+	ppad->pad = pad;
+
+	list_add_tail(&ppad->list, &pipe->pads);
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline: added pad '%s':%u\n",
+		pad->entity->name, pad->index);
+
+	return media_pipeline_walk_push(walk, pad);
+}
+
+/* Explore the next link of the entity at the top of the stack. */
+static int media_pipeline_explore_next_link(struct media_pipeline *pipe,
+					    struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry = media_pipeline_walk_top(walk);
+	struct media_pad *pad;
+	struct media_link *link;
+	struct media_pad *local;
+	struct media_pad *remote;
+	int ret;
+
+	pad = entry->pad;
+	link = list_entry(entry->links, typeof(*link), list);
+	media_pipeline_walk_pop(walk);
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: exploring link '%s':%u -> '%s':%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	/* Skip links that are not enabled. */
+	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (disabled)\n");
+		return 0;
+	}
+
+	/* Get the local pad and remote pad. */
+	if (link->source->entity == pad->entity) {
+		local = link->source;
+		remote = link->sink;
+	} else {
+		local = link->sink;
+		remote = link->source;
+	}
+
+	/*
+	 * Skip links that originate from a different pad than the incoming pad
+	 * that is not connected internally in the entity to the incoming pad.
+	 */
+	if (pad != local &&
+	    !media_entity_has_pad_interdep(pad->entity, pad->index, local->index)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (no route)\n");
+		return 0;
+	}
+
+	/*
+	 * Add the local and remote pads of the link to the pipeline and push
+	 * them to the stack, if they're not already present.
+	 */
+	ret = media_pipeline_add_pad(pipe, walk, local);
+	if (ret)
+		return ret;
+
+	ret = media_pipeline_add_pad(pipe, walk, remote);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void media_pipeline_cleanup(struct media_pipeline *pipe)
+{
+	while (!list_empty(&pipe->pads)) {
+		struct media_pipeline_pad *ppad;
+
+		ppad = list_first_entry(&pipe->pads, typeof(*ppad), list);
+		list_del(&ppad->list);
+		kfree(ppad);
+	}
+}
+
+static int media_pipeline_populate(struct media_pipeline *pipe,
+				   struct media_pad *pad)
+{
+	struct media_pipeline_walk walk = { };
+	struct media_pipeline_pad *ppad;
+	int ret;
+
+	/*
+	 * Populate the media pipeline by walking the media graph, starting
+	 * from @pad.
+	 */
+	INIT_LIST_HEAD(&pipe->pads);
+	pipe->mdev = pad->graph_obj.mdev;
+
+	walk.mdev = pipe->mdev;
+	walk.stack.top = -1;
+	ret = media_pipeline_add_pad(pipe, &walk, pad);
+	if (ret)
+		goto done;
+
+	/*
+	 * Use a depth-first search algorithm: as long as the stack is not
+	 * empty, explore the next link of the top entry. The
+	 * media_pipeline_explore_next_link() function will either move to the
+	 * next link, pop the entry if fully visited, or add new entries on
+	 * top.
+	 */
+	while (!media_pipeline_walk_empty(&walk)) {
+		ret = media_pipeline_explore_next_link(pipe, &walk);
+		if (ret)
+			goto done;
+	}
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline populated, found pads:\n");
+
+	list_for_each_entry(ppad, &pipe->pads, list)
+		dev_dbg(pad->graph_obj.mdev->dev, "- '%s':%u\n",
+			ppad->pad->entity->name, ppad->pad->index);
+
+	WARN_ON(walk.stack.top != -1);
+
+	ret = 0;
+
+done:
+	media_pipeline_walk_destroy(&walk);
+
+	if (ret)
+		media_pipeline_cleanup(pipe);
+
+	return ret;
+}
+
 __must_check int __media_pipeline_start(struct media_entity *entity,
 					struct media_pipeline *pipe)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
-	struct media_link *link;
+	struct media_pipeline_pad *err_ppad;
+	struct media_pipeline_pad *ppad;
 	int ret;
 
+	lockdep_assert_held(&mdev->graph_mutex);
+
+	/*
+	 * media_pipeline_start(entity) only makes sense with entities that have
+	 * a single pad.
+	 */
+
+	if (WARN_ON(entity->num_pads != 1))
+		return -EINVAL;
+
+	/*
+	 * If the entity is already part of a pipeline, that pipeline must
+	 * be the same as the pipe given to media_pipeline_start().
+	 */
+	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
+		return -EINVAL;
+
+	/*
+	 * If the pipeline has already been started, it is guaranteed to be
+	 * valid, so just increase the start count.
+	 */
 	if (pipe->start_count) {
 		pipe->start_count++;
 		return 0;
 	}
 
-	ret = media_graph_walk_init(&pipe->graph, mdev);
+	/*
+	 * Populate the pipeline. This populates the media_pipeline pads list
+	 * with media_pipeline_pad instances for each pad found during graph
+	 * walk.
+	 */
+	ret = media_pipeline_populate(pipe, entity->pads);
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&pipe->graph, entity);
+	/*
+	 * Now that all the pads in the pipeline have been gathered, perform
+	 * the validation steps.
+	 */
+
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		struct media_pad *pad = ppad->pad;
+		struct media_entity *entity = pad->entity;
+		bool has_enabled_link = false;
+		bool has_link = false;
+		struct media_link *link;
 
-	while ((entity = media_graph_walk_next(graph))) {
-		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
-		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
+		dev_dbg(mdev->dev, "Validating pad '%s':%u\n", pad->entity->name,
+			pad->index);
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				entity_err->name);
+		/*
+		 * 1. Ensure that the pad doesn't already belong to a different
+		 * pipeline.
+		 */
+		if (pad->pipe) {
+			dev_dbg(mdev->dev, "Failed to start pipeline: pad '%s':%u busy\n",
+				pad->entity->name, pad->index);
 			ret = -EBUSY;
 			goto error;
 		}
 
-		/* Already streaming --- no need to check. */
-		if (entity->pipe)
-			continue;
-
-		entity->pipe = pipe;
-
-		if (!entity->ops || !entity->ops->link_validate)
-			continue;
-
-		bitmap_zero(active, entity->num_pads);
-		bitmap_fill(has_no_links, entity->num_pads);
-
+		/*
+		 * 2. Validate all active links whose sink is the current pad.
+		 * Validation of the source pads is performed in the context of
+		 * the connected sink pad to avoid duplicating checks.
+		 */
 		for_each_media_entity_data_link(entity, link) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			/* Skip links unrelated to the current pad. */
+			if (link->sink != pad && link->source != pad)
+				continue;
 
-			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			/* Record if the pad has links and enabled links. */
+			if (link->flags & MEDIA_LNK_FL_ENABLED)
+				has_enabled_link = true;
+			has_link = true;
 
 			/*
-			 * Pads that either do not need to connect or
-			 * are connected through an enabled link are
-			 * fine.
+			 * Validate the link if it's enabled and has the
+			 * current pad as its sink.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
-			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+			if (!(link->flags & MEDIA_LNK_FL_ENABLED))
+				continue;
 
-			/*
-			 * Link validation will only take place for
-			 * sink ends of the link that are enabled.
-			 */
-			if (link->sink != pad ||
-			    !(link->flags & MEDIA_LNK_FL_ENABLED))
+			if (link->sink != pad)
+				continue;
+
+			if (!entity->ops || !entity->ops->link_validate)
 				continue;
 
 			ret = entity->ops->link_validate(link);
-			if (ret < 0 && ret != -ENOIOCTLCMD) {
-				dev_dbg(entity->graph_obj.mdev->dev,
-					"link validation failed for '%s':%u -> '%s':%u, error %d\n",
+			if (ret) {
+				dev_dbg(mdev->dev,
+					"Link '%s':%u -> '%s':%u failed validation: %d\n",
 					link->source->entity->name,
 					link->source->index,
-					entity->name, link->sink->index, ret);
+					link->sink->entity->name,
+					link->sink->index, ret);
 				goto error;
 			}
-		}
 
-		/* Either no links or validated links are fine. */
-		bitmap_or(active, active, has_no_links, entity->num_pads);
+			dev_dbg(mdev->dev,
+				"Link '%s':%u -> '%s':%u is valid\n",
+				link->source->entity->name,
+				link->source->index,
+				link->sink->entity->name,
+				link->sink->index);
+		}
 
-		if (!bitmap_full(active, entity->num_pads)) {
+		/*
+		 * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
+		 * ensure that it has either no link or an enabled link.
+		 */
+		if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) && has_link &&
+		    !has_enabled_link) {
+			dev_dbg(mdev->dev,
+				"Pad '%s':%u must be connected by an enabled link\n",
+				pad->entity->name, pad->index);
 			ret = -ENOLINK;
-			dev_dbg(entity->graph_obj.mdev->dev,
-				"'%s':%u must be connected by an enabled link\n",
-				entity->name,
-				(unsigned)find_first_zero_bit(
-					active, entity->num_pads));
 			goto error;
 		}
+
+		/* Validation passed, store the pipe pointer in the pad. */
+		pad->pipe = pipe;
 	}
 
 	pipe->start_count++;
@@ -476,20 +836,15 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
-
-	while ((entity_err = media_graph_walk_next(graph))) {
-		entity_err->pipe = NULL;
 
-		/*
-		 * We haven't started entities further than this so we quit
-		 * here.
-		 */
-		if (entity_err == entity)
+	list_for_each_entry(err_ppad, &pipe->pads, list) {
+		if (err_ppad == ppad)
 			break;
+
+		err_ppad->pad->pipe = NULL;
 	}
 
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 	return ret;
 }
@@ -510,8 +865,8 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline_pad *ppad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -523,12 +878,10 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (--pipe->start_count)
 		return;
 
-	media_graph_walk_start(graph, entity);
+	list_for_each_entry(ppad, &pipe->pads, list)
+		ppad->pad->pipe = NULL;
 
-	while ((entity = media_graph_walk_next(graph)))
-		entity->pipe = NULL;
-
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 	if (pipe->allocated)
 		kfree(pipe);
@@ -835,7 +1188,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -851,12 +1204,11 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
-	    (media_entity_is_streaming(source) ||
-	     media_entity_is_streaming(sink)))
+	    (media_pad_is_streaming(source) || media_pad_is_streaming(sink)))
 		return -EBUSY;
 
 	mdev = source->graph_obj.mdev;
@@ -1034,10 +1386,23 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
 struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
 {
-	return entity->pipe;
+	struct media_pad *pad;
+
+	media_entity_for_each_pad(entity, pad) {
+		if (pad->pipe)
+			return pad->pipe;
+	}
+
+	return NULL;
 }
 EXPORT_SYMBOL_GPL(media_entity_pipeline);
 
+struct media_pipeline *media_pad_pipeline(struct media_pad *pad)
+{
+	return pad->pipe;
+}
+EXPORT_SYMBOL_GPL(media_pad_pipeline);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a77933afaa48..00990b20b3d5 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -101,13 +101,33 @@ struct media_graph {
  * struct media_pipeline - Media pipeline related information
  *
  * @allocated:		Media pipeline allocated and freed by the framework
+ * @mdev:		The media device the pipeline is part of
+ * @pads:		List of media_pipeline_pad
  * @start_count:	Media pipeline start - stop count
- * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
 	bool allocated;
+	struct media_device *mdev;
+	struct list_head pads;
 	int start_count;
-	struct media_graph graph;
+};
+
+/**
+ * struct media_pipeline_pad - A pad part of a media pipeline
+ *
+ * @list:		Entry in the media_pad pads list
+ * @pipe:		The media_pipeline that the pad is part of
+ * @pad:		The media pad
+ *
+ * This structure associate a pad with a media pipeline. Instances of
+ * media_pipeline_pad are created by media_pipeline_start() when it builds the
+ * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
+ * removes the entries from the list and deletes them.
+ */
+struct media_pipeline_pad {
+	struct list_head list;
+	struct media_pipeline *pipe;
+	struct media_pad *pad;
 };
 
 /**
@@ -189,6 +209,8 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * @pipe:	Pipeline this pad belongs to. Use media_entity_pipeline() to
+ *		access this field.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -196,6 +218,12 @@ struct media_pad {
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+	struct media_pipeline *pipe;
 };
 
 /**
@@ -272,7 +300,6 @@ enum media_entity_type {
  * @links:	List of data links.
  * @ops:	Entity operations.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -308,8 +335,6 @@ struct media_entity {
 
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
@@ -938,6 +963,18 @@ media_entity_remote_source_pad_unique(const struct media_entity *entity)
 	return media_entity_remote_pad_unique(entity, MEDIA_PAD_FL_SOURCE);
 }
 
+/**
+ * media_pad_is_streaming - Test if a pad is part of a streaming pipeline
+ * @pad: The pad
+ *
+ * Return: True if the pad is part of a pipeline started with the
+ * media_pipeline_start() function, false otherwise.
+ */
+static inline bool media_pad_is_streaming(const struct media_pad *pad)
+{
+	return pad->pipe;
+}
+
 /**
  * media_entity_is_streaming - Test if an entity is part of a streaming pipeline
  * @entity: The entity
@@ -947,13 +984,22 @@ media_entity_remote_source_pad_unique(const struct media_entity *entity)
  */
 static inline bool media_entity_is_streaming(const struct media_entity *entity)
 {
-	return entity->pipe;
+	struct media_pad *pad;
+
+	media_entity_for_each_pad(entity, pad) {
+		if (media_pad_is_streaming(pad))
+			return true;
+	}
+
+	return false;
 }
 
 /**
  * media_entity_pipeline - Get the media pipeline an entity is part of
  * @entity: The entity
  *
+ * DEPRECATED: use media_pad_pipeline() instead.
+ *
  * This function returns the media pipeline that an entity has been associated
  * with when constructing the pipeline with media_pipeline_start(). The pointer
  * remains valid until media_pipeline_stop() is called.
@@ -968,6 +1014,19 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
  */
 struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
 
+/**
+ * media_pad_pipeline - Get the media pipeline a pad is part of
+ * @pad: The pad
+ *
+ * This function returns the media pipeline that a pad has been associated
+ * with when constructing the pipeline with media_pipeline_start(). The pointer
+ * remains valid until media_pipeline_stop() is called.
+ *
+ * Return: The media_pipeline the pad is part of, or NULL if the pad is
+ * not part of any pipeline.
+ */
+struct media_pipeline *media_pad_pipeline(struct media_pad *pad);
+
 /**
  * media_entity_get_fwnode_pad - Get pad number from fwnode
  *
-- 
2.34.1

[PATCH v14 15/34] media: mc: entity: Add has_pad_interdep entity operation

[Tomi Valkeinen]

Add a new media entity operation, has_pad_interdep. The optional op is
used to discover the pad interdependencies inside an entity during the
pipeline construction.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 10 ++++++++--
 include/media/media-entity.h | 10 ++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 0a5c92b8bbce..831076b36847 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -232,7 +232,10 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * and enabling one of the pads means that the other pad will become "locked"
  * and doesn't allow configuration changes.
  *
- * For the time being all pads are considered interdependent.
+ * This function uses the &media_entity_operations.has_pad_interdep() operation
+ * to check the dependency inside the entity between @pad0 and @pad1. If the
+ * has_pad_interdep operation is not implemented, all pads of the entity are
+ * considered to be interdependent.
  */
 static bool media_entity_has_pad_interdep(struct media_entity *entity,
 					  unsigned int pad0, unsigned int pad1)
@@ -244,7 +247,10 @@ static bool media_entity_has_pad_interdep(struct media_entity *entity,
 	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
 		return false;
 
-	return true;
+	if (!entity->ops || !entity->ops->has_pad_interdep)
+		return true;
+
+	return entity->ops->has_pad_interdep(entity, pad0, pad1);
 }
 
 static struct media_entity *
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 00990b20b3d5..8e9fd309aa65 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -237,6 +237,14 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_pad_interdep:	Return whether a two pads inside the entity are
+ *			interdependent. If two pads are interdependent they are
+ *			part of the same pipeline and enabling one of the pads
+ *			means that the other pad will become "locked" and
+ *			doesn't allow configuration changes. pad0 and pad1 are
+ *			guaranteed to not both be sinks or sources.
+ *			Optional: If the operation isn't implemented all pads
+ *			will be considered as interdependent.
  *
  * .. note::
  *
@@ -250,6 +258,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
+				 unsigned int pad1);
 };
 
 /**
-- 
2.34.1

[PATCH v14 16/34] media: mc: convert pipeline funcs to take media_pad

[Tomi Valkeinen]

Now that the pipeline is stored into pads instead of entities, we can
change the relevant functions to take pads instead of entities.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 40 ++++++++-----------
 .../samsung/s3c-camif/camif-capture.c         |  6 +--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++--
 drivers/media/v4l2-core/v4l2-dev.c            | 12 +++---
 drivers/staging/media/imx/imx-media-utils.c   |  8 ++--
 include/media/media-entity.h                  | 34 ++++++++--------
 include/media/v4l2-dev.h                      |  4 +-
 7 files changed, 52 insertions(+), 60 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 831076b36847..b8bcbc734eaf 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -700,29 +700,21 @@ static int media_pipeline_populate(struct media_pipeline *pipe,
 	return ret;
 }
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	struct media_pipeline_pad *err_ppad;
 	struct media_pipeline_pad *ppad;
 	int ret;
 
 	lockdep_assert_held(&mdev->graph_mutex);
 
-	/*
-	 * media_pipeline_start(entity) only makes sense with entities that have
-	 * a single pad.
-	 */
-
-	if (WARN_ON(entity->num_pads != 1))
-		return -EINVAL;
-
 	/*
 	 * If the entity is already part of a pipeline, that pipeline must
 	 * be the same as the pipe given to media_pipeline_start().
 	 */
-	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
+	if (WARN_ON(pad->pipe && pad->pipe != pipe))
 		return -EINVAL;
 
 	/*
@@ -739,7 +731,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * with media_pipeline_pad instances for each pad found during graph
 	 * walk.
 	 */
-	ret = media_pipeline_populate(pipe, entity->pads);
+	ret = media_pipeline_populate(pipe, pad);
 	if (ret)
 		return ret;
 
@@ -856,22 +848,22 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_pipeline_pad *ppad;
 
 	/*
@@ -894,19 +886,19 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
-__must_check int media_pipeline_alloc_start(struct media_entity *entity)
+__must_check int media_pipeline_alloc_start(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	struct media_pipeline *new_pipe = NULL;
 	struct media_pipeline *pipe;
 	int ret;
@@ -917,7 +909,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
 	 * Is the entity already part of a pipeline? If not, we need to allocate
 	 * a pipe.
 	 */
-	pipe = media_entity_pipeline(entity);
+	pipe = media_pad_pipeline(pad);
 	if (!pipe) {
 		new_pipe = kzalloc(sizeof(*new_pipe), GFP_KERNEL);
 		if (!new_pipe) {
@@ -929,7 +921,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
 		pipe->allocated = true;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	if (ret)
 		kfree(new_pipe);
 
diff --git a/drivers/media/platform/samsung/s3c-camif/camif-capture.c b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
index c2d8f1e425d8..db106ebdf870 100644
--- a/drivers/media/platform/samsung/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index caefac07af92..877e85a451cb 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 945bb867a4c1..397d553177fa 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1105,7 +1105,7 @@ __must_check int video_device_pipeline_start(struct video_device *vdev,
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return media_pipeline_start(entity, pipe);
+	return media_pipeline_start(&entity->pads[0], pipe);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_start);
 
@@ -1117,7 +1117,7 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return __media_pipeline_start(entity, pipe);
+	return __media_pipeline_start(&entity->pads[0], pipe);
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
 
@@ -1128,7 +1128,7 @@ void video_device_pipeline_stop(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return;
 
-	return media_pipeline_stop(entity);
+	return media_pipeline_stop(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
 
@@ -1139,7 +1139,7 @@ void __video_device_pipeline_stop(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return;
 
-	return __media_pipeline_stop(entity);
+	return __media_pipeline_stop(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
 
@@ -1150,7 +1150,7 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return media_pipeline_alloc_start(entity);
+	return media_pipeline_alloc_start(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
 
@@ -1161,7 +1161,7 @@ struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return NULL;
 
-	return media_entity_pipeline(entity);
+	return media_pad_pipeline(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index e9a3c6d2c66f..3e7462112649 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -863,16 +863,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (media_entity_pipeline(entity))
-			__media_pipeline_stop(entity);
+		if (media_pad_pipeline(entity->pads))
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 8e9fd309aa65..28c9de8a1f34 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1115,66 +1115,66 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
+ * Mark all pads connected to a given pad through enabled links, either
  * directly or indirectly, as streaming. The given pipeline object is assigned
- * to every entity in the pipeline and stored in the media_entity pipe field.
+ * to every pad in the pipeline and stored in the media_pad pipe field.
  *
  * Calls to this function can be nested, in which case the same number of
  * media_pipeline_stop() calls will be required to stop streaming. The
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as not streaming. The media_entity pipe field is
+ * Mark all pads connected to a given pads through enabled links, either
+ * directly or indirectly, as not streaming. The media_pad pipe field is
  * reset to %NULL.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
  * number of calls to this function are required to mark the pipeline as not
  * streaming.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_pipeline_alloc_start - Mark a pipeline as streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
  * of working on a given pipeline the function will use an existing pipeline if
- * the entity is already part of a pipeline, or allocate a new pipeline.
+ * the pad is already part of a pipeline, or allocate a new pipeline.
  *
  * Calls to media_pipeline_alloc_start() must be matched with
  * media_pipeline_stop().
  */
-__must_check int media_pipeline_alloc_start(struct media_entity *entity);
+__must_check int media_pipeline_alloc_start(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 643da0740ab0..e0a13505f88d 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -548,7 +548,7 @@ static inline int video_is_registered(struct video_device *vdev)
  *
  * Mark all entities connected to a given video device through enabled links,
  * either directly or indirectly, as streaming. The given pipeline object is
- * assigned to every entity in the pipeline and stored in the media_entity pipe
+ * assigned to every pad in the pipeline and stored in the media_pad pipe
  * field.
  *
  * Calls to this function can be nested, in which case the same number of
@@ -582,7 +582,7 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
  * @vdev: Starting video device
  *
  * Mark all entities connected to a given video device through enabled links,
- * either directly or indirectly, as not streaming. The media_entity pipe field
+ * either directly or indirectly, as not streaming. The media_pad pipe field
  * is reset to %NULL.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
-- 
2.34.1

[PATCH v14 17/34] media: add V4L2_SUBDEV_FL_STREAMS

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_STREAMS. It is used to indicate that the
subdev supports the new API with multiplexed streams (routing, stream
configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 3797b99bb408..c38de23b7f22 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -879,6 +879,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed subdev active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads is NULL)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_STREAMS			(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.34.1

[PATCH v14 18/34] media: add V4L2_SUBDEV_CAP_STREAMS

[Tomi Valkeinen]

Add a subdev capability flag to expose to userspace if a subdev supports
multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
 include/uapi/linux/v4l2-subdev.h      | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5c27bac772ea..90826b956693 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -439,7 +439,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			((sd->flags & V4L2_SUBDEV_FL_STREAMS) ? V4L2_SUBDEV_CAP_STREAMS : 0);
 
 		return 0;
 	}
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..89af27f50a41 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports routing and multiplexed streams. */
+#define V4L2_SUBDEV_CAP_STREAMS			0x00000002
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
-- 
2.34.1

[PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
 3 files changed, 153 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..a0d9c79e162f
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.34.1

[PATCH v14 20/34] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 87 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 +++++++
 include/uapi/linux/v4l2-subdev.h      | 52 ++++++++++++++++
 4 files changed, 185 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index c314025d977e..1c02f935cc6c 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/media-device.h> /* for media_set_bus_info() */
@@ -3151,6 +3152,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3414,8 +3430,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 90826b956693..af1f53d99507 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -23,6 +23,16 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 
+/*
+ * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
+ * of streams.
+ *
+ * Note that V4L2_FRAME_DESC_ENTRY_MAX is related: V4L2_FRAME_DESC_ENTRY_MAX
+ * restricts the total number of streams in a pad, although the stream ID is
+ * not restricted.
+ */
+#define V4L2_SUBDEV_MAX_STREAM_ID 63
+
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
@@ -417,6 +427,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 	case VIDIOC_SUBDEV_S_SELECTION:
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING:
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -732,6 +746,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
+			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
+				return -EINVAL;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -1015,6 +1101,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->_lock);
 
+	kfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index c38de23b7f22..f38943932cfe 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,12 +695,26 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @_lock: default for 'lock'
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -711,6 +725,7 @@ struct v4l2_subdev_state {
 	struct mutex _lock;
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -763,6 +778,9 @@ struct v4l2_subdev_state {
  *		     this operation as close as possible to stream on time. The
  *		     operation shall fail if the pad index it has been called on
  *		     is not valid or in case of unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -805,6 +823,10 @@ struct v4l2_subdev_pad_ops {
 			      struct v4l2_mbus_frame_desc *fd);
 	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 89af27f50a41..b63b80576dd3 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -24,6 +24,7 @@
 #ifndef __LINUX_V4L2_SUBDEV_H
 #define __LINUX_V4L2_SUBDEV_H
 
+#include <linux/const.h>
 #include <linux/ioctl.h>
 #include <linux/types.h>
 #include <linux/v4l2-common.h>
@@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device supports routing and multiplexed streams. */
 #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated by the subdevice (usually a sensor), and thus there is no
+ * sink-side endpoint for the route. The sink_pad and sink_stream fields are
+ * unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u32 num_routes;
+	__u64 routes;
+	__u32 reserved[6];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -206,6 +256,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.34.1

[PATCH v14 21/34] media: subdev: add v4l2_subdev_has_pad_interdep()

[Tomi Valkeinen]

Add a v4l2_subdev_has_pad_interdep() helper function which can be used
for media_entity_operations.has_pad_interdep op.

It considers two pads interdependent if there is an active route between
pad0 and pad1.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 +++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index af1f53d99507..1ce9a7dc0c6e 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1045,6 +1045,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_pad_interdep);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f38943932cfe..2d1509556ce0 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1235,6 +1235,23 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_pad_interdep - MC has_pad_interdep implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function is an implementation of the media_entity_operations.has_pad_interdep
+ * operation for subdevs that implement the multiplexed streams API (as
+ * indicated by the V4L2_SUBDEV_FL_STREAMS subdev flag).
+ *
+ * It considers two pads interdependent if there is an active route between pad0
+ * and pad1.
+ */
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.34.1

[PATCH v14 22/34] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 +++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 1ce9a7dc0c6e..f3f872c72180 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1180,6 +1180,34 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+	struct v4l2_subdev_krouting new_routing = { 0 };
+
+	lockdep_assert_held(state->lock);
+
+	if (src->num_routes > 0) {
+		new_routing.routes = kmemdup(src->routes,
+			src->num_routes * sizeof(*src->routes),
+			GFP_KERNEL);
+
+		if (!new_routing.routes)
+			return -ENOMEM;
+	}
+
+	new_routing.num_routes = src->num_routes;
+
+	kfree(dst->routes);
+	*dst = new_routing;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 2d1509556ce0..b29003de8b0a 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1412,6 +1412,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 23/34] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add a for_each_active_route() macro to replace the repeated pattern
of iterating on the active routes of a routing table.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .clang-format                         |  1 +
 drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
 include/media/v4l2-subdev.h           | 13 +++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/.clang-format b/.clang-format
index 1247d54f9e49..31f39ae78f7b 100644
--- a/.clang-format
+++ b/.clang-format
@@ -190,6 +190,7 @@ ForEachMacros:
   - 'for_each_active_dev_scope'
   - 'for_each_active_drhd_unit'
   - 'for_each_active_iommu'
+  - 'for_each_active_route'
   - 'for_each_aggr_pgid'
   - 'for_each_available_child_of_node'
   - 'for_each_bench'
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index f3f872c72180..da8550598673 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1208,6 +1208,26 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b29003de8b0a..03c836d65713 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1428,6 +1428,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    const struct v4l2_subdev_krouting *routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 24/34] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 173 ++++++++++++++++++
 2 files changed, 181 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 6f8d79926aa5..260cfa8c3f3d 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -593,6 +593,14 @@ before calling v4l2_subdev_init_finalize():
 
 This shares the driver's private mutex between the controls and the states.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_STREAMS subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..5075b1828b32 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,176 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, only a
+single stream can pass through a media link and a media pad. Thus each pad
+contains a format configuration for that single stream. In some cases a subdev
+can do stream processing and split a stream into two or compose two streams
+into one, but the inputs and outputs for the subdev are still a single stream
+per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
+data streams are transmitted on the same bus, which is represented by a media
+link connecting a transmitter source pad with a sink pad on the receiver. For
+example, a camera sensor can produce two distinct streams, a pixel stream and a
+metadata stream, which are transmitted on the multiplexed data bus, represented
+by a media link which connects the single sensor's source pad with the receiver
+sink pad. The stream-aware receiver will de-multiplex the streams received on
+the its sink pad and allows to route them individually to one of its source
+pads.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the enabled
+streams from one end of the link to the other, and subdevices have routing
+tables which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a stream.
+In other words, a particular stream ID must exist on both sides of a media
+link, but another stream ID can be used for the same stream at the other side
+of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations in a media
+entity.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source
+  pad (pad 0) which carries two streams, pixel data stream and metadata
+  stream.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver,
+using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
+stream endpoint in each subdev.
-- 
2.34.1

[PATCH v14 25/34] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 151 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  79 +++++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 272 insertions(+), 21 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index da8550598673..e752eb3df1a3 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -158,8 +158,19 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+		       u32 which, u32 pad, u32 stream)
 {
+#if defined(CONFIG_MEDIA_CONTROLLER)
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
+		if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
+			return -EINVAL;
+		return 0;
+	}
+#endif
+	if (stream != 0)
+		return -EINVAL;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
@@ -174,7 +185,7 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state(sd, state, format->which, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -201,7 +212,7 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state(sd, state, code->which, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -213,7 +224,7 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -248,7 +259,7 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -260,7 +271,7 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state(sd, state, sel->which, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -919,6 +930,59 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
+static int
+v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+				const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_stream_configs new_configs = { 0 };
+	struct v4l2_subdev_route *route;
+	u32 format_idx = 0;
+
+	/* Count number of formats needed */
+	for_each_active_route(routing, route) {
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		new_configs.num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (new_configs.num_configs) {
+		new_configs.configs = kvcalloc(new_configs.num_configs,
+						   sizeof(*new_configs.configs),
+						   GFP_KERNEL);
+
+		if (!new_configs.configs)
+			return -ENOMEM;
+	}
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for_each_active_route(routing, route) {
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			new_configs.configs[idx].pad = route->sink_pad;
+			new_configs.configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		new_configs.configs[idx].pad = route->source_pad;
+		new_configs.configs[idx].stream = route->source_stream;
+	}
+
+	kvfree(stream_configs->configs);
+	*stream_configs = new_configs;
+
+	return 0;
+}
+
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
 				      struct fwnode_endpoint *endpoint)
 {
@@ -1093,7 +1157,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 	else
 		state->lock = &state->_lock;
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
 		state->pads = kvcalloc(sd->entity.num_pads,
 				       sizeof(*state->pads), GFP_KERNEL);
 		if (!state->pads) {
@@ -1133,6 +1198,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->_lock);
 
 	kfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1187,6 +1253,7 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 	struct v4l2_subdev_krouting *dst = &state->routing;
 	const struct v4l2_subdev_krouting *src = routing;
 	struct v4l2_subdev_krouting new_routing = { 0 };
+	int r;
 
 	lockdep_assert_held(state->lock);
 
@@ -1201,6 +1268,13 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 
 	new_routing.num_routes = src->num_routes;
 
+	r = v4l2_subdev_init_stream_configs(&state->stream_configs,
+					    &new_routing);
+	if (r) {
+		kfree(new_routing.routes);
+		return r;
+	}
+
 	kfree(dst->routes);
 	*dst = new_routing;
 
@@ -1228,6 +1302,69 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_format);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].crop;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_crop);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].compose;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 03c836d65713..1f2663425568 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,6 +695,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -715,6 +746,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_STREAMS)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -726,6 +758,7 @@ struct v4l2_subdev_state {
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1441,6 +1474,52 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_crop() - Get pointer to a stream crop rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to crop rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_compose() - Get pointer to a stream compose
+ *					    rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to compose rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index b63b80576dd3..39c9b87f1e70 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -45,13 +45,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -59,13 +61,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -81,6 +85,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -89,7 +94,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -102,6 +108,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -113,19 +120,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -137,6 +147,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -147,7 +158,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -159,6 +171,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -171,7 +184,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.34.1

[PATCH v14 26/34] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 245 +++++++++++++++++++++++---
 1 file changed, 225 insertions(+), 20 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e752eb3df1a3..a46059cda541 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1062,7 +1063,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
-v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1071,7 +1072,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
+		fmt->stream = stream;
+
+		return v4l2_subdev_call(sd, pad, get_fmt,
+					v4l2_subdev_get_locked_active_state(sd),
+					fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
@@ -1081,31 +1086,231 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_pad *pad,
+					  u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev_route *route;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	struct v4l2_subdev_state *state;
+	int ret;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	subdev = media_entity_to_v4l2_subdev(pad->entity);
+
+	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_get_locked_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+	if (!streams)
+		return -ENOMEM;
+
+	num_streams = 0;
+
+	for_each_active_route(routing, route) {
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+		}
+
+		if (route_pad != pad->index)
+			continue;
+
+		/* look for duplicates */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream) {
+				ret = -EINVAL;
+				goto free_streams;
+			}
+		}
+
+		streams[num_streams++] = route_stream;
+	}
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
+
+	return 0;
+
+free_streams:
+	kfree(streams);
+
+	return ret;
+}
+
+static int v4l2_subdev_link_validate_locked(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link->source, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link->sink, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored by the receiver, but having extra
+	 * sink streams is an error as streams must have a source.
+	 */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto free_sink;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto free_sink;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+		if (ret != -ENOIOCTLCMD)
+			goto free_sink;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto free_sink;
+	}
+
+free_sink:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *source_sd, *sink_sd;
+	struct v4l2_subdev_state *source_state, *sink_state;
+	int ret;
+
+	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
+	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
+
+	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
+	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
+
+	if (sink_state)
+		v4l2_subdev_lock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_lock_state(source_state);
+
+	ret = v4l2_subdev_link_validate_locked(link);
+
+	if (sink_state)
+		v4l2_subdev_unlock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_unlock_state(source_state);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.34.1

[PATCH v14 27/34] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
+ stream, return the pad + stream on the opposite side of the subdev.

v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
format on the pad + stream on the opposite side from the given pad +
stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 48 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a46059cda541..a4ae11c27e24 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1570,6 +1570,54 @@ v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
 
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing, pad, stream,
+						    &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_subdev_state_get_stream_format(state, other_pad,
+						   other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1f2663425568..c46ca30f05d2 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1520,6 +1520,42 @@ struct v4l2_rect *
 v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 				     unsigned int pad, u32 stream);
 
+/**
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
+
+/**
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 28/34] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
drivers that do not need to do anything special in their get_fmt op can
use this helper directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a4ae11c27e24..76788798e976 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1438,10 +1438,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 {
 	struct v4l2_mbus_framefmt *fmt;
 
-	if (format->pad >= sd->entity.num_pads)
-		return -EINVAL;
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
+							  format->stream);
+	else if (format->pad < sd->entity.num_pads && format->stream == 0)
+		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
+	else
+		fmt = NULL;
 
-	fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
 	if (!fmt)
 		return -EINVAL;
 
-- 
2.34.1

[PATCH v14 29/34] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 76788798e976..92a3142fcf23 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1511,6 +1511,28 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
 				    unsigned int pad, u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index c46ca30f05d2..64397ce1cb4d 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1474,6 +1474,22 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.34.1

[PATCH v14 30/34] media: subdev: add v4l2_subdev_routing_validate() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a v4l2_subdev_routing_validate() helper for verifying routing for
common cases like only allowing non-overlapping 1-to-1 streams.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 102 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  39 ++++++++++
 2 files changed, 141 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 92a3142fcf23..4fdde9d8c07d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1644,6 +1644,108 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow)
+{
+	u32 *remote_pads = NULL;
+	unsigned int i, j;
+	int ret = -EINVAL;
+
+	if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+		remote_pads = kcalloc(sd->entity.num_pads, sizeof(*remote_pads),
+				      GFP_KERNEL);
+		if (!remote_pads)
+			return -ENOMEM;
+
+		for (i = 0; i < sd->entity.num_pads; ++i)
+			remote_pads[i] = U32_MAX;
+	}
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		/* Validate the sink and source pad numbers. */
+		if (route->sink_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
+				i, route->sink_pad);
+			goto out;
+		}
+
+		if (route->source_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->source_pad].flags & MEDIA_PAD_FL_SOURCE)) {
+			dev_dbg(sd->dev, "route %u source (%u) is not a source pad\n",
+				i, route->source_pad);
+			goto out;
+		}
+
+		/*
+		 * V4L2_SUBDEV_ROUTING_NO_STREAM_MIX: Streams on the same pad
+		 * may not be routed to streams on different pads.
+		 */
+		if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+			if (remote_pads[route->sink_pad] != U32_MAX &&
+			    remote_pads[route->sink_pad] != route->source_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "sink");
+				goto out;
+			}
+
+			if (remote_pads[route->source_pad] != U32_MAX &&
+			    remote_pads[route->source_pad] != route->sink_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "source");
+				goto out;
+			}
+
+			remote_pads[route->sink_pad] = route->source_pad;
+			remote_pads[route->source_pad] = route->sink_pad;
+		}
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_1_TO_N: No two routes can
+			 * originate from the same (sink) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_1_TO_N) &&
+			    route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u originate from same sink (%u/%u)\n",
+					i, j, route->sink_pad,
+					route->sink_stream);
+				goto out;
+			}
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_N_TO_1: No two routes can end
+			 * at the same (source) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_N_TO_1) &&
+			    route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u end at same source (%u/%u)\n",
+					i, j, route->source_pad,
+					route->source_stream);
+				goto out;
+			}
+		}
+	}
+
+	ret = 0;
+
+out:
+	kfree(remote_pads);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 64397ce1cb4d..1c66a8b9d918 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1572,6 +1572,45 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
+ *
+ * @V4L2_SUBDEV_ROUTING_NO_1_TO_N:
+ *	an input stream may not be routed to multiple output streams (stream
+ *	duplication)
+ * @V4L2_SUBDEV_ROUTING_NO_N_TO_1:
+ *	multiple input streams may not be routed to the same output stream
+ *	(stream merging)
+ * @V4L2_SUBDEV_ROUTING_NO_STREAM_MIX:
+ *	streams on the same pad may not be routed to streams on different pads
+ * @V4L2_SUBDEV_ROUTING_ONLY_1_TO_1:
+ *	only non-overlapping 1-to-1 stream routing is allowed (a combination of
+ *	@V4L2_SUBDEV_ROUTING_NO_1_TO_N and @V4L2_SUBDEV_ROUTING_NO_N_TO_1)
+ */
+enum v4l2_subdev_routing_restriction {
+	V4L2_SUBDEV_ROUTING_NO_1_TO_N = BIT(0),
+	V4L2_SUBDEV_ROUTING_NO_N_TO_1 = BIT(1),
+	V4L2_SUBDEV_ROUTING_NO_STREAM_MIX = BIT(2),
+	V4L2_SUBDEV_ROUTING_ONLY_1_TO_1 =
+		V4L2_SUBDEV_ROUTING_NO_1_TO_N |
+		V4L2_SUBDEV_ROUTING_NO_N_TO_1,
+};
+
+/**
+ * v4l2_subdev_routing_validate() - Verify that routes comply with driver
+ *				    constraints
+ * @sd: The subdevice
+ * @routing: Routing to verify
+ * @disallow: Restrictions on routes
+ *
+ * This verifies that the given routing complies with the @disallow constraints.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 31/34] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a helper function to translate streams between two pads of a subdev,
using the subdev's internal routing table.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 26 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 23 +++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4fdde9d8c07d..7d46c9a5a6c4 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1644,6 +1644,32 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams)
+{
+	const struct v4l2_subdev_krouting *routing = &state->routing;
+	struct v4l2_subdev_route *route;
+	u64 streams0 = 0;
+	u64 streams1 = 0;
+
+	for_each_active_route(routing, route) {
+		if (route->sink_pad == pad0 && route->source_pad == pad1 &&
+		    (*streams & BIT_ULL(route->sink_stream))) {
+			streams0 |= BIT_ULL(route->sink_stream);
+			streams1 |= BIT_ULL(route->source_stream);
+		}
+		if (route->source_pad == pad0 && route->sink_pad == pad1 &&
+		    (*streams & BIT_ULL(route->source_stream))) {
+			streams0 |= BIT_ULL(route->source_stream);
+			streams1 |= BIT_ULL(route->sink_stream);
+		}
+	}
+
+	*streams = streams0;
+	return streams1;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_xlate_streams);
+
 int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1c66a8b9d918..68cd8431c535 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1572,6 +1572,29 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * v4l2_subdev_state_xlate_streams() - Translate streams from one pad to another
+ *
+ * @state: Subdevice state
+ * @pad0: The first pad
+ * @pad1: The second pad
+ * @streams: Streams bitmask on the first pad
+ *
+ * Streams on sink pads of a subdev are routed to source pads as expressed in
+ * the subdev state routing table. Stream numbers don't necessarily match on
+ * the sink and source side of a route. This function translates stream numbers
+ * on @pad0, expressed as a bitmask in @streams, to the corresponding streams
+ * on @pad1 using the routing table from the @state. It returns the stream mask
+ * on @pad1, and updates @streams with the streams that have been found in the
+ * routing table.
+ *
+ * @pad0 and @pad1 must be a sink and a source, in any order.
+ *
+ * Return: The bitmask of streams of @pad1 that are routed to @streams on @pad0.
+ */
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams);
+
 /**
  * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
  *
-- 
2.34.1

[PATCH v14 32/34] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add two new subdev pad operations, .enable_streams() and
.disable_streams(), to allow control of individual streams per pad. This
is a superset of what the video .s_stream() operation implements.

To help with handling of backward compatibility, add two wrapper
functions around those operations, and require their usage in drivers.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 224 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  85 ++++++++++
 2 files changed, 309 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7d46c9a5a6c4..258f04968047 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1772,6 +1772,230 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
 
+static int v4l2_subdev_enable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+					       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * The subdev doesn't implement pad-based stream enable, fall back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if (sd->enabled_streams & streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already enabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Start streaming when the first streams are enabled. */
+	if (!sd->enabled_streams) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 1);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams |= streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .enable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->enable_streams)
+		return v4l2_subdev_enable_streams_fallback(sd, pad,
+							   streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already enabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (cfg->enabled) {
+			dev_dbg(dev, "stream %u already enabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .enable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, enable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as enabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = true;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_enable_streams);
+
+static int v4l2_subdev_disable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+						u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * If the subdev doesn't implement pad-based stream enable, fall  back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if ((sd->enabled_streams & streams_mask) != streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already disabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Stop streaming when the last streams are disabled. */
+	if (!(sd->enabled_streams & ~streams_mask)) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 0);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams &= ~streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .disable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->disable_streams)
+		return v4l2_subdev_disable_streams_fallback(sd, pad,
+							    streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already disabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (!cfg->enabled) {
+			dev_dbg(dev, "stream %u already disabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .disable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, disable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as disabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = false;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 68cd8431c535..0d660d4d17e1 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -700,6 +700,7 @@ struct v4l2_subdev_pad_config {
  *
  * @pad: pad number
  * @stream: stream number
+ * @enabled: has the stream been enabled with v4l2_subdev_enable_stream()
  * @fmt: &struct v4l2_mbus_framefmt
  * @crop: &struct v4l2_rect to be used for crop
  * @compose: &struct v4l2_rect to be used for compose
@@ -709,6 +710,7 @@ struct v4l2_subdev_pad_config {
 struct v4l2_subdev_stream_config {
 	u32 pad;
 	u32 stream;
+	bool enabled;
 
 	struct v4l2_mbus_framefmt fmt;
 	struct v4l2_rect crop;
@@ -814,6 +816,18 @@ struct v4l2_subdev_state {
  *
  * @set_routing: enable or disable data connection routes described in the
  *		 subdevice routing table.
+ *
+ * @enable_streams: Enable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_enable_streams() instead.
+ *
+ * @disable_streams: Disable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_disable_streams() instead.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -860,6 +874,12 @@ struct v4l2_subdev_pad_ops {
 			   struct v4l2_subdev_state *state,
 			   enum v4l2_subdev_format_whence which,
 			   struct v4l2_subdev_krouting *route);
+	int (*enable_streams)(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask);
+	int (*disable_streams)(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask);
 };
 
 /**
@@ -1005,6 +1025,10 @@ struct v4l2_subdev_platform_data {
  * @active_state: Active state for the subdev (NULL for subdevs tracking the
  *		  state internally). Initialized by calling
  *		  v4l2_subdev_init_finalize().
+ * @enabled_streams: Bitmask of enabled streams used by
+ *		     v4l2_subdev_enable_streams() and
+ *		     v4l2_subdev_disable_streams() helper functions for fallback
+ *		     cases.
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -1050,6 +1074,7 @@ struct v4l2_subdev {
 	 * doesn't support it.
 	 */
 	struct v4l2_subdev_state *active_state;
+	u64 enabled_streams;
 };
 
 
@@ -1634,6 +1659,66 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow);
 
+/**
+ * v4l2_subdev_enable_streams() - Enable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to enable
+ *
+ * This function enables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows enabling multiple streams on a pad at
+ * once.
+ *
+ * Enabling a stream that is already enabled isn't allowed. If @streams_mask
+ * contains an already enabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream enable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is already enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask);
+
+/**
+ * v4l2_subdev_disable_streams() - Disable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to disable
+ *
+ * This function disables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows disabling multiple streams on a pad at
+ * once.
+ *
+ * Disabling a streams that is not enabled isn't allowed. If @streams_mask
+ * contains a disabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream disable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is not enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 33/34] media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The v4l2_subdev_s_stream_helper() helper can be used by subdevs that
implement the stream-aware .enable_streams() and .disable_streams()
operations to implement .s_stream(). This is limited to subdevs that
have a single source pad.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 40 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 258f04968047..cfd7d795abef 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1996,6 +1996,46 @@ int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
 
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable)
+{
+	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_route *route;
+	struct media_pad *pad;
+	u64 source_mask = 0;
+	int pad_index = -1;
+
+	/*
+	 * Find the source pad. This helper is meant for subdevs that have a
+	 * single source pad, so failures shouldn't happen, but catch them
+	 * loudly nonetheless as they indicate a driver bug.
+	 */
+	media_entity_for_each_pad(&sd->entity, pad) {
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			pad_index = pad->index;
+			break;
+		}
+	}
+
+	if (WARN_ON(pad_index == -1))
+		return -EINVAL;
+
+	/*
+	 * As there's a single source pad, just collect all the source streams.
+	 */
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	for_each_active_route(&state->routing, route)
+		source_mask |= BIT_ULL(route->source_stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (enable)
+		return v4l2_subdev_enable_streams(sd, pad_index, source_mask);
+	else
+		return v4l2_subdev_disable_streams(sd, pad_index, source_mask);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_s_stream_helper);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 0d660d4d17e1..76d284c99484 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1719,6 +1719,23 @@ int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
 int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 				u64 streams_mask);
 
+/**
+ * v4l2_subdev_s_stream_helper() - Helper to implement the subdev s_stream
+ *	operation using enable_streams and disable_streams
+ * @sd: The subdevice
+ * @enable: Enable or disable streaming
+ *
+ * Subdevice drivers that implement the streams-aware
+ * &v4l2_subdev_pad_ops.enable_streams and &v4l2_subdev_pad_ops.disable_streams
+ * operations can use this helper to implement the legacy
+ * &v4l2_subdev_video_ops.s_stream operation.
+ *
+ * This helper can only be used by subdevs that have a single source pad.
+ *
+ * Return: 0 on success, or a negative error code otherwise.
+ */
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v14 34/34] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 76d284c99484..293b789bbc2e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.34.1

Re: [PATCH v14 25/34] media: subdev: add stream based configuration

[Tomi Valkeinen]

Hi,

On 31/08/2022 17:13, Tomi Valkeinen wrote:
> Add support to manage configurations (format, crop, compose) per stream,
> instead of per pad. This is accomplished with data structures that hold
> an array of all subdev's stream configurations.
> 
> The number of streams can vary at runtime based on routing. Every time
> the routing is changed, the stream configurations need to be
> re-initialized.

Intel's test bot says that this doesn't compile, and I can see that 
having CONFIG_MEDIA_CONTROLLER enabled but CONFIG_VIDEO_V4L2_SUBDEV_API 
disabled causes a build failure.

I need to find out (again) what exactly those defines mean, and do the 
necessary adjustments to the code.

  Tomi

Re: [PATCH v14 21/34] media: subdev: add v4l2_subdev_has_pad_interdep()

[Sakari Ailus]

Moi,

On Wed, Aug 31, 2022 at 05:13:44PM +0300, Tomi Valkeinen wrote:
> Add a v4l2_subdev_has_pad_interdep() helper function which can be used
> for media_entity_operations.has_pad_interdep op.
> 
> It considers two pads interdependent if there is an active route between
> pad0 and pad1.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 17 +++++++++++++++
>  2 files changed, 48 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index af1f53d99507..1ce9a7dc0c6e 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1045,6 +1045,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> +bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
> +				  unsigned int pad0, unsigned int pad1)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct v4l2_subdev_krouting *routing;
> +	unsigned int i;
> +	struct v4l2_subdev_state *state;

Please declare i after state.

> +
> +	state = v4l2_subdev_lock_and_get_active_state(sd);
> +
> +	routing = &state->routing;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
> +		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
> +			v4l2_subdev_unlock_state(state);
> +			return true;
> +		}
> +	}
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	return false;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_has_pad_interdep);
> +
>  struct v4l2_subdev_state *
>  __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
>  			  struct lock_class_key *lock_key)
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index f38943932cfe..2d1509556ce0 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1235,6 +1235,23 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>   */
>  int v4l2_subdev_link_validate(struct media_link *link);
>  
> +/**
> + * v4l2_subdev_has_pad_interdep - MC has_pad_interdep implementation for subdevs
> + *
> + * @entity: pointer to &struct media_entity
> + * @pad0: pad number for the first pad
> + * @pad1: pad number for the second pad
> + *
> + * This function is an implementation of the media_entity_operations.has_pad_interdep

Please run

$ scripts/checkpatch.pl --strict --max-line-length=80

on this.

> + * operation for subdevs that implement the multiplexed streams API (as
> + * indicated by the V4L2_SUBDEV_FL_STREAMS subdev flag).
> + *
> + * It considers two pads interdependent if there is an active route between pad0
> + * and pad1.
> + */
> +bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
> +				  unsigned int pad0, unsigned int pad1);
> +
>  /**
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *

-- 
Terveisin,
Sakari Ailus

Re: [PATCH v14 22/34] media: subdev: add v4l2_subdev_set_routing helper()

[Sakari Ailus]

Moi,

On Wed, Aug 31, 2022 at 05:13:45PM +0300, Tomi Valkeinen wrote:
> Add a helper function to set the subdev routing. The helper can be used
> from subdev driver's set_routing op to store the routing table.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 +++++++++++++++
>  2 files changed, 44 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 1ce9a7dc0c6e..f3f872c72180 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1180,6 +1180,34 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>  
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    const struct v4l2_subdev_krouting *routing)
> +{
> +	struct v4l2_subdev_krouting *dst = &state->routing;
> +	const struct v4l2_subdev_krouting *src = routing;
> +	struct v4l2_subdev_krouting new_routing = { 0 };
> +
> +	lockdep_assert_held(state->lock);
> +
> +	if (src->num_routes > 0) {
> +		new_routing.routes = kmemdup(src->routes,
> +			src->num_routes * sizeof(*src->routes),

This can overflow.

> +			GFP_KERNEL);

Apart from that:

		new_routing.routes =
			kmemdup(src->routes,
				src->num_routes * sizeof(*src->routes),
				GFP_KERNEL);

> +
> +		if (!new_routing.routes)
> +			return -ENOMEM;
> +	}
> +
> +	new_routing.num_routes = src->num_routes;
> +
> +	kfree(dst->routes);
> +	*dst = new_routing;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 2d1509556ce0..b29003de8b0a 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1412,6 +1412,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
>  int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  			struct v4l2_subdev_format *format);
>  
> +/**
> + * v4l2_subdev_set_routing() - Set given routing to subdev state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + *
> + * This will release old routing table (if any) from the state, allocate
> + * enough space for the given routing, and copy the routing.
> + *
> + * This can be used from the subdev driver's set_routing op, after validating
> + * the routing.
> + */
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    const struct v4l2_subdev_krouting *routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v14 22/34] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

On 25/09/2022 14:26, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Aug 31, 2022 at 05:13:45PM +0300, Tomi Valkeinen wrote:
>> Add a helper function to set the subdev routing. The helper can be used
>> from subdev driver's set_routing op to store the routing table.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 16 +++++++++++++++
>>   2 files changed, 44 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 1ce9a7dc0c6e..f3f872c72180 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1180,6 +1180,34 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>>   
>> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>> +			    struct v4l2_subdev_state *state,
>> +			    const struct v4l2_subdev_krouting *routing)
>> +{
>> +	struct v4l2_subdev_krouting *dst = &state->routing;
>> +	const struct v4l2_subdev_krouting *src = routing;
>> +	struct v4l2_subdev_krouting new_routing = { 0 };
>> +
>> +	lockdep_assert_held(state->lock);
>> +
>> +	if (src->num_routes > 0) {
>> +		new_routing.routes = kmemdup(src->routes,
>> +			src->num_routes * sizeof(*src->routes),
> 
> This can overflow.

Right. In practice it won't happen as the num_routes has been limited to 
256, but no harm to check for overflow here.

  Tomi

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Sakari Ailus]

Moi,

On Wed, Aug 31, 2022 at 05:13:42PM +0300, Tomi Valkeinen wrote:
> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> description of multiplexed media pads and internal routing to the
> V4L2-subdev documentation section.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
>  3 files changed, 153 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index fd1de0a73a9f..a67c2749089a 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>  
>  -  negotiate image formats on individual pads
>  
> +-  inspect and modify internal data routing between pads of the same entity
> +
>  Sub-device character device nodes, conventionally named
>  ``/dev/v4l-subdev*``, use major number 81.
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index 53e604bd7d60..228c1521f190 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -70,6 +70,7 @@ Function Reference
>      vidioc-subdev-g-crop
>      vidioc-subdev-g-fmt
>      vidioc-subdev-g-frame-interval
> +    vidioc-subdev-g-routing
>      vidioc-subdev-g-selection
>      vidioc-subdev-querycap
>      vidioc-subscribe-event
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> new file mode 100644
> index 000000000000..a0d9c79e162f
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -0,0 +1,150 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +.. c:namespace:: V4L
> +
> +.. _VIDIOC_SUBDEV_G_ROUTING:
> +
> +******************************************************
> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> +******************************************************
> +
> +Name
> +====
> +
> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_G_ROUTING
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_S_ROUTING
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to get and set the routing in a media entity.
> +The routing configuration determines the flows of data inside an entity.
> +
> +Drivers report their current routing tables using the
> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> +setting or clearing flags of the  ``flags`` field of a
> +struct :c:type:`v4l2_subdev_route`.

How about adding:

Routes that have V4L2_SUBDEV_ROUTE_FL_IMMUTABLE flag cannot be removed.
Depending on the driver, their V4L2_SUBDEV_ROUTE_FL_ACTIVE flag may be set
or reset.

Also see a comment later on.

> +
> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> +means that the userspace mut reconfigure all streams after calling the ioctl
> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.

How about this:

Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
formats being propagated from the sink pads towards the sources.

> +
> +A special case for routing are routes marked with
> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> +source endpoints on sensors and the sink fields are unused.
> +
> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> +provided ``num_routes`` is not big enough to contain all the available routes
> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> +value of the ``num_routes`` field. Application should then reserve enough memory
> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_routing
> +
> +.. flat-table:: struct v4l2_subdev_routing
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``which``
> +      - Format to modified, from enum
> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - struct :c:type:`v4l2_subdev_route`
> +      - ``routes[]``
> +      - Array of struct :c:type:`v4l2_subdev_route` entries
> +    * - __u32
> +      - ``num_routes``
> +      - Number of entries of the routes array
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_route
> +
> +.. flat-table:: struct v4l2_subdev_route
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``sink_pad``
> +      - Sink pad number.
> +    * - __u32
> +      - ``sink_stream``
> +      - Sink pad stream number.
> +    * - __u32
> +      - ``source_pad``
> +      - Source pad number.
> +    * - __u32
> +      - ``source_stream``
> +      - Source pad stream number.
> +    * - __u32
> +      - ``flags``
> +      - Route enable/disable flags
> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> +
> +.. _v4l2-subdev-routing-flags:
> +
> +.. flat-table:: enum v4l2_subdev_routing_flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - 0
> +      - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE

How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
link flag to mean a link that may not be changed in any way. In this case
we rather want to say that the route is always there, albeit you can still
enable or disable it.

> +      - 1
> +      - The route is immutable. Set by the driver.
> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - 2
> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> +        fields are unused. Set by the driver.
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOSPC
> +   The number of provided route entries is less than the available ones.

What does "available ones" mean in this context? More than is supported?
Wouldn't E2BIG be the appropriate code in that case?

> +
> +EINVAL
> +   The sink or source pad identifiers reference a non-existing pad, or reference
> +   pads of different types (ie. the sink_pad identifiers refers to a source pad)
> +   or the sink or source stream identifiers reference a non-existing stream on
> +   the sink or source pad.

-- 
Ystävällisin terveisin,

Sakari Ailus

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

On 27/09/2022 08:59, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Aug 31, 2022 at 05:13:42PM +0300, Tomi Valkeinen wrote:
>> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>
>> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
>> description of multiplexed media pads and internal routing to the
>> V4L2-subdev documentation section.
>>
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>>   .../userspace-api/media/v4l/user-func.rst     |   1 +
>>   .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
>>   3 files changed, 153 insertions(+)
>>   create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>>
>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> index fd1de0a73a9f..a67c2749089a 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>>   
>>   -  negotiate image formats on individual pads
>>   
>> +-  inspect and modify internal data routing between pads of the same entity
>> +
>>   Sub-device character device nodes, conventionally named
>>   ``/dev/v4l-subdev*``, use major number 81.
>>   
>> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
>> index 53e604bd7d60..228c1521f190 100644
>> --- a/Documentation/userspace-api/media/v4l/user-func.rst
>> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
>> @@ -70,6 +70,7 @@ Function Reference
>>       vidioc-subdev-g-crop
>>       vidioc-subdev-g-fmt
>>       vidioc-subdev-g-frame-interval
>> +    vidioc-subdev-g-routing
>>       vidioc-subdev-g-selection
>>       vidioc-subdev-querycap
>>       vidioc-subscribe-event
>> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>> new file mode 100644
>> index 000000000000..a0d9c79e162f
>> --- /dev/null
>> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>> @@ -0,0 +1,150 @@
>> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
>> +.. c:namespace:: V4L
>> +
>> +.. _VIDIOC_SUBDEV_G_ROUTING:
>> +
>> +******************************************************
>> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
>> +******************************************************
>> +
>> +Name
>> +====
>> +
>> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
>> +
>> +
>> +Synopsis
>> +========
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
>> +    :name: VIDIOC_SUBDEV_G_ROUTING
>> +
>> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
>> +    :name: VIDIOC_SUBDEV_S_ROUTING
>> +
>> +
>> +Arguments
>> +=========
>> +
>> +``fd``
>> +    File descriptor returned by :ref:`open() <func-open>`.
>> +
>> +``argp``
>> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
>> +
>> +
>> +Description
>> +===========
>> +
>> +These ioctls are used to get and set the routing in a media entity.
>> +The routing configuration determines the flows of data inside an entity.
>> +
>> +Drivers report their current routing tables using the
>> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
>> +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
>> +setting or clearing flags of the  ``flags`` field of a
>> +struct :c:type:`v4l2_subdev_route`.
> 
> How about adding:
> 
> Routes that have V4L2_SUBDEV_ROUTE_FL_IMMUTABLE flag cannot be removed.
> Depending on the driver, their V4L2_SUBDEV_ROUTE_FL_ACTIVE flag may be set
> or reset.

I have dropped the IMMUTABLE flag in my WIP branch, as I couldn't figure 
out a use for it. The only immutable routes are source routes, which are 
already special and there's no need for an extra flag.

> Also see a comment later on.
> 
>> +
>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>> +means that the userspace mut reconfigure all streams after calling the ioctl
>> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> 
> How about this:
> 
> Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
> formats being propagated from the sink pads towards the sources.

Hmm, but that's not true. The selections and formats will be zeroed, 
unless the driver initializes them to a value. There's no propagation done.

>> +
>> +A special case for routing are routes marked with
>> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
>> +source endpoints on sensors and the sink fields are unused.
>> +
>> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>> +provided ``num_routes`` is not big enough to contain all the available routes
>> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
>> +value of the ``num_routes`` field. Application should then reserve enough memory
>> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>> +
>> +.. c:type:: v4l2_subdev_routing
>> +
>> +.. flat-table:: struct v4l2_subdev_routing
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - __u32
>> +      - ``which``
>> +      - Format to modified, from enum
>> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>> +    * - struct :c:type:`v4l2_subdev_route`
>> +      - ``routes[]``
>> +      - Array of struct :c:type:`v4l2_subdev_route` entries
>> +    * - __u32
>> +      - ``num_routes``
>> +      - Number of entries of the routes array
>> +    * - __u32
>> +      - ``reserved``\ [5]
>> +      - Reserved for future extensions. Applications and drivers must set
>> +	the array to zero.
>> +
>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>> +
>> +.. c:type:: v4l2_subdev_route
>> +
>> +.. flat-table:: struct v4l2_subdev_route
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       1 1 2
>> +
>> +    * - __u32
>> +      - ``sink_pad``
>> +      - Sink pad number.
>> +    * - __u32
>> +      - ``sink_stream``
>> +      - Sink pad stream number.
>> +    * - __u32
>> +      - ``source_pad``
>> +      - Source pad number.
>> +    * - __u32
>> +      - ``source_stream``
>> +      - Source pad stream number.
>> +    * - __u32
>> +      - ``flags``
>> +      - Route enable/disable flags
>> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
>> +    * - __u32
>> +      - ``reserved``\ [5]
>> +      - Reserved for future extensions. Applications and drivers must set
>> +	the array to zero.
>> +
>> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
>> +
>> +.. _v4l2-subdev-routing-flags:
>> +
>> +.. flat-table:: enum v4l2_subdev_routing_flags
>> +    :header-rows:  0
>> +    :stub-columns: 0
>> +    :widths:       3 1 4
>> +
>> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - 0
>> +      - The route is enabled. Set by applications.
>> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> 
> How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
> link flag to mean a link that may not be changed in any way. In this case
> we rather want to say that the route is always there, albeit you can still
> enable or disable it.

If we think there's a need for this, I can add it back and name it 
static. I think what it then should mean is that the user can 
enable/disable it and also set the stream id, but the route must always 
exist.

But as I said above, I haven't figured out a use for this.

>> +      - 1
>> +      - The route is immutable. Set by the driver.
>> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
>> +      - 2
>> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
>> +        fields are unused. Set by the driver.
>> +
>> +Return Value
>> +============
>> +
>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>> +appropriately. The generic error codes are described at the
>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>> +
>> +ENOSPC
>> +   The number of provided route entries is less than the available ones.
> 
> What does "available ones" mean in this context? More than is supported?
> Wouldn't E2BIG be the appropriate code in that case?

Good question. I don't think I wrote this part =). ENOSPC refers to the 
case where VIDIOC_SUBDEV_G_ROUTING is called without enough space for 
the routing table. So "available ones" mean the routes in the subdev's 
routing table, and "provided route entries" refers to the userspace 
target routing table.

It sounds pretty odd, and obviously needs a clarification.

  Tomi

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Sakari Ailus]

Heipparallaa Tomi,

On Tue, Sep 27, 2022 at 12:32:43PM +0300, Tomi Valkeinen wrote:
> On 27/09/2022 08:59, Sakari Ailus wrote:
> > Moi,
> > 
> > On Wed, Aug 31, 2022 at 05:13:42PM +0300, Tomi Valkeinen wrote:
> > > From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > 
> > > Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> > > description of multiplexed media pads and internal routing to the
> > > V4L2-subdev documentation section.
> > > 
> > > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> > > ---
> > >   .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
> > >   .../userspace-api/media/v4l/user-func.rst     |   1 +
> > >   .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
> > >   3 files changed, 153 insertions(+)
> > >   create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > 
> > > diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > index fd1de0a73a9f..a67c2749089a 100644
> > > --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> > > @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
> > >   -  negotiate image formats on individual pads
> > > +-  inspect and modify internal data routing between pads of the same entity
> > > +
> > >   Sub-device character device nodes, conventionally named
> > >   ``/dev/v4l-subdev*``, use major number 81.
> > > diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> > > index 53e604bd7d60..228c1521f190 100644
> > > --- a/Documentation/userspace-api/media/v4l/user-func.rst
> > > +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> > > @@ -70,6 +70,7 @@ Function Reference
> > >       vidioc-subdev-g-crop
> > >       vidioc-subdev-g-fmt
> > >       vidioc-subdev-g-frame-interval
> > > +    vidioc-subdev-g-routing
> > >       vidioc-subdev-g-selection
> > >       vidioc-subdev-querycap
> > >       vidioc-subscribe-event
> > > diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > new file mode 100644
> > > index 000000000000..a0d9c79e162f
> > > --- /dev/null
> > > +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> > > @@ -0,0 +1,150 @@
> > > +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> > > +.. c:namespace:: V4L
> > > +
> > > +.. _VIDIOC_SUBDEV_G_ROUTING:
> > > +
> > > +******************************************************
> > > +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> > > +******************************************************
> > > +
> > > +Name
> > > +====
> > > +
> > > +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> > > +
> > > +
> > > +Synopsis
> > > +========
> > > +
> > > +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> > > +    :name: VIDIOC_SUBDEV_G_ROUTING
> > > +
> > > +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> > > +    :name: VIDIOC_SUBDEV_S_ROUTING
> > > +
> > > +
> > > +Arguments
> > > +=========
> > > +
> > > +``fd``
> > > +    File descriptor returned by :ref:`open() <func-open>`.
> > > +
> > > +``argp``
> > > +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> > > +
> > > +
> > > +Description
> > > +===========
> > > +
> > > +These ioctls are used to get and set the routing in a media entity.
> > > +The routing configuration determines the flows of data inside an entity.
> > > +
> > > +Drivers report their current routing tables using the
> > > +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> > > +with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
> > > +setting or clearing flags of the  ``flags`` field of a
> > > +struct :c:type:`v4l2_subdev_route`.
> > 
> > How about adding:
> > 
> > Routes that have V4L2_SUBDEV_ROUTE_FL_IMMUTABLE flag cannot be removed.
> > Depending on the driver, their V4L2_SUBDEV_ROUTE_FL_ACTIVE flag may be set
> > or reset.
> 
> I have dropped the IMMUTABLE flag in my WIP branch, as I couldn't figure out
> a use for it. The only immutable routes are source routes, which are already
> special and there's no need for an extra flag.

The two flags still mean different things... but let's continue the
discussion later in the message.

> 
> > Also see a comment later on.
> > 
> > > +
> > > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > > +means that the userspace mut reconfigure all streams after calling the ioctl
> > > +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > 
> > How about this:
> > 
> > Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
> > formats being propagated from the sink pads towards the sources.
> 
> Hmm, but that's not true. The selections and formats will be zeroed, unless
> the driver initializes them to a value. There's no propagation done.

They need to be propagated. The driver is responsible for maintaining a
valid configuration for the processing steps in a sub-device, and with
routes that must apply to routes as well.

> 
> > > +
> > > +A special case for routing are routes marked with
> > > +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> > > +source endpoints on sensors and the sink fields are unused.
> > > +
> > > +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > > +provided ``num_routes`` is not big enough to contain all the available routes
> > > +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > > +value of the ``num_routes`` field. Application should then reserve enough memory
> > > +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > > +
> > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > +
> > > +.. c:type:: v4l2_subdev_routing
> > > +
> > > +.. flat-table:: struct v4l2_subdev_routing
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths:       1 1 2
> > > +
> > > +    * - __u32
> > > +      - ``which``
> > > +      - Format to modified, from enum
> > > +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > > +    * - struct :c:type:`v4l2_subdev_route`
> > > +      - ``routes[]``
> > > +      - Array of struct :c:type:`v4l2_subdev_route` entries
> > > +    * - __u32
> > > +      - ``num_routes``
> > > +      - Number of entries of the routes array
> > > +    * - __u32
> > > +      - ``reserved``\ [5]
> > > +      - Reserved for future extensions. Applications and drivers must set
> > > +	the array to zero.
> > > +
> > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > +
> > > +.. c:type:: v4l2_subdev_route
> > > +
> > > +.. flat-table:: struct v4l2_subdev_route
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths:       1 1 2
> > > +
> > > +    * - __u32
> > > +      - ``sink_pad``
> > > +      - Sink pad number.
> > > +    * - __u32
> > > +      - ``sink_stream``
> > > +      - Sink pad stream number.
> > > +    * - __u32
> > > +      - ``source_pad``
> > > +      - Source pad number.
> > > +    * - __u32
> > > +      - ``source_stream``
> > > +      - Source pad stream number.
> > > +    * - __u32
> > > +      - ``flags``
> > > +      - Route enable/disable flags
> > > +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> > > +    * - __u32
> > > +      - ``reserved``\ [5]
> > > +      - Reserved for future extensions. Applications and drivers must set
> > > +	the array to zero.
> > > +
> > > +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> > > +
> > > +.. _v4l2-subdev-routing-flags:
> > > +
> > > +.. flat-table:: enum v4l2_subdev_routing_flags
> > > +    :header-rows:  0
> > > +    :stub-columns: 0
> > > +    :widths:       3 1 4
> > > +
> > > +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > +      - 0
> > > +      - The route is enabled. Set by applications.
> > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > 
> > How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
> > link flag to mean a link that may not be changed in any way. In this case
> > we rather want to say that the route is always there, albeit you can still
> > enable or disable it.
> 
> If we think there's a need for this, I can add it back and name it static. I
> think what it then should mean is that the user can enable/disable it and
> also set the stream id, but the route must always exist.

But the static routes are recognised by the stream ID only, aren't they?

I think we'll definitely need a way to determine which routes are always
there and which ones can be removed at will.

> 
> But as I said above, I haven't figured out a use for this.
> 
> > > +      - 1
> > > +      - The route is immutable. Set by the driver.
> > > +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> > > +      - 2
> > > +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> > > +        fields are unused. Set by the driver.
> > > +
> > > +Return Value
> > > +============
> > > +
> > > +On success 0 is returned, on error -1 and the ``errno`` variable is set
> > > +appropriately. The generic error codes are described at the
> > > +:ref:`Generic Error Codes <gen-errors>` chapter.
> > > +
> > > +ENOSPC
> > > +   The number of provided route entries is less than the available ones.
> > 
> > What does "available ones" mean in this context? More than is supported?
> > Wouldn't E2BIG be the appropriate code in that case?
> 
> Good question. I don't think I wrote this part =). ENOSPC refers to the case
> where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
> table. So "available ones" mean the routes in the subdev's routing table,
> and "provided route entries" refers to the userspace target routing table.
> 
> It sounds pretty odd, and obviously needs a clarification.

I think I actually can think what this did mean. It means that the
num_routes is smaller than the number of routes in a routing table when
G_ROUTING is called. For that I think ENOSPC is the right code actually.

But also I think we need to document what happens when there are too many
routes. For that E2BIG would be appropriate.

Should we have a field for telling which route was the bad one, if it was
one of them? That can be done later, too, if we'll ever need it.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

On 27/09/2022 13:23, Sakari Ailus wrote:

<snip>

>>>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>>>> +means that the userspace mut reconfigure all streams after calling the ioctl
>>>> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>>>
>>> How about this:
>>>
>>> Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
>>> formats being propagated from the sink pads towards the sources.
>>
>> Hmm, but that's not true. The selections and formats will be zeroed, unless
>> the driver initializes them to a value. There's no propagation done.
> 
> They need to be propagated. The driver is responsible for maintaining a
> valid configuration for the processing steps in a sub-device, and with
> routes that must apply to routes as well.

Hmm, no, they don't need to be propagated. The driver needs to 
initialize the formats and selections to valid configuration, that is 
true, but it doesn't mean the driver propagates settings from the sink 
pads to the source pads. In theory the formats on sink and source sides 
could be different.

I guess you could say that the driver initializes the sink side streams 
and then propagates those to the source side, but at least to me that 
gives the wrong impression. What the driver does is initialize the 
configuration, or reset the configuration to defaults (which it could do 
with propagation if it so wants).

The framework provides v4l2_subdev_set_routing_with_fmt() helper to 
achieve this, which sets the given format to all streams.

Anyway, I think we can talk about propagation if that helps, but I think 
the main point there is that the settings are reset or initialized.

>>>> +
>>>> +A special case for routing are routes marked with
>>>> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
>>>> +source endpoints on sensors and the sink fields are unused.
>>>> +
>>>> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>>>> +provided ``num_routes`` is not big enough to contain all the available routes
>>>> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
>>>> +value of the ``num_routes`` field. Application should then reserve enough memory
>>>> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>>>> +
>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>> +
>>>> +.. c:type:: v4l2_subdev_routing
>>>> +
>>>> +.. flat-table:: struct v4l2_subdev_routing
>>>> +    :header-rows:  0
>>>> +    :stub-columns: 0
>>>> +    :widths:       1 1 2
>>>> +
>>>> +    * - __u32
>>>> +      - ``which``
>>>> +      - Format to modified, from enum
>>>> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>>> +    * - struct :c:type:`v4l2_subdev_route`
>>>> +      - ``routes[]``
>>>> +      - Array of struct :c:type:`v4l2_subdev_route` entries
>>>> +    * - __u32
>>>> +      - ``num_routes``
>>>> +      - Number of entries of the routes array
>>>> +    * - __u32
>>>> +      - ``reserved``\ [5]
>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>> +	the array to zero.
>>>> +
>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>> +
>>>> +.. c:type:: v4l2_subdev_route
>>>> +
>>>> +.. flat-table:: struct v4l2_subdev_route
>>>> +    :header-rows:  0
>>>> +    :stub-columns: 0
>>>> +    :widths:       1 1 2
>>>> +
>>>> +    * - __u32
>>>> +      - ``sink_pad``
>>>> +      - Sink pad number.
>>>> +    * - __u32
>>>> +      - ``sink_stream``
>>>> +      - Sink pad stream number.
>>>> +    * - __u32
>>>> +      - ``source_pad``
>>>> +      - Source pad number.
>>>> +    * - __u32
>>>> +      - ``source_stream``
>>>> +      - Source pad stream number.
>>>> +    * - __u32
>>>> +      - ``flags``
>>>> +      - Route enable/disable flags
>>>> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
>>>> +    * - __u32
>>>> +      - ``reserved``\ [5]
>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>> +	the array to zero.
>>>> +
>>>> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
>>>> +
>>>> +.. _v4l2-subdev-routing-flags:
>>>> +
>>>> +.. flat-table:: enum v4l2_subdev_routing_flags
>>>> +    :header-rows:  0
>>>> +    :stub-columns: 0
>>>> +    :widths:       3 1 4
>>>> +
>>>> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>>> +      - 0
>>>> +      - The route is enabled. Set by applications.
>>>> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
>>>
>>> How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
>>> link flag to mean a link that may not be changed in any way. In this case
>>> we rather want to say that the route is always there, albeit you can still
>>> enable or disable it.
>>
>> If we think there's a need for this, I can add it back and name it static. I
>> think what it then should mean is that the user can enable/disable it and
>> also set the stream id, but the route must always exist.
> 
> But the static routes are recognised by the stream ID only, aren't they?
> 
> I think we'll definitely need a way to determine which routes are always
> there and which ones can be removed at will.

That's the V4L2_SUBDEV_ROUTE_FL_SOURCE. V4L2_SUBDEV_ROUTE_FL_SOURCE 
routes are always there, and the user can only enable or disable them.

This is why I dropped IMMUTABLE, as SOURCE is already immutable (or 
rather, static), and I don't see a need for a non-source route to be static.

>>
>> But as I said above, I haven't figured out a use for this.
>>
>>>> +      - 1
>>>> +      - The route is immutable. Set by the driver.
>>>> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
>>>> +      - 2
>>>> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
>>>> +        fields are unused. Set by the driver.
>>>> +
>>>> +Return Value
>>>> +============
>>>> +
>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>> +appropriately. The generic error codes are described at the
>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>> +
>>>> +ENOSPC
>>>> +   The number of provided route entries is less than the available ones.
>>>
>>> What does "available ones" mean in this context? More than is supported?
>>> Wouldn't E2BIG be the appropriate code in that case?
>>
>> Good question. I don't think I wrote this part =). ENOSPC refers to the case
>> where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
>> table. So "available ones" mean the routes in the subdev's routing table,
>> and "provided route entries" refers to the userspace target routing table.
>>
>> It sounds pretty odd, and obviously needs a clarification.
> 
> I think I actually can think what this did mean. It means that the
> num_routes is smaller than the number of routes in a routing table when
> G_ROUTING is called. For that I think ENOSPC is the right code actually.
> 
> But also I think we need to document what happens when there are too many
> routes. For that E2BIG would be appropriate.

v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers 
should, of course, do additional check if needed. In v4l2-ioctl.c it 
seems common to return EINVAL if there's too much data, but we can of 
course define E2BIG for routing ioctls.

> Should we have a field for telling which route was the bad one, if it was
> one of them? That can be done later, too, if we'll ever need it.

Hmm maybe, although I wonder how often the drivers can say that this 
particular route is the problem, and what would the userspace do with 
that information...

Do you have any examples in mind?

  Tomi

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Sakari Ailus]

Moi,

On Tue, Sep 27, 2022 at 03:33:15PM +0300, Tomi Valkeinen wrote:
> On 27/09/2022 13:23, Sakari Ailus wrote:
> 
> <snip>
> 
> > > > > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > > > > +means that the userspace mut reconfigure all streams after calling the ioctl
> > > > > +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > > > 
> > > > How about this:
> > > > 
> > > > Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
> > > > formats being propagated from the sink pads towards the sources.
> > > 
> > > Hmm, but that's not true. The selections and formats will be zeroed, unless
> > > the driver initializes them to a value. There's no propagation done.
> > 
> > They need to be propagated. The driver is responsible for maintaining a
> > valid configuration for the processing steps in a sub-device, and with
> > routes that must apply to routes as well.
> 
> Hmm, no, they don't need to be propagated. The driver needs to initialize
> the formats and selections to valid configuration, that is true, but it
> doesn't mean the driver propagates settings from the sink pads to the source
> pads. In theory the formats on sink and source sides could be different.

After propagation, the user may set the format (or selection) later on in
the processing steps. The propagation is required by the spec and I don't
see why it would be different for drivers with support for streams. Of
course this needs to take place taking hardware limitations into account.

> 
> I guess you could say that the driver initializes the sink side streams and
> then propagates those to the source side, but at least to me that gives the
> wrong impression. What the driver does is initialize the configuration, or
> reset the configuration to defaults (which it could do with propagation if
> it so wants).
> 
> The framework provides v4l2_subdev_set_routing_with_fmt() helper to achieve
> this, which sets the given format to all streams.
> 
> Anyway, I think we can talk about propagation if that helps, but I think the
> main point there is that the settings are reset or initialized.
> 
> > > > > +
> > > > > +A special case for routing are routes marked with
> > > > > +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> > > > > +source endpoints on sensors and the sink fields are unused.
> > > > > +
> > > > > +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > > > > +provided ``num_routes`` is not big enough to contain all the available routes
> > > > > +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > > > > +value of the ``num_routes`` field. Application should then reserve enough memory
> > > > > +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > > > > +
> > > > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > > > +
> > > > > +.. c:type:: v4l2_subdev_routing
> > > > > +
> > > > > +.. flat-table:: struct v4l2_subdev_routing
> > > > > +    :header-rows:  0
> > > > > +    :stub-columns: 0
> > > > > +    :widths:       1 1 2
> > > > > +
> > > > > +    * - __u32
> > > > > +      - ``which``
> > > > > +      - Format to modified, from enum
> > > > > +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > > > > +    * - struct :c:type:`v4l2_subdev_route`
> > > > > +      - ``routes[]``
> > > > > +      - Array of struct :c:type:`v4l2_subdev_route` entries
> > > > > +    * - __u32
> > > > > +      - ``num_routes``
> > > > > +      - Number of entries of the routes array
> > > > > +    * - __u32
> > > > > +      - ``reserved``\ [5]
> > > > > +      - Reserved for future extensions. Applications and drivers must set
> > > > > +	the array to zero.
> > > > > +
> > > > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > > > +
> > > > > +.. c:type:: v4l2_subdev_route
> > > > > +
> > > > > +.. flat-table:: struct v4l2_subdev_route
> > > > > +    :header-rows:  0
> > > > > +    :stub-columns: 0
> > > > > +    :widths:       1 1 2
> > > > > +
> > > > > +    * - __u32
> > > > > +      - ``sink_pad``
> > > > > +      - Sink pad number.
> > > > > +    * - __u32
> > > > > +      - ``sink_stream``
> > > > > +      - Sink pad stream number.
> > > > > +    * - __u32
> > > > > +      - ``source_pad``
> > > > > +      - Source pad number.
> > > > > +    * - __u32
> > > > > +      - ``source_stream``
> > > > > +      - Source pad stream number.
> > > > > +    * - __u32
> > > > > +      - ``flags``
> > > > > +      - Route enable/disable flags
> > > > > +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> > > > > +    * - __u32
> > > > > +      - ``reserved``\ [5]
> > > > > +      - Reserved for future extensions. Applications and drivers must set
> > > > > +	the array to zero.
> > > > > +
> > > > > +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> > > > > +
> > > > > +.. _v4l2-subdev-routing-flags:
> > > > > +
> > > > > +.. flat-table:: enum v4l2_subdev_routing_flags
> > > > > +    :header-rows:  0
> > > > > +    :stub-columns: 0
> > > > > +    :widths:       3 1 4
> > > > > +
> > > > > +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > > +      - 0
> > > > > +      - The route is enabled. Set by applications.
> > > > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > > > 
> > > > How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
> > > > link flag to mean a link that may not be changed in any way. In this case
> > > > we rather want to say that the route is always there, albeit you can still
> > > > enable or disable it.
> > > 
> > > If we think there's a need for this, I can add it back and name it static. I
> > > think what it then should mean is that the user can enable/disable it and
> > > also set the stream id, but the route must always exist.
> > 
> > But the static routes are recognised by the stream ID only, aren't they?
> > 
> > I think we'll definitely need a way to determine which routes are always
> > there and which ones can be removed at will.
> 
> That's the V4L2_SUBDEV_ROUTE_FL_SOURCE. V4L2_SUBDEV_ROUTE_FL_SOURCE routes
> are always there, and the user can only enable or disable them.
> 
> This is why I dropped IMMUTABLE, as SOURCE is already immutable (or rather,
> static), and I don't see a need for a non-source route to be static.

We don't know of such a device at the moment but I'm fairly certain they
exist. A number of older CSI-2 receivers do not support directing different
VC/DT pairs to different buffers in any generic sort of way.

Using one flag for two different purposes may thus prove problematic over
time. I'd thus define another for the other purpose. In the worst case it
won't be needed and we can make it obsolete later on.

> 
> > > 
> > > But as I said above, I haven't figured out a use for this.
> > > 
> > > > > +      - 1
> > > > > +      - The route is immutable. Set by the driver.
> > > > > +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> > > > > +      - 2
> > > > > +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> > > > > +        fields are unused. Set by the driver.
> > > > > +
> > > > > +Return Value
> > > > > +============
> > > > > +
> > > > > +On success 0 is returned, on error -1 and the ``errno`` variable is set
> > > > > +appropriately. The generic error codes are described at the
> > > > > +:ref:`Generic Error Codes <gen-errors>` chapter.
> > > > > +
> > > > > +ENOSPC
> > > > > +   The number of provided route entries is less than the available ones.
> > > > 
> > > > What does "available ones" mean in this context? More than is supported?
> > > > Wouldn't E2BIG be the appropriate code in that case?
> > > 
> > > Good question. I don't think I wrote this part =). ENOSPC refers to the case
> > > where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
> > > table. So "available ones" mean the routes in the subdev's routing table,
> > > and "provided route entries" refers to the userspace target routing table.
> > > 
> > > It sounds pretty odd, and obviously needs a clarification.
> > 
> > I think I actually can think what this did mean. It means that the
> > num_routes is smaller than the number of routes in a routing table when
> > G_ROUTING is called. For that I think ENOSPC is the right code actually.
> > 
> > But also I think we need to document what happens when there are too many
> > routes. For that E2BIG would be appropriate.
> 
> v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers
> should, of course, do additional check if needed. In v4l2-ioctl.c it seems
> common to return EINVAL if there's too much data, but we can of course
> define E2BIG for routing ioctls.

The number (256) is just the current limit. I don't expect more though.

But the user space could know the number is too large if we have a proper
error code for it. Up to you. However at least documentation needs to be
amended since this case remains undocumented.

> 
> > Should we have a field for telling which route was the bad one, if it was
> > one of them? That can be done later, too, if we'll ever need it.
> 
> Hmm maybe, although I wonder how often the drivers can say that this
> particular route is the problem, and what would the userspace do with that
> information...
> 
> Do you have any examples in mind?

I think it would be mainly useful for debugging purposes, software as such
probably wouldn't need it. Say, if you have a problem somewhere in your,
say, 256 routes, it could be hard to figure out which one of them is the
faulty one.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

Hi,

On 28/09/2022 00:13, Sakari Ailus wrote:
> Moi,
> 
> On Tue, Sep 27, 2022 at 03:33:15PM +0300, Tomi Valkeinen wrote:
>> On 27/09/2022 13:23, Sakari Ailus wrote:
>>
>> <snip>
>>
>>>>>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>>>>>> +means that the userspace mut reconfigure all streams after calling the ioctl
>>>>>> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>>>>>
>>>>> How about this:
>>>>>
>>>>> Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
>>>>> formats being propagated from the sink pads towards the sources.
>>>>
>>>> Hmm, but that's not true. The selections and formats will be zeroed, unless
>>>> the driver initializes them to a value. There's no propagation done.
>>>
>>> They need to be propagated. The driver is responsible for maintaining a
>>> valid configuration for the processing steps in a sub-device, and with
>>> routes that must apply to routes as well.
>>
>> Hmm, no, they don't need to be propagated. The driver needs to initialize
>> the formats and selections to valid configuration, that is true, but it
>> doesn't mean the driver propagates settings from the sink pads to the source
>> pads. In theory the formats on sink and source sides could be different.
> 
> After propagation, the user may set the format (or selection) later on in
> the processing steps. The propagation is required by the spec and I don't
> see why it would be different for drivers with support for streams. Of
> course this needs to take place taking hardware limitations into account.

I don't disagree with the above, but I still don't see why it matters here.

So do you suggest replacing the current paragraph with your version? Or 
adding a new paragraph?

Why is propagation important here? Isn't the resetting of the 
configuration the important thing?

For me, the propagation concept is important in the case where the user 
configures the sink pads, as then the settings need to be propagated. 
Here it doesn't matter.

>> I guess you could say that the driver initializes the sink side streams and
>> then propagates those to the source side, but at least to me that gives the
>> wrong impression. What the driver does is initialize the configuration, or
>> reset the configuration to defaults (which it could do with propagation if
>> it so wants).
>>
>> The framework provides v4l2_subdev_set_routing_with_fmt() helper to achieve
>> this, which sets the given format to all streams.
>>
>> Anyway, I think we can talk about propagation if that helps, but I think the
>> main point there is that the settings are reset or initialized.
>>
>>>>>> +
>>>>>> +A special case for routing are routes marked with
>>>>>> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
>>>>>> +source endpoints on sensors and the sink fields are unused.
>>>>>> +
>>>>>> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>>>>>> +provided ``num_routes`` is not big enough to contain all the available routes
>>>>>> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
>>>>>> +value of the ``num_routes`` field. Application should then reserve enough memory
>>>>>> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>>>>>> +
>>>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>>>> +
>>>>>> +.. c:type:: v4l2_subdev_routing
>>>>>> +
>>>>>> +.. flat-table:: struct v4l2_subdev_routing
>>>>>> +    :header-rows:  0
>>>>>> +    :stub-columns: 0
>>>>>> +    :widths:       1 1 2
>>>>>> +
>>>>>> +    * - __u32
>>>>>> +      - ``which``
>>>>>> +      - Format to modified, from enum
>>>>>> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>>>>> +    * - struct :c:type:`v4l2_subdev_route`
>>>>>> +      - ``routes[]``
>>>>>> +      - Array of struct :c:type:`v4l2_subdev_route` entries
>>>>>> +    * - __u32
>>>>>> +      - ``num_routes``
>>>>>> +      - Number of entries of the routes array
>>>>>> +    * - __u32
>>>>>> +      - ``reserved``\ [5]
>>>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>>>> +	the array to zero.
>>>>>> +
>>>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>>>> +
>>>>>> +.. c:type:: v4l2_subdev_route
>>>>>> +
>>>>>> +.. flat-table:: struct v4l2_subdev_route
>>>>>> +    :header-rows:  0
>>>>>> +    :stub-columns: 0
>>>>>> +    :widths:       1 1 2
>>>>>> +
>>>>>> +    * - __u32
>>>>>> +      - ``sink_pad``
>>>>>> +      - Sink pad number.
>>>>>> +    * - __u32
>>>>>> +      - ``sink_stream``
>>>>>> +      - Sink pad stream number.
>>>>>> +    * - __u32
>>>>>> +      - ``source_pad``
>>>>>> +      - Source pad number.
>>>>>> +    * - __u32
>>>>>> +      - ``source_stream``
>>>>>> +      - Source pad stream number.
>>>>>> +    * - __u32
>>>>>> +      - ``flags``
>>>>>> +      - Route enable/disable flags
>>>>>> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
>>>>>> +    * - __u32
>>>>>> +      - ``reserved``\ [5]
>>>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>>>> +	the array to zero.
>>>>>> +
>>>>>> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
>>>>>> +
>>>>>> +.. _v4l2-subdev-routing-flags:
>>>>>> +
>>>>>> +.. flat-table:: enum v4l2_subdev_routing_flags
>>>>>> +    :header-rows:  0
>>>>>> +    :stub-columns: 0
>>>>>> +    :widths:       3 1 4
>>>>>> +
>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>>>>> +      - 0
>>>>>> +      - The route is enabled. Set by applications.
>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
>>>>>
>>>>> How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
>>>>> link flag to mean a link that may not be changed in any way. In this case
>>>>> we rather want to say that the route is always there, albeit you can still
>>>>> enable or disable it.
>>>>
>>>> If we think there's a need for this, I can add it back and name it static. I
>>>> think what it then should mean is that the user can enable/disable it and
>>>> also set the stream id, but the route must always exist.
>>>
>>> But the static routes are recognised by the stream ID only, aren't they?
>>>
>>> I think we'll definitely need a way to determine which routes are always
>>> there and which ones can be removed at will.
>>
>> That's the V4L2_SUBDEV_ROUTE_FL_SOURCE. V4L2_SUBDEV_ROUTE_FL_SOURCE routes
>> are always there, and the user can only enable or disable them.
>>
>> This is why I dropped IMMUTABLE, as SOURCE is already immutable (or rather,
>> static), and I don't see a need for a non-source route to be static.
> 
> We don't know of such a device at the moment but I'm fairly certain they
> exist. A number of older CSI-2 receivers do not support directing different
> VC/DT pairs to different buffers in any generic sort of way.

That is true, but the problem there is that the driver often does not 
know the number of streams.

For example, if we have a CSI-2 bridge which, say, has a single input 
and two output pads. It routes VC0 & 2 to the first output and VC1 & 3 
to the second output.

Here it would be easy to say that there are 4 static streams, going as 
described above. But that's not true, as data-types also define streams, 
so we can actually have a lot more streams there.

This, I think, essentially means that static routes can never be defined 
for any subdevice in the middle of the pipeline. The only places where 
we can have static routes are the very beginning and very end of the 
pipeline. For the beginning, i.e. the sensors, we already have source 
streams. But can we have static routes on the end side, basically just 
before the DMA?

If we have a CSI-2 receiver that has a hardcoded handling of the VC & 
DT, how does the userspace configure the routes? The userspace doesn't 
see the VCs or DTs. We could have static routes defined in the receiver 
subdevice, but does that help?

The HW I use, TI's CAL, has the means to configure VC/DT freely. But it 
has 8 DMA engines, and, of course, each stream has to go to a single DMA 
engine. So I think we could say that it has 8 static streams, and the 
user can only enable or disable them. But I'm not sure how much adding a 
new flag for this helps.

> Using one flag for two different purposes may thus prove problematic over
> time. I'd thus define another for the other purpose. In the worst case it
> won't be needed and we can make it obsolete later on.

I'd like to have a clear example of a setup where we need this flag and 
benefit from it before adding it.

In the CAL case I don't see much benefit. I think the only thing it 
gives us is a minimal discovery mechanism for the userspace to 
understand how CAL routes can be configured. I say minimal, as it still 
won't cover it fully as the validity of the routing depends on the 
actual VCs and DTs too (which will be found out only at the stream start 
time).

And this would only give us discovery for the receiver and wouldn't help 
with the bridges.

>>>>
>>>> But as I said above, I haven't figured out a use for this.
>>>>
>>>>>> +      - 1
>>>>>> +      - The route is immutable. Set by the driver.
>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
>>>>>> +      - 2
>>>>>> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
>>>>>> +        fields are unused. Set by the driver.
>>>>>> +
>>>>>> +Return Value
>>>>>> +============
>>>>>> +
>>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>>> +appropriately. The generic error codes are described at the
>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>>> +
>>>>>> +ENOSPC
>>>>>> +   The number of provided route entries is less than the available ones.
>>>>>
>>>>> What does "available ones" mean in this context? More than is supported?
>>>>> Wouldn't E2BIG be the appropriate code in that case?
>>>>
>>>> Good question. I don't think I wrote this part =). ENOSPC refers to the case
>>>> where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
>>>> table. So "available ones" mean the routes in the subdev's routing table,
>>>> and "provided route entries" refers to the userspace target routing table.
>>>>
>>>> It sounds pretty odd, and obviously needs a clarification.
>>>
>>> I think I actually can think what this did mean. It means that the
>>> num_routes is smaller than the number of routes in a routing table when
>>> G_ROUTING is called. For that I think ENOSPC is the right code actually.
>>>
>>> But also I think we need to document what happens when there are too many
>>> routes. For that E2BIG would be appropriate.
>>
>> v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers
>> should, of course, do additional check if needed. In v4l2-ioctl.c it seems
>> common to return EINVAL if there's too much data, but we can of course
>> define E2BIG for routing ioctls.
> 
> The number (256) is just the current limit. I don't expect more though.
> 
> But the user space could know the number is too large if we have a proper
> error code for it. Up to you. However at least documentation needs to be
> amended since this case remains undocumented.

I can change the returned error from EINVAL to E2BIG and document it. 
But everything else in check_array_args return EINVAL, so it would be 
going into different direction.

>>> Should we have a field for telling which route was the bad one, if it was
>>> one of them? That can be done later, too, if we'll ever need it.
>>
>> Hmm maybe, although I wonder how often the drivers can say that this
>> particular route is the problem, and what would the userspace do with that
>> information...
>>
>> Do you have any examples in mind?
> 
> I think it would be mainly useful for debugging purposes, software as such
> probably wouldn't need it. Say, if you have a problem somewhere in your,
> say, 256 routes, it could be hard to figure out which one of them is the
> faulty one.

Maybe... I'm a bit cautious about this, as I think it may be often 
difficult to return a good "bad route" number, and for the user do any 
proper debugging based only on it.

Similar to the pipeline validation, you need to turn on the kernel 
debugs to see where things go wrong. Which is, of course, bad as a 
normal user cannot do that, but I think for real debugging aid we'd need 
to return more than just a single number.

I wonder if something like that would work... A videodev/subdev file 
specific buffer for error strings (with a max size), which the user 
could fetch if it gets an error.

Or perhaps not. Maybe the people who work on v4l2 level things also have 
the means to enable kernel debugs.

  Tomi

Re: [PATCH v14 03/34] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Bingbu Cao]

On 8/31/22 10:13 PM, Tomi Valkeinen wrote:
> V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
> to have an arbitrary amount of streams in a single pad, so preferably
> there should be no hardcoded maximum number.
> 
> However, I believe a reasonable max is 8, which would cover a CSI-2 pad
> with 4 streams of pixel data and 4 streams of metadata.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> ---
>  include/media/v4l2-subdev.h | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 9689f38a0af1..3797b99bb408 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -358,7 +358,11 @@ struct v4l2_mbus_frame_desc_entry {
>  	} bus;
>  };
>  
> -#define V4L2_FRAME_DESC_ENTRY_MAX	4
> + /*
> +  * If this number is too small, it should be dropped altogether and the
> +  * API switched to a dynamic number of frame descriptor entries.
> +  */
> +#define V4L2_FRAME_DESC_ENTRY_MAX	8

The number 8 here is still not enough I think, CSI2 specification already
extended the VC identifier to be at most 5 bits, which support a max of
32 VCs.

Considering the metadata, the number should be larger, it looks like that
we have to switch using dynamic number?

BTW, does this change break the uAPI?

>  
>  /**
>   * enum v4l2_mbus_frame_desc_type - media bus frame description type
> 

-- 
Best regards,
Bingbu Cao

Re: [PATCH v14 20/34] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Sakari Ailus]

Moi,

On Wed, Aug 31, 2022 at 05:13:43PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 87 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 +++++++
>  include/uapi/linux/v4l2-subdev.h      | 52 ++++++++++++++++
>  4 files changed, 185 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index c314025d977e..1c02f935cc6c 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>  
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>  
>  #include <media/media-device.h> /* for media_set_bus_info() */
> @@ -3151,6 +3152,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>  
>  	return ret;
> @@ -3414,8 +3430,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>  
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 90826b956693..af1f53d99507 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -23,6 +23,16 @@
>  #include <media/v4l2-fh.h>
>  #include <media/v4l2-event.h>
>  
> +/*
> + * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
> + * of streams.
> + *
> + * Note that V4L2_FRAME_DESC_ENTRY_MAX is related: V4L2_FRAME_DESC_ENTRY_MAX
> + * restricts the total number of streams in a pad, although the stream ID is
> + * not restricted.
> + */
> +#define V4L2_SUBDEV_MAX_STREAM_ID 63
> +
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
> @@ -417,6 +427,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  	case VIDIOC_SUBDEV_S_SELECTION:
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING:
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
>  	}
>  
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -732,6 +746,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
> +			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
> +				return -EINVAL;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -1015,6 +1101,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  
>  	mutex_destroy(&state->_lock);
>  
> +	kfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index c38de23b7f22..f38943932cfe 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -695,12 +695,26 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @_lock: default for 'lock'
>   * @lock: mutex for the state. May be replaced by the user.
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -711,6 +725,7 @@ struct v4l2_subdev_state {
>  	struct mutex _lock;
>  	struct mutex *lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>  
>  /**
> @@ -763,6 +778,9 @@ struct v4l2_subdev_state {
>   *		     this operation as close as possible to stream on time. The
>   *		     operation shall fail if the pad index it has been called on
>   *		     is not valid or in case of unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -805,6 +823,10 @@ struct v4l2_subdev_pad_ops {
>  			      struct v4l2_mbus_frame_desc *fd);
>  	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);
>  };
>  
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 89af27f50a41..b63b80576dd3 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -24,6 +24,7 @@
>  #ifndef __LINUX_V4L2_SUBDEV_H
>  #define __LINUX_V4L2_SUBDEV_H
>  
> +#include <linux/const.h>
>  #include <linux/ioctl.h>
>  #include <linux/types.h>
>  #include <linux/v4l2-common.h>
> @@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device supports routing and multiplexed streams. */
>  #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
>  
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)

_BITUL() will produce an unsigned long. This isn't necessary.

Please use (1U << 0) instead. Same below.

> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated by the subdevice (usually a sensor), and thus there is no
> + * sink-side endpoint for the route. The sink_pad and sink_stream fields are
> + * unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @num_routes: the total number of routes in the routes array
> + * @routes: pointer to the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u32 num_routes;
> +	__u64 routes;
> +	__u32 reserved[6];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>  
> @@ -206,6 +256,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Sakari Ailus]

Moi,

On Wed, Sep 28, 2022 at 10:54:44AM +0300, Tomi Valkeinen wrote:
> Hi,
> 
> On 28/09/2022 00:13, Sakari Ailus wrote:
> > Moi,
> > 
> > On Tue, Sep 27, 2022 at 03:33:15PM +0300, Tomi Valkeinen wrote:
> > > On 27/09/2022 13:23, Sakari Ailus wrote:
> > > 
> > > <snip>
> > > 
> > > > > > > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > > > > > > +means that the userspace mut reconfigure all streams after calling the ioctl
> > > > > > > +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > > > > > 
> > > > > > How about this:
> > > > > > 
> > > > > > Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
> > > > > > formats being propagated from the sink pads towards the sources.
> > > > > 
> > > > > Hmm, but that's not true. The selections and formats will be zeroed, unless
> > > > > the driver initializes them to a value. There's no propagation done.
> > > > 
> > > > They need to be propagated. The driver is responsible for maintaining a
> > > > valid configuration for the processing steps in a sub-device, and with
> > > > routes that must apply to routes as well.
> > > 
> > > Hmm, no, they don't need to be propagated. The driver needs to initialize
> > > the formats and selections to valid configuration, that is true, but it
> > > doesn't mean the driver propagates settings from the sink pads to the source
> > > pads. In theory the formats on sink and source sides could be different.
> > 
> > After propagation, the user may set the format (or selection) later on in
> > the processing steps. The propagation is required by the spec and I don't
> > see why it would be different for drivers with support for streams. Of
> > course this needs to take place taking hardware limitations into account.
> 
> I don't disagree with the above, but I still don't see why it matters here.

It does. The user needs to be able to rely on the ability of the driver to
maintain valid internal configuration. That user generally has less
information on this than the driver.

> 
> So do you suggest replacing the current paragraph with your version? Or
> adding a new paragraph?

I'm preparing some more documentation related to interaction between
routing, formats and selections. I can include this as well.

> 
> Why is propagation important here? Isn't the resetting of the configuration
> the important thing?
> 
> For me, the propagation concept is important in the case where the user
> configures the sink pads, as then the settings need to be propagated. Here
> it doesn't matter.

In general, if the user has configured something, it should not be changed
by configuring independent albeit somewhat related things. I'm saying that
the routing configuration shouldn't be an exception to how the interface
works in general.

> 
> > > I guess you could say that the driver initializes the sink side streams and
> > > then propagates those to the source side, but at least to me that gives the
> > > wrong impression. What the driver does is initialize the configuration, or
> > > reset the configuration to defaults (which it could do with propagation if
> > > it so wants).
> > > 
> > > The framework provides v4l2_subdev_set_routing_with_fmt() helper to achieve
> > > this, which sets the given format to all streams.
> > > 
> > > Anyway, I think we can talk about propagation if that helps, but I think the
> > > main point there is that the settings are reset or initialized.
> > > 
> > > > > > > +
> > > > > > > +A special case for routing are routes marked with
> > > > > > > +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> > > > > > > +source endpoints on sensors and the sink fields are unused.
> > > > > > > +
> > > > > > > +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
> > > > > > > +provided ``num_routes`` is not big enough to contain all the available routes
> > > > > > > +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> > > > > > > +value of the ``num_routes`` field. Application should then reserve enough memory
> > > > > > > +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
> > > > > > > +
> > > > > > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > > > > > +
> > > > > > > +.. c:type:: v4l2_subdev_routing
> > > > > > > +
> > > > > > > +.. flat-table:: struct v4l2_subdev_routing
> > > > > > > +    :header-rows:  0
> > > > > > > +    :stub-columns: 0
> > > > > > > +    :widths:       1 1 2
> > > > > > > +
> > > > > > > +    * - __u32
> > > > > > > +      - ``which``
> > > > > > > +      - Format to modified, from enum
> > > > > > > +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> > > > > > > +    * - struct :c:type:`v4l2_subdev_route`
> > > > > > > +      - ``routes[]``
> > > > > > > +      - Array of struct :c:type:`v4l2_subdev_route` entries
> > > > > > > +    * - __u32
> > > > > > > +      - ``num_routes``
> > > > > > > +      - Number of entries of the routes array
> > > > > > > +    * - __u32
> > > > > > > +      - ``reserved``\ [5]
> > > > > > > +      - Reserved for future extensions. Applications and drivers must set
> > > > > > > +	the array to zero.
> > > > > > > +
> > > > > > > +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> > > > > > > +
> > > > > > > +.. c:type:: v4l2_subdev_route
> > > > > > > +
> > > > > > > +.. flat-table:: struct v4l2_subdev_route
> > > > > > > +    :header-rows:  0
> > > > > > > +    :stub-columns: 0
> > > > > > > +    :widths:       1 1 2
> > > > > > > +
> > > > > > > +    * - __u32
> > > > > > > +      - ``sink_pad``
> > > > > > > +      - Sink pad number.
> > > > > > > +    * - __u32
> > > > > > > +      - ``sink_stream``
> > > > > > > +      - Sink pad stream number.
> > > > > > > +    * - __u32
> > > > > > > +      - ``source_pad``
> > > > > > > +      - Source pad number.
> > > > > > > +    * - __u32
> > > > > > > +      - ``source_stream``
> > > > > > > +      - Source pad stream number.
> > > > > > > +    * - __u32
> > > > > > > +      - ``flags``
> > > > > > > +      - Route enable/disable flags
> > > > > > > +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> > > > > > > +    * - __u32
> > > > > > > +      - ``reserved``\ [5]
> > > > > > > +      - Reserved for future extensions. Applications and drivers must set
> > > > > > > +	the array to zero.
> > > > > > > +
> > > > > > > +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> > > > > > > +
> > > > > > > +.. _v4l2-subdev-routing-flags:
> > > > > > > +
> > > > > > > +.. flat-table:: enum v4l2_subdev_routing_flags
> > > > > > > +    :header-rows:  0
> > > > > > > +    :stub-columns: 0
> > > > > > > +    :widths:       3 1 4
> > > > > > > +
> > > > > > > +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> > > > > > > +      - 0
> > > > > > > +      - The route is enabled. Set by applications.
> > > > > > > +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> > > > > > 
> > > > > > How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
> > > > > > link flag to mean a link that may not be changed in any way. In this case
> > > > > > we rather want to say that the route is always there, albeit you can still
> > > > > > enable or disable it.
> > > > > 
> > > > > If we think there's a need for this, I can add it back and name it static. I
> > > > > think what it then should mean is that the user can enable/disable it and
> > > > > also set the stream id, but the route must always exist.
> > > > 
> > > > But the static routes are recognised by the stream ID only, aren't they?
> > > > 
> > > > I think we'll definitely need a way to determine which routes are always
> > > > there and which ones can be removed at will.
> > > 
> > > That's the V4L2_SUBDEV_ROUTE_FL_SOURCE. V4L2_SUBDEV_ROUTE_FL_SOURCE routes
> > > are always there, and the user can only enable or disable them.
> > > 
> > > This is why I dropped IMMUTABLE, as SOURCE is already immutable (or rather,
> > > static), and I don't see a need for a non-source route to be static.
> > 
> > We don't know of such a device at the moment but I'm fairly certain they
> > exist. A number of older CSI-2 receivers do not support directing different
> > VC/DT pairs to different buffers in any generic sort of way.
> 
> That is true, but the problem there is that the driver often does not know
> the number of streams.
> 
> For example, if we have a CSI-2 bridge which, say, has a single input and
> two output pads. It routes VC0 & 2 to the first output and VC1 & 3 to the
> second output.
> 
> Here it would be easy to say that there are 4 static streams, going as
> described above. But that's not true, as data-types also define streams, so
> we can actually have a lot more streams there.
> 
> This, I think, essentially means that static routes can never be defined for
> any subdevice in the middle of the pipeline. The only places where we can
> have static routes are the very beginning and very end of the pipeline. For
> the beginning, i.e. the sensors, we already have source streams. But can we
> have static routes on the end side, basically just before the DMA?

If you can have a single route only, isn't that effectively static?
Although in that case the sub-device should not support routing.

> 
> If we have a CSI-2 receiver that has a hardcoded handling of the VC & DT,
> how does the userspace configure the routes? The userspace doesn't see the
> VCs or DTs. We could have static routes defined in the receiver subdevice,
> but does that help?

Good point. I think not.

I guess we would then leave the routes for the user to create and driver to
try to configure the hardware accordingly or fail in link validation?

Perhaps we won't need a static route flag then after all.

> 
> The HW I use, TI's CAL, has the means to configure VC/DT freely. But it has
> 8 DMA engines, and, of course, each stream has to go to a single DMA engine.
> So I think we could say that it has 8 static streams, and the user can only
> enable or disable them. But I'm not sure how much adding a new flag for this
> helps.

Could this be limited by only allowing to create eight routes?

> 
> > Using one flag for two different purposes may thus prove problematic over
> > time. I'd thus define another for the other purpose. In the worst case it
> > won't be needed and we can make it obsolete later on.
> 
> I'd like to have a clear example of a setup where we need this flag and
> benefit from it before adding it.
> 
> In the CAL case I don't see much benefit. I think the only thing it gives us
> is a minimal discovery mechanism for the userspace to understand how CAL
> routes can be configured. I say minimal, as it still won't cover it fully as
> the validity of the routing depends on the actual VCs and DTs too (which
> will be found out only at the stream start time).
> 
> And this would only give us discovery for the receiver and wouldn't help
> with the bridges.
> 
> > > > > 
> > > > > But as I said above, I haven't figured out a use for this.
> > > > > 
> > > > > > > +      - 1
> > > > > > > +      - The route is immutable. Set by the driver.
> > > > > > > +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> > > > > > > +      - 2
> > > > > > > +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> > > > > > > +        fields are unused. Set by the driver.
> > > > > > > +
> > > > > > > +Return Value
> > > > > > > +============
> > > > > > > +
> > > > > > > +On success 0 is returned, on error -1 and the ``errno`` variable is set
> > > > > > > +appropriately. The generic error codes are described at the
> > > > > > > +:ref:`Generic Error Codes <gen-errors>` chapter.
> > > > > > > +
> > > > > > > +ENOSPC
> > > > > > > +   The number of provided route entries is less than the available ones.
> > > > > > 
> > > > > > What does "available ones" mean in this context? More than is supported?
> > > > > > Wouldn't E2BIG be the appropriate code in that case?
> > > > > 
> > > > > Good question. I don't think I wrote this part =). ENOSPC refers to the case
> > > > > where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
> > > > > table. So "available ones" mean the routes in the subdev's routing table,
> > > > > and "provided route entries" refers to the userspace target routing table.
> > > > > 
> > > > > It sounds pretty odd, and obviously needs a clarification.
> > > > 
> > > > I think I actually can think what this did mean. It means that the
> > > > num_routes is smaller than the number of routes in a routing table when
> > > > G_ROUTING is called. For that I think ENOSPC is the right code actually.
> > > > 
> > > > But also I think we need to document what happens when there are too many
> > > > routes. For that E2BIG would be appropriate.
> > > 
> > > v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers
> > > should, of course, do additional check if needed. In v4l2-ioctl.c it seems
> > > common to return EINVAL if there's too much data, but we can of course
> > > define E2BIG for routing ioctls.
> > 
> > The number (256) is just the current limit. I don't expect more though.
> > 
> > But the user space could know the number is too large if we have a proper
> > error code for it. Up to you. However at least documentation needs to be
> > amended since this case remains undocumented.
> 
> I can change the returned error from EINVAL to E2BIG and document it. But
> everything else in check_array_args return EINVAL, so it would be going into
> different direction.

Could this be beneficial in telling the user too many routes have been
configured (as I wrote above)?

> 
> > > > Should we have a field for telling which route was the bad one, if it was
> > > > one of them? That can be done later, too, if we'll ever need it.
> > > 
> > > Hmm maybe, although I wonder how often the drivers can say that this
> > > particular route is the problem, and what would the userspace do with that
> > > information...
> > > 
> > > Do you have any examples in mind?
> > 
> > I think it would be mainly useful for debugging purposes, software as such
> > probably wouldn't need it. Say, if you have a problem somewhere in your,
> > say, 256 routes, it could be hard to figure out which one of them is the
> > faulty one.
> 
> Maybe... I'm a bit cautious about this, as I think it may be often difficult
> to return a good "bad route" number, and for the user do any proper
> debugging based only on it.
> 
> Similar to the pipeline validation, you need to turn on the kernel debugs to
> see where things go wrong. Which is, of course, bad as a normal user cannot
> do that, but I think for real debugging aid we'd need to return more than
> just a single number.
> 
> I wonder if something like that would work... A videodev/subdev file
> specific buffer for error strings (with a max size), which the user could
> fetch if it gets an error.
> 
> Or perhaps not. Maybe the people who work on v4l2 level things also have the
> means to enable kernel debugs.

That's the other way. I haven't checked if there are appropriate debug
prints there currently but it's always easy to add them later.

This is something that could be also added later if needed. I'm not
requesting it to be added now, as I wrote earlier.

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v14 20/34] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hei,

On 30/09/2022 14:05, Sakari Ailus wrote:

>> @@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
>>   /* The v4l2 sub-device supports routing and multiplexed streams. */
>>   #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
>>   
>> +/*
>> + * Is the route active? An active route will start when streaming is enabled
>> + * on a video node.
>> + */
>> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
> 
> _BITUL() will produce an unsigned long. This isn't necessary.
> 
> Please use (1U << 0) instead. Same below.

Ok. I used BIT() originally, but that, of course, isn't available in 
userspace...

You might also have noticed that the next flag is bit 2, not bit 1. Bit 
1 was immutable route, which I dropped, but for the time being I kept 
the numbering to keep the ABI compatibility with my already compiled 
tools. I'll change that one to bit 1 in the next series, or if we decide 
to add a static flag, I'll add the static flag as bit 1.

  Tomi

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

Hei,

On 30/09/2022 14:21, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Sep 28, 2022 at 10:54:44AM +0300, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 28/09/2022 00:13, Sakari Ailus wrote:
>>> Moi,
>>>
>>> On Tue, Sep 27, 2022 at 03:33:15PM +0300, Tomi Valkeinen wrote:
>>>> On 27/09/2022 13:23, Sakari Ailus wrote:
>>>>
>>>> <snip>
>>>>
>>>>>>>> +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
>>>>>>>> +means that the userspace mut reconfigure all streams after calling the ioctl
>>>>>>>> +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
>>>>>>>
>>>>>>> How about this:
>>>>>>>
>>>>>>> Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
>>>>>>> formats being propagated from the sink pads towards the sources.
>>>>>>
>>>>>> Hmm, but that's not true. The selections and formats will be zeroed, unless
>>>>>> the driver initializes them to a value. There's no propagation done.
>>>>>
>>>>> They need to be propagated. The driver is responsible for maintaining a
>>>>> valid configuration for the processing steps in a sub-device, and with
>>>>> routes that must apply to routes as well.
>>>>
>>>> Hmm, no, they don't need to be propagated. The driver needs to initialize
>>>> the formats and selections to valid configuration, that is true, but it
>>>> doesn't mean the driver propagates settings from the sink pads to the source
>>>> pads. In theory the formats on sink and source sides could be different.
>>>
>>> After propagation, the user may set the format (or selection) later on in
>>> the processing steps. The propagation is required by the spec and I don't
>>> see why it would be different for drivers with support for streams. Of
>>> course this needs to take place taking hardware limitations into account.
>>
>> I don't disagree with the above, but I still don't see why it matters here.
> 
> It does. The user needs to be able to rely on the ability of the driver to
> maintain valid internal configuration. That user generally has less
> information on this than the driver.

I had a short chat with Sakari, and it seems there's no real issue here, 
mostly just a misunderstanding.

The action point is to clarify what "reset" means (it means resetting to 
driver default values, keeping the subdev configuration valid). And 
perhaps also highlighting somewhere that when e.g. setting the formats, 
propagation happens with subdevices using routes too.

>> So do you suggest replacing the current paragraph with your version? Or
>> adding a new paragraph?
> 
> I'm preparing some more documentation related to interaction between
> routing, formats and selections. I can include this as well.
> 
>>
>> Why is propagation important here? Isn't the resetting of the configuration
>> the important thing?
>>
>> For me, the propagation concept is important in the case where the user
>> configures the sink pads, as then the settings need to be propagated. Here
>> it doesn't matter.
> 
> In general, if the user has configured something, it should not be changed
> by configuring independent albeit somewhat related things. I'm saying that
> the routing configuration shouldn't be an exception to how the interface
> works in general.
> 
>>
>>>> I guess you could say that the driver initializes the sink side streams and
>>>> then propagates those to the source side, but at least to me that gives the
>>>> wrong impression. What the driver does is initialize the configuration, or
>>>> reset the configuration to defaults (which it could do with propagation if
>>>> it so wants).
>>>>
>>>> The framework provides v4l2_subdev_set_routing_with_fmt() helper to achieve
>>>> this, which sets the given format to all streams.
>>>>
>>>> Anyway, I think we can talk about propagation if that helps, but I think the
>>>> main point there is that the settings are reset or initialized.
>>>>
>>>>>>>> +
>>>>>>>> +A special case for routing are routes marked with
>>>>>>>> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
>>>>>>>> +source endpoints on sensors and the sink fields are unused.
>>>>>>>> +
>>>>>>>> +When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
>>>>>>>> +provided ``num_routes`` is not big enough to contain all the available routes
>>>>>>>> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
>>>>>>>> +value of the ``num_routes`` field. Application should then reserve enough memory
>>>>>>>> +for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
>>>>>>>> +
>>>>>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>>>>>> +
>>>>>>>> +.. c:type:: v4l2_subdev_routing
>>>>>>>> +
>>>>>>>> +.. flat-table:: struct v4l2_subdev_routing
>>>>>>>> +    :header-rows:  0
>>>>>>>> +    :stub-columns: 0
>>>>>>>> +    :widths:       1 1 2
>>>>>>>> +
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``which``
>>>>>>>> +      - Format to modified, from enum
>>>>>>>> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>>>>>>>> +    * - struct :c:type:`v4l2_subdev_route`
>>>>>>>> +      - ``routes[]``
>>>>>>>> +      - Array of struct :c:type:`v4l2_subdev_route` entries
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``num_routes``
>>>>>>>> +      - Number of entries of the routes array
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``reserved``\ [5]
>>>>>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>>>>>> +	the array to zero.
>>>>>>>> +
>>>>>>>> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
>>>>>>>> +
>>>>>>>> +.. c:type:: v4l2_subdev_route
>>>>>>>> +
>>>>>>>> +.. flat-table:: struct v4l2_subdev_route
>>>>>>>> +    :header-rows:  0
>>>>>>>> +    :stub-columns: 0
>>>>>>>> +    :widths:       1 1 2
>>>>>>>> +
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``sink_pad``
>>>>>>>> +      - Sink pad number.
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``sink_stream``
>>>>>>>> +      - Sink pad stream number.
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``source_pad``
>>>>>>>> +      - Source pad number.
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``source_stream``
>>>>>>>> +      - Source pad stream number.
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``flags``
>>>>>>>> +      - Route enable/disable flags
>>>>>>>> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
>>>>>>>> +    * - __u32
>>>>>>>> +      - ``reserved``\ [5]
>>>>>>>> +      - Reserved for future extensions. Applications and drivers must set
>>>>>>>> +	the array to zero.
>>>>>>>> +
>>>>>>>> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
>>>>>>>> +
>>>>>>>> +.. _v4l2-subdev-routing-flags:
>>>>>>>> +
>>>>>>>> +.. flat-table:: enum v4l2_subdev_routing_flags
>>>>>>>> +    :header-rows:  0
>>>>>>>> +    :stub-columns: 0
>>>>>>>> +    :widths:       3 1 4
>>>>>>>> +
>>>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>>>>>>>> +      - 0
>>>>>>>> +      - The route is enabled. Set by applications.
>>>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
>>>>>>>
>>>>>>> How about calling this STATIC instead of IMMUTABLE? IMMUTABLE is used as a
>>>>>>> link flag to mean a link that may not be changed in any way. In this case
>>>>>>> we rather want to say that the route is always there, albeit you can still
>>>>>>> enable or disable it.
>>>>>>
>>>>>> If we think there's a need for this, I can add it back and name it static. I
>>>>>> think what it then should mean is that the user can enable/disable it and
>>>>>> also set the stream id, but the route must always exist.
>>>>>
>>>>> But the static routes are recognised by the stream ID only, aren't they?
>>>>>
>>>>> I think we'll definitely need a way to determine which routes are always
>>>>> there and which ones can be removed at will.
>>>>
>>>> That's the V4L2_SUBDEV_ROUTE_FL_SOURCE. V4L2_SUBDEV_ROUTE_FL_SOURCE routes
>>>> are always there, and the user can only enable or disable them.
>>>>
>>>> This is why I dropped IMMUTABLE, as SOURCE is already immutable (or rather,
>>>> static), and I don't see a need for a non-source route to be static.
>>>
>>> We don't know of such a device at the moment but I'm fairly certain they
>>> exist. A number of older CSI-2 receivers do not support directing different
>>> VC/DT pairs to different buffers in any generic sort of way.
>>
>> That is true, but the problem there is that the driver often does not know
>> the number of streams.
>>
>> For example, if we have a CSI-2 bridge which, say, has a single input and
>> two output pads. It routes VC0 & 2 to the first output and VC1 & 3 to the
>> second output.
>>
>> Here it would be easy to say that there are 4 static streams, going as
>> described above. But that's not true, as data-types also define streams, so
>> we can actually have a lot more streams there.
>>
>> This, I think, essentially means that static routes can never be defined for
>> any subdevice in the middle of the pipeline. The only places where we can
>> have static routes are the very beginning and very end of the pipeline. For
>> the beginning, i.e. the sensors, we already have source streams. But can we
>> have static routes on the end side, basically just before the DMA?
> 
> If you can have a single route only, isn't that effectively static?
> Although in that case the sub-device should not support routing.

Yes, I think that's effectively static (but only in a end side, just 
before DMA, not in a subdev along the pipeline).

>> If we have a CSI-2 receiver that has a hardcoded handling of the VC & DT,
>> how does the userspace configure the routes? The userspace doesn't see the
>> VCs or DTs. We could have static routes defined in the receiver subdevice,
>> but does that help?
> 
> Good point. I think not.
> 
> I guess we would then leave the routes for the user to create and driver to
> try to configure the hardware accordingly or fail in link validation?
> 
> Perhaps we won't need a static route flag then after all.
> 
>>
>> The HW I use, TI's CAL, has the means to configure VC/DT freely. But it has
>> 8 DMA engines, and, of course, each stream has to go to a single DMA engine.
>> So I think we could say that it has 8 static streams, and the user can only
>> enable or disable them. But I'm not sure how much adding a new flag for this
>> helps.
> 
> Could this be limited by only allowing to create eight routes?

Yes, and the driver should do that. But the driver should also verify 
the routing, so that each DMA engine only gets one route.

So here it might be possible to use a STATIC flag for the routes. 
Afaics, the only benefit of that would be to give a hint to the 
userspace about the possible routes. It's difficult to say if it's worth 
the trouble.

>>> Using one flag for two different purposes may thus prove problematic over
>>> time. I'd thus define another for the other purpose. In the worst case it
>>> won't be needed and we can make it obsolete later on.
>>
>> I'd like to have a clear example of a setup where we need this flag and
>> benefit from it before adding it.
>>
>> In the CAL case I don't see much benefit. I think the only thing it gives us
>> is a minimal discovery mechanism for the userspace to understand how CAL
>> routes can be configured. I say minimal, as it still won't cover it fully as
>> the validity of the routing depends on the actual VCs and DTs too (which
>> will be found out only at the stream start time).
>>
>> And this would only give us discovery for the receiver and wouldn't help
>> with the bridges.
>>
>>>>>>
>>>>>> But as I said above, I haven't figured out a use for this.
>>>>>>
>>>>>>>> +      - 1
>>>>>>>> +      - The route is immutable. Set by the driver.
>>>>>>>> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
>>>>>>>> +      - 2
>>>>>>>> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
>>>>>>>> +        fields are unused. Set by the driver.
>>>>>>>> +
>>>>>>>> +Return Value
>>>>>>>> +============
>>>>>>>> +
>>>>>>>> +On success 0 is returned, on error -1 and the ``errno`` variable is set
>>>>>>>> +appropriately. The generic error codes are described at the
>>>>>>>> +:ref:`Generic Error Codes <gen-errors>` chapter.
>>>>>>>> +
>>>>>>>> +ENOSPC
>>>>>>>> +   The number of provided route entries is less than the available ones.
>>>>>>>
>>>>>>> What does "available ones" mean in this context? More than is supported?
>>>>>>> Wouldn't E2BIG be the appropriate code in that case?
>>>>>>
>>>>>> Good question. I don't think I wrote this part =). ENOSPC refers to the case
>>>>>> where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
>>>>>> table. So "available ones" mean the routes in the subdev's routing table,
>>>>>> and "provided route entries" refers to the userspace target routing table.
>>>>>>
>>>>>> It sounds pretty odd, and obviously needs a clarification.
>>>>>
>>>>> I think I actually can think what this did mean. It means that the
>>>>> num_routes is smaller than the number of routes in a routing table when
>>>>> G_ROUTING is called. For that I think ENOSPC is the right code actually.
>>>>>
>>>>> But also I think we need to document what happens when there are too many
>>>>> routes. For that E2BIG would be appropriate.
>>>>
>>>> v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers
>>>> should, of course, do additional check if needed. In v4l2-ioctl.c it seems
>>>> common to return EINVAL if there's too much data, but we can of course
>>>> define E2BIG for routing ioctls.
>>>
>>> The number (256) is just the current limit. I don't expect more though.
>>>
>>> But the user space could know the number is too large if we have a proper
>>> error code for it. Up to you. However at least documentation needs to be
>>> amended since this case remains undocumented.
>>
>> I can change the returned error from EINVAL to E2BIG and document it. But
>> everything else in check_array_args return EINVAL, so it would be going into
>> different direction.
> 
> Could this be beneficial in telling the user too many routes have been
> configured (as I wrote above)?

Yes, I think the driver should return E2BIG if there are too many routes.

But my question is, should v4l2-ioctl.c return E2BIG for > 256 routes? 
That's not how it works for all the other ioctls there, they return 
EINVAL. I don't mind changing that to E2BIG, but usually it's nice if 
the code is consistent.

  Tomi

Re: [PATCH v14 19/34] media: Documentation: Add GS_ROUTING documentation

[Sakari Ailus]

Moi,

On Fri, Sep 30, 2022 at 03:10:16PM +0300, Tomi Valkeinen wrote:
> Hei,
> 
> On 30/09/2022 14:21, Sakari Ailus wrote:
> > Moi,
> > 
> > On Wed, Sep 28, 2022 at 10:54:44AM +0300, Tomi Valkeinen wrote:
> > > Hi,
> > > 
> > > On 28/09/2022 00:13, Sakari Ailus wrote:
> > > > Moi,
> > > > 
> > > > On Tue, Sep 27, 2022 at 03:33:15PM +0300, Tomi Valkeinen wrote:
> > > > > On 27/09/2022 13:23, Sakari Ailus wrote:
> > > > > 
> > > > > <snip>
> > > > > 
> > > > > > > > > +All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
> > > > > > > > > +means that the userspace mut reconfigure all streams after calling the ioctl
> > > > > > > > > +with e.g. ``VIDIOC_SUBDEV_S_FMT``.
> > > > > > > > 
> > > > > > > > How about this:
> > > > > > > > 
> > > > > > > > Calling ``VIDIOC_SUBDEV_S_ROUTING`` will cause the selections and subdev
> > > > > > > > formats being propagated from the sink pads towards the sources.
> > > > > > > 
> > > > > > > Hmm, but that's not true. The selections and formats will be zeroed, unless
> > > > > > > the driver initializes them to a value. There's no propagation done.
> > > > > > 
> > > > > > They need to be propagated. The driver is responsible for maintaining a
> > > > > > valid configuration for the processing steps in a sub-device, and with
> > > > > > routes that must apply to routes as well.
> > > > > 
> > > > > Hmm, no, they don't need to be propagated. The driver needs to initialize
> > > > > the formats and selections to valid configuration, that is true, but it
> > > > > doesn't mean the driver propagates settings from the sink pads to the source
> > > > > pads. In theory the formats on sink and source sides could be different.
> > > > 
> > > > After propagation, the user may set the format (or selection) later on in
> > > > the processing steps. The propagation is required by the spec and I don't
> > > > see why it would be different for drivers with support for streams. Of
> > > > course this needs to take place taking hardware limitations into account.
> > > 
> > > I don't disagree with the above, but I still don't see why it matters here.
> > 
> > It does. The user needs to be able to rely on the ability of the driver to
> > maintain valid internal configuration. That user generally has less
> > information on this than the driver.
> 
> I had a short chat with Sakari, and it seems there's no real issue here,
> mostly just a misunderstanding.
> 
> The action point is to clarify what "reset" means (it means resetting to
> driver default values, keeping the subdev configuration valid). And perhaps
> also highlighting somewhere that when e.g. setting the formats, propagation
> happens with subdevices using routes too.

Thanks. Sometimes e-mail just isn't enough. :-)

...

> > > If we have a CSI-2 receiver that has a hardcoded handling of the VC & DT,
> > > how does the userspace configure the routes? The userspace doesn't see the
> > > VCs or DTs. We could have static routes defined in the receiver subdevice,
> > > but does that help?
> > 
> > Good point. I think not.
> > 
> > I guess we would then leave the routes for the user to create and driver to
> > try to configure the hardware accordingly or fail in link validation?
> > 
> > Perhaps we won't need a static route flag then after all.
> > 
> > > 
> > > The HW I use, TI's CAL, has the means to configure VC/DT freely. But it has
> > > 8 DMA engines, and, of course, each stream has to go to a single DMA engine.
> > > So I think we could say that it has 8 static streams, and the user can only
> > > enable or disable them. But I'm not sure how much adding a new flag for this
> > > helps.
> > 
> > Could this be limited by only allowing to create eight routes?
> 
> Yes, and the driver should do that. But the driver should also verify the
> routing, so that each DMA engine only gets one route.
> 
> So here it might be possible to use a STATIC flag for the routes. Afaics,
> the only benefit of that would be to give a hint to the userspace about the
> possible routes. It's difficult to say if it's worth the trouble.

I'm fine with leaving out the flag. It seems it wouldn't have been as
useful in practice as I originally thought.

> 
> > > > Using one flag for two different purposes may thus prove problematic over
> > > > time. I'd thus define another for the other purpose. In the worst case it
> > > > won't be needed and we can make it obsolete later on.
> > > 
> > > I'd like to have a clear example of a setup where we need this flag and
> > > benefit from it before adding it.
> > > 
> > > In the CAL case I don't see much benefit. I think the only thing it gives us
> > > is a minimal discovery mechanism for the userspace to understand how CAL
> > > routes can be configured. I say minimal, as it still won't cover it fully as
> > > the validity of the routing depends on the actual VCs and DTs too (which
> > > will be found out only at the stream start time).
> > > 
> > > And this would only give us discovery for the receiver and wouldn't help
> > > with the bridges.
> > > 
> > > > > > > 
> > > > > > > But as I said above, I haven't figured out a use for this.
> > > > > > > 
> > > > > > > > > +      - 1
> > > > > > > > > +      - The route is immutable. Set by the driver.
> > > > > > > > > +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> > > > > > > > > +      - 2
> > > > > > > > > +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> > > > > > > > > +        fields are unused. Set by the driver.
> > > > > > > > > +
> > > > > > > > > +Return Value
> > > > > > > > > +============
> > > > > > > > > +
> > > > > > > > > +On success 0 is returned, on error -1 and the ``errno`` variable is set
> > > > > > > > > +appropriately. The generic error codes are described at the
> > > > > > > > > +:ref:`Generic Error Codes <gen-errors>` chapter.
> > > > > > > > > +
> > > > > > > > > +ENOSPC
> > > > > > > > > +   The number of provided route entries is less than the available ones.
> > > > > > > > 
> > > > > > > > What does "available ones" mean in this context? More than is supported?
> > > > > > > > Wouldn't E2BIG be the appropriate code in that case?
> > > > > > > 
> > > > > > > Good question. I don't think I wrote this part =). ENOSPC refers to the case
> > > > > > > where VIDIOC_SUBDEV_G_ROUTING is called without enough space for the routing
> > > > > > > table. So "available ones" mean the routes in the subdev's routing table,
> > > > > > > and "provided route entries" refers to the userspace target routing table.
> > > > > > > 
> > > > > > > It sounds pretty odd, and obviously needs a clarification.
> > > > > > 
> > > > > > I think I actually can think what this did mean. It means that the
> > > > > > num_routes is smaller than the number of routes in a routing table when
> > > > > > G_ROUTING is called. For that I think ENOSPC is the right code actually.
> > > > > > 
> > > > > > But also I think we need to document what happens when there are too many
> > > > > > routes. For that E2BIG would be appropriate.
> > > > > 
> > > > > v4l2-ioctl.c returns EINVAL if there are over 256 routes. The drivers
> > > > > should, of course, do additional check if needed. In v4l2-ioctl.c it seems
> > > > > common to return EINVAL if there's too much data, but we can of course
> > > > > define E2BIG for routing ioctls.
> > > > 
> > > > The number (256) is just the current limit. I don't expect more though.
> > > > 
> > > > But the user space could know the number is too large if we have a proper
> > > > error code for it. Up to you. However at least documentation needs to be
> > > > amended since this case remains undocumented.
> > > 
> > > I can change the returned error from EINVAL to E2BIG and document it. But
> > > everything else in check_array_args return EINVAL, so it would be going into
> > > different direction.
> > 
> > Could this be beneficial in telling the user too many routes have been
> > configured (as I wrote above)?
> 
> Yes, I think the driver should return E2BIG if there are too many routes.
> 
> But my question is, should v4l2-ioctl.c return E2BIG for > 256 routes?
> That's not how it works for all the other ioctls there, they return EINVAL.
> I don't mind changing that to E2BIG, but usually it's nice if the code is
> consistent.

From the user's point of view the "too many routes" condition is the same
independently of whether the information comes from the framework or the
driver. So I think using -E2BIG in the framework is the right thing to do.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v14 20/34] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Sakari Ailus]

On Fri, Sep 30, 2022 at 02:22:40PM +0300, Tomi Valkeinen wrote:
> Hei,
> 
> On 30/09/2022 14:05, Sakari Ailus wrote:
> 
> > > @@ -191,6 +192,55 @@ struct v4l2_subdev_capability {
> > >   /* The v4l2 sub-device supports routing and multiplexed streams. */
> > >   #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
> > > +/*
> > > + * Is the route active? An active route will start when streaming is enabled
> > > + * on a video node.
> > > + */
> > > +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
> > 
> > _BITUL() will produce an unsigned long. This isn't necessary.
> > 
> > Please use (1U << 0) instead. Same below.
> 
> Ok. I used BIT() originally, but that, of course, isn't available in
> userspace...
> 
> You might also have noticed that the next flag is bit 2, not bit 1. Bit 1
> was immutable route, which I dropped, but for the time being I kept the
> numbering to keep the ABI compatibility with my already compiled tools. I'll
> change that one to bit 1 in the next series, or if we decide to add a static
> flag, I'll add the static flag as bit 1.

Sounds good.

-- 
Sakari Ailus

Re: [PATCH v14 03/34] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

Hi,

On 29/09/2022 09:48, Bingbu Cao wrote:
> 
> 
> On 8/31/22 10:13 PM, Tomi Valkeinen wrote:
>> V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
>> to have an arbitrary amount of streams in a single pad, so preferably
>> there should be no hardcoded maximum number.
>>
>> However, I believe a reasonable max is 8, which would cover a CSI-2 pad
>> with 4 streams of pixel data and 4 streams of metadata.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> ---
>>   include/media/v4l2-subdev.h | 6 +++++-
>>   1 file changed, 5 insertions(+), 1 deletion(-)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 9689f38a0af1..3797b99bb408 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -358,7 +358,11 @@ struct v4l2_mbus_frame_desc_entry {
>>   	} bus;
>>   };
>>   
>> -#define V4L2_FRAME_DESC_ENTRY_MAX	4
>> + /*
>> +  * If this number is too small, it should be dropped altogether and the
>> +  * API switched to a dynamic number of frame descriptor entries.
>> +  */
>> +#define V4L2_FRAME_DESC_ENTRY_MAX	8
> 
> The number 8 here is still not enough I think, CSI2 specification already
> extended the VC identifier to be at most 5 bits, which support a max of
> 32 VCs.

Well, 8 frame desc entries is not enough for the "old" CSI2 either as 
there can be a lot of data-types, each a separate stream ("stream" as 
defined by this series) which needs a frame desc.

> Considering the metadata, the number should be larger, it looks like that
> we have to switch using dynamic number?

Do we have a current use case which needs more than 8 streams per pad? 
If not, I'd stay away from this for the time being. This can be changed 
later.

> BTW, does this change break the uAPI?

No, the frame desc is fully internal to the kernel.

  Tomi

Re: [PATCH v14 03/34] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Bingbu Cao]

Tomi, 

On 10/3/22 7:32 PM, Tomi Valkeinen wrote:
> Hi,
> 
> On 29/09/2022 09:48, Bingbu Cao wrote:
>>
>>
>> On 8/31/22 10:13 PM, Tomi Valkeinen wrote:
>>> V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
>>> to have an arbitrary amount of streams in a single pad, so preferably
>>> there should be no hardcoded maximum number.
>>>
>>> However, I believe a reasonable max is 8, which would cover a CSI-2 pad
>>> with 4 streams of pixel data and 4 streams of metadata.
>>>
>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>>> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>> ---
>>>   include/media/v4l2-subdev.h | 6 +++++-
>>>   1 file changed, 5 insertions(+), 1 deletion(-)
>>>
>>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>>> index 9689f38a0af1..3797b99bb408 100644
>>> --- a/include/media/v4l2-subdev.h
>>> +++ b/include/media/v4l2-subdev.h
>>> @@ -358,7 +358,11 @@ struct v4l2_mbus_frame_desc_entry {
>>>       } bus;
>>>   };
>>>   -#define V4L2_FRAME_DESC_ENTRY_MAX    4
>>> + /*
>>> +  * If this number is too small, it should be dropped altogether and the
>>> +  * API switched to a dynamic number of frame descriptor entries.
>>> +  */
>>> +#define V4L2_FRAME_DESC_ENTRY_MAX    8
>>
>> The number 8 here is still not enough I think, CSI2 specification already
>> extended the VC identifier to be at most 5 bits, which support a max of
>> 32 VCs.
> 
> Well, 8 frame desc entries is not enough for the "old" CSI2 either as there can be a lot of data-types, each a separate stream ("stream" as defined by this series) which needs a frame desc.
> 
>> Considering the metadata, the number should be larger, it looks like that
>> we have to switch using dynamic number?
> 
> Do we have a current use case which needs more than 8 streams per pad? If not, I'd stay away from this for the time being. This can be changed later.

Yes, we have a use case which run at maximum 6 streams + 4 meta-data.

> 
>> BTW, does this change break the uAPI?
> 
> No, the frame desc is fully internal to the kernel.

Got it, thanks.
> 
>  Tomi

-- 
Best regards,
Bingbu Cao