[PATCH v12 00/30] v4l: routing and streams support

[PATCH v12 00/30] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

This is v12 of the streams series. The v11 can be found from:

https://lore.kernel.org/all/20220301161156.1119557-1-tomi.valkeinen@ideasonboard.com/

My work branch with additional drivers can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git streams/work-v12

And there's also the v4l-utils series v2 to add support to v4l2-ctl and
media-ctl:

https://lore.kernel.org/all/20220714132116.132498-1-tomi.valkeinen@ideasonboard.com/

Changes in v12:

- Rebased on top of upstream (media graph walk changes, subdev active
  state changes)

- V4L2_SUBDEV_FL_MULTIPLEXED renamed to V4L2_SUBDEV_FL_STREAMS

- V4L2_SUBDEV_CAP_MPLEXED renamed to V4L2_SUBDEV_CAP_STREAMS

- Limit Stream ID number to max 63

- Added v4l2_subdev_state_get_stream_crop() and
  v4l2_subdev_state_get_stream_compose()

- Use BIT_ULL() instead of BIT() for 64bit masks.

- Some documentation improvements

- All the patches related to modifying the graph walk have been dropped
  and there is a new implementation of the pipeline construction in
  __media_pipeline_start.

- The change which made media_pipeline_start() take a pad instead of an
  entity has been dropped.

- New pipeline start/stop functions, media_pipeline_alloc_start() &
  media_pipeline_stop_free()

There are still open discussion items wrt. the uAPI. I will post a mail
in reply to this cover letter about those items.

 Tomi

Jacopo Mondi (3):
  media: mc: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (10):
  media: mc: entity: Merge media_entity_enum_init and
    __media_entity_enum_init
  media: mc: entity: Move media_entity_get_fwnode_pad() out of graph
    walk section
  media: mc: entity: Add media_entity_pipeline() to access the media
    pipeline
  media: mc: entity: Add has_route entity operation and
    media_entity_has_route() helper
  media: mc: entity: Rewrite media_pipeline_start() to support routes
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations
  media: subdev: add v4l2_subdev_routing_validate() helper
  media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper
  media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations
  media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

Sakari Ailus (1):
  media: Add stream to frame descriptor

Tomi Valkeinen (16):
  media: Documentation: mc: add definitions for stream and pipeline
  media: mc: entity: Rename streaming_count -> start_count
  media: mc: entity: add media_pipeline_alloc_start &
    media_pipeline_stop_free
  media: add V4L2_SUBDEV_FL_STREAMS
  media: add V4L2_SUBDEV_CAP_MPLEXED
  media: subdev: add v4l2_subdev_has_route()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add streams to v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
  media: subdev: use for_each_active_route() in
    v4l2_subdev_init_stream_configs()
  media: subdev: use for_each_active_route() in
    v4l2_link_validate_get_streams()
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

 .clang-format                                 |    1 +
 Documentation/driver-api/media/mc-core.rst    |   19 +-
 .../driver-api/media/v4l2-subdev.rst          |    8 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  175 +++
 .../userspace-api/media/v4l/user-func.rst     |    1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |    5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |    5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |    5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |    5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |    5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |    5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |  150 +++
 .../media/v4l/vidioc-subdev-g-selection.rst   |    5 +-
 drivers/media/mc/mc-device.c                  |   13 +-
 drivers/media/mc/mc-entity.c                  |  646 +++++++++--
 .../platform/renesas/rcar-vin/rcar-core.c     |    5 +-
 .../platform/renesas/rcar-vin/rcar-dma.c      |    2 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |    4 +-
 drivers/media/platform/ti/omap3isp/isp.c      |    4 +-
 drivers/media/platform/ti/omap3isp/ispvideo.c |    3 +-
 drivers/media/platform/ti/omap3isp/ispvideo.h |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |    3 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |    7 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 1033 ++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |    2 +-
 drivers/staging/media/omap4iss/iss.c          |    4 +-
 drivers/staging/media/omap4iss/iss_video.c    |    3 +-
 drivers/staging/media/omap4iss/iss_video.h    |   11 +-
 include/media/media-device.h                  |   15 -
 include/media/media-entity.h                  |  130 ++-
 include/media/v4l2-subdev.h                   |  381 +++++-
 include/uapi/linux/v4l2-subdev.h              |   88 +-
 33 files changed, 2559 insertions(+), 220 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.34.1

[PATCH v12 01/30] media: Documentation: mc: add definitions for stream and pipeline

[Tomi Valkeinen]

The doc talks about streams and pipelines, but doesn't really define
them. This is an attempt to define them according to my understanding.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 02481a2513b9..2a0c0aeec5f2 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,6 +213,18 @@ Link properties can be modified at runtime by calling
 Pipelines and media streams
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+A media stream is a stream of pixels or metadata originating from one or more
+source devices (such as a sensors) and flowing through media entity pads
+towards the final sinks. The stream can be modified on the route by the
+devices (e.g. scaling or pixel format conversions), or it can be split into
+multiple branches, or multiple branches can be merged.
+
+A media pipeline is a set of media streams which are interdependent. This
+interdependency can be caused by the hardware (e.g. configuration of a second
+stream cannot be changed if the first stream has been enabled) or by the driver
+due to the software design. Most commonly a media pipeline consists of a single
+stream which does not branch.
+
 When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
-- 
2.34.1

[PATCH v12 02/30] media: mc: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index b8176a3b76d3..25020d58eb06 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -610,7 +610,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -639,9 +639,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 11f5207f73aa..35a1aa218c51 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -192,7 +192,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -203,12 +204,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					&iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a9a1c0ec5d1c..6b6fea6a713e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -315,6 +315,18 @@ struct media_entity {
 	} info;
 };
 
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
+
 /**
  * struct media_interface - A media interface graph object.
  *
-- 
2.34.1

[PATCH v12 03/30] media: mc: entity: Merge media_entity_enum_init and __media_entity_enum_init

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_enum_init() function is a wrapper around
__media_entity_enum_init() that turns a media_device pointer argument
into the maximum entity ID in the corresponding media graph.
__media_entity_enum_init() is never used outside of
media_entity_enum_init(), so the two functions can be merged together.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 10 ++++++----
 include/media/media-device.h | 15 ---------------
 include/media/media-entity.h | 10 +++++-----
 3 files changed, 11 insertions(+), 24 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 35a1aa218c51..c810b5f45d82 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -58,10 +58,12 @@ static inline const char *link_type_name(struct media_link *link)
 	}
 }
 
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max)
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev)
 {
-	idx_max = ALIGN(idx_max, BITS_PER_LONG);
+	int idx_max;
+
+	idx_max = ALIGN(mdev->entity_internal_idx_max + 1, BITS_PER_LONG);
 	ent_enum->bmap = bitmap_zalloc(idx_max, GFP_KERNEL);
 	if (!ent_enum->bmap)
 		return -ENOMEM;
@@ -70,7 +72,7 @@ __must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(__media_entity_enum_init);
+EXPORT_SYMBOL_GPL(media_entity_enum_init);
 
 void media_entity_enum_cleanup(struct media_entity_enum *ent_enum)
 {
diff --git a/include/media/media-device.h b/include/media/media-device.h
index a10b30507524..86716ee7cc6c 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -191,21 +191,6 @@ struct usb_device;
 #define MEDIA_DEV_NOTIFY_PRE_LINK_CH	0
 #define MEDIA_DEV_NOTIFY_POST_LINK_CH	1
 
-/**
- * media_entity_enum_init - Initialise an entity enumeration
- *
- * @ent_enum: Entity enumeration to be initialised
- * @mdev: The related media device
- *
- * Return: zero on success or a negative error code.
- */
-static inline __must_check int media_entity_enum_init(
-	struct media_entity_enum *ent_enum, struct media_device *mdev)
-{
-	return __media_entity_enum_init(ent_enum,
-					mdev->entity_internal_idx_max + 1);
-}
-
 /**
  * media_device_init() - Initializes a media device element
  *
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 6b6fea6a713e..4e4fec60bf97 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -438,15 +438,15 @@ static inline bool is_media_entity_v4l2_subdev(struct media_entity *entity)
 }
 
 /**
- * __media_entity_enum_init - Initialise an entity enumeration
+ * media_entity_enum_init - Initialise an entity enumeration
  *
  * @ent_enum: Entity enumeration to be initialised
- * @idx_max: Maximum number of entities in the enumeration
+ * @mdev: The related media device
  *
- * Return: Returns zero on success or a negative error code.
+ * Return: zero on success or a negative error code.
  */
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max);
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev);
 
 /**
  * media_entity_enum_cleanup - Release resources of an entity enumeration
-- 
2.34.1

[PATCH v12 04/30] media: mc: entity: Move media_entity_get_fwnode_pad() out of graph walk section

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_get_fwnode_pad() function is unrelated to the graph
traversal code that it is currently bundled with. Move it with the
media_entity_remote_pad() function.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 70 ++++++++++++++++++------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index c810b5f45d82..9f4a1c98dc43 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -369,41 +369,6 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
-int media_entity_get_fwnode_pad(struct media_entity *entity,
-				struct fwnode_handle *fwnode,
-				unsigned long direction_flags)
-{
-	struct fwnode_endpoint endpoint;
-	unsigned int i;
-	int ret;
-
-	if (!entity->ops || !entity->ops->get_fwnode_pad) {
-		for (i = 0; i < entity->num_pads; i++) {
-			if (entity->pads[i].flags & direction_flags)
-				return i;
-		}
-
-		return -ENXIO;
-	}
-
-	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
-	if (ret)
-		return ret;
-
-	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
-	if (ret < 0)
-		return ret;
-
-	if (ret >= entity->num_pads)
-		return -ENXIO;
-
-	if (!(entity->pads[ret].flags & direction_flags))
-		return -ENXIO;
-
-	return ret;
-}
-EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
-
 /* -----------------------------------------------------------------------------
  * Pipeline management
  */
@@ -923,6 +888,41 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad)
 }
 EXPORT_SYMBOL_GPL(media_entity_remote_pad);
 
+int media_entity_get_fwnode_pad(struct media_entity *entity,
+				struct fwnode_handle *fwnode,
+				unsigned long direction_flags)
+{
+	struct fwnode_endpoint endpoint;
+	unsigned int i;
+	int ret;
+
+	if (!entity->ops || !entity->ops->get_fwnode_pad) {
+		for (i = 0; i < entity->num_pads; i++) {
+			if (entity->pads[i].flags & direction_flags)
+				return i;
+		}
+
+		return -ENXIO;
+	}
+
+	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
+	if (ret)
+		return ret;
+
+	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
+	if (ret < 0)
+		return ret;
+
+	if (ret >= entity->num_pads)
+		return -ENXIO;
+
+	if (!(entity->pads[ret].flags & direction_flags))
+		return -ENXIO;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
-- 
2.34.1

[PATCH v12 05/30] media: mc: entity: Add media_entity_pipeline() to access the media pipeline

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Replace direct access to the pipe field in drivers with a new helper
function. This will allow easier refactoring of media pipeline handling
in the MC core behind the scenes without affecting drivers.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                   |  6 ++++++
 .../platform/renesas/rcar-vin/rcar-core.c      |  5 ++---
 .../media/platform/renesas/rcar-vin/rcar-dma.c |  2 +-
 drivers/media/platform/ti/omap3isp/isp.c       |  4 +---
 drivers/media/platform/ti/omap3isp/ispvideo.c  |  3 +--
 drivers/media/platform/ti/omap3isp/ispvideo.h  | 11 +++++++++--
 drivers/media/platform/xilinx/xilinx-dma.c     |  3 +--
 drivers/media/platform/xilinx/xilinx-dma.h     |  7 ++++++-
 drivers/staging/media/imx/imx-media-utils.c    |  2 +-
 drivers/staging/media/omap4iss/iss.c           |  4 +---
 drivers/staging/media/omap4iss/iss_video.c     |  3 +--
 drivers/staging/media/omap4iss/iss_video.h     | 11 +++++++++--
 include/media/media-entity.h                   | 18 ++++++++++++++++++
 13 files changed, 57 insertions(+), 22 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 9f4a1c98dc43..50872d953cf9 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -923,6 +923,12 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
+{
+	return entity->pipe;
+}
+EXPORT_SYMBOL_GPL(media_entity_pipeline);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-core.c b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
index 49bdcfba010b..7b12af3ed8fb 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
@@ -786,9 +786,8 @@ static int rvin_csi2_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
 	media_device_for_each_entity(entity, &group->mdev)
 		if (media_entity_is_streaming(entity))
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 6644b498929d..924907b71263 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1281,7 +1281,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/isp.c b/drivers/media/platform/ti/omap3isp/isp.c
index 4c937f3f323e..2b5310b07bc3 100644
--- a/drivers/media/platform/ti/omap3isp/isp.c
+++ b/drivers/media/platform/ti/omap3isp/isp.c
@@ -937,10 +937,8 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_isp_pipeline(me);
-	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_entity_remote_pad(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
index 8811d6dd4ee7..44b0d55ee5d8 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.c
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
@@ -1093,8 +1093,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
-	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
+	pipe = to_isp_pipeline(&video->video.entity) ? : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.h b/drivers/media/platform/ti/omap3isp/ispvideo.h
index a0908670c0cf..1d23df576e6b 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.h
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.h
@@ -99,8 +99,15 @@ struct isp_pipeline {
 	unsigned int external_width;
 };
 
-#define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+static inline struct isp_pipeline *to_isp_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct isp_pipeline, pipe);
+}
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 338c3661d809..72eff5ef626b 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,8 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
-	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
+	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
 	if (ret < 0)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..3ea10f6b0bb9 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,12 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	struct media_pipeline *pipe = media_entity_pipeline(e);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 94bc866ca28c..5d9c6fc6f2e0 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -871,7 +871,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (media_entity_pipeline(entity))
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..77760d913b49 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,10 +548,8 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_iss_pipeline(me);
-	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_entity_remote_pad(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..67d63a400fa2 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -870,8 +870,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = to_iss_pipeline(&video->video.entity) ? : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..ca2d5edb6261 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -90,8 +90,15 @@ struct iss_pipeline {
 	int external_bpp;
 };
 
-#define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+static inline struct iss_pipeline *to_iss_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct iss_pipeline, pipe);
+}
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 4e4fec60bf97..96f5fcda1985 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -883,6 +883,24 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
 	return entity->pipe;
 }
 
+/**
+ * media_entity_pipeline - Get the media pipeline an entity is part of
+ * @entity: The entity
+ *
+ * This function returns the media pipeline that an entity has been associated
+ * with when constructing the pipeline with media_pipeline_start(). The pointer
+ * remains valid until media_pipeline_stop() is called.
+ *
+ * In general, entities can be part of multiple pipelines, when carrying
+ * multiple streams (either on different pads, or on the same pad using
+ * multiplexed streams). This function is ill-defined in that case. It
+ * currently returns the pipeline associated with the first pad of the entity.
+ *
+ * Return: The media_pipeline the entity is part of, or NULL if the entity is
+ * not part of any pipeline.
+ */
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
+
 /**
  * media_entity_get_fwnode_pad - Get pad number from fwnode
  *
-- 
2.34.1

[PATCH v12 06/30] media: mc: entity: Add has_route entity operation and media_entity_has_route() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The optional operation can be used by entities to report whether a
source pad and a sink pad are internally connected.

media_entity_has_route() is a wrapper around the media entity has_route operation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 37 ++++++++++++++++++++++++++++++++++++
 include/media/media-entity.h |  8 ++++++++
 2 files changed, 45 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 50872d953cf9..4f02f1247762 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -225,6 +225,43 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+/**
+ * media_entity_has_route - Check if two entity pads are connected internally
+ *
+ * @entity: The entity
+ * @pad0: The first pad index
+ * @pad1: The second pad index
+ *
+ * This function uses the &media_entity_operations.has_route() operation to
+ * check connectivity inside the entity between @pad0 and @pad1.
+ *
+ * One of @pad0 and @pad1 must be a sink pad and the other one a source pad.
+ * The function returns 0 if both pads are sinks or sources.
+ *
+ * If the has_route operation is not implemented, all pads of the entity are
+ * considered as connected.
+ *
+ * The caller must hold entity->graph_obj.mdev->mutex.
+ *
+ * Return: true if the pads are connected internally and false otherwise.
+ */
+__maybe_unused
+static bool media_entity_has_route(struct media_entity *entity,
+				   unsigned int pad0, unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (entity->pads[pad0].flags & entity->pads[pad1].flags &
+	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
+		return false;
+
+	if (!entity->ops || !entity->ops->has_route)
+		return true;
+
+	return entity->ops->has_route(entity, pad0, pad1);
+}
+
 static struct media_entity *
 media_entity_other(struct media_entity *entity, struct media_link *link)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 96f5fcda1985..5764aadd2c42 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -206,6 +206,12 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_route:		Return whether a route exists inside the entity between
+ *			pad0 and pad1. pad0 and pad1 are guaranteed to not both
+ *			be sinks or sources. Never call the .has_route()
+ *			operation directly, always use media_entity_has_route().
+ *			Optional: If the operation isn't implemented all pads
+ *			will be considered as connected, with the same streams.
  *
  * .. note::
  *
@@ -219,6 +225,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
+			  unsigned int pad1);
 };
 
 /**
-- 
2.34.1

[PATCH v12 07/30] media: mc: entity: Rename streaming_count -> start_count

[Tomi Valkeinen]

'streaming_count' is a bit misleading name, as the count is increased
with media_pipeline_start(). Let's rename it to 'start_count' instead.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                            | 8 ++++----
 drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c | 4 ++--
 include/media/media-entity.h                            | 4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 4f02f1247762..0d0d6c0dda16 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -419,8 +419,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	struct media_link *link;
 	int ret;
 
-	if (pipe->streaming_count) {
-		pipe->streaming_count++;
+	if (pipe->start_count) {
+		pipe->start_count++;
 		return 0;
 	}
 
@@ -503,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		}
 	}
 
-	pipe->streaming_count++;
+	pipe->start_count++;
 
 	return 0;
 
@@ -556,7 +556,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	if (--pipe->streaming_count)
+	if (--pipe->start_count)
 		return;
 
 	media_graph_walk_start(graph, entity);
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index fee2aaacb26b..0ebd70275e23 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -931,7 +931,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes shouldn't
 	 * be disabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count < 2) {
+	if (rkisp1->pipe.start_count < 2) {
 		v4l2_subdev_call(rkisp1->active_sensor->sd, video, s_stream,
 				 false);
 		v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, false);
@@ -964,7 +964,7 @@ static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes are already
 	 * enabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count > 1)
+	if (rkisp1->pipe.start_count > 1)
 		return 0;
 
 	ret = v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, true);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5764aadd2c42..6c0d0a00d58e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -99,11 +99,11 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
- * @streaming_count:	Streaming start count - streaming stop count
+ * @start_count:	Media pipeline start - stop count
  * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
-	int streaming_count;
+	int start_count;
 	struct media_graph graph;
 };
 
-- 
2.34.1

[PATCH v12 08/30] media: mc: entity: add media_pipeline_alloc_start & media_pipeline_stop_free

[Tomi Valkeinen]

Add new versions of media_pipeline_start() and media_pipeline_stop().
The new functions can be used by drivers that do not need to extend the
media_pipeline with their own structs, and the new functions will handle
allocating and freeing the media_pipeline as needed.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 49 ++++++++++++++++++++++++++++++++++++
 include/media/media-entity.h | 22 ++++++++++++++++
 2 files changed, 71 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 0d0d6c0dda16..b7b6c6411aa7 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -579,6 +579,55 @@ void media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
+__must_check int media_pipeline_alloc_start(struct media_entity *entity)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pipeline *pipe;
+	int ret;
+	bool new_pipe = false;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	/*
+	 * Is the entity already part of a pipeline? If not, we need to allocate
+	 * a pipe.
+	 */
+	pipe = media_entity_pipeline(entity);
+	if (!pipe) {
+		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
+		new_pipe = true;
+	}
+
+	ret = __media_pipeline_start(entity, pipe);
+	if (ret) {
+		if (new_pipe)
+			kfree(pipe);
+	}
+
+	mutex_unlock(&mdev->graph_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
+
+void media_pipeline_stop_free(struct media_entity *entity)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pipeline *pipe;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	pipe = media_entity_pipeline(entity);
+
+	__media_pipeline_stop(entity);
+
+	if (pipe && pipe->start_count == 0)
+		kfree(pipe);
+
+	mutex_unlock(&mdev->graph_mutex);
+}
+EXPORT_SYMBOL_GPL(media_pipeline_stop_free);
+
 /* -----------------------------------------------------------------------------
  * Links management
  */
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 6c0d0a00d58e..13a882a7839c 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1035,6 +1035,28 @@ void media_pipeline_stop(struct media_entity *entity);
  */
 void __media_pipeline_stop(struct media_entity *entity);
 
+/**
+ * media_pipeline_alloc_start - Mark a pipeline as streaming
+ * @entity: Starting entity
+ *
+ * media_pipeline_alloc_start() is similar to media_pipeline_start() but
+ * instead of working on a given pipeline the function will allocate a new
+ * pipeline if needed.
+ *
+ * Calls to media_pipeline_alloc_start() must be matched with
+ * media_pipeline_stop_free().
+ */
+__must_check int media_pipeline_alloc_start(struct media_entity *entity);
+
+/**
+ * media_pipeline_stop_free - Mark a pipeline as not streaming
+ * @entity: Starting entity
+ *
+ * media_pipeline_stop_free() is similar to media_pipeline_stop() but will
+ * also free the pipeline when the start_count drops to 0.
+ */
+void media_pipeline_stop_free(struct media_entity *entity);
+
 /**
  * media_devnode_create() - creates and initializes a device node interface
  *
-- 
2.34.1

[PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

[Note: the code is mostly from Laurent but the patch description is from Tomi]

The media_pipeline_start() and media_pipeline_stop() functions use the
media graph walk API to traverse the graph and validate the pipeline.
The graph walk traverses the media graph following links between the
entities.

Also, while the pipeline can't change between the start and stop calls,
the graph is walked again from scratch at stop time, or any time a
driver needs to inspect the pipeline.

With the upcoming multiplexed streams support we will need a bit more
intelligent pipeline construction, as e.g. an entity may be passing
through two independent streams via separate pads, in which case those
pads should not be part of the same pipeline.

This patch essentially rewrites the media_pipeline_start/stop so that
a pipeline is defined as a set of pads instead of entities and the media
graph traversal uses the newly introduced has_route operation.

If an entity does not define has_route operation, all the entity's pads
are considered as connected. This means that the behavior with all the
current drivers stays the same, but new drivers can define a more
fine-grained pipeline construction.

Additionally the media pipeline's pads are cached at
media_pipeline_start() time, and re-used at media_pipeline_stop() which
avoid the need to re-walk the whole graph as the previous implementation
did.

Additionally, caching pads in the pipeline can serve in the future as
the foundation to provide a better API than the media graph walk to
drivers to iterate over pads and entities in the pipeline.

Note that the old media_pipeline_start/stop used the media graph walk
API. The new version does not use the media graph walk API, but instead
a new implementation.

There are two reason for not changing the graph walk: it proved to be
rather difficult to change the graph walk to have the features
implemented in this patch, and second, this keeps the backward
compatibility of the graph walk as there are users of the graph walk API

The long term plan is that all the existing code would be converted to
use the new cached pipeline, thus allowing us to remove the graph walk.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst |   7 +-
 drivers/media/mc/mc-entity.c               | 483 +++++++++++++++++----
 include/media/media-entity.h               |  56 ++-
 3 files changed, 460 insertions(+), 86 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 2a0c0aeec5f2..9c0b09745c0a 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -229,14 +229,13 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads which are part of the pipeline as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
+pipeline through the struct media_pad
 pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b7b6c6411aa7..ca8845bf8d48 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -410,97 +410,436 @@ EXPORT_SYMBOL_GPL(media_graph_walk_next);
  * Pipeline management
  */
 
+/*
+ * The pipeline traversal stack stores pads that are reached during graph
+ * traversal, with a list of links to be visited to continue the traversal.
+ * When a new pad is reached, an entry is pushed on the top of the stack and
+ * points to the incoming pad and the first link of the entity.
+ *
+ * To find further pads in the pipeline, the traversal algorithm follows
+ * internal routes in the entity, and then links in the graph. It does so by
+ * iterating over all links of the entity, and following enabled links that
+ * originate from a pad that is internally connected to the incoming pad, as
+ * reported by the media_entity_has_route() function.
+ */
+
+/**
+ * struct media_pipeline_walk_entry - Entry in the pipeline traversal stack
+ *
+ * @pad: The media pad being visited
+ * @links: Links left to be visited
+ */
+struct media_pipeline_walk_entry {
+	struct media_pad *pad;
+	struct list_head *links;
+};
+
+/**
+ * struct media_pipeline_walk - State used by the media pipeline traversal
+ * 				algorithm
+ *
+ * @mdev: The media device
+ * @stack: Depth-first search stack
+ * @stack.size: Number of allocated entries in @stack.entries
+ * @stack.top: Index of the top stack entry (-1 if the stack is empty)
+ * @stack.entries: Stack entries
+ */
+struct media_pipeline_walk {
+	struct media_device *mdev;
+
+	struct {
+		unsigned int size;
+		int top;
+		struct media_pipeline_walk_entry *entries;
+	} stack;
+};
+
+#define MEDIA_PIPELINE_STACK_GROW_STEP		16
+
+static struct media_pipeline_walk_entry *
+media_pipeline_walk_top(struct media_pipeline_walk *walk)
+{
+	return &walk->stack.entries[walk->stack.top];
+}
+
+static bool media_pipeline_walk_empty(struct media_pipeline_walk *walk)
+{
+	return walk->stack.top == -1;
+}
+
+/* Increase the stack size by MEDIA_PIPELINE_STACK_GROW_STEP elements. */
+static int media_pipeline_walk_resize(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entries;
+	unsigned int new_size;
+
+	/* Safety check, to avoid stack overflows in case of bugs. */
+	if (walk->stack.size >= 256)
+		return -E2BIG;
+
+	new_size = walk->stack.size + MEDIA_PIPELINE_STACK_GROW_STEP;
+
+	entries = krealloc(walk->stack.entries,
+			   new_size * sizeof(*walk->stack.entries),
+			   GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	walk->stack.entries = entries;
+	walk->stack.size = new_size;
+
+	return 0;
+}
+
+/* Push a new entry on the stack. */
+static int media_pipeline_walk_push(struct media_pipeline_walk *walk,
+				    struct media_pad *pad)
+{
+	struct media_pipeline_walk_entry *entry;
+	int ret;
+
+	if (walk->stack.top + 1 >= walk->stack.size) {
+		ret = media_pipeline_walk_resize(walk);
+		if (ret)
+			return ret;
+	}
+
+	walk->stack.top++;
+	entry = media_pipeline_walk_top(walk);
+	entry->pad = pad;
+	entry->links = pad->entity->links.next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: pushed entry %u: '%s':%u\n",
+		walk->stack.top, pad->entity->name, pad->index);
+
+	return 0;
+}
+
+/*
+ * Move the top entry link cursor to the next link. If all links of the entry
+ * have been visited, pop the entry itself.
+ */
+static void media_pipeline_walk_pop(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry;
+
+	if (WARN_ON(walk->stack.top < 0))
+		return;
+
+	entry = media_pipeline_walk_top(walk);
+
+	if (entry->links->next == &entry->pad->entity->links) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: entry %u has no more links, popping\n",
+			walk->stack.top);
+
+		walk->stack.top--;
+		return;
+	}
+
+	entry->links = entry->links->next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: moved entry %u to next link\n",
+		walk->stack.top);
+}
+
+/* Free all memory allocated while walking the pipeline. */
+static void media_pipeline_walk_destroy(struct media_pipeline_walk *walk)
+{
+	kfree(walk->stack.entries);
+}
+
+/* Add a pad to the pipeline and push it to the stack. */
+static int media_pipeline_add_pad(struct media_pipeline *pipe,
+				  struct media_pipeline_walk *walk,
+				  struct media_pad *pad)
+{
+	struct media_pipeline_pad *ppad;
+
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		if (ppad->pad == pad) {
+			dev_dbg(pad->graph_obj.mdev->dev,
+				"media pipeline: already contains pad '%s':%u\n",
+				pad->entity->name, pad->index);
+			return 0;
+		}
+	}
+
+	ppad = kzalloc(sizeof(*ppad), GFP_KERNEL);
+	if (!ppad)
+		return -ENOMEM;
+
+	ppad->pipe = pipe;
+	ppad->pad = pad;
+
+	list_add_tail(&ppad->list, &pipe->pads);
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline: added pad '%s':%u\n",
+		pad->entity->name, pad->index);
+
+	return media_pipeline_walk_push(walk, pad);
+}
+
+/* Explore the next link of the entity at the top of the stack. */
+static int media_pipeline_explore_next_link(struct media_pipeline *pipe,
+					    struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry = media_pipeline_walk_top(walk);
+	struct media_pad *pad;
+	struct media_link *link;
+	struct media_pad *local;
+	struct media_pad *remote;
+	int ret;
+
+	pad = entry->pad;
+	link = list_entry(entry->links, typeof(*link), list);
+	media_pipeline_walk_pop(walk);
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: exploring link '%s':%u -> '%s':%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	/* Skip links that are not enabled. */
+	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (disabled)\n");
+		return 0;
+	}
+
+	/* Get the local pad and remote pad. */
+	if (link->source->entity == pad->entity) {
+		local = link->source;
+		remote = link->sink;
+	} else {
+		local = link->sink;
+		remote = link->source;
+	}
+
+	/*
+	 * Skip links that originate from a different pad than the incoming pad
+	 * that is not connected internally in the entity to the incoming pad.
+	 */
+	if (pad != local &&
+	    !media_entity_has_route(pad->entity, pad->index, local->index)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (no route)\n");
+		return 0;
+	}
+
+	/*
+	 * Add the local and remote pads of the link to the pipeline and push
+	 * them to the stack, if they're not already present.
+	 */
+	ret = media_pipeline_add_pad(pipe, walk, local);
+	if (ret)
+		return ret;
+
+	ret = media_pipeline_add_pad(pipe, walk, remote);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void media_pipeline_cleanup(struct media_pipeline *pipe)
+{
+	while (!list_empty(&pipe->pads)) {
+		struct media_pipeline_pad *ppad;
+
+		ppad = list_first_entry(&pipe->pads, typeof(*ppad), list);
+		list_del(&ppad->list);
+		kfree(ppad);
+	}
+}
+
+static int media_pipeline_populate(struct media_pipeline *pipe,
+				   struct media_pad *pad)
+{
+	struct media_pipeline_walk walk = { };
+	struct media_pipeline_pad *ppad;
+	int ret;
+
+	/*
+	 * Populate the media pipeline by walking the media graph, starting
+	 * from @pad.
+	 */
+	INIT_LIST_HEAD(&pipe->pads);
+	pipe->mdev = pad->graph_obj.mdev;
+
+	walk.mdev = pipe->mdev;
+	walk.stack.top = -1;
+	ret = media_pipeline_add_pad(pipe, &walk, pad);
+	if (ret)
+		goto done;
+
+	/*
+	 * Use a depth-first search algorithm: as long as the stack is not
+	 * empty, explore the next link of the top entry. The
+	 * media_pipeline_explore_next_link() function will either move to the
+	 * next link, pop the entry if fully visited, or add new entries on
+	 * top.
+	 */
+	while (!media_pipeline_walk_empty(&walk)) {
+		ret = media_pipeline_explore_next_link(pipe, &walk);
+		if (ret)
+			goto done;
+	}
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline populated, found pads:\n");
+
+	list_for_each_entry(ppad, &pipe->pads, list)
+		dev_dbg(pad->graph_obj.mdev->dev, "- '%s':%u\n",
+			ppad->pad->entity->name, ppad->pad->index);
+
+	WARN_ON(walk.stack.top != -1);
+
+	ret = 0;
+
+done:
+	media_pipeline_walk_destroy(&walk);
+
+	if (ret)
+		media_pipeline_cleanup(pipe);
+
+	return ret;
+}
+
 __must_check int __media_pipeline_start(struct media_entity *entity,
 					struct media_pipeline *pipe)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
-	struct media_link *link;
+	struct media_pipeline_pad *err_ppad;
+	struct media_pipeline_pad *ppad;
 	int ret;
 
+	lockdep_assert_held(&mdev->graph_mutex);
+
+	/*
+	 * media_pipeline_start(entity) only makes sense with entities that have
+	 * a single pad.
+	 */
+
+	if (WARN_ON(entity->num_pads != 1))
+		return -EINVAL;
+
+	/*
+	 * If the entity is already part of a pipeline, that pipeline must
+	 * be the same as the pipe given to media_pipeline_start().
+	 */
+	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
+		return -EINVAL;
+
+	/*
+	 * If the pipeline has already been started, it is guaranteed to be
+	 * valid, so just increase the start count.
+	 */
 	if (pipe->start_count) {
 		pipe->start_count++;
 		return 0;
 	}
 
-	ret = media_graph_walk_init(&pipe->graph, mdev);
+	/*
+	 * Populate the pipeline. This populates the media_pipeline pads list
+	 * with media_pipeline_pad instances for each pad found during graph
+	 * walk.
+	 */
+	ret = media_pipeline_populate(pipe, entity->pads);
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&pipe->graph, entity);
+	/*
+	 * Now that all the pads in the pipeline have been gathered, perform
+	 * the validation steps.
+	 */
 
-	while ((entity = media_graph_walk_next(graph))) {
-		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
-		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		struct media_pad *pad = ppad->pad;
+		struct media_entity *entity = pad->entity;
+		bool has_enabled_link = false;
+		bool has_link = false;
+		struct media_link *link;
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				entity_err->name);
+		dev_dbg(mdev->dev, "Validating pad '%s':%u\n", pad->entity->name,
+			pad->index);
+
+		/*
+		 * 1. Ensure that the pad doesn't already belong to a different
+		 * pipeline.
+		 */
+		if (pad->pipe) {
+			dev_dbg(mdev->dev, "Failed to start pipeline: pad '%s':%u busy\n",
+				pad->entity->name, pad->index);
 			ret = -EBUSY;
 			goto error;
 		}
 
-		/* Already streaming --- no need to check. */
-		if (entity->pipe)
-			continue;
-
-		entity->pipe = pipe;
-
-		if (!entity->ops || !entity->ops->link_validate)
-			continue;
-
-		bitmap_zero(active, entity->num_pads);
-		bitmap_fill(has_no_links, entity->num_pads);
-
+		/*
+		 * 2. Validate all active links whose sink is the current pad.
+		 * Validation of the source pads is performed in the context of
+		 * the connected sink pad to avoid duplicating checks.
+		 */
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			/* Skip links unrelated to the current pad. */
+			if (link->sink != pad && link->source != pad)
+				continue;
 
-			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			/* Record if the pad has links and enabled links. */
+			if (link->flags & MEDIA_LNK_FL_ENABLED)
+				has_enabled_link = true;
+			has_link = true;
 
 			/*
-			 * Pads that either do not need to connect or
-			 * are connected through an enabled link are
-			 * fine.
+			 * Validate the link if it's enabled and has the
+			 * current pad as its sink.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
-			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+			if (!(link->flags & MEDIA_LNK_FL_ENABLED))
+				continue;
 
-			/*
-			 * Link validation will only take place for
-			 * sink ends of the link that are enabled.
-			 */
-			if (link->sink != pad ||
-			    !(link->flags & MEDIA_LNK_FL_ENABLED))
+			if (link->sink != pad)
+				continue;
+
+			if (!entity->ops || !entity->ops->link_validate)
 				continue;
 
 			ret = entity->ops->link_validate(link);
-			if (ret < 0 && ret != -ENOIOCTLCMD) {
-				dev_dbg(entity->graph_obj.mdev->dev,
-					"link validation failed for '%s':%u -> '%s':%u, error %d\n",
+			if (ret) {
+				dev_dbg(mdev->dev,
+					"Link '%s':%u -> '%s':%u failed validation: %d\n",
 					link->source->entity->name,
 					link->source->index,
-					entity->name, link->sink->index, ret);
+					link->sink->entity->name,
+					link->sink->index, ret);
 				goto error;
 			}
-		}
 
-		/* Either no links or validated links are fine. */
-		bitmap_or(active, active, has_no_links, entity->num_pads);
+			dev_dbg(mdev->dev,
+				"Link '%s':%u -> '%s':%u is valid\n",
+				link->source->entity->name,
+				link->source->index,
+				link->sink->entity->name,
+				link->sink->index);
+		}
 
-		if (!bitmap_full(active, entity->num_pads)) {
+		/*
+		 * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
+		 * ensure that it has either no link or an enabled link.
+		 */
+		if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) && has_link &&
+		    !has_enabled_link) {
+			dev_dbg(mdev->dev,
+				"Pad '%s':%u must be connected by an enabled link\n",
+				pad->entity->name, pad->index);
 			ret = -ENOLINK;
-			dev_dbg(entity->graph_obj.mdev->dev,
-				"'%s':%u must be connected by an enabled link\n",
-				entity->name,
-				(unsigned)find_first_zero_bit(
-					active, entity->num_pads));
 			goto error;
 		}
+
+		/* Validation passed, store the pipe pointer in the pad. */
+		pad->pipe = pipe;
 	}
 
 	pipe->start_count++;
@@ -512,20 +851,15 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		entity_err->pipe = NULL;
-
-		/*
-		 * We haven't started entities further than this so we quit
-		 * here.
-		 */
-		if (entity_err == entity)
+	list_for_each_entry(err_ppad, &pipe->pads, list) {
+		if (err_ppad == ppad)
 			break;
+
+		err_ppad->pad->pipe = NULL;
 	}
 
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 	return ret;
 }
@@ -546,8 +880,8 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline_pad *ppad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -559,12 +893,10 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (--pipe->start_count)
 		return;
 
-	media_graph_walk_start(graph, entity);
-
-	while ((entity = media_graph_walk_next(graph)))
-		entity->pipe = NULL;
+	list_for_each_entry(ppad, &pipe->pads, list)
+		ppad->pad->pipe = NULL;
 
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
@@ -882,7 +1214,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -898,12 +1230,11 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
-	    (media_entity_is_streaming(source) ||
-	     media_entity_is_streaming(sink)))
+	    (media_pad_is_streaming(source) || media_pad_is_streaming(sink)))
 		return -EBUSY;
 
 	mdev = source->graph_obj.mdev;
@@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
 struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
 {
-	return entity->pipe;
+	return entity->pads->pipe;
 }
 EXPORT_SYMBOL_GPL(media_entity_pipeline);
 
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 13a882a7839c..c50c3980201e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -99,12 +99,32 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
+ * @mdev:		The media device the pipeline is part of
+ * @pads:		List of media_pipeline_pad
  * @start_count:	Media pipeline start - stop count
- * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
+	struct media_device *mdev;
+	struct list_head pads;
 	int start_count;
-	struct media_graph graph;
+};
+
+/**
+ * struct media_pipeline_pad - A pad part of a media pipeline
+ *
+ * @list:		Entry in the media_pad pads list
+ * @pipe:		The media_pipeline that the pad is part of
+ * @pad:		The media pad
+ *
+ * This structure associate a pad with a media pipeline. Instances of
+ * media_pipeline_pad are created by media_pipeline_start() when it builds the
+ * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
+ * removes the entries from the list and deletes them.
+ */
+struct media_pipeline_pad {
+	struct list_head list;
+	struct media_pipeline *pipe;
+	struct media_pad *pad;
 };
 
 /**
@@ -186,6 +206,8 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * @pipe:	Pipeline this pad belongs to. Use media_entity_pipeline() to
+ *		access this field.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -193,6 +215,12 @@ struct media_pad {
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+	struct media_pipeline *pipe;
 };
 
 /**
@@ -277,7 +305,6 @@ enum media_entity_type {
  * @links:	List of data links.
  * @ops:	Entity operations.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -313,8 +340,6 @@ struct media_entity {
 
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
@@ -879,6 +904,18 @@ struct media_link *media_entity_find_link(struct media_pad *source,
  */
 struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
 
+/**
+ * media_pad_is_streaming - Test if a pad is part of a streaming pipeline
+ * @pad: The pad
+ *
+ * Return: True if the pad is part of a pipeline started with the
+ * media_pipeline_start() function, false otherwise.
+ */
+static inline bool media_pad_is_streaming(const struct media_pad *pad)
+{
+	return pad->pipe;
+}
+
 /**
  * media_entity_is_streaming - Test if an entity is part of a streaming pipeline
  * @entity: The entity
@@ -888,7 +925,14 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
  */
 static inline bool media_entity_is_streaming(const struct media_entity *entity)
 {
-	return entity->pipe;
+	struct media_pad *pad;
+
+	media_entity_for_each_pad(entity, pad) {
+		if (media_pad_is_streaming(pad))
+			return true;
+	}
+
+	return false;
 }
 
 /**
-- 
2.34.1

[PATCH v12 10/30] media: add V4L2_SUBDEV_FL_STREAMS

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_STREAMS. It is used to indicate that the
subdev supports the new API with multiplexed streams (routing, stream
configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b661e1817470..56735dfbb6b5 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -875,6 +875,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed subdev active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads is NULL)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_STREAMS			(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.34.1

[PATCH v12 11/30] media: add V4L2_SUBDEV_CAP_MPLEXED

[Tomi Valkeinen]

Add a subdev capability flag to expose to userspace if a subdev supports
multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
 include/uapi/linux/v4l2-subdev.h      | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5c27bac772ea..90826b956693 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -439,7 +439,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			((sd->flags & V4L2_SUBDEV_FL_STREAMS) ? V4L2_SUBDEV_CAP_STREAMS : 0);
 
 		return 0;
 	}
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..89af27f50a41 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports routing and multiplexed streams. */
+#define V4L2_SUBDEV_CAP_STREAMS			0x00000002
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
-- 
2.34.1

[PATCH v12 12/30] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
 3 files changed, 153 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..a0d9c79e162f
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.34.1

[PATCH v12 13/30] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 83 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 +++++++
 include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++
 4 files changed, 186 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 21470de62d72..f42f7ffcc247 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/media-device.h> /* for media_set_bus_info() */
@@ -3102,6 +3103,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3365,8 +3381,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 90826b956693..7badd693f36c 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -23,6 +23,12 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 
+/*
+ * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
+ * of streams.
+ */
+#define V4L2_SUBDEV_MAX_STREAM_ID 63
+
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
@@ -417,6 +423,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 	case VIDIOC_SUBDEV_S_SELECTION:
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING:
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -732,6 +742,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
+			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
+				return -EINVAL;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -1015,6 +1097,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->_lock);
 
+	kfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 56735dfbb6b5..a683d911e44c 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -691,12 +691,26 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @_lock: default for 'lock'
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -707,6 +721,7 @@ struct v4l2_subdev_state {
 	struct mutex _lock;
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -759,6 +774,9 @@ struct v4l2_subdev_state {
  *		     this operation as close as possible to stream on time. The
  *		     operation shall fail if the pad index it has been called on
  *		     is not valid or in case of unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -801,6 +819,10 @@ struct v4l2_subdev_pad_ops {
 			      struct v4l2_mbus_frame_desc *fd);
 	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 89af27f50a41..ae393f9f17c1 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -191,6 +191,61 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device supports routing and multiplexed streams. */
 #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
+
+/*
+ * Is the route immutable, i.e. can it be activated and inactivated?
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated internally by the subdevice (usually a sensor), and thus there
+ * is no sink-side endpoint for the route. The sink_pad and sink_stream
+ * fields are unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u32 num_routes;
+	__u64 routes;
+	__u32 reserved[6];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -206,6 +261,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.34.1

[PATCH v12 14/30] media: subdev: add v4l2_subdev_has_route()

[Tomi Valkeinen]

Add a v4l2_subdev_has_route() helper function which can be used for
media_entity_operations.has_route op.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7badd693f36c..befd00ebf381 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1041,6 +1041,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a683d911e44c..decd1e1c3d39 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1231,6 +1231,22 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_route - MC has_route implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function is an implementation of the media_entity_operations.has_route
+ * operation for subdevs that implement the multiplexed streams API (as
+ * indicated by the V4L2_SUBDEV_FL_STREAMS subdev flag).
+ *
+ * It returns true if any active route exists between pad0 and pad1.
+ */
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.34.1

[PATCH v12 15/30] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 27 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 43 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index befd00ebf381..0b841cf74c74 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1176,6 +1176,33 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+
+	lockdep_assert_held(state->lock);
+
+	kfree(dst->routes);
+	dst->routes = NULL;
+	dst->num_routes = 0;
+
+	if (src->num_routes > 0) {
+		dst->routes = kmemdup(src->routes,
+				      src->num_routes * sizeof(*src->routes),
+				      GFP_KERNEL);
+		if (!dst->routes)
+			return -ENOMEM;
+
+		dst->num_routes = src->num_routes;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index decd1e1c3d39..37081a2e6697 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1407,6 +1407,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 16/30] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 173 ++++++++++++++++++
 2 files changed, 181 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index cf3b52bdbfb9..8bf5b3f9acfd 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -587,6 +587,14 @@ before calling v4l2_subdev_init_finalize():
 
 This shares the driver's private mutex between the controls and the states.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_STREAMS subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..5075b1828b32 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,176 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, only a
+single stream can pass through a media link and a media pad. Thus each pad
+contains a format configuration for that single stream. In some cases a subdev
+can do stream processing and split a stream into two or compose two streams
+into one, but the inputs and outputs for the subdev are still a single stream
+per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
+data streams are transmitted on the same bus, which is represented by a media
+link connecting a transmitter source pad with a sink pad on the receiver. For
+example, a camera sensor can produce two distinct streams, a pixel stream and a
+metadata stream, which are transmitted on the multiplexed data bus, represented
+by a media link which connects the single sensor's source pad with the receiver
+sink pad. The stream-aware receiver will de-multiplex the streams received on
+the its sink pad and allows to route them individually to one of its source
+pads.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the enabled
+streams from one end of the link to the other, and subdevices have routing
+tables which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a stream.
+In other words, a particular stream ID must exist on both sides of a media
+link, but another stream ID can be used for the same stream at the other side
+of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations in a media
+entity.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source
+  pad (pad 0) which carries two streams, pixel data stream and metadata
+  stream.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver,
+using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
+stream endpoint in each subdev.
-- 
2.34.1

[PATCH v12 17/30] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 156 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  79 +++++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 276 insertions(+), 22 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 0b841cf74c74..d26715cbd955 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -154,8 +154,17 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+		       u32 which, u32 pad, u32 stream)
 {
+#if defined(CONFIG_MEDIA_CONTROLLER)
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
+		if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
+			return -EINVAL;
+		return 0;
+	}
+#endif
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
@@ -170,7 +179,7 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state(sd, state, format->which, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -197,7 +206,7 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state(sd, state, code->which, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -209,7 +218,7 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -244,7 +253,7 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -256,7 +265,7 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state(sd, state, sel->which, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -915,6 +924,72 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
+static int
+v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+				const struct v4l2_subdev_krouting *routing)
+{
+	u32 num_configs = 0;
+	unsigned int i;
+	u32 format_idx = 0;
+
+	kvfree(stream_configs->configs);
+	stream_configs->configs = NULL;
+	stream_configs->num_configs = 0;
+
+	/* Count number of formats needed */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (!num_configs)
+		return 0;
+
+	stream_configs->configs = kvcalloc(num_configs,
+					   sizeof(*stream_configs->configs),
+					   GFP_KERNEL);
+
+	if (!stream_configs->configs)
+		return -ENOMEM;
+
+	stream_configs->num_configs = num_configs;
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			stream_configs->configs[idx].pad = route->sink_pad;
+			stream_configs->configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->source_pad;
+		stream_configs->configs[idx].stream = route->source_stream;
+	}
+
+	return 0;
+}
+
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
 				      struct fwnode_endpoint *endpoint)
 {
@@ -1089,7 +1164,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 	else
 		state->lock = &state->_lock;
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
 		state->pads = kvcalloc(sd->entity.num_pads,
 				       sizeof(*state->pads), GFP_KERNEL);
 		if (!state->pads) {
@@ -1129,6 +1205,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->_lock);
 
 	kfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1199,10 +1276,73 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 		dst->num_routes = src->num_routes;
 	}
 
-	return 0;
+	return v4l2_subdev_init_stream_configs(&state->stream_configs, dst);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_format);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].crop;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_crop);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].compose;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 37081a2e6697..6d177fc609f7 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -691,6 +691,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -711,6 +742,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_STREAMS)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -722,6 +754,7 @@ struct v4l2_subdev_state {
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1423,6 +1456,52 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_crop() - Get pointer to a stream crop rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to crop rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_compose() - Get pointer to a stream compose
+ *					    rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to compose rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index ae393f9f17c1..429390c2f8a0 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -58,13 +60,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.34.1

[PATCH v12 18/30] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 255 ++++++++++++++++++++++++--
 1 file changed, 235 insertions(+), 20 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index d26715cbd955..7ca2337ca8d3 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1069,7 +1070,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
-v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1078,7 +1079,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
+		fmt->stream = stream;
+
+		return v4l2_subdev_call(sd, pad, get_fmt,
+					v4l2_subdev_get_locked_active_state(sd),
+					fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
@@ -1088,31 +1093,241 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_link *link,
+					  bool is_source, u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+	int ret;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	if (is_source)
+		subdev = media_entity_to_v4l2_subdev(link->source->entity);
+	else
+		subdev = media_entity_to_v4l2_subdev(link->sink->entity);
+
+	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_get_locked_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+	if (!streams)
+		return -ENOMEM;
+
+	num_streams = 0;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+		u32 link_pad;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (is_source) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+			link_pad = link->source->index;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+			link_pad = link->sink->index;
+		}
+
+		if (route_pad != link_pad)
+			continue;
+
+		/* look for duplicates */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream) {
+				ret = -EINVAL;
+				goto free_streams;
+			}
+		}
+
+		streams[num_streams++] = route_stream;
+	}
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
+
+	return 0;
+
+free_streams:
+	kfree(streams);
+
+	return ret;
+}
+
+static int v4l2_subdev_link_validate_locked(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored by the receiver, but having extra
+	 * sink streams is an error as streams must have a source.
+	 */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto free_sink;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto free_sink;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			goto free_sink;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto free_sink;
+	}
+
+free_sink:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *source_sd, *sink_sd;
+	struct v4l2_subdev_state *source_state, *sink_state;
+	int ret;
+
+	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
+	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
+
+	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
+	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
+
+	if (sink_state)
+		v4l2_subdev_lock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_lock_state(source_state);
+
+	ret = v4l2_subdev_link_validate_locked(link);
+
+	if (sink_state)
+		v4l2_subdev_unlock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_unlock_state(source_state);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.34.1

[PATCH v12 19/30] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
+ stream, return the pad + stream on the opposite side of the subdev.

v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
format on the pad + stream on the opposite side from the given pad +
stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 48 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7ca2337ca8d3..0dd612f09f35 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1558,6 +1558,54 @@ v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
 
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing, pad, stream,
+						    &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_subdev_state_get_stream_format(state, other_pad,
+						   other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 6d177fc609f7..69831967fb6f 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1502,6 +1502,42 @@ struct v4l2_rect *
 v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 				     unsigned int pad, u32 stream);
 
+/**
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
+
+/**
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 20/30] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
drivers that do not need to do anything special in their get_fmt op can
use this helper directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 0dd612f09f35..95fd2e07d69f 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1455,10 +1455,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 {
 	struct v4l2_mbus_framefmt *fmt;
 
-	if (format->pad >= sd->entity.num_pads)
-		return -EINVAL;
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
+							  format->stream);
+	else if (format->pad < sd->entity.num_pads && format->stream == 0)
+		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
+	else
+		fmt = NULL;
 
-	fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
 	if (!fmt)
 		return -EINVAL;
 
-- 
2.34.1

[PATCH v12 21/30] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 95fd2e07d69f..4a8e2b944b57 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1499,6 +1499,28 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
 				    unsigned int pad, u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 69831967fb6f..134e81a66f9f 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1456,6 +1456,22 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.34.1

[PATCH v12 22/30] media: subdev: add v4l2_subdev_routing_validate() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a v4l2_subdev_routing_validate() helper for verifying routing for
common cases like only allowing non-overlapping 1-to-1 streams.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 102 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  39 ++++++++++
 2 files changed, 141 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4a8e2b944b57..c9f3d1a0ede1 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1632,6 +1632,108 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow)
+{
+	u32 *remote_pads = NULL;
+	unsigned int i, j;
+	int ret = -EINVAL;
+
+	if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+		remote_pads = kcalloc(sd->entity.num_pads, sizeof(*remote_pads),
+				      GFP_KERNEL);
+		if (!remote_pads)
+			return -ENOMEM;
+
+		for (i = 0; i < sd->entity.num_pads; ++i)
+			remote_pads[i] = U32_MAX;
+	}
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		/* Validate the sink and source pad numbers. */
+		if (route->sink_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
+				i, route->sink_pad);
+			goto out;
+		}
+
+		if (route->source_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->source_pad].flags & MEDIA_PAD_FL_SOURCE)) {
+			dev_dbg(sd->dev, "route %u source (%u) is not a source pad\n",
+				i, route->source_pad);
+			goto out;
+		}
+
+		/*
+		 * V4L2_SUBDEV_ROUTING_NO_STREAM_MIX: Streams on the same pad
+		 * may not be routed to streams on different pads.
+		 */
+		if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+			if (remote_pads[route->sink_pad] != U32_MAX &&
+			    remote_pads[route->sink_pad] != route->source_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "sink");
+				goto out;
+			}
+
+			if (remote_pads[route->source_pad] != U32_MAX &&
+			    remote_pads[route->source_pad] != route->sink_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "source");
+				goto out;
+			}
+
+			remote_pads[route->sink_pad] = route->source_pad;
+			remote_pads[route->source_pad] = route->sink_pad;
+		}
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_1_TO_N: No two routes can
+			 * originate from the same (sink) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_1_TO_N) &&
+			    route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u originate from same sink (%u/%u)\n",
+					i, j, route->sink_pad,
+					route->sink_stream);
+				goto out;
+			}
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_N_TO_1: No two routes can end
+			 * at the same (source) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_N_TO_1) &&
+			    route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u end at same source (%u/%u)\n",
+					i, j, route->source_pad,
+					route->source_stream);
+				goto out;
+			}
+		}
+	}
+
+	ret = 0;
+
+out:
+	kfree(remote_pads);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 134e81a66f9f..a088ff1ecdc1 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1554,6 +1554,45 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
+ *
+ * @V4L2_SUBDEV_ROUTING_NO_1_TO_N:
+ *	an input stream may not be routed to multiple output streams (stream
+ *	duplication)
+ * @V4L2_SUBDEV_ROUTING_NO_N_TO_1:
+ *	multiple input streams may not be routed to the same output stream
+ *	(stream merging)
+ * @V4L2_SUBDEV_ROUTING_NO_STREAM_MIX:
+ *	streams on the same pad may not be routed to streams on different pads
+ * @V4L2_SUBDEV_ROUTING_ONLY_1_TO_1:
+ *	only non-overlapping 1-to-1 stream routing is allowed (a combination of
+ *	@V4L2_SUBDEV_ROUTING_NO_1_TO_N and @V4L2_SUBDEV_ROUTING_NO_N_TO_1)
+ */
+enum v4l2_subdev_routing_restriction {
+	V4L2_SUBDEV_ROUTING_NO_1_TO_N = BIT(0),
+	V4L2_SUBDEV_ROUTING_NO_N_TO_1 = BIT(1),
+	V4L2_SUBDEV_ROUTING_NO_STREAM_MIX = BIT(2),
+	V4L2_SUBDEV_ROUTING_ONLY_1_TO_1 =
+		V4L2_SUBDEV_ROUTING_NO_1_TO_N |
+		V4L2_SUBDEV_ROUTING_NO_N_TO_1,
+};
+
+/**
+ * v4l2_subdev_routing_validate() - Verify that routes comply with driver
+ *				    constraints
+ * @sd: The subdevice
+ * @routing: Routing to verify
+ * @disallow: Restrictions on routes
+ *
+ * This verifies that the given routing complies with the @disallow constraints.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 23/30] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add a for_each_active_route() macro to replace the repeated pattern
of iterating on the active routes of a routing table.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .clang-format                         |  1 +
 drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
 include/media/v4l2-subdev.h           | 13 +++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/.clang-format b/.clang-format
index 9b87ea1fc16e..2629a5051bf8 100644
--- a/.clang-format
+++ b/.clang-format
@@ -190,6 +190,7 @@ ForEachMacros:
   - 'for_each_active_dev_scope'
   - 'for_each_active_drhd_unit'
   - 'for_each_active_iommu'
+  - 'for_each_active_route'
   - 'for_each_aggr_pgid'
   - 'for_each_available_child_of_node'
   - 'for_each_bench'
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index c9f3d1a0ede1..f3219c8a6317 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1734,6 +1734,26 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a088ff1ecdc1..f749effadde2 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1593,6 +1593,19 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 24/30] media: subdev: use for_each_active_route() in v4l2_subdev_init_stream_configs()

[Tomi Valkeinen]

Use for_each_active_route() in v4l2_subdev_init_stream_configs().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 15 +++------------
 1 file changed, 3 insertions(+), 12 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index f3219c8a6317..97c77b32ca1d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -929,8 +929,8 @@ static int
 v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
 				const struct v4l2_subdev_krouting *routing)
 {
+	struct v4l2_subdev_route *route;
 	u32 num_configs = 0;
-	unsigned int i;
 	u32 format_idx = 0;
 
 	kvfree(stream_configs->configs);
@@ -938,12 +938,7 @@ v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_config
 	stream_configs->num_configs = 0;
 
 	/* Count number of formats needed */
-	for (i = 0; i < routing->num_routes; ++i) {
-		struct v4l2_subdev_route *route = &routing->routes[i];
-
-		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
-			continue;
-
+	for_each_active_route(routing, route) {
 		/*
 		 * Each route needs a format on both ends of the route, except
 		 * for source streams which only need one format.
@@ -968,13 +963,9 @@ v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_config
 	 * Fill in the 'pad' and stream' value for each item in the array from
 	 * the routing table
 	 */
-	for (i = 0; i < routing->num_routes; ++i) {
-		struct v4l2_subdev_route *route = &routing->routes[i];
+	for_each_active_route(routing, route) {
 		u32 idx;
 
-		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
-			continue;
-
 		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
 			idx = format_idx++;
 
-- 
2.34.1

[PATCH v12 25/30] media: subdev: use for_each_active_route() in v4l2_link_validate_get_streams()

[Tomi Valkeinen]

Use for_each_active_route() in v4l2_link_validate_get_streams().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 8 ++------
 1 file changed, 2 insertions(+), 6 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 97c77b32ca1d..2fc999b1b6c1 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1099,10 +1099,10 @@ static int v4l2_link_validate_get_streams(struct media_link *link,
 {
 	static const u32 default_streams[] = { 0 };
 	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev_route *route;
 	struct v4l2_subdev *subdev;
 	u32 num_streams;
 	u32 *streams;
-	unsigned int i;
 	struct v4l2_subdev_state *state;
 	int ret;
 
@@ -1128,16 +1128,12 @@ static int v4l2_link_validate_get_streams(struct media_link *link,
 
 	num_streams = 0;
 
-	for (i = 0; i < routing->num_routes; ++i) {
-		struct v4l2_subdev_route *route = &routing->routes[i];
+	for_each_active_route(routing, route) {
 		int j;
 		u32 route_pad;
 		u32 route_stream;
 		u32 link_pad;
 
-		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
-			continue;
-
 		if (is_source) {
 			route_pad = route->source_pad;
 			route_stream = route->source_stream;
-- 
2.34.1

[PATCH v12 26/30] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a helper function to translate streams between two pads of a subdev,
using the subdev's internal routing table.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 26 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 23 +++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 2fc999b1b6c1..35ddf4dbe3a9 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1619,6 +1619,32 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams)
+{
+	const struct v4l2_subdev_krouting *routing = &state->routing;
+	struct v4l2_subdev_route *route;
+	u64 streams0 = 0;
+	u64 streams1 = 0;
+
+	for_each_active_route(routing, route) {
+		if (route->sink_pad == pad0 && route->source_pad == pad1 &&
+		    (*streams & BIT_ULL(route->sink_stream))) {
+			streams0 |= BIT_ULL(route->sink_stream);
+			streams1 |= BIT_ULL(route->source_stream);
+		}
+		if (route->source_pad == pad0 && route->sink_pad == pad1 &&
+		    (*streams & BIT_ULL(route->source_stream))) {
+			streams0 |= BIT_ULL(route->source_stream);
+			streams1 |= BIT_ULL(route->sink_stream);
+		}
+	}
+
+	*streams = streams0;
+	return streams1;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_xlate_streams);
+
 int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f749effadde2..dd5cccc827ef 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1554,6 +1554,29 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * v4l2_subdev_state_xlate_streams() - Translate streams from one pad to another
+ *
+ * @state: Subdevice state
+ * @pad0: The first pad
+ * @pad1: The second pad
+ * @streams: Streams bitmask on the first pad
+ *
+ * Streams on sink pads of a subdev are routed to source pads as expressed in
+ * the subdev state routing table. Stream numbers don't necessarily match on
+ * the sink and source side of a route. This function translates stream numbers
+ * on @pad0, expressed as a bitmask in @streams, to the corresponding streams
+ * on @pad1 using the routing table from the @state. It returns the stream mask
+ * on @pad1, and updates @streams with the streams that have been found in the
+ * routing table.
+ *
+ * @pad0 and @pad1 must be a sink and a source, in any order.
+ *
+ * Return: The bitmask of streams of @pad1 that are routed to @streams on @pad0.
+ */
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams);
+
 /**
  * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
  *
-- 
2.34.1

[PATCH v12 27/30] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add two new subdev pad operations, .enable_streams() and
.disable_streams(), to allow control of individual streams per pad. This
is a superset of what the video .s_stream() operation implements.

To help with handling of backward compatibility, add two wrapper
functions around those operations, and require their usage in drivers.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 224 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  85 ++++++++++
 2 files changed, 309 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 35ddf4dbe3a9..6ed062b6be31 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1767,6 +1767,230 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+static int v4l2_subdev_enable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+					       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * The subdev doesn't implement pad-based stream enable, fall back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if (sd->enabled_streams & streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already enabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Start streaming when the first streams are enabled. */
+	if (!sd->enabled_streams) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 1);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams |= streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .enable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->enable_streams)
+		return v4l2_subdev_enable_streams_fallback(sd, pad,
+							   streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already enabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (cfg->enabled) {
+			dev_dbg(dev, "stream %u already enabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .enable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, enable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as enabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = true;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_enable_streams);
+
+static int v4l2_subdev_disable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+						u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * If the subdev doesn't implement pad-based stream enable, fall  back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if ((sd->enabled_streams & streams_mask) != streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already disabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Stop streaming when the last streams are disabled. */
+	if (!(sd->enabled_streams & ~streams_mask)) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 0);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams &= ~streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .disable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->disable_streams)
+		return v4l2_subdev_disable_streams_fallback(sd, pad,
+							    streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already disabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (!cfg->enabled) {
+			dev_dbg(dev, "stream %u already disabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .disable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, disable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as disabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = false;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index dd5cccc827ef..becf2ec0373f 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -696,6 +696,7 @@ struct v4l2_subdev_pad_config {
  *
  * @pad: pad number
  * @stream: stream number
+ * @enabled: has the stream been enabled with v4l2_subdev_enable_stream()
  * @fmt: &struct v4l2_mbus_framefmt
  * @crop: &struct v4l2_rect to be used for crop
  * @compose: &struct v4l2_rect to be used for compose
@@ -705,6 +706,7 @@ struct v4l2_subdev_pad_config {
 struct v4l2_subdev_stream_config {
 	u32 pad;
 	u32 stream;
+	bool enabled;
 
 	struct v4l2_mbus_framefmt fmt;
 	struct v4l2_rect crop;
@@ -810,6 +812,18 @@ struct v4l2_subdev_state {
  *
  * @set_routing: enable or disable data connection routes described in the
  *		 subdevice routing table.
+ *
+ * @enable_streams: Enable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_enable_streams() instead.
+ *
+ * @disable_streams: Disable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_disable_streams() instead.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -856,6 +870,12 @@ struct v4l2_subdev_pad_ops {
 			   struct v4l2_subdev_state *state,
 			   enum v4l2_subdev_format_whence which,
 			   struct v4l2_subdev_krouting *route);
+	int (*enable_streams)(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask);
+	int (*disable_streams)(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask);
 };
 
 /**
@@ -1001,6 +1021,10 @@ struct v4l2_subdev_platform_data {
  * @active_state: Active state for the subdev (NULL for subdevs tracking the
  *		  state internally). Initialized by calling
  *		  v4l2_subdev_init_finalize().
+ * @enabled_streams: Bitmask of enabled streams used by
+ *		     v4l2_subdev_enable_streams() and
+ *		     v4l2_subdev_disable_streams() helper functions for fallback
+ *		     cases.
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -1046,6 +1070,7 @@ struct v4l2_subdev {
 	 * doesn't support it.
 	 */
 	struct v4l2_subdev_state *active_state;
+	u64 enabled_streams;
 };
 
 
@@ -1629,6 +1654,66 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_enable_streams() - Enable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to enable
+ *
+ * This function enables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows enabling multiple streams on a pad at
+ * once.
+ *
+ * Enabling a stream that is already enabled isn't allowed. If @streams_mask
+ * contains an already enabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream enable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is already enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask);
+
+/**
+ * v4l2_subdev_disable_streams() - Disable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to disable
+ *
+ * This function disables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows disabling multiple streams on a pad at
+ * once.
+ *
+ * Disabling a streams that is not enabled isn't allowed. If @streams_mask
+ * contains a disabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream disable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is not enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 28/30] media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The v4l2_subdev_s_stream_helper() helper can be used by subdevs that
implement the stream-aware .enable_streams() and .disable_streams()
operations to implement .s_stream(). This is limited to subdevs that
have a single source pad.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 40 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 6ed062b6be31..010df1564b3c 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1991,6 +1991,46 @@ int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
 
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable)
+{
+	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_route *route;
+	struct media_pad *pad;
+	u64 source_mask = 0;
+	int pad_index = -1;
+
+	/*
+	 * Find the source pad. This helper is meant for subdevs that have a
+	 * single source pad, so failures shouldn't happen, but catch them
+	 * loudly nonetheless as they indicate a driver bug.
+	 */
+	media_entity_for_each_pad(&sd->entity, pad) {
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			pad_index = pad->index;
+			break;
+		}
+	}
+
+	if (WARN_ON(pad_index == -1))
+		return -EINVAL;
+
+	/*
+	 * As there's a single source pad, just collect all the source streams.
+	 */
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	for_each_active_route(&state->routing, route)
+		source_mask |= BIT_ULL(route->source_stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (enable)
+		return v4l2_subdev_enable_streams(sd, pad_index, source_mask);
+	else
+		return v4l2_subdev_disable_streams(sd, pad_index, source_mask);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_s_stream_helper);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index becf2ec0373f..e39e61d8f431 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1714,6 +1714,23 @@ int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
 int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 				u64 streams_mask);
 
+/**
+ * v4l2_subdev_s_stream_helper() - Helper to implement the subdev s_stream
+ *	operation using enable_streams and disable_streams
+ * @sd: The subdevice
+ * @enable: Enable or disable streaming
+ *
+ * Subdevice drivers that implement the streams-aware
+ * &v4l2_subdev_pad_ops.enable_streams and &v4l2_subdev_pad_ops.disable_streams
+ * operations can use this helper to implement the legacy
+ * &v4l2_subdev_video_ops.s_stream operation.
+ *
+ * This helper can only be used by subdevs that have a single source pad.
+ *
+ * Return: 0 on success, or a negative error code otherwise.
+ */
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v12 29/30] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index e39e61d8f431..631e6784fb76 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.34.1

[PATCH v12 30/30] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 631e6784fb76..6d11ec783a58 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * FIXME: If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.34.1

Re: [PATCH v12 00/30] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

# uAPI

This is both a summary of the uAPI changes for routing and streams 
support and a list of the open questions.

# Subdev Cap

Subdev capability V4L2_SUBDEV_CAP_STREAMS tells the userspace that the 
subdev supports routing and streams.

# Stream ID

We use u64 bitmask inside the kernel to represent a set of streams in a 
pad, each bit representing a stream ID, so the stream ID number is 
currently limited to a max of 63 (i.e. max 64  streams per pad). The 
userspace API doesn't have this limitation at the moment, as the stream 
ID number is passed as u32, and we don't have stream masks in the uAPI 
structs. However, the kernel checks that the Stream IDs given in 
S_ROUTING are < 64.

In theory we can increase this number later, as it won't change the 
uAPI. In practice there will be some challenges in how to do the change 
and how to find all the places in the kernel which depend on this.

If it's clear that 64 streams won't be enough, perhaps we already need 
to take some steps toward this. Maybe hiding the type behind a typedef 
and creating a set of macros/functions which need to be used to access 
the data.

# Routing

```
#define VIDIOC_SUBDEV_G_ROUTING                        _IOWR('V', 38, 
struct v4l2_subdev_routing)
#define VIDIOC_SUBDEV_S_ROUTING                        _IOWR('V', 39, 
struct v4l2_subdev_routing)

#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		_BITUL(1)
#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)

struct v4l2_subdev_route {
	__u32 sink_pad;
	__u32 sink_stream;
	__u32 source_pad;
	__u32 source_stream;
	__u32 flags;
	__u32 reserved[5];
};

struct v4l2_subdev_routing {
	__u32 which;
	__u32 num_routes;
	__u64 routes;
	__u32 reserved[6];
};
```

- There's a maximum of 256 for the number of routes in the routing 
table, but there's nothing in the code that relies on this limit. It is 
there just to limit the memory allocations the kernel will do based on 
the routing table we get from the userspace.

- Using S_ROUTING will overwrite the old routing table. All the 
configurations (e.g. formats) for the streams on that subdevice are lost 
(reset to defaults), so the userspace must reconfigure the subdev after 
S_ROUTING.

- If V4L2_SUBDEV_ROUTE_FL_SOURCE flag is present, the sink_pad and 
sink_stream are ignored. This is used e.g. for sensors.

- I think V4L2_SUBDEV_ROUTE_FL_IMMUTABLE should be dropped. I don't 
really know what it means or how to use it. It was there in the original 
routing patches and so far I have kept it.

- Usually V4L2_SUBDEV_ROUTE_FL_ACTIVE is not strictly needed, as the 
existence of a route in the routing table implies that the route is 
active. However, it can be used for cases where the routing table is not 
freely modifiable. E.g. a sensor that supports both a pixel stream and a 
metadata stream could always have both in its routing table, metadta 
stream inactive by default, and the user could use 
V4L2_SUBDEV_ROUTE_FL_ACTIVE to enable/disable the metadata stream.

# Stream/routing discovery of a device

We currently don't have much for this.

For example:

- How many streams a device can support?
- What kind of routing can the device do?
- What are the streams? If a sensor supports a pixel stream and a 
metadata stream, how does the userspace know which stream ID represents 
which?

I believe this is a very complex issue, as it is perhaps impossible to 
somehow describe complex device's capabilities in a form that the 
userspace could actually use.

Options:
- Just document the device's & driver's behavior and let the userspace 
deal with it. I think this is acceptable, as when streams and routing 
are involved the userspace more or less has to have detailed 
understanding of the HW in question.
- In addition to the above, for source devices we can use the (absence 
of) V4L2_SUBDEV_ROUTE_FL_ACTIVE as described above to inform the 
userspace about the available streams.
- Try to add some kind of discovery mechanism that covers the HW we have 
now, or perhaps only for "simple" pipelines, understanding that it won't 
possibly work for future HWs.

# Selection

I admit I'm not very familiar with the cropping, scaling and composition 
in v4l2 subdevs. I believe the straightforward cases should work fine 
with streams, as there should be no practical difference between a 
legacy case of one sink pad + one source pad (and implicit routing 
between those two), and a stream coming to one pad and routed to a 
source pad.

Possibly problems could arise on cases where a stream is split into two 
streams, or two streams are merged into one stream.

I'd appreciate if someone familiar with the subdev selection could study 
this and if there are problems, explain the setup where it arises.

# Streams

`__u32 stream` field has been added to the following structs:
- v4l2_subdev_format
- v4l2_subdev_crop
- v4l2_subdev_mbus_code_enum
- v4l2_subdev_frame_size_enum
- v4l2_subdev_frame_interval
- v4l2_subdev_frame_interval_enum
- v4l2_subdev_selection

In other words, all the above operations support streams, and for the 
subdevs which do not support streams, they operate on the implicit 
stream ID 0.

  Tomi

Re: [EXT] [PATCH v12 08/30] media: mc: entity: add media_pipeline_alloc_start & media_pipeline_stop_free

[Satish Nagireddy]

Hi Tomi,

Thanks for the patch.

On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> Add new versions of media_pipeline_start() and media_pipeline_stop().
> The new functions can be used by drivers that do not need to extend the
> media_pipeline with their own structs, and the new functions will handle
> allocating and freeing the media_pipeline as needed.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c | 49 ++++++++++++++++++++++++++++++++++++
>  include/media/media-entity.h | 22 ++++++++++++++++
>  2 files changed, 71 insertions(+)
>
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 0d0d6c0dda16..b7b6c6411aa7 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -579,6 +579,55 @@ void media_pipeline_stop(struct media_entity *entity)
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_stop);
>
> +__must_check int media_pipeline_alloc_start(struct media_entity *entity)
> +{
> +       struct media_device *mdev = entity->graph_obj.mdev;
> +       struct media_pipeline *pipe;
> +       int ret;
> +       bool new_pipe = false;
> +
> +       mutex_lock(&mdev->graph_mutex);
> +
> +       /*
> +        * Is the entity already part of a pipeline? If not, we need to allocate
> +        * a pipe.
> +        */
> +       pipe = media_entity_pipeline(entity);
> +       if (!pipe) {
> +               pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);

Please add NULL check here to handle the allocation failure.

> +               new_pipe = true;
> +       }
> +
> +       ret = __media_pipeline_start(entity, pipe);
> +       if (ret) {
> +               if (new_pipe)
> +                       kfree(pipe);
> +       }
> +
> +       mutex_unlock(&mdev->graph_mutex);
> +
> +       return ret;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);

Just a suggestion. It would be nice to extend the
media_pipeline_start() instead of adding a new function. The caller
can pass pipe as NULL as below.
      media_pipeline_start(entity, NULL)
And allocation can happen inside media_pipeline_start() when pipe is NULL.

Regards,
Satish

> +
> +void media_pipeline_stop_free(struct media_entity *entity)
> +{
> +       struct media_device *mdev = entity->graph_obj.mdev;
> +       struct media_pipeline *pipe;
> +
> +       mutex_lock(&mdev->graph_mutex);
> +
> +       pipe = media_entity_pipeline(entity);
> +
> +       __media_pipeline_stop(entity);
> +
> +       if (pipe && pipe->start_count == 0)
> +               kfree(pipe);
> +
> +       mutex_unlock(&mdev->graph_mutex);
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_stop_free);
> +
>  /* -----------------------------------------------------------------------------
>   * Links management
>   */
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 6c0d0a00d58e..13a882a7839c 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -1035,6 +1035,28 @@ void media_pipeline_stop(struct media_entity *entity);
>   */
>  void __media_pipeline_stop(struct media_entity *entity);
>
> +/**
> + * media_pipeline_alloc_start - Mark a pipeline as streaming
> + * @entity: Starting entity
> + *
> + * media_pipeline_alloc_start() is similar to media_pipeline_start() but
> + * instead of working on a given pipeline the function will allocate a new
> + * pipeline if needed.
> + *
> + * Calls to media_pipeline_alloc_start() must be matched with
> + * media_pipeline_stop_free().
> + */
> +__must_check int media_pipeline_alloc_start(struct media_entity *entity);
> +
> +/**
> + * media_pipeline_stop_free - Mark a pipeline as not streaming
> + * @entity: Starting entity
> + *
> + * media_pipeline_stop_free() is similar to media_pipeline_stop() but will
> + * also free the pipeline when the start_count drops to 0.
> + */
> +void media_pipeline_stop_free(struct media_entity *entity);
> +
>  /**
>   * media_devnode_create() - creates and initializes a device node interface
>   *
> --
> 2.34.1
>

Re: [EXT] [PATCH v12 08/30] media: mc: entity: add media_pipeline_alloc_start & media_pipeline_stop_free

[Tomi Valkeinen]

On 29/07/2022 11:30, Satish Nagireddy wrote:
> Hi Tomi,
> 
> Thanks for the patch.
> 
> On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> Add new versions of media_pipeline_start() and media_pipeline_stop().
>> The new functions can be used by drivers that do not need to extend the
>> media_pipeline with their own structs, and the new functions will handle
>> allocating and freeing the media_pipeline as needed.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c | 49 ++++++++++++++++++++++++++++++++++++
>>   include/media/media-entity.h | 22 ++++++++++++++++
>>   2 files changed, 71 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 0d0d6c0dda16..b7b6c6411aa7 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -579,6 +579,55 @@ void media_pipeline_stop(struct media_entity *entity)
>>   }
>>   EXPORT_SYMBOL_GPL(media_pipeline_stop);
>>
>> +__must_check int media_pipeline_alloc_start(struct media_entity *entity)
>> +{
>> +       struct media_device *mdev = entity->graph_obj.mdev;
>> +       struct media_pipeline *pipe;
>> +       int ret;
>> +       bool new_pipe = false;
>> +
>> +       mutex_lock(&mdev->graph_mutex);
>> +
>> +       /*
>> +        * Is the entity already part of a pipeline? If not, we need to allocate
>> +        * a pipe.
>> +        */
>> +       pipe = media_entity_pipeline(entity);
>> +       if (!pipe) {
>> +               pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
> 
> Please add NULL check here to handle the allocation failure.

Oops...

>> +               new_pipe = true;
>> +       }
>> +
>> +       ret = __media_pipeline_start(entity, pipe);
>> +       if (ret) {
>> +               if (new_pipe)
>> +                       kfree(pipe);
>> +       }
>> +
>> +       mutex_unlock(&mdev->graph_mutex);
>> +
>> +       return ret;
>> +}
>> +EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
> 
> Just a suggestion. It would be nice to extend the
> media_pipeline_start() instead of adding a new function. The caller
> can pass pipe as NULL as below.
>        media_pipeline_start(entity, NULL)
> And allocation can happen inside media_pipeline_start() when pipe is NULL.

Hmm, yes, that is an option, although I'm not quite happy with such a 
call either. Passing a NULL there could be a driver bug, and with such a 
change that would not be caught.

But we also need to free the memory, for which I have the 
media_pipeline_stop_free(). If we go with your suggestion, I think we 
can add a boolean to struct media_pipeline which tells if it was 
allocated by the media_pipeline_start(), and then media_pipeline_stop() 
can free it.

This would allow us to keep the existing calls and not add any new ones.

  Tomi

Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Satish Nagireddy]

() Hi Laurent,

Thanks for the patch.

On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> [Note: the code is mostly from Laurent but the patch description is from Tomi]
>
> The media_pipeline_start() and media_pipeline_stop() functions use the
> media graph walk API to traverse the graph and validate the pipeline.
> The graph walk traverses the media graph following links between the
> entities.
>
> Also, while the pipeline can't change between the start and stop calls,
> the graph is walked again from scratch at stop time, or any time a
> driver needs to inspect the pipeline.
>
> With the upcoming multiplexed streams support we will need a bit more
> intelligent pipeline construction, as e.g. an entity may be passing
> through two independent streams via separate pads, in which case those
> pads should not be part of the same pipeline.
>
> This patch essentially rewrites the media_pipeline_start/stop so that
> a pipeline is defined as a set of pads instead of entities and the media
> graph traversal uses the newly introduced has_route operation.
>
> If an entity does not define has_route operation, all the entity's pads
> are considered as connected. This means that the behavior with all the
> current drivers stays the same, but new drivers can define a more
> fine-grained pipeline construction.
>
> Additionally the media pipeline's pads are cached at
> media_pipeline_start() time, and re-used at media_pipeline_stop() which
> avoid the need to re-walk the whole graph as the previous implementation
> did.
>
> Additionally, caching pads in the pipeline can serve in the future as
> the foundation to provide a better API than the media graph walk to
> drivers to iterate over pads and entities in the pipeline.
>
> Note that the old media_pipeline_start/stop used the media graph walk
> API. The new version does not use the media graph walk API, but instead
> a new implementation.
>
> There are two reason for not changing the graph walk: it proved to be
> rather difficult to change the graph walk to have the features
> implemented in this patch, and second, this keeps the backward
> compatibility of the graph walk as there are users of the graph walk API
>
> The long term plan is that all the existing code would be converted to
> use the new cached pipeline, thus allowing us to remove the graph walk.
>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst |   7 +-
>  drivers/media/mc/mc-entity.c               | 483 +++++++++++++++++----
>  include/media/media-entity.h               |  56 ++-
>  3 files changed, 460 insertions(+), 86 deletions(-)
>
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 2a0c0aeec5f2..9c0b09745c0a 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -229,14 +229,13 @@ When starting streaming, drivers must notify all entities in the pipeline to
>  prevent link states from being modified during streaming by calling
>  :c:func:`media_pipeline_start()`.
>
> -The function will mark all entities connected to the given entity through
> -enabled links, either directly or indirectly, as streaming.
> +The function will mark all the pads which are part of the pipeline as streaming.
>
>  The struct media_pipeline instance pointed to by
> -the pipe argument will be stored in every entity in the pipeline.
> +the pipe argument will be stored in every pad in the pipeline.
>  Drivers should embed the struct media_pipeline
>  in higher-level pipeline structures and can then access the
> -pipeline through the struct media_entity
> +pipeline through the struct media_pad
>  pipe field.
>
>  Calls to :c:func:`media_pipeline_start()` can be nested.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index b7b6c6411aa7..ca8845bf8d48 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -410,97 +410,436 @@ EXPORT_SYMBOL_GPL(media_graph_walk_next);
>   * Pipeline management
>   */
>
> +/*
> + * The pipeline traversal stack stores pads that are reached during graph
> + * traversal, with a list of links to be visited to continue the traversal.
> + * When a new pad is reached, an entry is pushed on the top of the stack and
> + * points to the incoming pad and the first link of the entity.
> + *
> + * To find further pads in the pipeline, the traversal algorithm follows
> + * internal routes in the entity, and then links in the graph. It does so by
> + * iterating over all links of the entity, and following enabled links that
> + * originate from a pad that is internally connected to the incoming pad, as
> + * reported by the media_entity_has_route() function.
> + */
> +
> +/**
> + * struct media_pipeline_walk_entry - Entry in the pipeline traversal stack
> + *
> + * @pad: The media pad being visited
> + * @links: Links left to be visited
> + */
> +struct media_pipeline_walk_entry {
> +       struct media_pad *pad;
> +       struct list_head *links;
> +};
> +
> +/**
> + * struct media_pipeline_walk - State used by the media pipeline traversal
> + *                             algorithm
> + *
> + * @mdev: The media device
> + * @stack: Depth-first search stack
> + * @stack.size: Number of allocated entries in @stack.entries
> + * @stack.top: Index of the top stack entry (-1 if the stack is empty)
> + * @stack.entries: Stack entries
> + */
> +struct media_pipeline_walk {
> +       struct media_device *mdev;
> +
> +       struct {
> +               unsigned int size;
> +               int top;
> +               struct media_pipeline_walk_entry *entries;
> +       } stack;
> +};
> +
> +#define MEDIA_PIPELINE_STACK_GROW_STEP         16
> +
> +static struct media_pipeline_walk_entry *
> +media_pipeline_walk_top(struct media_pipeline_walk *walk)
> +{
> +       return &walk->stack.entries[walk->stack.top];
> +}
> +
> +static bool media_pipeline_walk_empty(struct media_pipeline_walk *walk)
> +{
> +       return walk->stack.top == -1;
> +}
> +
> +/* Increase the stack size by MEDIA_PIPELINE_STACK_GROW_STEP elements. */
> +static int media_pipeline_walk_resize(struct media_pipeline_walk *walk)
> +{
> +       struct media_pipeline_walk_entry *entries;
> +       unsigned int new_size;
> +
> +       /* Safety check, to avoid stack overflows in case of bugs. */
> +       if (walk->stack.size >= 256)
> +               return -E2BIG;
> +
> +       new_size = walk->stack.size + MEDIA_PIPELINE_STACK_GROW_STEP;
> +
> +       entries = krealloc(walk->stack.entries,
> +                          new_size * sizeof(*walk->stack.entries),
> +                          GFP_KERNEL);
> +       if (!entries)
> +               return -ENOMEM;
> +
> +       walk->stack.entries = entries;
> +       walk->stack.size = new_size;
> +
> +       return 0;
> +}
> +
> +/* Push a new entry on the stack. */
> +static int media_pipeline_walk_push(struct media_pipeline_walk *walk,
> +                                   struct media_pad *pad)
> +{
> +       struct media_pipeline_walk_entry *entry;
> +       int ret;
> +
> +       if (walk->stack.top + 1 >= walk->stack.size) {
> +               ret = media_pipeline_walk_resize(walk);
> +               if (ret)
> +                       return ret;
> +       }
> +
> +       walk->stack.top++;
> +       entry = media_pipeline_walk_top(walk);
> +       entry->pad = pad;
> +       entry->links = pad->entity->links.next;
> +
> +       dev_dbg(walk->mdev->dev,
> +               "media pipeline: pushed entry %u: '%s':%u\n",
> +               walk->stack.top, pad->entity->name, pad->index);
> +
> +       return 0;
> +}
> +
> +/*
> + * Move the top entry link cursor to the next link. If all links of the entry
> + * have been visited, pop the entry itself.
> + */
> +static void media_pipeline_walk_pop(struct media_pipeline_walk *walk)
> +{
> +       struct media_pipeline_walk_entry *entry;
> +
> +       if (WARN_ON(walk->stack.top < 0))
> +               return;
> +
> +       entry = media_pipeline_walk_top(walk);
> +
> +       if (entry->links->next == &entry->pad->entity->links) {
> +               dev_dbg(walk->mdev->dev,
> +                       "media pipeline: entry %u has no more links, popping\n",
> +                       walk->stack.top);
> +
> +               walk->stack.top--;
> +               return;
> +       }
> +
> +       entry->links = entry->links->next;
> +
> +       dev_dbg(walk->mdev->dev,
> +               "media pipeline: moved entry %u to next link\n",
> +               walk->stack.top);
> +}
> +
> +/* Free all memory allocated while walking the pipeline. */
> +static void media_pipeline_walk_destroy(struct media_pipeline_walk *walk)
> +{
> +       kfree(walk->stack.entries);
> +}
> +
> +/* Add a pad to the pipeline and push it to the stack. */
> +static int media_pipeline_add_pad(struct media_pipeline *pipe,
> +                                 struct media_pipeline_walk *walk,
> +                                 struct media_pad *pad)
> +{
> +       struct media_pipeline_pad *ppad;
> +
> +       list_for_each_entry(ppad, &pipe->pads, list) {
> +               if (ppad->pad == pad) {
> +                       dev_dbg(pad->graph_obj.mdev->dev,
> +                               "media pipeline: already contains pad '%s':%u\n",
> +                               pad->entity->name, pad->index);
> +                       return 0;
> +               }
> +       }
> +
> +       ppad = kzalloc(sizeof(*ppad), GFP_KERNEL);
> +       if (!ppad)
> +               return -ENOMEM;
> +
> +       ppad->pipe = pipe;
> +       ppad->pad = pad;
> +
> +       list_add_tail(&ppad->list, &pipe->pads);
> +
> +       dev_dbg(pad->graph_obj.mdev->dev,
> +               "media pipeline: added pad '%s':%u\n",
> +               pad->entity->name, pad->index);
> +
> +       return media_pipeline_walk_push(walk, pad);
> +}
> +
> +/* Explore the next link of the entity at the top of the stack. */
> +static int media_pipeline_explore_next_link(struct media_pipeline *pipe,
> +                                           struct media_pipeline_walk *walk)
> +{
> +       struct media_pipeline_walk_entry *entry = media_pipeline_walk_top(walk);
> +       struct media_pad *pad;
> +       struct media_link *link;
> +       struct media_pad *local;
> +       struct media_pad *remote;
> +       int ret;
> +
> +       pad = entry->pad;
> +       link = list_entry(entry->links, typeof(*link), list);
> +       media_pipeline_walk_pop(walk);
> +
> +       dev_dbg(walk->mdev->dev,
> +               "media pipeline: exploring link '%s':%u -> '%s':%u\n",
> +               link->source->entity->name, link->source->index,
> +               link->sink->entity->name, link->sink->index);
> +
> +       /* Skip links that are not enabled. */
> +       if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
> +               dev_dbg(walk->mdev->dev,
> +                       "media pipeline: skipping link (disabled)\n");
> +               return 0;
> +       }
> +
> +       /* Get the local pad and remote pad. */
> +       if (link->source->entity == pad->entity) {
> +               local = link->source;
> +               remote = link->sink;
> +       } else {
> +               local = link->sink;
> +               remote = link->source;
> +       }
> +
> +       /*
> +        * Skip links that originate from a different pad than the incoming pad
> +        * that is not connected internally in the entity to the incoming pad.
> +        */
> +       if (pad != local &&
> +           !media_entity_has_route(pad->entity, pad->index, local->index)) {
> +               dev_dbg(walk->mdev->dev,
> +                       "media pipeline: skipping link (no route)\n");
> +               return 0;
> +       }
> +
> +       /*
> +        * Add the local and remote pads of the link to the pipeline and push
> +        * them to the stack, if they're not already present.
> +        */
> +       ret = media_pipeline_add_pad(pipe, walk, local);
> +       if (ret)
> +               return ret;
> +
> +       ret = media_pipeline_add_pad(pipe, walk, remote);
> +       if (ret)
> +               return ret;
> +
> +       return 0;
> +}
> +
> +static void media_pipeline_cleanup(struct media_pipeline *pipe)
> +{
> +       while (!list_empty(&pipe->pads)) {
> +               struct media_pipeline_pad *ppad;
> +
> +               ppad = list_first_entry(&pipe->pads, typeof(*ppad), list);
> +               list_del(&ppad->list);
> +               kfree(ppad);
> +       }
> +}
> +
> +static int media_pipeline_populate(struct media_pipeline *pipe,
> +                                  struct media_pad *pad)
> +{
> +       struct media_pipeline_walk walk = { };
> +       struct media_pipeline_pad *ppad;
> +       int ret;
> +
> +       /*
> +        * Populate the media pipeline by walking the media graph, starting
> +        * from @pad.
> +        */
> +       INIT_LIST_HEAD(&pipe->pads);
> +       pipe->mdev = pad->graph_obj.mdev;
> +
> +       walk.mdev = pipe->mdev;
> +       walk.stack.top = -1;
> +       ret = media_pipeline_add_pad(pipe, &walk, pad);
> +       if (ret)
> +               goto done;
> +
> +       /*
> +        * Use a depth-first search algorithm: as long as the stack is not
> +        * empty, explore the next link of the top entry. The
> +        * media_pipeline_explore_next_link() function will either move to the
> +        * next link, pop the entry if fully visited, or add new entries on
> +        * top.
> +        */
> +       while (!media_pipeline_walk_empty(&walk)) {
> +               ret = media_pipeline_explore_next_link(pipe, &walk);
> +               if (ret)
> +                       goto done;
> +       }
> +
> +       dev_dbg(pad->graph_obj.mdev->dev,
> +               "media pipeline populated, found pads:\n");
> +
> +       list_for_each_entry(ppad, &pipe->pads, list)
> +               dev_dbg(pad->graph_obj.mdev->dev, "- '%s':%u\n",
> +                       ppad->pad->entity->name, ppad->pad->index);
> +
> +       WARN_ON(walk.stack.top != -1);
> +
> +       ret = 0;
> +
> +done:
> +       media_pipeline_walk_destroy(&walk);
> +
> +       if (ret)
> +               media_pipeline_cleanup(pipe);
> +
> +       return ret;
> +}
> +
>  __must_check int __media_pipeline_start(struct media_entity *entity,
>                                         struct media_pipeline *pipe)
>  {
>         struct media_device *mdev = entity->graph_obj.mdev;
> -       struct media_graph *graph = &pipe->graph;
> -       struct media_entity *entity_err = entity;
> -       struct media_link *link;
> +       struct media_pipeline_pad *err_ppad;
> +       struct media_pipeline_pad *ppad;
>         int ret;
>
> +       lockdep_assert_held(&mdev->graph_mutex);
> +
> +       /*
> +        * media_pipeline_start(entity) only makes sense with entities that have
> +        * a single pad.
> +        */
> +
> +       if (WARN_ON(entity->num_pads != 1))
> +               return -EINVAL;
> +
> +       /*
> +        * If the entity is already part of a pipeline, that pipeline must
> +        * be the same as the pipe given to media_pipeline_start().
> +        */
> +       if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
> +               return -EINVAL;
> +
> +       /*
> +        * If the pipeline has already been started, it is guaranteed to be
> +        * valid, so just increase the start count.
> +        */
>         if (pipe->start_count) {
>                 pipe->start_count++;
>                 return 0;
>         }
>
> -       ret = media_graph_walk_init(&pipe->graph, mdev);
> +       /*
> +        * Populate the pipeline. This populates the media_pipeline pads list
> +        * with media_pipeline_pad instances for each pad found during graph
> +        * walk.
> +        */
> +       ret = media_pipeline_populate(pipe, entity->pads);
>         if (ret)
>                 return ret;
>
> -       media_graph_walk_start(&pipe->graph, entity);
> +       /*
> +        * Now that all the pads in the pipeline have been gathered, perform
> +        * the validation steps.
> +        */
>
> -       while ((entity = media_graph_walk_next(graph))) {
> -               DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
> -               DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
> +       list_for_each_entry(ppad, &pipe->pads, list) {
> +               struct media_pad *pad = ppad->pad;
> +               struct media_entity *entity = pad->entity;
> +               bool has_enabled_link = false;
> +               bool has_link = false;
> +               struct media_link *link;
>
> -               if (entity->pipe && entity->pipe != pipe) {
> -                       pr_err("Pipe active for %s. Can't start for %s\n",
> -                               entity->name,
> -                               entity_err->name);
> +               dev_dbg(mdev->dev, "Validating pad '%s':%u\n", pad->entity->name,
> +                       pad->index);
> +
> +               /*
> +                * 1. Ensure that the pad doesn't already belong to a different
> +                * pipeline.
> +                */
> +               if (pad->pipe) {
> +                       dev_dbg(mdev->dev, "Failed to start pipeline: pad '%s':%u busy\n",
> +                               pad->entity->name, pad->index);
>                         ret = -EBUSY;
>                         goto error;
>                 }
>
> -               /* Already streaming --- no need to check. */
> -               if (entity->pipe)
> -                       continue;
> -
> -               entity->pipe = pipe;
> -
> -               if (!entity->ops || !entity->ops->link_validate)
> -                       continue;
> -
> -               bitmap_zero(active, entity->num_pads);
> -               bitmap_fill(has_no_links, entity->num_pads);
> -
> +               /*
> +                * 2. Validate all active links whose sink is the current pad.
> +                * Validation of the source pads is performed in the context of
> +                * the connected sink pad to avoid duplicating checks.
> +                */
>                 list_for_each_entry(link, &entity->links, list) {
> -                       struct media_pad *pad = link->sink->entity == entity
> -                                               ? link->sink : link->source;
> +                       /* Skip links unrelated to the current pad. */
> +                       if (link->sink != pad && link->source != pad)
> +                               continue;
>
> -                       /* Mark that a pad is connected by a link. */
> -                       bitmap_clear(has_no_links, pad->index, 1);
> +                       /* Record if the pad has links and enabled links. */
> +                       if (link->flags & MEDIA_LNK_FL_ENABLED)
> +                               has_enabled_link = true;
> +                       has_link = true;
>
>                         /*
> -                        * Pads that either do not need to connect or
> -                        * are connected through an enabled link are
> -                        * fine.
> +                        * Validate the link if it's enabled and has the
> +                        * current pad as its sink.
>                          */
> -                       if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
> -                           link->flags & MEDIA_LNK_FL_ENABLED)
> -                               bitmap_set(active, pad->index, 1);
> +                       if (!(link->flags & MEDIA_LNK_FL_ENABLED))
> +                               continue;
>
> -                       /*
> -                        * Link validation will only take place for
> -                        * sink ends of the link that are enabled.
> -                        */
> -                       if (link->sink != pad ||
> -                           !(link->flags & MEDIA_LNK_FL_ENABLED))
> +                       if (link->sink != pad)
> +                               continue;
> +
> +                       if (!entity->ops || !entity->ops->link_validate)
>                                 continue;
>
>                         ret = entity->ops->link_validate(link);
> -                       if (ret < 0 && ret != -ENOIOCTLCMD) {
> -                               dev_dbg(entity->graph_obj.mdev->dev,
> -                                       "link validation failed for '%s':%u -> '%s':%u, error %d\n",
> +                       if (ret) {
> +                               dev_dbg(mdev->dev,
> +                                       "Link '%s':%u -> '%s':%u failed validation: %d\n",
>                                         link->source->entity->name,
>                                         link->source->index,
> -                                       entity->name, link->sink->index, ret);
> +                                       link->sink->entity->name,
> +                                       link->sink->index, ret);
>                                 goto error;
>                         }
> -               }
>
> -               /* Either no links or validated links are fine. */
> -               bitmap_or(active, active, has_no_links, entity->num_pads);
> +                       dev_dbg(mdev->dev,
> +                               "Link '%s':%u -> '%s':%u is valid\n",
> +                               link->source->entity->name,
> +                               link->source->index,
> +                               link->sink->entity->name,
> +                               link->sink->index);
> +               }
>
> -               if (!bitmap_full(active, entity->num_pads)) {
> +               /*
> +                * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
> +                * ensure that it has either no link or an enabled link.
> +                */
> +               if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) && has_link &&
> +                   !has_enabled_link) {
> +                       dev_dbg(mdev->dev,
> +                               "Pad '%s':%u must be connected by an enabled link\n",
> +                               pad->entity->name, pad->index);
>                         ret = -ENOLINK;
> -                       dev_dbg(entity->graph_obj.mdev->dev,
> -                               "'%s':%u must be connected by an enabled link\n",
> -                               entity->name,
> -                               (unsigned)find_first_zero_bit(
> -                                       active, entity->num_pads));
>                         goto error;
>                 }
> +
> +               /* Validation passed, store the pipe pointer in the pad. */
> +               pad->pipe = pipe;
>         }
>
>         pipe->start_count++;
> @@ -512,20 +851,15 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>          * Link validation on graph failed. We revert what we did and
>          * return the error.
>          */
> -       media_graph_walk_start(graph, entity_err);
>
> -       while ((entity_err = media_graph_walk_next(graph))) {
> -               entity_err->pipe = NULL;
> -
> -               /*
> -                * We haven't started entities further than this so we quit
> -                * here.
> -                */
> -               if (entity_err == entity)
> +       list_for_each_entry(err_ppad, &pipe->pads, list) {
> +               if (err_ppad == ppad)
>                         break;
> +
> +               err_ppad->pad->pipe = NULL;
>         }
>
> -       media_graph_walk_cleanup(graph);
> +       media_pipeline_cleanup(pipe);
>
>         return ret;
>  }
> @@ -546,8 +880,8 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
>
>  void __media_pipeline_stop(struct media_entity *entity)
>  {
> -       struct media_graph *graph = &entity->pipe->graph;
> -       struct media_pipeline *pipe = entity->pipe;
> +       struct media_pipeline *pipe = entity->pads->pipe;
> +       struct media_pipeline_pad *ppad;
>
>         /*
>          * If the following check fails, the driver has performed an
> @@ -559,12 +893,10 @@ void __media_pipeline_stop(struct media_entity *entity)
>         if (--pipe->start_count)
>                 return;
>
> -       media_graph_walk_start(graph, entity);
> -
> -       while ((entity = media_graph_walk_next(graph)))
> -               entity->pipe = NULL;
> +       list_for_each_entry(ppad, &pipe->pads, list)
> +               ppad->pad->pipe = NULL;
>
> -       media_graph_walk_cleanup(graph);
> +       media_pipeline_cleanup(pipe);
>
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_stop);
> @@ -882,7 +1214,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  {
>         const u32 mask = MEDIA_LNK_FL_ENABLED;
>         struct media_device *mdev;
> -       struct media_entity *source, *sink;
> +       struct media_pad *source, *sink;
>         int ret = -EBUSY;
>
>         if (link == NULL)
> @@ -898,12 +1230,11 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>         if (link->flags == flags)
>                 return 0;
>
> -       source = link->source->entity;
> -       sink = link->sink->entity;
> +       source = link->source;
> +       sink = link->sink;
>
>         if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
> -           (media_entity_is_streaming(source) ||
> -            media_entity_is_streaming(sink)))
> +           (media_pad_is_streaming(source) || media_pad_is_streaming(sink)))
>                 return -EBUSY;
>
>         mdev = source->graph_obj.mdev;
> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>
>  struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>  {
> -       return entity->pipe;
> +       return entity->pads->pipe;

I am not sure If it is always safe to return the pipe associated with
the first pad. I think this will work with all the existing drivers.
Let's say If pads of an entity are associated with different pipes,
this function might require extending the support of returning
pipe based on pad index. Please let me know your opinion.

>  }
>  EXPORT_SYMBOL_GPL(media_entity_pipeline);

nit, It would be nice to rename this function to media_entity_get_pipe
or media_entity_get_pipeline for better readability.

Regards,
Satish

>
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 13a882a7839c..c50c3980201e 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -99,12 +99,32 @@ struct media_graph {
>  /**
>   * struct media_pipeline - Media pipeline related information
>   *
> + * @mdev:              The media device the pipeline is part of
> + * @pads:              List of media_pipeline_pad
>   * @start_count:       Media pipeline start - stop count
> - * @graph:             Media graph walk during pipeline start / stop
>   */
>  struct media_pipeline {
> +       struct media_device *mdev;
> +       struct list_head pads;
>         int start_count;
> -       struct media_graph graph;
> +};
> +
> +/**
> + * struct media_pipeline_pad - A pad part of a media pipeline
> + *
> + * @list:              Entry in the media_pad pads list
> + * @pipe:              The media_pipeline that the pad is part of
> + * @pad:               The media pad
> + *
> + * This structure associate a pad with a media pipeline. Instances of
> + * media_pipeline_pad are created by media_pipeline_start() when it builds the
> + * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
> + * removes the entries from the list and deletes them.
> + */
> +struct media_pipeline_pad {
> +       struct list_head list;
> +       struct media_pipeline *pipe;
> +       struct media_pad *pad;
>  };
>
>  /**
> @@ -186,6 +206,8 @@ enum media_pad_signal_type {
>   * @flags:     Pad flags, as defined in
>   *             :ref:`include/uapi/linux/media.h <media_header>`
>   *             (seek for ``MEDIA_PAD_FL_*``)
> + * @pipe:      Pipeline this pad belongs to. Use media_entity_pipeline() to
> + *             access this field.
>   */
>  struct media_pad {
>         struct media_gobj graph_obj;    /* must be first field in struct */
> @@ -193,6 +215,12 @@ struct media_pad {
>         u16 index;
>         enum media_pad_signal_type sig_type;
>         unsigned long flags;
> +
> +       /*
> +        * The fields below are private, and should only be accessed via
> +        * appropriate functions.
> +        */
> +       struct media_pipeline *pipe;
>  };
>
>  /**
> @@ -277,7 +305,6 @@ enum media_entity_type {
>   * @links:     List of data links.
>   * @ops:       Entity operations.
>   * @use_count: Use count for the entity.
> - * @pipe:      Pipeline this entity belongs to.
>   * @info:      Union with devnode information.  Kept just for backward
>   *             compatibility.
>   * @info.dev:  Contains device major and minor info.
> @@ -313,8 +340,6 @@ struct media_entity {
>
>         int use_count;
>
> -       struct media_pipeline *pipe;
> -
>         union {
>                 struct {
>                         u32 major;
> @@ -879,6 +904,18 @@ struct media_link *media_entity_find_link(struct media_pad *source,
>   */
>  struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
>
> +/**
> + * media_pad_is_streaming - Test if a pad is part of a streaming pipeline
> + * @pad: The pad
> + *
> + * Return: True if the pad is part of a pipeline started with the
> + * media_pipeline_start() function, false otherwise.
> + */
> +static inline bool media_pad_is_streaming(const struct media_pad *pad)
> +{
> +       return pad->pipe;
> +}
> +
>  /**
>   * media_entity_is_streaming - Test if an entity is part of a streaming pipeline
>   * @entity: The entity
> @@ -888,7 +925,14 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
>   */
>  static inline bool media_entity_is_streaming(const struct media_entity *entity)
>  {
> -       return entity->pipe;
> +       struct media_pad *pad;
> +
> +       media_entity_for_each_pad(entity, pad) {
> +               if (media_pad_is_streaming(pad))
> +                       return true;
> +       }
> +
> +       return false;
>  }
>
>  /**
> --
> 2.34.1
>

Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Tomi Valkeinen]

On 29/07/2022 11:45, Satish Nagireddy wrote:

>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>
>>   struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>   {
>> -       return entity->pipe;
>> +       return entity->pads->pipe;
> 
> I am not sure If it is always safe to return the pipe associated with
> the first pad. I think this will work with all the existing drivers.
> Let's say If pads of an entity are associated with different pipes,
> this function might require extending the support of returning
> pipe based on pad index. Please let me know your opinion.

That's true. The kdoc for this function says:

  * In general, entities can be part of multiple pipelines, when carrying
  * multiple streams (either on different pads, or on the same pad using
  * multiplexed streams). This function is ill-defined in that case. It
  * currently returns the pipeline associated with the first pad of the 
entity.

I did consider adding a warning if the function is called for entities 
with more than one pad. But that probably would give false warnings, 
e.g. for a simple entity with one sink and one source pad. In that case 
both pads are always part of the same pipeline, and 
media_entity_pipeline() works correctly.

We could perhaps add a check here which verifies that all the pads in 
the entity have the same pipe.

>>   }
>>   EXPORT_SYMBOL_GPL(media_entity_pipeline);
> 
> nit, It would be nice to rename this function to media_entity_get_pipe
> or media_entity_get_pipeline for better readability.

I'm ok with that, but we do have other functions with this style: 
media_entity_remote_pad(), media_entity_id(), ...

  Tomi

Re: [EXT] [PATCH v12 18/30] media: subdev: use streams in v4l2_subdev_link_validate()

[Satish Nagireddy]

Hi Tomi,

Thanks for the patch.

On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> Update v4l2_subdev_link_validate() to use routing and streams for
> validation.
>
> Instead of just looking at the format on the pad on both ends of the
> link, the routing tables are used to collect all the streams going from
> the source to the sink over the link, and the streams' formats on both
> ends of the link are verified.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 255 ++++++++++++++++++++++++--
>  1 file changed, 235 insertions(+), 20 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index d26715cbd955..7ca2337ca8d3 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -16,6 +16,7 @@
>  #include <linux/videodev2.h>
>  #include <linux/export.h>
>  #include <linux/version.h>
> +#include <linux/sort.h>
>
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
> @@ -1069,7 +1070,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>
>  static int
> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
>                                      struct v4l2_subdev_format *fmt)
>  {
>         if (is_media_entity_v4l2_subdev(pad->entity)) {
> @@ -1078,7 +1079,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>
>                 fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>                 fmt->pad = pad->index;
> -               return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
> +               fmt->stream = stream;
> +
> +               return v4l2_subdev_call(sd, pad, get_fmt,
> +                                       v4l2_subdev_get_locked_active_state(sd),
> +                                       fmt);
>         }
>
>         WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> @@ -1088,31 +1093,241 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>         return -EINVAL;
>  }
>
> -int v4l2_subdev_link_validate(struct media_link *link)
> +static int cmp_u32(const void *a, const void *b)
>  {
> -       struct v4l2_subdev *sink;
> -       struct v4l2_subdev_format sink_fmt, source_fmt;
> -       int rval;
> +       u32 a32 = *(u32 *)a;
> +       u32 b32 = *(u32 *)b;
>
> -       rval = v4l2_subdev_link_validate_get_format(
> -               link->source, &source_fmt);
> -       if (rval < 0)
> -               return 0;
> +       return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
> +}
> +
> +static int v4l2_link_validate_get_streams(struct media_link *link,
> +                                         bool is_source, u32 *out_num_streams,
> +                                         const u32 **out_streams,
> +                                         bool *allocated)

I have a suggestion to refactor this function a bit to avoid the usage
of is_source boolean, hopefully that makes this implementation look
better.
Here is my idea: Pass the media_pad (source or sink) to the function
directly and drop the parameters link and is_source. Then...

> +{
> +       static const u32 default_streams[] = { 0 };
> +       struct v4l2_subdev_krouting *routing;
> +       struct v4l2_subdev *subdev;
> +       u32 num_streams;
> +       u32 *streams;
> +       unsigned int i;
> +       struct v4l2_subdev_state *state;
> +       int ret;
>
> -       rval = v4l2_subdev_link_validate_get_format(
> -               link->sink, &sink_fmt);
> -       if (rval < 0)
> +       if (is_source)
> +               subdev = media_entity_to_v4l2_subdev(link->source->entity);
> +       else
> +               subdev = media_entity_to_v4l2_subdev(link->sink->entity);

... we can avoid the if check here and directly assign subdev as below.
     subdev = media_entity_to_v4l2_subdev(pad->entity);
Then...

> +
> +       if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
> +               *out_num_streams = 1;
> +               *out_streams = default_streams;
> +               *allocated = false;
>                 return 0;
> +       }
>
> -       sink = media_entity_to_v4l2_subdev(link->sink->entity);
> +       state = v4l2_subdev_get_locked_active_state(subdev);
>
> -       rval = v4l2_subdev_call(sink, pad, link_validate, link,
> -                               &source_fmt, &sink_fmt);
> -       if (rval != -ENOIOCTLCMD)
> -               return rval;
> +       routing = &state->routing;
> +
> +       streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
> +       if (!streams)
> +               return -ENOMEM;
> +
> +       num_streams = 0;
> +
> +       for (i = 0; i < routing->num_routes; ++i) {
> +               struct v4l2_subdev_route *route = &routing->routes[i];
> +               int j;
> +               u32 route_pad;
> +               u32 route_stream;
> +               u32 link_pad;
> +
> +               if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +                       continue;
> +
> +               if (is_source) {
> +                       route_pad = route->source_pad;
> +                       route_stream = route->source_stream;
> +                       link_pad = link->source->index;
> +               } else {
> +                       route_pad = route->sink_pad;
> +                       route_stream = route->sink_stream;
> +                       link_pad = link->sink->index;
> +               }

This can be achieved by checking pad->flags against
MEDIA_PAD_FL_SOURCE or MEDIA_PAD_FL_SINK.
Please let me know your opinion.

> +
> +               if (route_pad != link_pad)
> +                       continue;
> +
> +               /* look for duplicates */
> +               for (j = 0; j < num_streams; ++j) {
> +                       if (streams[j] == route_stream) {
> +                               ret = -EINVAL;
> +                               goto free_streams;
> +                       }
> +               }

This is good logic, but seems to be a bit inefficient as it is
repeatedly checking the array from the start.
I do not have a better idea :)

> +
> +               streams[num_streams++] = route_stream;
> +       }
> +
> +       sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
> +
> +       *out_num_streams = num_streams;
> +       *out_streams = streams;
> +       *allocated = true;
> +
> +       return 0;
> +
> +free_streams:
> +       kfree(streams);
> +
> +       return ret;
> +}
> +
> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
> +{
> +       struct v4l2_subdev *sink_subdev =
> +               media_entity_to_v4l2_subdev(link->sink->entity);
> +       struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> +       u32 num_source_streams;
> +       const u32 *source_streams;
> +       bool source_allocated;
> +       u32 num_sink_streams;
> +       const u32 *sink_streams;
> +       bool sink_allocated;
> +       unsigned int sink_idx;
> +       unsigned int source_idx;
> +       int ret;
> +
> +       dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> +               link->source->entity->name, link->source->index,
> +               link->sink->entity->name, link->sink->index);
> +
> +       ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,

This function call can be modified as below
    v4l2_link_validate_get_streams(link->source, &num_source_streams,

> +                                            &source_streams,
> +                                            &source_allocated);
> +       if (ret)
> +               return ret;
> +
> +       ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,

This function call can be modified as below
    v4l2_link_validate_get_streams(link->sink, &num_source_streams,


> +                                            &sink_streams, &sink_allocated);
> +       if (ret)
> +               goto free_source;
> +
> +       /*
> +        * It is ok to have more source streams than sink streams as extra
> +        * source streams can just be ignored by the receiver, but having extra
> +        * sink streams is an error as streams must have a source.
> +        */
> +       if (num_source_streams < num_sink_streams) {
> +               dev_err(dev,
> +                       "Not enough source streams: %d < %d\n",
> +                       num_source_streams, num_sink_streams);
> +               ret = -EINVAL;
> +               goto free_sink;
> +       }
> +
> +       /* Validate source and sink stream formats */
> +
> +       source_idx = 0;

Nit, This can be assigned at the declaration.

Regards,
Satish

> +
> +       for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
> +               struct v4l2_subdev_format sink_fmt, source_fmt;
> +               u32 stream;
> +
> +               stream = sink_streams[sink_idx];
> +
> +               for (; source_idx < num_source_streams; ++source_idx) {
> +                       if (source_streams[source_idx] == stream)
> +                               break;
> +               }
> +
> +               if (source_idx == num_source_streams) {
> +                       dev_err(dev, "No source stream for sink stream %u\n",
> +                               stream);
> +                       ret = -EINVAL;
> +                       goto free_sink;
> +               }
> +
> +               dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
> +                       link->source->entity->name, link->source->index, stream,
> +                       link->sink->entity->name, link->sink->index, stream);
> +
> +               ret = v4l2_subdev_link_validate_get_format(link->source, stream,
> +                                                          &source_fmt);
> +               if (ret < 0) {
> +                       dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +                               link->source->entity->name, link->source->index,
> +                               stream);
> +                       ret = 0;
> +                       continue;
> +               }
> +
> +               ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
> +                                                          &sink_fmt);
> +               if (ret < 0) {
> +                       dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +                               link->sink->entity->name, link->sink->index,
> +                               stream);
> +                       ret = 0;
> +                       continue;
> +               }
>
> -       return v4l2_subdev_link_validate_default(
> -               sink, link, &source_fmt, &sink_fmt);
> +               /* TODO: add stream number to link_validate() */
> +               ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
> +                                      &source_fmt, &sink_fmt);
> +               if (!ret)
> +                       continue;
> +
> +               if (ret != -ENOIOCTLCMD)
> +                       goto free_sink;
> +
> +               ret = v4l2_subdev_link_validate_default(sink_subdev, link,
> +                                                       &source_fmt, &sink_fmt);
> +
> +               if (ret)
> +                       goto free_sink;
> +       }
> +
> +free_sink:
> +       if (sink_allocated)
> +               kfree(sink_streams);
> +
> +free_source:
> +       if (source_allocated)
> +               kfree(source_streams);
> +
> +       return ret;
> +}
> +
> +int v4l2_subdev_link_validate(struct media_link *link)
> +{
> +       struct v4l2_subdev *source_sd, *sink_sd;
> +       struct v4l2_subdev_state *source_state, *sink_state;
> +       int ret;
> +
> +       sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
> +       source_sd = media_entity_to_v4l2_subdev(link->source->entity);
> +
> +       sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
> +       source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
> +
> +       if (sink_state)
> +               v4l2_subdev_lock_state(sink_state);
> +
> +       if (source_state)
> +               v4l2_subdev_lock_state(source_state);
> +
> +       ret = v4l2_subdev_link_validate_locked(link);
> +
> +       if (sink_state)
> +               v4l2_subdev_unlock_state(sink_state);
> +
> +       if (source_state)
> +               v4l2_subdev_unlock_state(source_state);
> +
> +       return ret;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>
> --
> 2.34.1
>

Re: [EXT] [PATCH v12 20/30] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Satish Nagireddy]

Hi Tomi,

Thanks for the patch.

On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
> drivers that do not need to do anything special in their get_fmt op can
> use this helper directly for v4l2_subdev_pad_ops.get_fmt.
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
>  1 file changed, 7 insertions(+), 3 deletions(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 0dd612f09f35..95fd2e07d69f 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1455,10 +1455,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  {
>         struct v4l2_mbus_framefmt *fmt;
>
> -       if (format->pad >= sd->entity.num_pads)
> -               return -EINVAL;
> +       if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> +               fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
> +                                                         format->stream);
> +       else if (format->pad < sd->entity.num_pads && format->stream == 0)
> +               fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
> +       else
> +               fmt = NULL;

fmt can be initialized to NULL at the declaration, so that the above 2
lines can be removed.

Regards,
Satish

>
> -       fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
>         if (!fmt)
>                 return -EINVAL;
>
> --
> 2.34.1
>

Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Satish Nagireddy]

On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> On 29/07/2022 11:45, Satish Nagireddy wrote:
>
> >> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> >>
> >>   struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> >>   {
> >> -       return entity->pipe;
> >> +       return entity->pads->pipe;
> >
> > I am not sure If it is always safe to return the pipe associated with
> > the first pad. I think this will work with all the existing drivers.
> > Let's say If pads of an entity are associated with different pipes,
> > this function might require extending the support of returning
> > pipe based on pad index. Please let me know your opinion.
>
> That's true. The kdoc for this function says:
>
>   * In general, entities can be part of multiple pipelines, when carrying
>   * multiple streams (either on different pads, or on the same pad using
>   * multiplexed streams). This function is ill-defined in that case. It
>   * currently returns the pipeline associated with the first pad of the
> entity.
>
> I did consider adding a warning if the function is called for entities
> with more than one pad. But that probably would give false warnings,
> e.g. for a simple entity with one sink and one source pad. In that case
> both pads are always part of the same pipeline, and
> media_entity_pipeline() works correctly.
>
> We could perhaps add a check here which verifies that all the pads in
> the entity have the same pipe.
>
> >>   }
> >>   EXPORT_SYMBOL_GPL(media_entity_pipeline);
> >
> > nit, It would be nice to rename this function to media_entity_get_pipe
> > or media_entity_get_pipeline for better readability.
>
> I'm ok with that, but we do have other functions with this style:
> media_entity_remote_pad(), media_entity_id(), ...
>
>   Tomi

I could only see one function with the similar style ==>
media_entity_get_fwnode_pad

-Satish

Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Tomi Valkeinen]

On 29/07/2022 12:19, Satish Nagireddy wrote:
> On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> On 29/07/2022 11:45, Satish Nagireddy wrote:
>>
>>>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>>>
>>>>    struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>>>    {
>>>> -       return entity->pipe;
>>>> +       return entity->pads->pipe;
>>>
>>> I am not sure If it is always safe to return the pipe associated with
>>> the first pad. I think this will work with all the existing drivers.
>>> Let's say If pads of an entity are associated with different pipes,
>>> this function might require extending the support of returning
>>> pipe based on pad index. Please let me know your opinion.
>>
>> That's true. The kdoc for this function says:
>>
>>    * In general, entities can be part of multiple pipelines, when carrying
>>    * multiple streams (either on different pads, or on the same pad using
>>    * multiplexed streams). This function is ill-defined in that case. It
>>    * currently returns the pipeline associated with the first pad of the
>> entity.
>>
>> I did consider adding a warning if the function is called for entities
>> with more than one pad. But that probably would give false warnings,
>> e.g. for a simple entity with one sink and one source pad. In that case
>> both pads are always part of the same pipeline, and
>> media_entity_pipeline() works correctly.
>>
>> We could perhaps add a check here which verifies that all the pads in
>> the entity have the same pipe.

Perhaps something like:

struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
{
	struct media_pipeline *pipe;
	struct media_pad *pad;

	if (entity->num_pads == 0)
		return NULL;

	pipe = entity->pads->pipe;

	media_entity_for_each_pad(entity, pad) {
		if (WARN_ON(pad->pipe != pipe))
			return NULL;
	}

	return pipe;
}

>>>>    }
>>>>    EXPORT_SYMBOL_GPL(media_entity_pipeline);
>>>
>>> nit, It would be nice to rename this function to media_entity_get_pipe
>>> or media_entity_get_pipeline for better readability.
>>
>> I'm ok with that, but we do have other functions with this style:
>> media_entity_remote_pad(), media_entity_id(), ...
>>
>>    Tomi
> 
> I could only see one function with the similar style ==>
> media_entity_get_fwnode_pad

Right, so, do you agree that we should keep the name as 
media_entity_pipeline, as we already have many other functions with 
similar style?

  Tomi

Re: [EXT] [PATCH v12 20/30] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

On 29/07/2022 12:16, Satish Nagireddy wrote:
> Hi Tomi,
> 
> Thanks for the patch.
> 
> On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
>> drivers that do not need to do anything special in their get_fmt op can
>> use this helper directly for v4l2_subdev_pad_ops.get_fmt.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
>>   1 file changed, 7 insertions(+), 3 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 0dd612f09f35..95fd2e07d69f 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1455,10 +1455,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   {
>>          struct v4l2_mbus_framefmt *fmt;
>>
>> -       if (format->pad >= sd->entity.num_pads)
>> -               return -EINVAL;
>> +       if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
>> +               fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
>> +                                                         format->stream);
>> +       else if (format->pad < sd->entity.num_pads && format->stream == 0)
>> +               fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
>> +       else
>> +               fmt = NULL;
> 
> fmt can be initialized to NULL at the declaration, so that the above 2
> lines can be removed.

That is true, but I personally like the above better, as it handles all 
the cases in the same if-else-sequence.

  Tomi

Re: [EXT] [PATCH v12 18/30] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

On 29/07/2022 12:12, Satish Nagireddy wrote:
> Hi Tomi,
> 
> Thanks for the patch.
> 
> On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> Update v4l2_subdev_link_validate() to use routing and streams for
>> validation.
>>
>> Instead of just looking at the format on the pad on both ends of the
>> link, the routing tables are used to collect all the streams going from
>> the source to the sink over the link, and the streams' formats on both
>> ends of the link are verified.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 255 ++++++++++++++++++++++++--
>>   1 file changed, 235 insertions(+), 20 deletions(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index d26715cbd955..7ca2337ca8d3 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -16,6 +16,7 @@
>>   #include <linux/videodev2.h>
>>   #include <linux/export.h>
>>   #include <linux/version.h>
>> +#include <linux/sort.h>
>>
>>   #include <media/v4l2-ctrls.h>
>>   #include <media/v4l2-device.h>
>> @@ -1069,7 +1070,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>>
>>   static int
>> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
>>                                       struct v4l2_subdev_format *fmt)
>>   {
>>          if (is_media_entity_v4l2_subdev(pad->entity)) {
>> @@ -1078,7 +1079,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>
>>                  fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>                  fmt->pad = pad->index;
>> -               return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
>> +               fmt->stream = stream;
>> +
>> +               return v4l2_subdev_call(sd, pad, get_fmt,
>> +                                       v4l2_subdev_get_locked_active_state(sd),
>> +                                       fmt);
>>          }
>>
>>          WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
>> @@ -1088,31 +1093,241 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>          return -EINVAL;
>>   }
>>
>> -int v4l2_subdev_link_validate(struct media_link *link)
>> +static int cmp_u32(const void *a, const void *b)
>>   {
>> -       struct v4l2_subdev *sink;
>> -       struct v4l2_subdev_format sink_fmt, source_fmt;
>> -       int rval;
>> +       u32 a32 = *(u32 *)a;
>> +       u32 b32 = *(u32 *)b;
>>
>> -       rval = v4l2_subdev_link_validate_get_format(
>> -               link->source, &source_fmt);
>> -       if (rval < 0)
>> -               return 0;
>> +       return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
>> +}
>> +
>> +static int v4l2_link_validate_get_streams(struct media_link *link,
>> +                                         bool is_source, u32 *out_num_streams,
>> +                                         const u32 **out_streams,
>> +                                         bool *allocated)
> 
> I have a suggestion to refactor this function a bit to avoid the usage
> of is_source boolean, hopefully that makes this implementation look
> better.
> Here is my idea: Pass the media_pad (source or sink) to the function
> directly and drop the parameters link and is_source. Then...
> 
>> +{
>> +       static const u32 default_streams[] = { 0 };
>> +       struct v4l2_subdev_krouting *routing;
>> +       struct v4l2_subdev *subdev;
>> +       u32 num_streams;
>> +       u32 *streams;
>> +       unsigned int i;
>> +       struct v4l2_subdev_state *state;
>> +       int ret;
>>
>> -       rval = v4l2_subdev_link_validate_get_format(
>> -               link->sink, &sink_fmt);
>> -       if (rval < 0)
>> +       if (is_source)
>> +               subdev = media_entity_to_v4l2_subdev(link->source->entity);
>> +       else
>> +               subdev = media_entity_to_v4l2_subdev(link->sink->entity);
> 
> ... we can avoid the if check here and directly assign subdev as below.
>       subdev = media_entity_to_v4l2_subdev(pad->entity);
> Then...
> 
>> +
>> +       if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
>> +               *out_num_streams = 1;
>> +               *out_streams = default_streams;
>> +               *allocated = false;
>>                  return 0;
>> +       }
>>
>> -       sink = media_entity_to_v4l2_subdev(link->sink->entity);
>> +       state = v4l2_subdev_get_locked_active_state(subdev);
>>
>> -       rval = v4l2_subdev_call(sink, pad, link_validate, link,
>> -                               &source_fmt, &sink_fmt);
>> -       if (rval != -ENOIOCTLCMD)
>> -               return rval;
>> +       routing = &state->routing;
>> +
>> +       streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
>> +       if (!streams)
>> +               return -ENOMEM;
>> +
>> +       num_streams = 0;
>> +
>> +       for (i = 0; i < routing->num_routes; ++i) {
>> +               struct v4l2_subdev_route *route = &routing->routes[i];
>> +               int j;
>> +               u32 route_pad;
>> +               u32 route_stream;
>> +               u32 link_pad;
>> +
>> +               if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +                       continue;
>> +
>> +               if (is_source) {
>> +                       route_pad = route->source_pad;
>> +                       route_stream = route->source_stream;
>> +                       link_pad = link->source->index;
>> +               } else {
>> +                       route_pad = route->sink_pad;
>> +                       route_stream = route->sink_stream;
>> +                       link_pad = link->sink->index;
>> +               }
> 
> This can be achieved by checking pad->flags against
> MEDIA_PAD_FL_SOURCE or MEDIA_PAD_FL_SINK.
> Please let me know your opinion.

Yes, I like this. It gets rid of that annoying 'is_source' =).

>> +
>> +               if (route_pad != link_pad)
>> +                       continue;
>> +
>> +               /* look for duplicates */
>> +               for (j = 0; j < num_streams; ++j) {
>> +                       if (streams[j] == route_stream) {
>> +                               ret = -EINVAL;
>> +                               goto free_streams;
>> +                       }
>> +               }
> 
> This is good logic, but seems to be a bit inefficient as it is
> repeatedly checking the array from the start.
> I do not have a better idea :)
> 
>> +
>> +               streams[num_streams++] = route_stream;
>> +       }
>> +
>> +       sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
>> +
>> +       *out_num_streams = num_streams;
>> +       *out_streams = streams;
>> +       *allocated = true;
>> +
>> +       return 0;
>> +
>> +free_streams:
>> +       kfree(streams);
>> +
>> +       return ret;
>> +}
>> +
>> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
>> +{
>> +       struct v4l2_subdev *sink_subdev =
>> +               media_entity_to_v4l2_subdev(link->sink->entity);
>> +       struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
>> +       u32 num_source_streams;
>> +       const u32 *source_streams;
>> +       bool source_allocated;
>> +       u32 num_sink_streams;
>> +       const u32 *sink_streams;
>> +       bool sink_allocated;
>> +       unsigned int sink_idx;
>> +       unsigned int source_idx;
>> +       int ret;
>> +
>> +       dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
>> +               link->source->entity->name, link->source->index,
>> +               link->sink->entity->name, link->sink->index);
>> +
>> +       ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
> 
> This function call can be modified as below
>      v4l2_link_validate_get_streams(link->source, &num_source_streams,
> 
>> +                                            &source_streams,
>> +                                            &source_allocated);
>> +       if (ret)
>> +               return ret;
>> +
>> +       ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
> 
> This function call can be modified as below
>      v4l2_link_validate_get_streams(link->sink, &num_source_streams,
> 
> 
>> +                                            &sink_streams, &sink_allocated);
>> +       if (ret)
>> +               goto free_source;
>> +
>> +       /*
>> +        * It is ok to have more source streams than sink streams as extra
>> +        * source streams can just be ignored by the receiver, but having extra
>> +        * sink streams is an error as streams must have a source.
>> +        */
>> +       if (num_source_streams < num_sink_streams) {
>> +               dev_err(dev,
>> +                       "Not enough source streams: %d < %d\n",
>> +                       num_source_streams, num_sink_streams);
>> +               ret = -EINVAL;
>> +               goto free_sink;
>> +       }
>> +
>> +       /* Validate source and sink stream formats */
>> +
>> +       source_idx = 0;
> 
> Nit, This can be assigned at the declaration.

True, but I often like to initialize the variables closer to the
use if the function is a bit on the longer side.

Although nowadays we could do:

for (unsigned int sink_idx = 0, source_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {

  Tomi

Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Satish Nagireddy]

On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> On 29/07/2022 12:19, Satish Nagireddy wrote:
> > On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
> > <tomi.valkeinen@ideasonboard.com> wrote:
> >>
> >> On 29/07/2022 11:45, Satish Nagireddy wrote:
> >>
> >>>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> >>>>
> >>>>    struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> >>>>    {
> >>>> -       return entity->pipe;
> >>>> +       return entity->pads->pipe;
> >>>
> >>> I am not sure If it is always safe to return the pipe associated with
> >>> the first pad. I think this will work with all the existing drivers.
> >>> Let's say If pads of an entity are associated with different pipes,
> >>> this function might require extending the support of returning
> >>> pipe based on pad index. Please let me know your opinion.
> >>
> >> That's true. The kdoc for this function says:
> >>
> >>    * In general, entities can be part of multiple pipelines, when carrying
> >>    * multiple streams (either on different pads, or on the same pad using
> >>    * multiplexed streams). This function is ill-defined in that case. It
> >>    * currently returns the pipeline associated with the first pad of the
> >> entity.
> >>
> >> I did consider adding a warning if the function is called for entities
> >> with more than one pad. But that probably would give false warnings,
> >> e.g. for a simple entity with one sink and one source pad. In that case
> >> both pads are always part of the same pipeline, and
> >> media_entity_pipeline() works correctly.
> >>
> >> We could perhaps add a check here which verifies that all the pads in
> >> the entity have the same pipe.
>
> Perhaps something like:
>
> struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> {
>         struct media_pipeline *pipe;
>         struct media_pad *pad;
>
>         if (entity->num_pads == 0)
>                 return NULL;
>
>         pipe = entity->pads->pipe;
>
>         media_entity_for_each_pad(entity, pad) {
>                 if (WARN_ON(pad->pipe != pipe))
>                         return NULL;
>         }
>
>         return pipe;
> }

The above code means that we do not support multiple pipelines per
entity. Or leave the implementation as is now and
this can be done as a different patch later, I will leave it to you.
He is what I'm thinking, assuming that every media_pad has it's own pipe.

struct media_pipeline *media_entity_pipeline(struct media_entity
*entity, u32 pad_index)
{
         if (pad_index >= entity->num_pads)
                 return NULL;

         return entity->pads[pad_index].pipe;
}

- Satish

>
> >>>>    }
> >>>>    EXPORT_SYMBOL_GPL(media_entity_pipeline);
> >>>
> >>> nit, It would be nice to rename this function to media_entity_get_pipe
> >>> or media_entity_get_pipeline for better readability.
> >>
> >> I'm ok with that, but we do have other functions with this style:
> >> media_entity_remote_pad(), media_entity_id(), ...
> >>
> >>    Tomi
> >
> > I could only see one function with the similar style ==>
> > media_entity_get_fwnode_pad
>
> Right, so, do you agree that we should keep the name as
> media_entity_pipeline, as we already have many other functions with
> similar style?
>
>   Tomi

Sure, we can go with the same function name.

- Satish

Re: [EXT] Re: [EXT] [PATCH v12 20/30] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Satish Nagireddy]

On Fri, Jul 29, 2022 at 3:30 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> On 29/07/2022 12:16, Satish Nagireddy wrote:
> > Hi Tomi,
> >
> > Thanks for the patch.
> >
> > On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
> > <tomi.valkeinen@ideasonboard.com> wrote:
> >>
> >> Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
> >> drivers that do not need to do anything special in their get_fmt op can
> >> use this helper directly for v4l2_subdev_pad_ops.get_fmt.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
> >>   1 file changed, 7 insertions(+), 3 deletions(-)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index 0dd612f09f35..95fd2e07d69f 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -1455,10 +1455,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> >>   {
> >>          struct v4l2_mbus_framefmt *fmt;
> >>
> >> -       if (format->pad >= sd->entity.num_pads)
> >> -               return -EINVAL;
> >> +       if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
> >> +               fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
> >> +                                                         format->stream);
> >> +       else if (format->pad < sd->entity.num_pads && format->stream == 0)
> >> +               fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
> >> +       else
> >> +               fmt = NULL;
> >
> > fmt can be initialized to NULL at the declaration, so that the above 2
> > lines can be removed.
>
> That is true, but I personally like the above better, as it handles all
> the cases in the same if-else-sequence.
>
>   Tomi

Sure.

- Satish

Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Tomi Valkeinen]

On 29/07/2022 20:00, Satish Nagireddy wrote:
> On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
> <tomi.valkeinen@ideasonboard.com> wrote:
>>
>> On 29/07/2022 12:19, Satish Nagireddy wrote:
>>> On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
>>> <tomi.valkeinen@ideasonboard.com> wrote:
>>>>
>>>> On 29/07/2022 11:45, Satish Nagireddy wrote:
>>>>
>>>>>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>>>>>
>>>>>>     struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>>>>>     {
>>>>>> -       return entity->pipe;
>>>>>> +       return entity->pads->pipe;
>>>>>
>>>>> I am not sure If it is always safe to return the pipe associated with
>>>>> the first pad. I think this will work with all the existing drivers.
>>>>> Let's say If pads of an entity are associated with different pipes,
>>>>> this function might require extending the support of returning
>>>>> pipe based on pad index. Please let me know your opinion.
>>>>
>>>> That's true. The kdoc for this function says:
>>>>
>>>>     * In general, entities can be part of multiple pipelines, when carrying
>>>>     * multiple streams (either on different pads, or on the same pad using
>>>>     * multiplexed streams). This function is ill-defined in that case. It
>>>>     * currently returns the pipeline associated with the first pad of the
>>>> entity.
>>>>
>>>> I did consider adding a warning if the function is called for entities
>>>> with more than one pad. But that probably would give false warnings,
>>>> e.g. for a simple entity with one sink and one source pad. In that case
>>>> both pads are always part of the same pipeline, and
>>>> media_entity_pipeline() works correctly.
>>>>
>>>> We could perhaps add a check here which verifies that all the pads in
>>>> the entity have the same pipe.
>>
>> Perhaps something like:
>>
>> struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>> {
>>          struct media_pipeline *pipe;
>>          struct media_pad *pad;
>>
>>          if (entity->num_pads == 0)
>>                  return NULL;
>>
>>          pipe = entity->pads->pipe;
>>
>>          media_entity_for_each_pad(entity, pad) {
>>                  if (WARN_ON(pad->pipe != pipe))
>>                          return NULL;
>>          }
>>
>>          return pipe;
>> }
> 
> The above code means that we do not support multiple pipelines per
> entity. Or leave the implementation as is now and
> this can be done as a different patch later, I will leave it to you.
> He is what I'm thinking, assuming that every media_pad has it's own pipe.
> 
> struct media_pipeline *media_entity_pipeline(struct media_entity
> *entity, u32 pad_index)
> {
>           if (pad_index >= entity->num_pads)
>                   return NULL;
> 
>           return entity->pads[pad_index].pipe;
> }

Oh, I see now where the confusion is.

media_entity_pipeline() is a convenience helper for the currently 
existing drivers. They access entity->pipe directly (or when we move the 
pipe to pad, entity->pads->pipe). So this function is just refactoring 
the direct access away from the drivers, which makes it easier in the 
future to find those drivers accessing the pipe, or change where the 
pipe is stored without doing any changes to the drivers.

A driver which supports streams should not use this function, as there's 
no "pipe for entity". We seem to be missing a similar helper function 
for these cases, media_pad_pipeline() or such. We can add when needed.

  Tomi

Re: [EXT] Re: [EXT] [PATCH v12 18/30] media: subdev: use streams in v4l2_subdev_link_validate()

[Satish Nagireddy]

On Fri, Jul 29, 2022 at 4:00 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> On 29/07/2022 12:12, Satish Nagireddy wrote:
> > Hi Tomi,
> >
> > Thanks for the patch.
> >
> > On Wed, Jul 27, 2022 at 3:37 AM Tomi Valkeinen
> > <tomi.valkeinen@ideasonboard.com> wrote:
> >>
> >> Update v4l2_subdev_link_validate() to use routing and streams for
> >> validation.
> >>
> >> Instead of just looking at the format on the pad on both ends of the
> >> link, the routing tables are used to collect all the streams going from
> >> the source to the sink over the link, and the streams' formats on both
> >> ends of the link are verified.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 255 ++++++++++++++++++++++++--
> >>   1 file changed, 235 insertions(+), 20 deletions(-)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index d26715cbd955..7ca2337ca8d3 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -16,6 +16,7 @@
> >>   #include <linux/videodev2.h>
> >>   #include <linux/export.h>
> >>   #include <linux/version.h>
> >> +#include <linux/sort.h>
> >>
> >>   #include <media/v4l2-ctrls.h>
> >>   #include <media/v4l2-device.h>
> >> @@ -1069,7 +1070,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
> >>
> >>   static int
> >> -v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> >> +v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
> >>                                       struct v4l2_subdev_format *fmt)
> >>   {
> >>          if (is_media_entity_v4l2_subdev(pad->entity)) {
> >> @@ -1078,7 +1079,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> >>
> >>                  fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >>                  fmt->pad = pad->index;
> >> -               return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
> >> +               fmt->stream = stream;
> >> +
> >> +               return v4l2_subdev_call(sd, pad, get_fmt,
> >> +                                       v4l2_subdev_get_locked_active_state(sd),
> >> +                                       fmt);
> >>          }
> >>
> >>          WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> >> @@ -1088,31 +1093,241 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> >>          return -EINVAL;
> >>   }
> >>
> >> -int v4l2_subdev_link_validate(struct media_link *link)
> >> +static int cmp_u32(const void *a, const void *b)
> >>   {
> >> -       struct v4l2_subdev *sink;
> >> -       struct v4l2_subdev_format sink_fmt, source_fmt;
> >> -       int rval;
> >> +       u32 a32 = *(u32 *)a;
> >> +       u32 b32 = *(u32 *)b;
> >>
> >> -       rval = v4l2_subdev_link_validate_get_format(
> >> -               link->source, &source_fmt);
> >> -       if (rval < 0)
> >> -               return 0;
> >> +       return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
> >> +}
> >> +
> >> +static int v4l2_link_validate_get_streams(struct media_link *link,
> >> +                                         bool is_source, u32 *out_num_streams,
> >> +                                         const u32 **out_streams,
> >> +                                         bool *allocated)
> >
> > I have a suggestion to refactor this function a bit to avoid the usage
> > of is_source boolean, hopefully that makes this implementation look
> > better.
> > Here is my idea: Pass the media_pad (source or sink) to the function
> > directly and drop the parameters link and is_source. Then...
> >
> >> +{
> >> +       static const u32 default_streams[] = { 0 };
> >> +       struct v4l2_subdev_krouting *routing;
> >> +       struct v4l2_subdev *subdev;
> >> +       u32 num_streams;
> >> +       u32 *streams;
> >> +       unsigned int i;
> >> +       struct v4l2_subdev_state *state;
> >> +       int ret;
> >>
> >> -       rval = v4l2_subdev_link_validate_get_format(
> >> -               link->sink, &sink_fmt);
> >> -       if (rval < 0)
> >> +       if (is_source)
> >> +               subdev = media_entity_to_v4l2_subdev(link->source->entity);
> >> +       else
> >> +               subdev = media_entity_to_v4l2_subdev(link->sink->entity);
> >
> > ... we can avoid the if check here and directly assign subdev as below.
> >       subdev = media_entity_to_v4l2_subdev(pad->entity);
> > Then...
> >
> >> +
> >> +       if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
> >> +               *out_num_streams = 1;
> >> +               *out_streams = default_streams;
> >> +               *allocated = false;
> >>                  return 0;
> >> +       }
> >>
> >> -       sink = media_entity_to_v4l2_subdev(link->sink->entity);
> >> +       state = v4l2_subdev_get_locked_active_state(subdev);
> >>
> >> -       rval = v4l2_subdev_call(sink, pad, link_validate, link,
> >> -                               &source_fmt, &sink_fmt);
> >> -       if (rval != -ENOIOCTLCMD)
> >> -               return rval;
> >> +       routing = &state->routing;
> >> +
> >> +       streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
> >> +       if (!streams)
> >> +               return -ENOMEM;
> >> +
> >> +       num_streams = 0;
> >> +
> >> +       for (i = 0; i < routing->num_routes; ++i) {
> >> +               struct v4l2_subdev_route *route = &routing->routes[i];
> >> +               int j;
> >> +               u32 route_pad;
> >> +               u32 route_stream;
> >> +               u32 link_pad;
> >> +
> >> +               if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> >> +                       continue;
> >> +
> >> +               if (is_source) {
> >> +                       route_pad = route->source_pad;
> >> +                       route_stream = route->source_stream;
> >> +                       link_pad = link->source->index;
> >> +               } else {
> >> +                       route_pad = route->sink_pad;
> >> +                       route_stream = route->sink_stream;
> >> +                       link_pad = link->sink->index;
> >> +               }
> >
> > This can be achieved by checking pad->flags against
> > MEDIA_PAD_FL_SOURCE or MEDIA_PAD_FL_SINK.
> > Please let me know your opinion.
>
> Yes, I like this. It gets rid of that annoying 'is_source' =).
>
> >> +
> >> +               if (route_pad != link_pad)
> >> +                       continue;
> >> +
> >> +               /* look for duplicates */
> >> +               for (j = 0; j < num_streams; ++j) {
> >> +                       if (streams[j] == route_stream) {
> >> +                               ret = -EINVAL;
> >> +                               goto free_streams;
> >> +                       }
> >> +               }
> >
> > This is good logic, but seems to be a bit inefficient as it is
> > repeatedly checking the array from the start.
> > I do not have a better idea :)
> >
> >> +
> >> +               streams[num_streams++] = route_stream;
> >> +       }
> >> +
> >> +       sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
> >> +
> >> +       *out_num_streams = num_streams;
> >> +       *out_streams = streams;
> >> +       *allocated = true;
> >> +
> >> +       return 0;
> >> +
> >> +free_streams:
> >> +       kfree(streams);
> >> +
> >> +       return ret;
> >> +}
> >> +
> >> +static int v4l2_subdev_link_validate_locked(struct media_link *link)
> >> +{
> >> +       struct v4l2_subdev *sink_subdev =
> >> +               media_entity_to_v4l2_subdev(link->sink->entity);
> >> +       struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> >> +       u32 num_source_streams;
> >> +       const u32 *source_streams;
> >> +       bool source_allocated;
> >> +       u32 num_sink_streams;
> >> +       const u32 *sink_streams;
> >> +       bool sink_allocated;
> >> +       unsigned int sink_idx;
> >> +       unsigned int source_idx;
> >> +       int ret;
> >> +
> >> +       dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> >> +               link->source->entity->name, link->source->index,
> >> +               link->sink->entity->name, link->sink->index);
> >> +
> >> +       ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
> >
> > This function call can be modified as below
> >      v4l2_link_validate_get_streams(link->source, &num_source_streams,
> >
> >> +                                            &source_streams,
> >> +                                            &source_allocated);
> >> +       if (ret)
> >> +               return ret;
> >> +
> >> +       ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
> >
> > This function call can be modified as below
> >      v4l2_link_validate_get_streams(link->sink, &num_source_streams,
> >
> >
> >> +                                            &sink_streams, &sink_allocated);
> >> +       if (ret)
> >> +               goto free_source;
> >> +
> >> +       /*
> >> +        * It is ok to have more source streams than sink streams as extra
> >> +        * source streams can just be ignored by the receiver, but having extra
> >> +        * sink streams is an error as streams must have a source.
> >> +        */
> >> +       if (num_source_streams < num_sink_streams) {
> >> +               dev_err(dev,
> >> +                       "Not enough source streams: %d < %d\n",
> >> +                       num_source_streams, num_sink_streams);
> >> +               ret = -EINVAL;
> >> +               goto free_sink;
> >> +       }
> >> +
> >> +       /* Validate source and sink stream formats */
> >> +
> >> +       source_idx = 0;
> >
> > Nit, This can be assigned at the declaration.
>
> True, but I often like to initialize the variables closer to the
> use if the function is a bit on the longer side.
>
> Although nowadays we could do:
>
> for (unsigned int sink_idx = 0, source_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
>
>   Tomi

This looks good, thanks!

- Satish

Re: [EXT] Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Satish Nagireddy]

On Fri, Jul 29, 2022 at 10:07 AM Tomi Valkeinen
<tomi.valkeinen@ideasonboard.com> wrote:
>
> On 29/07/2022 20:00, Satish Nagireddy wrote:
> > On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
> > <tomi.valkeinen@ideasonboard.com> wrote:
> >>
> >> On 29/07/2022 12:19, Satish Nagireddy wrote:
> >>> On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
> >>> <tomi.valkeinen@ideasonboard.com> wrote:
> >>>>
> >>>> On 29/07/2022 11:45, Satish Nagireddy wrote:
> >>>>
> >>>>>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> >>>>>>
> >>>>>>     struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> >>>>>>     {
> >>>>>> -       return entity->pipe;
> >>>>>> +       return entity->pads->pipe;
> >>>>>
> >>>>> I am not sure If it is always safe to return the pipe associated with
> >>>>> the first pad. I think this will work with all the existing drivers.
> >>>>> Let's say If pads of an entity are associated with different pipes,
> >>>>> this function might require extending the support of returning
> >>>>> pipe based on pad index. Please let me know your opinion.
> >>>>
> >>>> That's true. The kdoc for this function says:
> >>>>
> >>>>     * In general, entities can be part of multiple pipelines, when carrying
> >>>>     * multiple streams (either on different pads, or on the same pad using
> >>>>     * multiplexed streams). This function is ill-defined in that case. It
> >>>>     * currently returns the pipeline associated with the first pad of the
> >>>> entity.
> >>>>
> >>>> I did consider adding a warning if the function is called for entities
> >>>> with more than one pad. But that probably would give false warnings,
> >>>> e.g. for a simple entity with one sink and one source pad. In that case
> >>>> both pads are always part of the same pipeline, and
> >>>> media_entity_pipeline() works correctly.
> >>>>
> >>>> We could perhaps add a check here which verifies that all the pads in
> >>>> the entity have the same pipe.
> >>
> >> Perhaps something like:
> >>
> >> struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> >> {
> >>          struct media_pipeline *pipe;
> >>          struct media_pad *pad;
> >>
> >>          if (entity->num_pads == 0)
> >>                  return NULL;
> >>
> >>          pipe = entity->pads->pipe;
> >>
> >>          media_entity_for_each_pad(entity, pad) {
> >>                  if (WARN_ON(pad->pipe != pipe))
> >>                          return NULL;
> >>          }
> >>
> >>          return pipe;
> >> }
> >
> > The above code means that we do not support multiple pipelines per
> > entity. Or leave the implementation as is now and
> > this can be done as a different patch later, I will leave it to you.
> > He is what I'm thinking, assuming that every media_pad has it's own pipe.
> >
> > struct media_pipeline *media_entity_pipeline(struct media_entity
> > *entity, u32 pad_index)
> > {
> >           if (pad_index >= entity->num_pads)
> >                   return NULL;
> >
> >           return entity->pads[pad_index].pipe;
> > }
>
> Oh, I see now where the confusion is.
>
> media_entity_pipeline() is a convenience helper for the currently
> existing drivers. They access entity->pipe directly (or when we move the
> pipe to pad, entity->pads->pipe). So this function is just refactoring
> the direct access away from the drivers, which makes it easier in the
> future to find those drivers accessing the pipe, or change where the
> pipe is stored without doing any changes to the drivers.
>
> A driver which supports streams should not use this function, as there's
> no "pipe for entity". We seem to be missing a similar helper function
> for these cases, media_pad_pipeline() or such. We can add when needed.
>
>   Tomi

Sure, great!

- Satish

Re: [PATCH v12 02/30] media: mc: entity: Add iterator helper for entity pads

[Sakari Ailus]

Moi,

Thanks for the update!

On Wed, Jul 27, 2022 at 01:36:11PM +0300, Tomi Valkeinen wrote:
> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> Add an iterator helper to easily cycle through all pads in an entity and
> use it in media-entity and media-device code where appropriate.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-device.c | 13 ++++++-------
>  drivers/media/mc/mc-entity.c | 11 ++++++-----
>  include/media/media-entity.h | 12 ++++++++++++
>  3 files changed, 24 insertions(+), 12 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
> index b8176a3b76d3..25020d58eb06 100644
> --- a/drivers/media/mc/mc-device.c
> +++ b/drivers/media/mc/mc-device.c
> @@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
>  	struct media_device *mdev = entity->graph_obj.mdev;
>  	struct media_link *link, *tmp;
>  	struct media_interface *intf;
> -	unsigned int i;
> +	struct media_pad *iter;
>  
>  	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
>  
> @@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
>  	__media_entity_remove_links(entity);
>  
>  	/* Remove all pads that belong to this entity */
> -	for (i = 0; i < entity->num_pads; i++)
> -		media_gobj_destroy(&entity->pads[i].graph_obj);
> +	media_entity_for_each_pad(entity, iter)
> +		media_gobj_destroy(&iter->graph_obj);
>  
>  	/* Remove the entity */
>  	media_gobj_destroy(&entity->graph_obj);
> @@ -610,7 +610,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
>  					      struct media_entity *entity)
>  {
>  	struct media_entity_notify *notify, *next;
> -	unsigned int i;
> +	struct media_pad *iter;
>  	int ret;
>  
>  	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
> @@ -639,9 +639,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
>  	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
>  
>  	/* Initialize objects at the pads */
> -	for (i = 0; i < entity->num_pads; i++)
> -		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
> -			       &entity->pads[i].graph_obj);
> +	media_entity_for_each_pad(entity, iter)
> +		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
>  
>  	/* invoke entity_notify callbacks */
>  	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 11f5207f73aa..35a1aa218c51 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -192,7 +192,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  			   struct media_pad *pads)
>  {
>  	struct media_device *mdev = entity->graph_obj.mdev;
> -	unsigned int i;
> +	struct media_pad *iter;
> +	unsigned int i = 0;
>  
>  	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
>  		return -E2BIG;
> @@ -203,12 +204,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>  	if (mdev)
>  		mutex_lock(&mdev->graph_mutex);
>  
> -	for (i = 0; i < num_pads; i++) {
> -		pads[i].entity = entity;
> -		pads[i].index = i;
> +	media_entity_for_each_pad(entity, iter) {
> +		iter->entity = entity;
> +		iter->index = i++;
>  		if (mdev)
>  			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
> -					&entity->pads[i].graph_obj);
> +					&iter->graph_obj);

I know it was wrongly aligned but fixing it would be nice here.

>  	}
>  
>  	if (mdev)
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index a9a1c0ec5d1c..6b6fea6a713e 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -315,6 +315,18 @@ struct media_entity {
>  	} info;
>  };
>  
> +/**
> + * media_entity_for_each_pad - Iterate on all pads in an entity
> + * @entity: The entity the pads belong to
> + * @iter: The iterator pad
> + *
> + * Iterate on all pads in a media entity.
> + */
> +#define media_entity_for_each_pad(entity, iter)			\
> +	for (iter = (entity)->pads;				\
> +	     iter < &(entity)->pads[(entity)->num_pads];	\
> +	     ++iter)

It'd be nice to align the backslashes.

> +
>  /**
>   * struct media_interface - A media interface graph object.
>   *

-- 
Kind regards,

Sakari Ailus

Re: [PATCH v12 05/30] media: mc: entity: Add media_entity_pipeline() to access the media pipeline

[Sakari Ailus]

Moi,

On Wed, Jul 27, 2022 at 01:36:14PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Replace direct access to the pipe field in drivers with a new helper
> function. This will allow easier refactoring of media pipeline handling
> in the MC core behind the scenes without affecting drivers.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c                   |  6 ++++++
>  .../platform/renesas/rcar-vin/rcar-core.c      |  5 ++---
>  .../media/platform/renesas/rcar-vin/rcar-dma.c |  2 +-
>  drivers/media/platform/ti/omap3isp/isp.c       |  4 +---
>  drivers/media/platform/ti/omap3isp/ispvideo.c  |  3 +--
>  drivers/media/platform/ti/omap3isp/ispvideo.h  | 11 +++++++++--
>  drivers/media/platform/xilinx/xilinx-dma.c     |  3 +--
>  drivers/media/platform/xilinx/xilinx-dma.h     |  7 ++++++-
>  drivers/staging/media/imx/imx-media-utils.c    |  2 +-
>  drivers/staging/media/omap4iss/iss.c           |  4 +---
>  drivers/staging/media/omap4iss/iss_video.c     |  3 +--
>  drivers/staging/media/omap4iss/iss_video.h     | 11 +++++++++--
>  include/media/media-entity.h                   | 18 ++++++++++++++++++
>  13 files changed, 57 insertions(+), 22 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 9f4a1c98dc43..50872d953cf9 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -923,6 +923,12 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
>  }
>  EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>  
> +struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> +{
> +	return entity->pipe;

I'd make this an inline function. Or do you plan to add more code later?

> +}
> +EXPORT_SYMBOL_GPL(media_entity_pipeline);
> +
>  static void media_interface_init(struct media_device *mdev,
>  				 struct media_interface *intf,
>  				 u32 gobj_type,
> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-core.c b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
> index 49bdcfba010b..7b12af3ed8fb 100644
> --- a/drivers/media/platform/renesas/rcar-vin/rcar-core.c
> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
> @@ -786,9 +786,8 @@ static int rvin_csi2_link_notify(struct media_link *link, u32 flags,
>  		return 0;
>  
>  	/*
> -	 * Don't allow link changes if any entity in the graph is
> -	 * streaming, modifying the CHSEL register fields can disrupt
> -	 * running streams.
> +	 * Don't allow link changes if any stream in the graph is active as
> +	 * modifying the CHSEL register fields can disrupt running streams.
>  	 */
>  	media_device_for_each_entity(entity, &group->mdev)
>  		if (media_entity_is_streaming(entity))
> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> index 6644b498929d..924907b71263 100644
> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> @@ -1281,7 +1281,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	 */
>  	mdev = vin->vdev.entity.graph_obj.mdev;
>  	mutex_lock(&mdev->graph_mutex);
> -	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
> +	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
>  	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	if (ret)
> diff --git a/drivers/media/platform/ti/omap3isp/isp.c b/drivers/media/platform/ti/omap3isp/isp.c
> index 4c937f3f323e..2b5310b07bc3 100644
> --- a/drivers/media/platform/ti/omap3isp/isp.c
> +++ b/drivers/media/platform/ti/omap3isp/isp.c
> @@ -937,10 +937,8 @@ static int isp_pipeline_is_last(struct media_entity *me)
>  	struct isp_pipeline *pipe;
>  	struct media_pad *pad;
>  
> -	if (!me->pipe)
> -		return 0;
>  	pipe = to_isp_pipeline(me);
> -	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
> +	if (!pipe || pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
>  		return 0;
>  	pad = media_entity_remote_pad(&pipe->output->pad);
>  	return pad->entity == me;
> diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
> index 8811d6dd4ee7..44b0d55ee5d8 100644
> --- a/drivers/media/platform/ti/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
> @@ -1093,8 +1093,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	/* Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = video->video.entity.pipe
> -	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
> +	pipe = to_isp_pipeline(&video->video.entity) ? : &video->pipe;
>  
>  	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
>  	if (ret)
> diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.h b/drivers/media/platform/ti/omap3isp/ispvideo.h
> index a0908670c0cf..1d23df576e6b 100644
> --- a/drivers/media/platform/ti/omap3isp/ispvideo.h
> +++ b/drivers/media/platform/ti/omap3isp/ispvideo.h
> @@ -99,8 +99,15 @@ struct isp_pipeline {
>  	unsigned int external_width;
>  };
>  
> -#define to_isp_pipeline(__e) \
> -	container_of((__e)->pipe, struct isp_pipeline, pipe)
> +static inline struct isp_pipeline *to_isp_pipeline(struct media_entity *entity)
> +{
> +	struct media_pipeline *pipe = media_entity_pipeline(entity);
> +
> +	if (!pipe)
> +		return NULL;
> +
> +	return container_of(pipe, struct isp_pipeline, pipe);
> +}
>  
>  static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
>  {
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 338c3661d809..72eff5ef626b 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -402,8 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	 * Use the pipeline object embedded in the first DMA object that starts
>  	 * streaming.
>  	 */
> -	pipe = dma->video.entity.pipe
> -	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
> +	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
>  
>  	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
>  	if (ret < 0)
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
> index 2378bdae57ae..3ea10f6b0bb9 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.h
> +++ b/drivers/media/platform/xilinx/xilinx-dma.h
> @@ -47,7 +47,12 @@ struct xvip_pipeline {
>  
>  static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
>  {
> -	return container_of(e->pipe, struct xvip_pipeline, pipe);
> +	struct media_pipeline *pipe = media_entity_pipeline(e);
> +
> +	if (!pipe)
> +		return NULL;
> +
> +	return container_of(pipe, struct xvip_pipeline, pipe);
>  }
>  
>  /**
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 94bc866ca28c..5d9c6fc6f2e0 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -871,7 +871,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  			__media_pipeline_stop(entity);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
> -		if (entity->pipe)
> +		if (media_entity_pipeline(entity))
>  			__media_pipeline_stop(entity);
>  	}
>  
> diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
> index 68588e9dab0b..77760d913b49 100644
> --- a/drivers/staging/media/omap4iss/iss.c
> +++ b/drivers/staging/media/omap4iss/iss.c
> @@ -548,10 +548,8 @@ static int iss_pipeline_is_last(struct media_entity *me)
>  	struct iss_pipeline *pipe;
>  	struct media_pad *pad;
>  
> -	if (!me->pipe)
> -		return 0;
>  	pipe = to_iss_pipeline(me);
> -	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
> +	if (!pipe || pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
>  		return 0;
>  	pad = media_entity_remote_pad(&pipe->output->pad);
>  	return pad->entity == me;
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index d0da083deed5..67d63a400fa2 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -870,8 +870,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	 * Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = entity->pipe
> -	     ? to_iss_pipeline(entity) : &video->pipe;
> +	pipe = to_iss_pipeline(&video->video.entity) ? : &video->pipe;

s/&video->video\.//

>  	pipe->external = NULL;
>  	pipe->external_rate = 0;
>  	pipe->external_bpp = 0;
> diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
> index 526281bf0051..ca2d5edb6261 100644
> --- a/drivers/staging/media/omap4iss/iss_video.h
> +++ b/drivers/staging/media/omap4iss/iss_video.h
> @@ -90,8 +90,15 @@ struct iss_pipeline {
>  	int external_bpp;
>  };
>  
> -#define to_iss_pipeline(__e) \
> -	container_of((__e)->pipe, struct iss_pipeline, pipe)
> +static inline struct iss_pipeline *to_iss_pipeline(struct media_entity *entity)
> +{
> +	struct media_pipeline *pipe = media_entity_pipeline(entity);
> +
> +	if (!pipe)
> +		return NULL;
> +
> +	return container_of(pipe, struct iss_pipeline, pipe);
> +}
>  
>  static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
>  {
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 4e4fec60bf97..96f5fcda1985 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -883,6 +883,24 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
>  	return entity->pipe;
>  }
>  
> +/**
> + * media_entity_pipeline - Get the media pipeline an entity is part of
> + * @entity: The entity
> + *
> + * This function returns the media pipeline that an entity has been associated
> + * with when constructing the pipeline with media_pipeline_start(). The pointer
> + * remains valid until media_pipeline_stop() is called.
> + *
> + * In general, entities can be part of multiple pipelines, when carrying
> + * multiple streams (either on different pads, or on the same pad using
> + * multiplexed streams). This function is ill-defined in that case. It
> + * currently returns the pipeline associated with the first pad of the entity.
> + *
> + * Return: The media_pipeline the entity is part of, or NULL if the entity is
> + * not part of any pipeline.
> + */
> +struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
> +
>  /**
>   * media_entity_get_fwnode_pad - Get pad number from fwnode
>   *

-- 
Terveisin,

Sakari Ailus

Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Sakari Ailus]

Moi,

On Fri, Jul 29, 2022 at 08:07:19PM +0300, Tomi Valkeinen wrote:
> On 29/07/2022 20:00, Satish Nagireddy wrote:
> > On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
> > <tomi.valkeinen@ideasonboard.com> wrote:
> > > 
> > > On 29/07/2022 12:19, Satish Nagireddy wrote:
> > > > On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
> > > > <tomi.valkeinen@ideasonboard.com> wrote:
> > > > > 
> > > > > On 29/07/2022 11:45, Satish Nagireddy wrote:
> > > > > 
> > > > > > > @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> > > > > > > 
> > > > > > >     struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> > > > > > >     {
> > > > > > > -       return entity->pipe;
> > > > > > > +       return entity->pads->pipe;
> > > > > > 
> > > > > > I am not sure If it is always safe to return the pipe associated with
> > > > > > the first pad. I think this will work with all the existing drivers.
> > > > > > Let's say If pads of an entity are associated with different pipes,
> > > > > > this function might require extending the support of returning
> > > > > > pipe based on pad index. Please let me know your opinion.
> > > > > 
> > > > > That's true. The kdoc for this function says:
> > > > > 
> > > > >     * In general, entities can be part of multiple pipelines, when carrying
> > > > >     * multiple streams (either on different pads, or on the same pad using
> > > > >     * multiplexed streams). This function is ill-defined in that case. It
> > > > >     * currently returns the pipeline associated with the first pad of the
> > > > > entity.
> > > > > 
> > > > > I did consider adding a warning if the function is called for entities
> > > > > with more than one pad. But that probably would give false warnings,
> > > > > e.g. for a simple entity with one sink and one source pad. In that case
> > > > > both pads are always part of the same pipeline, and
> > > > > media_entity_pipeline() works correctly.
> > > > > 
> > > > > We could perhaps add a check here which verifies that all the pads in
> > > > > the entity have the same pipe.
> > > 
> > > Perhaps something like:
> > > 
> > > struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> > > {
> > >          struct media_pipeline *pipe;
> > >          struct media_pad *pad;
> > > 
> > >          if (entity->num_pads == 0)
> > >                  return NULL;
> > > 
> > >          pipe = entity->pads->pipe;
> > > 
> > >          media_entity_for_each_pad(entity, pad) {
> > >                  if (WARN_ON(pad->pipe != pipe))
> > >                          return NULL;
> > >          }
> > > 
> > >          return pipe;
> > > }
> > 
> > The above code means that we do not support multiple pipelines per
> > entity. Or leave the implementation as is now and
> > this can be done as a different patch later, I will leave it to you.
> > He is what I'm thinking, assuming that every media_pad has it's own pipe.
> > 
> > struct media_pipeline *media_entity_pipeline(struct media_entity
> > *entity, u32 pad_index)
> > {
> >           if (pad_index >= entity->num_pads)
> >                   return NULL;
> > 
> >           return entity->pads[pad_index].pipe;
> > }
> 
> Oh, I see now where the confusion is.
> 
> media_entity_pipeline() is a convenience helper for the currently existing
> drivers. They access entity->pipe directly (or when we move the pipe to pad,
> entity->pads->pipe). So this function is just refactoring the direct access
> away from the drivers, which makes it easier in the future to find those
> drivers accessing the pipe, or change where the pipe is stored without doing
> any changes to the drivers.
> 
> A driver which supports streams should not use this function, as there's no
> "pipe for entity". We seem to be missing a similar helper function for these
> cases, media_pad_pipeline() or such. We can add when needed.

I think this could be done now, for consistency. Either the existing
drivers could call it on entity->pads, thus assuming they only use a single
pipeline for that entity. Or keep media_entity_pipeline(), but there's some
chance for confusion on where the pipeline is bound. At least there should
be a warning it's just for old existing drivers.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v12 00/30] v4l: routing and streams support

[Sakari Ailus]

Moi,

On Wed, Jul 27, 2022 at 01:36:09PM +0300, Tomi Valkeinen wrote:
> Hi,
> 
> This is v12 of the streams series. The v11 can be found from:

Thanks for the update. This continues to be a set that will need changes
before merging, hopefully less so than in the past. But at the same time I
think there are a number of patches early in the set that could be merged
now. Specifically I mean patches 1--5 and 7.

I'll pick those if there are no objections once we have rc1 in media tree.
That should at least make it a little easier to work with the rest.

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v12 02/30] media: mc: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

On 30/07/2022 14:11, Sakari Ailus wrote:

>> @@ -203,12 +204,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
>>   	if (mdev)
>>   		mutex_lock(&mdev->graph_mutex);
>>   
>> -	for (i = 0; i < num_pads; i++) {
>> -		pads[i].entity = entity;
>> -		pads[i].index = i;
>> +	media_entity_for_each_pad(entity, iter) {
>> +		iter->entity = entity;
>> +		iter->index = i++;
>>   		if (mdev)
>>   			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
>> -					&entity->pads[i].graph_obj);
>> +					&iter->graph_obj);
> 
> I know it was wrongly aligned but fixing it would be nice here.

Ok.

>>   	}
>>   
>>   	if (mdev)
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index a9a1c0ec5d1c..6b6fea6a713e 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -315,6 +315,18 @@ struct media_entity {
>>   	} info;
>>   };
>>   
>> +/**
>> + * media_entity_for_each_pad - Iterate on all pads in an entity
>> + * @entity: The entity the pads belong to
>> + * @iter: The iterator pad
>> + *
>> + * Iterate on all pads in a media entity.
>> + */
>> +#define media_entity_for_each_pad(entity, iter)			\
>> +	for (iter = (entity)->pads;				\
>> +	     iter < &(entity)->pads[(entity)->num_pads];	\
>> +	     ++iter)
> 
> It'd be nice to align the backslashes.

They are aligned. But aligning with tabs can get messed up visually in 
patches as there's a +/- at the front of the line.

  Tomi

Re: [PATCH v12 05/30] media: mc: entity: Add media_entity_pipeline() to access the media pipeline

[Tomi Valkeinen]

On 30/07/2022 14:21, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Jul 27, 2022 at 01:36:14PM +0300, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> Replace direct access to the pipe field in drivers with a new helper
>> function. This will allow easier refactoring of media pipeline handling
>> in the MC core behind the scenes without affecting drivers.
>>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c                   |  6 ++++++
>>   .../platform/renesas/rcar-vin/rcar-core.c      |  5 ++---
>>   .../media/platform/renesas/rcar-vin/rcar-dma.c |  2 +-
>>   drivers/media/platform/ti/omap3isp/isp.c       |  4 +---
>>   drivers/media/platform/ti/omap3isp/ispvideo.c  |  3 +--
>>   drivers/media/platform/ti/omap3isp/ispvideo.h  | 11 +++++++++--
>>   drivers/media/platform/xilinx/xilinx-dma.c     |  3 +--
>>   drivers/media/platform/xilinx/xilinx-dma.h     |  7 ++++++-
>>   drivers/staging/media/imx/imx-media-utils.c    |  2 +-
>>   drivers/staging/media/omap4iss/iss.c           |  4 +---
>>   drivers/staging/media/omap4iss/iss_video.c     |  3 +--
>>   drivers/staging/media/omap4iss/iss_video.h     | 11 +++++++++--
>>   include/media/media-entity.h                   | 18 ++++++++++++++++++
>>   13 files changed, 57 insertions(+), 22 deletions(-)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 9f4a1c98dc43..50872d953cf9 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -923,6 +923,12 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
>>   }
>>   EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>   
>> +struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>> +{
>> +	return entity->pipe;
> 
> I'd make this an inline function. Or do you plan to add more code later?

I have this in my work tree due to comments from Satish:

struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
{
	struct media_pipeline *pipe;
	struct media_pad *pad;

	if (entity->num_pads == 0)
		return NULL;

	pipe = entity->pads->pipe;

	media_entity_for_each_pad(entity, pad) {
		if (WARN_ON(pad->pipe != pipe))
			return NULL;
	}

	return pipe;
}

  Tomi

Re: [PATCH v12 00/30] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

On 31/07/2022 23:47, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Jul 27, 2022 at 01:36:09PM +0300, Tomi Valkeinen wrote:
>> Hi,
>>
>> This is v12 of the streams series. The v11 can be found from:
> 
> Thanks for the update. This continues to be a set that will need changes
> before merging, hopefully less so than in the past. But at the same time I
> think there are a number of patches early in the set that could be merged
> now. Specifically I mean patches 1--5 and 7.
> 
> I'll pick those if there are no objections once we have rc1 in media tree.
> That should at least make it a little easier to work with the rest.

I'm fine with that.

Everything up to, and including, patch 9 are kernel-only patches, they 
don't change the uAPI.

  Tomi

Re: [PATCH v12 15/30] media: subdev: add v4l2_subdev_set_routing helper()

[Sakari Ailus]

Moi,

On Wed, Jul 27, 2022 at 01:36:24PM +0300, Tomi Valkeinen wrote:
> Add a helper function to set the subdev routing. The helper can be used
> from subdev driver's set_routing op to store the routing table.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 27 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 ++++++++++++++++
>  2 files changed, 43 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index befd00ebf381..0b841cf74c74 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1176,6 +1176,33 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>  
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing)
> +{
> +	struct v4l2_subdev_krouting *dst = &state->routing;
> +	const struct v4l2_subdev_krouting *src = routing;
> +
> +	lockdep_assert_held(state->lock);
> +
> +	kfree(dst->routes);
> +	dst->routes = NULL;
> +	dst->num_routes = 0;

Shouldn't you keep the old routing around until memory allocation (and
anything else that can fail) has succeeded?

> +
> +	if (src->num_routes > 0) {
> +		dst->routes = kmemdup(src->routes,
> +				      src->num_routes * sizeof(*src->routes),
> +				      GFP_KERNEL);
> +		if (!dst->routes)
> +			return -ENOMEM;
> +
> +		dst->num_routes = src->num_routes;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index decd1e1c3d39..37081a2e6697 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1407,6 +1407,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
>  int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  			struct v4l2_subdev_format *format);
>  
> +/**
> + * v4l2_subdev_set_routing() - Set given routing to subdev state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + *
> + * This will release old routing table (if any) from the state, allocate
> + * enough space for the given routing, and copy the routing.
> + *
> + * This can be used from the subdev driver's set_routing op, after validating
> + * the routing.
> + */
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing);
> +
>  #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */

-- 
Terveisin,

Sakari Ailus

Re: [PATCH v12 15/30] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

On 01/08/2022 09:59, Sakari Ailus wrote:
> Moi,
> 
> On Wed, Jul 27, 2022 at 01:36:24PM +0300, Tomi Valkeinen wrote:
>> Add a helper function to set the subdev routing. The helper can be used
>> from subdev driver's set_routing op to store the routing table.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 27 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 16 ++++++++++++++++
>>   2 files changed, 43 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index befd00ebf381..0b841cf74c74 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1176,6 +1176,33 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>>   
>> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>> +			    struct v4l2_subdev_state *state,
>> +			    struct v4l2_subdev_krouting *routing)
>> +{
>> +	struct v4l2_subdev_krouting *dst = &state->routing;
>> +	const struct v4l2_subdev_krouting *src = routing;
>> +
>> +	lockdep_assert_held(state->lock);
>> +
>> +	kfree(dst->routes);
>> +	dst->routes = NULL;
>> +	dst->num_routes = 0;
> 
> Shouldn't you keep the old routing around until memory allocation (and
> anything else that can fail) has succeeded?

Indeed, good catch! I think v4l2_subdev_init_stream_configs() also needs 
improving.

  Tomi

Re: [PATCH v12 23/30] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

On 27/07/2022 13:36, Tomi Valkeinen wrote:

> @@ -1593,6 +1593,19 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
>   				 const struct v4l2_subdev_krouting *routing,
>   				 enum v4l2_subdev_routing_restriction disallow);
>   
> +struct v4l2_subdev_route *
> +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
> +				struct v4l2_subdev_route *route);
> +
> +/**
> + * for_each_active_route - iterate on all active routes of a routing table
> + * @routing: The routing table
> + * @route: The route iterator
> + */
> +#define for_each_active_route(routing, route) \
> +	for ((route) = NULL;                  \
> +	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
> +

By the way, now that we can do it, how do people feel about changing the
above (and other similar macros) to something like:

#define for_each_active_route(routing, route)        \
	for (struct v4l2_subdev_route *route = NULL; \
	     (route = __v4l2_subdev_next_active_route((routing), route));)

  Tomi

Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Tomi Valkeinen]

On 30/07/2022 14:56, Sakari Ailus wrote:
> Moi,
> 
> On Fri, Jul 29, 2022 at 08:07:19PM +0300, Tomi Valkeinen wrote:
>> On 29/07/2022 20:00, Satish Nagireddy wrote:
>>> On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
>>> <tomi.valkeinen@ideasonboard.com> wrote:
>>>>
>>>> On 29/07/2022 12:19, Satish Nagireddy wrote:
>>>>> On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
>>>>> <tomi.valkeinen@ideasonboard.com> wrote:
>>>>>>
>>>>>> On 29/07/2022 11:45, Satish Nagireddy wrote:
>>>>>>
>>>>>>>> @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>>>>>>>
>>>>>>>>      struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>>>>>>>      {
>>>>>>>> -       return entity->pipe;
>>>>>>>> +       return entity->pads->pipe;
>>>>>>>
>>>>>>> I am not sure If it is always safe to return the pipe associated with
>>>>>>> the first pad. I think this will work with all the existing drivers.
>>>>>>> Let's say If pads of an entity are associated with different pipes,
>>>>>>> this function might require extending the support of returning
>>>>>>> pipe based on pad index. Please let me know your opinion.
>>>>>>
>>>>>> That's true. The kdoc for this function says:
>>>>>>
>>>>>>      * In general, entities can be part of multiple pipelines, when carrying
>>>>>>      * multiple streams (either on different pads, or on the same pad using
>>>>>>      * multiplexed streams). This function is ill-defined in that case. It
>>>>>>      * currently returns the pipeline associated with the first pad of the
>>>>>> entity.
>>>>>>
>>>>>> I did consider adding a warning if the function is called for entities
>>>>>> with more than one pad. But that probably would give false warnings,
>>>>>> e.g. for a simple entity with one sink and one source pad. In that case
>>>>>> both pads are always part of the same pipeline, and
>>>>>> media_entity_pipeline() works correctly.
>>>>>>
>>>>>> We could perhaps add a check here which verifies that all the pads in
>>>>>> the entity have the same pipe.
>>>>
>>>> Perhaps something like:
>>>>
>>>> struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>>> {
>>>>           struct media_pipeline *pipe;
>>>>           struct media_pad *pad;
>>>>
>>>>           if (entity->num_pads == 0)
>>>>                   return NULL;
>>>>
>>>>           pipe = entity->pads->pipe;
>>>>
>>>>           media_entity_for_each_pad(entity, pad) {
>>>>                   if (WARN_ON(pad->pipe != pipe))
>>>>                           return NULL;
>>>>           }
>>>>
>>>>           return pipe;
>>>> }
>>>
>>> The above code means that we do not support multiple pipelines per
>>> entity. Or leave the implementation as is now and
>>> this can be done as a different patch later, I will leave it to you.
>>> He is what I'm thinking, assuming that every media_pad has it's own pipe.
>>>
>>> struct media_pipeline *media_entity_pipeline(struct media_entity
>>> *entity, u32 pad_index)
>>> {
>>>            if (pad_index >= entity->num_pads)
>>>                    return NULL;
>>>
>>>            return entity->pads[pad_index].pipe;
>>> }
>>
>> Oh, I see now where the confusion is.
>>
>> media_entity_pipeline() is a convenience helper for the currently existing
>> drivers. They access entity->pipe directly (or when we move the pipe to pad,
>> entity->pads->pipe). So this function is just refactoring the direct access
>> away from the drivers, which makes it easier in the future to find those
>> drivers accessing the pipe, or change where the pipe is stored without doing
>> any changes to the drivers.
>>
>> A driver which supports streams should not use this function, as there's no
>> "pipe for entity". We seem to be missing a similar helper function for these
>> cases, media_pad_pipeline() or such. We can add when needed.
> 
> I think this could be done now, for consistency. Either the existing
> drivers could call it on entity->pads, thus assuming they only use a single
> pipeline for that entity. Or keep media_entity_pipeline(), but there's some
> chance for confusion on where the pipeline is bound. At least there should
> be a warning it's just for old existing drivers.
> 

I clarified in the kdoc that media_entity_pipeline() is only to be used 
by drivers that do not support multiple pipelines, and I added 
media_pad_pipeline().

  Tomi

Re: [EXT] Re: [EXT] Re: [EXT] [PATCH v12 09/30] media: mc: entity: Rewrite media_pipeline_start() to support routes

[Sakari Ailus]

On Mon, Aug 01, 2022 at 12:33:32PM +0300, Tomi Valkeinen wrote:
> On 30/07/2022 14:56, Sakari Ailus wrote:
> > Moi,
> > 
> > On Fri, Jul 29, 2022 at 08:07:19PM +0300, Tomi Valkeinen wrote:
> > > On 29/07/2022 20:00, Satish Nagireddy wrote:
> > > > On Fri, Jul 29, 2022 at 3:27 AM Tomi Valkeinen
> > > > <tomi.valkeinen@ideasonboard.com> wrote:
> > > > > 
> > > > > On 29/07/2022 12:19, Satish Nagireddy wrote:
> > > > > > On Fri, Jul 29, 2022 at 1:53 AM Tomi Valkeinen
> > > > > > <tomi.valkeinen@ideasonboard.com> wrote:
> > > > > > > 
> > > > > > > On 29/07/2022 11:45, Satish Nagireddy wrote:
> > > > > > > 
> > > > > > > > > @@ -1011,7 +1342,7 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> > > > > > > > > 
> > > > > > > > >      struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> > > > > > > > >      {
> > > > > > > > > -       return entity->pipe;
> > > > > > > > > +       return entity->pads->pipe;
> > > > > > > > 
> > > > > > > > I am not sure If it is always safe to return the pipe associated with
> > > > > > > > the first pad. I think this will work with all the existing drivers.
> > > > > > > > Let's say If pads of an entity are associated with different pipes,
> > > > > > > > this function might require extending the support of returning
> > > > > > > > pipe based on pad index. Please let me know your opinion.
> > > > > > > 
> > > > > > > That's true. The kdoc for this function says:
> > > > > > > 
> > > > > > >      * In general, entities can be part of multiple pipelines, when carrying
> > > > > > >      * multiple streams (either on different pads, or on the same pad using
> > > > > > >      * multiplexed streams). This function is ill-defined in that case. It
> > > > > > >      * currently returns the pipeline associated with the first pad of the
> > > > > > > entity.
> > > > > > > 
> > > > > > > I did consider adding a warning if the function is called for entities
> > > > > > > with more than one pad. But that probably would give false warnings,
> > > > > > > e.g. for a simple entity with one sink and one source pad. In that case
> > > > > > > both pads are always part of the same pipeline, and
> > > > > > > media_entity_pipeline() works correctly.
> > > > > > > 
> > > > > > > We could perhaps add a check here which verifies that all the pads in
> > > > > > > the entity have the same pipe.
> > > > > 
> > > > > Perhaps something like:
> > > > > 
> > > > > struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
> > > > > {
> > > > >           struct media_pipeline *pipe;
> > > > >           struct media_pad *pad;
> > > > > 
> > > > >           if (entity->num_pads == 0)
> > > > >                   return NULL;
> > > > > 
> > > > >           pipe = entity->pads->pipe;
> > > > > 
> > > > >           media_entity_for_each_pad(entity, pad) {
> > > > >                   if (WARN_ON(pad->pipe != pipe))
> > > > >                           return NULL;
> > > > >           }
> > > > > 
> > > > >           return pipe;
> > > > > }
> > > > 
> > > > The above code means that we do not support multiple pipelines per
> > > > entity. Or leave the implementation as is now and
> > > > this can be done as a different patch later, I will leave it to you.
> > > > He is what I'm thinking, assuming that every media_pad has it's own pipe.
> > > > 
> > > > struct media_pipeline *media_entity_pipeline(struct media_entity
> > > > *entity, u32 pad_index)
> > > > {
> > > >            if (pad_index >= entity->num_pads)
> > > >                    return NULL;
> > > > 
> > > >            return entity->pads[pad_index].pipe;
> > > > }
> > > 
> > > Oh, I see now where the confusion is.
> > > 
> > > media_entity_pipeline() is a convenience helper for the currently existing
> > > drivers. They access entity->pipe directly (or when we move the pipe to pad,
> > > entity->pads->pipe). So this function is just refactoring the direct access
> > > away from the drivers, which makes it easier in the future to find those
> > > drivers accessing the pipe, or change where the pipe is stored without doing
> > > any changes to the drivers.
> > > 
> > > A driver which supports streams should not use this function, as there's no
> > > "pipe for entity". We seem to be missing a similar helper function for these
> > > cases, media_pad_pipeline() or such. We can add when needed.
> > 
> > I think this could be done now, for consistency. Either the existing
> > drivers could call it on entity->pads, thus assuming they only use a single
> > pipeline for that entity. Or keep media_entity_pipeline(), but there's some
> > chance for confusion on where the pipeline is bound. At least there should
> > be a warning it's just for old existing drivers.
> > 
> 
> I clarified in the kdoc that media_entity_pipeline() is only to be used by
> drivers that do not support multiple pipelines, and I added
> media_pad_pipeline().

Sounds good, thanks!

-- 
Sakari Ailus

Re: [PATCH v12 23/30] media: subdev: Add for_each_active_route() macro

[Sakari Ailus]

Moi,

On Mon, Aug 01, 2022 at 11:40:21AM +0300, Tomi Valkeinen wrote:
> On 27/07/2022 13:36, Tomi Valkeinen wrote:
> 
> > @@ -1593,6 +1593,19 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
> >   				 const struct v4l2_subdev_krouting *routing,
> >   				 enum v4l2_subdev_routing_restriction disallow);
> > +struct v4l2_subdev_route *
> > +__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
> > +				struct v4l2_subdev_route *route);
> > +
> > +/**
> > + * for_each_active_route - iterate on all active routes of a routing table
> > + * @routing: The routing table
> > + * @route: The route iterator
> > + */
> > +#define for_each_active_route(routing, route) \
> > +	for ((route) = NULL;                  \
> > +	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
> > +
> 
> By the way, now that we can do it, how do people feel about changing the
> above (and other similar macros) to something like:
> 
> #define for_each_active_route(routing, route)        \
> 	for (struct v4l2_subdev_route *route = NULL; \
> 	     (route = __v4l2_subdev_next_active_route((routing), route));)

Will this compile? :-)

Every such macro works that way and I'd think this is expected by the
developers in general. You also often need to break out of the loop and
keep that variable intact, this also supports doing it to usual way.

I.e. I'd keep it as-is.

-- 
Sakari Ailus

Re: [PATCH v12 26/30] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Sakari Ailus]

Moi,

On Wed, Jul 27, 2022 at 01:36:35PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add a helper function to translate streams between two pads of a subdev,
> using the subdev's internal routing table.
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 26 ++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 23 +++++++++++++++++++++++
>  2 files changed, 49 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 2fc999b1b6c1..35ddf4dbe3a9 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1619,6 +1619,32 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
>  
> +u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
> +				    u32 pad0, u32 pad1, u64 *streams)
> +{
> +	const struct v4l2_subdev_krouting *routing = &state->routing;
> +	struct v4l2_subdev_route *route;
> +	u64 streams0 = 0;
> +	u64 streams1 = 0;
> +
> +	for_each_active_route(routing, route) {
> +		if (route->sink_pad == pad0 && route->source_pad == pad1 &&
> +		    (*streams & BIT_ULL(route->sink_stream))) {
> +			streams0 |= BIT_ULL(route->sink_stream);
> +			streams1 |= BIT_ULL(route->source_stream);
> +		}
> +		if (route->source_pad == pad0 && route->sink_pad == pad1 &&
> +		    (*streams & BIT_ULL(route->source_stream))) {
> +			streams0 |= BIT_ULL(route->source_stream);
> +			streams1 |= BIT_ULL(route->sink_stream);
> +		}
> +	}
> +
> +	*streams = streams0;
> +	return streams1;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_state_xlate_streams);
> +
>  int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
>  				 const struct v4l2_subdev_krouting *routing,
>  				 enum v4l2_subdev_routing_restriction disallow)
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index f749effadde2..dd5cccc827ef 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1554,6 +1554,29 @@ struct v4l2_mbus_framefmt *
>  v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
>  					     u32 pad, u32 stream);
>  
> +/**
> + * v4l2_subdev_state_xlate_streams() - Translate streams from one pad to another
> + *
> + * @state: Subdevice state
> + * @pad0: The first pad
> + * @pad1: The second pad
> + * @streams: Streams bitmask on the first pad
> + *
> + * Streams on sink pads of a subdev are routed to source pads as expressed in
> + * the subdev state routing table. Stream numbers don't necessarily match on
> + * the sink and source side of a route. This function translates stream numbers
> + * on @pad0, expressed as a bitmask in @streams, to the corresponding streams
> + * on @pad1 using the routing table from the @state. It returns the stream mask
> + * on @pad1, and updates @streams with the streams that have been found in the
> + * routing table.
> + *
> + * @pad0 and @pad1 must be a sink and a source, in any order.
> + *
> + * Return: The bitmask of streams of @pad1 that are routed to @streams on @pad0.
> + */
> +u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
> +				    u32 pad0, u32 pad1, u64 *streams);
> +
>  /**
>   * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
>   *

How and where is this meant to be used?

-- 
Sakari Ailus

Re: [PATCH v12 26/30] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Tomi Valkeinen]

Hi,

On 01/08/2022 15:37, Sakari Ailus wrote:

>> +u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
>> +				    u32 pad0, u32 pad1, u64 *streams);
>> +
>>   /**
>>    * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
>>    *
> 
> How and where is this meant to be used?

I use it in subdev driver's .enable_streams(). E.g.:

static int ub953_enable_streams(struct v4l2_subdev *sd,
				struct v4l2_subdev_state *state, u32 pad,
				u64 streams_mask)
{
	struct ub953_data *priv = sd_to_ub953(sd);
	struct media_pad *remote_pad;
	u64 sink_streams;
	int ret;

	if (streams_mask & priv->enabled_source_streams)
		return -EALREADY;

	sink_streams = v4l2_subdev_state_xlate_streams(
		state, UB953_PAD_SOURCE, UB953_PAD_SINK, &streams_mask);

	remote_pad = media_entity_remote_pad(&priv->pads[UB953_PAD_SINK]);

	ret = v4l2_subdev_enable_streams(priv->source_sd, remote_pad->index,
					 sink_streams);
	if (ret)
		return ret;

	priv->enabled_source_streams |= streams_mask;

	return 0;
}

The driver gets the source pad & stream mask, and must get the
sink pad & stream mask so that it can then call enable_streams
on its source subdev.

  Tomi

Re: [PATCH v12 00/30] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

On 03/08/2022 12:03, Milen Mitkov (Consultant) wrote:
> Hey Tomi,
> 
> thank you for providing v12 of the routing and streams support patches! 
> We are using v11 of these to add CSI virtual channels support to the 
> Qualcomm Titan platform (a.k.a qcom-camss driver) and will be moving to 
> the brand new fresh v12 now.
> 
> I ponder several questions with regards to this newly added 
> functionality (not just v12, but all versions in general):
> 
> 1. What are the main benefits of having multiple streams, that can be 
> enabled/disabled/configured, on the same pad vs. say having multiple 
> pads, each of which can be enabled/disabled/configured?

The streams and pads are kind of similar, but also different.

One difference is a conceptual one, that a pad represents some kind of 
physical connector, streams are logical "virtual" pads or connectors.

But perhaps the main practical difference is that you had a hardcoded 
amount of pads, but the amount of streams is dynamic, adjusted based on 
the routing table.

> As far as I understood, in the user-space world each stream on the end 
> of the pipeline gets exposed as a /dev/video%d node. Each node 
> corresponds to a videodev which is wrapped in a media entity that has a 

Yes.

It would be nice to have a videodev that supports multiple streams, but 
at the moment there's no API for that kind of thing. Perhaps in the future.

> single sink pad. So in the end, all streams end up as multiple "stream 
> -> pad" correspondences. I am sure there is some benefit of having 
> multiple streams vs having multiple pads, but I am struggling to put it 
> into exact words.

Consider a bridge device with, say, 2 CSI2 inputs and 2 CSI2 outputs. 
The device can route streams from either input to either output, and 
possibly modify them (say, change virtual channel number).

How many pads would have there? You would need a predefined amount of 
pads, one for each stream. So how many streams? What's a stream? With 
CSI2, we can easily define at least that we can have streams identified 
with the VC and the DT. So, 4 VCs at max, but there can be many DTs. I 
don't remember how wide the DT field is, but lets say 10 DTs. That would 
be max 40 streams per input.

So the above device would need to have 40 * 4 pads to cover "all" 
possible uses. I say "all", because it's not clear how to define a 
stream. If the device in question can, say, split the incoming frame per 
line, and somehow output each of those lines separately, then, 
effectively, there would be as many streams as there are lines. That's a 
silly example, but I just want to highlight the dynamic nature of streams.

> 2. What is your userspace modus operandi with regards to testing these 

I have my own python scripts built on top of kms++. They're really not 
in such a condition that I could share them to others. Maybe I'll at 
some point find the time to clean them up...

> changes? For example, I figured out this much with regards to media-ctl:

Yes, it should be all doable with media-ctl, v4l2-ctl.

> media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->1/1[0x1]]'
> 
> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1 
> stream on the sink pad and route it to 2 streams on its source pad. Is 
> my thinking correct?

Yes, if your HW can do that. I don't have HW that can split (or clone) a 
stream, so it's possible that the use case doesn't work.

> And I also wonder what is your preferred method to open each /dev/video 
> node in userspace concurrently? Are you, say, running 2 or more parallel 
> instances of yavta?

I do it with my script, but yes, I think running multiple yavtas should 
do the trick.

> 3. I assume, that with these changes, it's _strongly_ preferred that the 
> subdevice's active state management is left to the V4L2 API and not kept 
> internally like older drivers do?

A subdev that uses routing/streams _must_ use the subdev active state.

  Tomi

Re: [PATCH v12 00/30] v4l: routing and streams support

[Tomi Valkeinen]

On 05/08/2022 18:14, Milen Mitkov (Consultant) wrote:

>  >> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1
>  >> stream on the sink pad and route it to 2 streams on its source pad. Is
>  >> my thinking correct?
>  >
>  >Yes, if your HW can do that. I don't have HW that can split (or clone) a
>  >stream, so it's possible that the use case doesn't work.
> 
> Now here's the main question. We use the CSI decoder (CSID) hardware to 
> split
> one stream from the sensor into 2 or more streams based on datatype or 
> CSI virtual channel.
> 
> Basically, the complete pipeline is something like this, for 2 virtual 
> channels:
> 
>                                             -> ISP line 0 -> videodev 
> /dev/video0
>                                           /
> sensor -> CSIPHY -> CSID ->   
>                                           \
>                                           -> ISP line 1 -> videodev 
> /dev/video1
>                                     
>                                     
> So my idea was to make the CSID subdevice multistream API compliant
> (e.g. V4L2_SUBDEV_FL_STREAMS, manage the active state with the
> V4L2 API v4l2_subdev_get_locked_active_state, take care of routing setup 
> etc.),
> but keep the rest of the devices the way they are.

That's now how the streams support has been designed. Your sensor 
provides two streams, and all the drivers that pass through multiple 
streams needs to be ported to streams API. So in your case, I believe 
everything but the "ISP line" needs to support streams.

> The CSID subdev must take 1 stream on the sink pad and output on 2 
> source pads.
> 
> The routing configuration I use for the CSID subdev looks like this:
> 
> media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->2/0[0x1]]'
> 
> 0 - sink pad, 1 - first source pad, 2 - second source pad
> 
> However, this routing setup fails with the validation in 
> v4l2_link_validate_get_streams().
> The logic there figures these are duplicate streams because because they 
> start at the same sink pad.
> 
> To summarize my questions:
> 
> 1. Is there some sort of restriction that the same sink pad can't be 
> used for more than 1 stream starting from it?

In theory no, but it hasn't been tested. I think this case would mainly 
be cloning of the stream, not really splitting it.

> 2. Is it ok to migrate only one subdevice to the multistream API
> or should all possible subdevices in the pipeline be migrated?

It's ok to mix streams and non-streams subdevices, but the non-streams 
subdevs must only use a single stream. E.g. You could have 4 non-streams 
cameras, each providing a single stream to a bridge. The bridge would 
support streams, and the bridge would send the 4 streams in a single 
CSI-2 bus.

Now, that said, I don't think anything strictly prevents from supporting 
stream splitting, but as I mentioned above, it's not been tested or 
really even considered very much. It's also a bit ambiguous and unclear 
and I'd stay away from it if full streams support makes sense.

I think if a source subdev (sensor) knows that it's providing multiple 
streams, then it should use streams API to provide those. I.e. if the 
sensor is providing different types of data, using VCs or DTs, then 
those are clearly separate streams and the sensor driver must be aware 
of them.

Stream splitting might came into play in situations where the sensor 
provides just a single stream, but a bridge subdev splits it based on 
information the sensor can't be aware of. For example, the sensor 
provides a normal pixel stream, and the bridge subdev splits the frames 
into two halves, sending upper half to output 1 and lower half to output 2.

  Tomi

Re: [PATCH v12 00/30] v4l: routing and streams support

[Laurent Pinchart]

On Mon, Aug 08, 2022 at 09:45:38AM +0300, Tomi Valkeinen wrote:
> On 05/08/2022 18:14, Milen Mitkov (Consultant) wrote:
> 
> > >> If I want to configure the CSI decoder subdev (msm_csid0) to receive 1
> > >> stream on the sink pad and route it to 2 streams on its source pad. Is
> > >> my thinking correct?
> > >
> > > Yes, if your HW can do that. I don't have HW that can split (or clone) a
> > > stream, so it's possible that the use case doesn't work.
> > 
> > Now here's the main question. We use the CSI decoder (CSID) hardware to 
> > split
> > one stream from the sensor into 2 or more streams based on datatype or 
> > CSI virtual channel.
> > 
> > Basically, the complete pipeline is something like this, for 2 virtual 
> > channels:
> > 
> >                               -> ISP line 0 -> videodev /dev/video0
> >                             /
> > sensor -> CSIPHY -> CSID ->
> >                             \
> >                               -> ISP line 1 -> videodev /dev/video1
> >                                     
> >                                     
> > So my idea was to make the CSID subdevice multistream API compliant
> > (e.g. V4L2_SUBDEV_FL_STREAMS, manage the active state with the
> > V4L2 API v4l2_subdev_get_locked_active_state, take care of routing setup 
> > etc.),
> > but keep the rest of the devices the way they are.
> 
> That's now how the streams support has been designed. Your sensor 
> provides two streams, and all the drivers that pass through multiple 
> streams needs to be ported to streams API. So in your case, I believe 
> everything but the "ISP line" needs to support streams.

To add a bit of information here, the important thing to understand is
that streams and physical links are two different concepts. The above
diagram describes the physical links (both outside the SoC, and inside
it). Streams are carried by physical links, and a link can carry
multiple streams (hence the name "multiplexed streams" used in this
patch series). If the sensor outputs image data and embedded data with
two CSI-2 DT on one VC, that's two streams carried over the sensor ->
CSIPHY link, and the same two streams going over the CSIPHY -> CSID
link. The CSID demultiplexes the streams, with one stream going to ISP
line 0 and the other one to ISP line 1. As Tomi explained, every subdev
that deals with multiple streams has to implement the new API. This
includes, in this case, the sensor, the CSIPHY and the CSID.

If the sensor were to output two images in different resolutions over
two VCs, it would conceptually be the same, with two streams. If it were
to output image data, embedded data and black level lines over with 3
DTs over one VC, that would be three streams. And so on.

> > The CSID subdev must take 1 stream on the sink pad and output on 2 
> > source pads.
> > 
> > The routing configuration I use for the CSID subdev looks like this:
> > 
> > media-ctl -R '"msm_csid0"[0/0->1/0[0x1],0/0->2/0[0x1]]'
> > 
> > 0 - sink pad, 1 - first source pad, 2 - second source pad
> > 
> > However, this routing setup fails with the validation in 
> > v4l2_link_validate_get_streams().
> > The logic there figures these are duplicate streams because because they 
> > start at the same sink pad.
> > 
> > To summarize my questions:
> > 
> > 1. Is there some sort of restriction that the same sink pad can't be 
> > used for more than 1 stream starting from it?
> 
> In theory no, but it hasn't been tested. I think this case would mainly 
> be cloning of the stream, not really splitting it.
> 
> > 2. Is it ok to migrate only one subdevice to the multistream API
> > or should all possible subdevices in the pipeline be migrated?
> 
> It's ok to mix streams and non-streams subdevices, but the non-streams 
> subdevs must only use a single stream. E.g. You could have 4 non-streams 
> cameras, each providing a single stream to a bridge. The bridge would 
> support streams, and the bridge would send the 4 streams in a single 
> CSI-2 bus.
> 
> Now, that said, I don't think anything strictly prevents from supporting 
> stream splitting, but as I mentioned above, it's not been tested or 
> really even considered very much. It's also a bit ambiguous and unclear 
> and I'd stay away from it if full streams support makes sense.
> 
> I think if a source subdev (sensor) knows that it's providing multiple 
> streams, then it should use streams API to provide those. I.e. if the 
> sensor is providing different types of data, using VCs or DTs, then 
> those are clearly separate streams and the sensor driver must be aware 
> of them.
> 
> Stream splitting might came into play in situations where the sensor 
> provides just a single stream, but a bridge subdev splits it based on 
> information the sensor can't be aware of. For example, the sensor 
> provides a normal pixel stream, and the bridge subdev splits the frames 
> into two halves, sending upper half to output 1 and lower half to output 2.

We've tested splitting on an i.MX8MP, with two different processing
pipelines capturing the stream produced by a single YUV sensor, in
different resolutions and formats. It works (or at least worked with v11
of the streams series, I'll update the code to v13 and retest).

-- 
Regards,

Laurent Pinchart