[PATCH v13 00/34] v4l: routing and streams support

[PATCH v13 00/34] v4l: routing and streams support

[Tomi Valkeinen]

Hi,

This is v13 of the streams series. The v12 can be found from:

https://lore.kernel.org/all/20220727103639.581567-1-tomi.valkeinen@ideasonboard.com/

My work branch with additional drivers can be found from:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git streams/work-v13

And there's also the v4l-utils series v2 to add support to v4l2-ctl and
media-ctl:

https://lore.kernel.org/all/20220714132116.132498-1-tomi.valkeinen@ideasonboard.com/

Changes in v13 to v12

* media_pipeline_alloc_start: handle no mem error
* Automatically free the allocated pipeline in media_pipeline_stop()
* sanity checks for media_entity_pipeline
* improve v4l2_link_validate_get_streams (drop 'is_source')
* indent Add iterator helper for entity pads
* v4l2_subdev_set_routing: leave the current routing untouched in case of error
* v4l2_subdev_init_stream_configs: leave the current routing untouched in case of error
* Drop V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
* In subdev ioctls, check that stream == 0 if subdev doesn't support streams
* Squash for_each_active_route() uses
* Add media_pad_pipeline()
* Add video device pipeline helpers and use them in the drivers
* Rename has_route to has_pad_interdep

 Tomi

Jacopo Mondi (3):
  media: mc: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation
  media: subdev: Add for_each_active_route() macro

Laurent Pinchart (9):
  media: mc: entity: Merge media_entity_enum_init and
    __media_entity_enum_init
  media: mc: entity: Move media_entity_get_fwnode_pad() out of graph
    walk section
  media: mc: entity: Add media_entity_pipeline() to access the media
    pipeline
  media: mc: entity: Rewrite media_pipeline_start()
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations
  media: subdev: add v4l2_subdev_routing_validate() helper
  media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper
  media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations
  media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

Sakari Ailus (1):
  media: Add stream to frame descriptor

Tomi Valkeinen (21):
  media: Documentation: mc: add definitions for stream and pipeline
  media: media-entity.h: add include for min()
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  media: mc: entity: Rename streaming_count -> start_count
  media: v4l2-dev: Add videodev wrappers for media pipelines
  media: drivers: use video device pipeline start/stop
  media: drivers: use video_device_pipeline()
  media: mc: entity: add alloc variant of pipeline_start
  media: drivers: use video_device_pipeline_alloc_start()
  media: mc: entity: Add has_pad_interdep entity operation
  media: mc: convert pipeline funcs to take media_pad
  media: add V4L2_SUBDEV_FL_STREAMS
  media: add V4L2_SUBDEV_CAP_STREAMS
  media: subdev: add v4l2_subdev_has_pad_interdep()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add streams to v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

 .clang-format                                 |    1 +
 Documentation/driver-api/media/mc-core.rst    |   19 +-
 .../driver-api/media/v4l2-subdev.rst          |    8 +
 .../userspace-api/media/v4l/dev-subdev.rst    |  175 +++
 .../userspace-api/media/v4l/user-func.rst     |    1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |    5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |    5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |    5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |    5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |    5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |    5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     |  150 +++
 .../media/v4l/vidioc-subdev-g-selection.rst   |    5 +-
 drivers/media/mc/mc-device.c                  |   13 +-
 drivers/media/mc/mc-entity.c                  |  657 +++++++++--
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |    6 +-
 .../media/platform/qcom/camss/camss-video.c   |    6 +-
 .../platform/renesas/rcar-vin/rcar-core.c     |    5 +-
 .../platform/renesas/rcar-vin/rcar-dma.c      |   18 +-
 .../media/platform/renesas/vsp1/vsp1_video.c  |    6 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   14 +-
 .../samsung/exynos4-is/fimc-capture.c         |    9 +-
 .../samsung/exynos4-is/fimc-isp-video.c       |    9 +-
 .../platform/samsung/exynos4-is/fimc-lite.c   |    9 +-
 .../samsung/s3c-camif/camif-capture.c         |    6 +-
 drivers/media/platform/st/stm32/stm32-dcmi.c  |    6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |    6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |    6 +-
 drivers/media/platform/ti/cal/cal-video.c     |    6 +-
 drivers/media/platform/ti/cal/cal.h           |    1 -
 drivers/media/platform/ti/omap3isp/isp.c      |    4 +-
 drivers/media/platform/ti/omap3isp/ispvideo.c |    9 +-
 drivers/media/platform/ti/omap3isp/ispvideo.h |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |   11 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |    9 +-
 .../media/test-drivers/vimc/vimc-capture.c    |    7 +-
 drivers/media/usb/au0828/au0828-core.c        |    8 +-
 drivers/media/v4l2-core/v4l2-dev.c            |   72 ++
 drivers/media/v4l2-core/v4l2-ioctl.c          |   25 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 1038 ++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |    8 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |    6 +-
 drivers/staging/media/omap4iss/iss.c          |    4 +-
 drivers/staging/media/omap4iss/iss_video.c    |    9 +-
 drivers/staging/media/omap4iss/iss_video.h    |   11 +-
 drivers/staging/media/tegra-video/tegra210.c  |    6 +-
 include/media/media-device.h                  |   15 -
 include/media/media-entity.h                  |  162 ++-
 include/media/v4l2-dev.h                      |  104 ++
 include/media/v4l2-subdev.h                   |  382 +++++-
 include/uapi/linux/v4l2-subdev.h              |   82 +-
 51 files changed, 2827 insertions(+), 328 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.34.1

[PATCH v13 01/34] media: Documentation: mc: add definitions for stream and pipeline

[Tomi Valkeinen]

The doc talks about streams and pipelines, but doesn't really define
them. This is an attempt to define them according to my understanding.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 02481a2513b9..2a0c0aeec5f2 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,6 +213,18 @@ Link properties can be modified at runtime by calling
 Pipelines and media streams
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^
 
+A media stream is a stream of pixels or metadata originating from one or more
+source devices (such as a sensors) and flowing through media entity pads
+towards the final sinks. The stream can be modified on the route by the
+devices (e.g. scaling or pixel format conversions), or it can be split into
+multiple branches, or multiple branches can be merged.
+
+A media pipeline is a set of media streams which are interdependent. This
+interdependency can be caused by the hardware (e.g. configuration of a second
+stream cannot be changed if the first stream has been enabled) or by the driver
+due to the software design. Most commonly a media pipeline consists of a single
+stream which does not branch.
+
 When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
-- 
2.34.1

[PATCH v13 02/34] media: media-entity.h: add include for min()

[Tomi Valkeinen]

./include/media/media-entity.h:595:34: error: implicit declaration of function ‘min’

Include kernel.h to get the definition for min().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/media-entity.h | 1 +
 1 file changed, 1 insertion(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index a9a1c0ec5d1c..3e3d06c1ec36 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -15,6 +15,7 @@
 #include <linux/bug.h>
 #include <linux/container_of.h>
 #include <linux/fwnode.h>
+#include <linux/kernel.h>
 #include <linux/list.h>
 #include <linux/media.h>
 #include <linux/types.h>
-- 
2.34.1

[PATCH v13 03/34] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b661e1817470..dbab00f79527 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -358,7 +358,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.34.1

[PATCH v13 04/34] media: mc: entity: Rename streaming_count -> start_count

[Tomi Valkeinen]

'streaming_count' is a bit misleading name, as the count is increased
with media_pipeline_start(). Let's rename it to 'start_count' instead.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                            | 8 ++++----
 drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c | 4 ++--
 include/media/media-entity.h                            | 4 ++--
 3 files changed, 8 insertions(+), 8 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 11f5207f73aa..d284f616a456 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -414,8 +414,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	struct media_link *link;
 	int ret;
 
-	if (pipe->streaming_count) {
-		pipe->streaming_count++;
+	if (pipe->start_count) {
+		pipe->start_count++;
 		return 0;
 	}
 
@@ -498,7 +498,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		}
 	}
 
-	pipe->streaming_count++;
+	pipe->start_count++;
 
 	return 0;
 
@@ -551,7 +551,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	if (--pipe->streaming_count)
+	if (--pipe->start_count)
 		return;
 
 	media_graph_walk_start(graph, entity);
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index fee2aaacb26b..0ebd70275e23 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -931,7 +931,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes shouldn't
 	 * be disabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count < 2) {
+	if (rkisp1->pipe.start_count < 2) {
 		v4l2_subdev_call(rkisp1->active_sensor->sd, video, s_stream,
 				 false);
 		v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, false);
@@ -964,7 +964,7 @@ static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
 	 * If the other capture is streaming, isp and sensor nodes are already
 	 * enabled, skip them.
 	 */
-	if (rkisp1->pipe.streaming_count > 1)
+	if (rkisp1->pipe.start_count > 1)
 		return 0;
 
 	ret = v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, true);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 3e3d06c1ec36..68a8ecb2b397 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -100,11 +100,11 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
- * @streaming_count:	Streaming start count - streaming stop count
+ * @start_count:	Media pipeline start - stop count
  * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
-	int streaming_count;
+	int start_count;
 	struct media_graph graph;
 };
 
-- 
2.34.1

[PATCH v13 05/34] media: mc: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index b8176a3b76d3..25020d58eb06 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -610,7 +610,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -639,9 +639,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index d284f616a456..8ffc964b42c6 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -192,7 +192,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -203,12 +204,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					  &iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 68a8ecb2b397..00f2f0c98e68 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -316,6 +316,18 @@ struct media_entity {
 	} info;
 };
 
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
+
 /**
  * struct media_interface - A media interface graph object.
  *
-- 
2.34.1

[PATCH v13 06/34] media: mc: entity: Merge media_entity_enum_init and __media_entity_enum_init

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_enum_init() function is a wrapper around
__media_entity_enum_init() that turns a media_device pointer argument
into the maximum entity ID in the corresponding media graph.
__media_entity_enum_init() is never used outside of
media_entity_enum_init(), so the two functions can be merged together.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 10 ++++++----
 include/media/media-device.h | 15 ---------------
 include/media/media-entity.h | 10 +++++-----
 3 files changed, 11 insertions(+), 24 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 8ffc964b42c6..d048c3661421 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -58,10 +58,12 @@ static inline const char *link_type_name(struct media_link *link)
 	}
 }
 
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max)
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev)
 {
-	idx_max = ALIGN(idx_max, BITS_PER_LONG);
+	int idx_max;
+
+	idx_max = ALIGN(mdev->entity_internal_idx_max + 1, BITS_PER_LONG);
 	ent_enum->bmap = bitmap_zalloc(idx_max, GFP_KERNEL);
 	if (!ent_enum->bmap)
 		return -ENOMEM;
@@ -70,7 +72,7 @@ __must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(__media_entity_enum_init);
+EXPORT_SYMBOL_GPL(media_entity_enum_init);
 
 void media_entity_enum_cleanup(struct media_entity_enum *ent_enum)
 {
diff --git a/include/media/media-device.h b/include/media/media-device.h
index a10b30507524..86716ee7cc6c 100644
--- a/include/media/media-device.h
+++ b/include/media/media-device.h
@@ -191,21 +191,6 @@ struct usb_device;
 #define MEDIA_DEV_NOTIFY_PRE_LINK_CH	0
 #define MEDIA_DEV_NOTIFY_POST_LINK_CH	1
 
-/**
- * media_entity_enum_init - Initialise an entity enumeration
- *
- * @ent_enum: Entity enumeration to be initialised
- * @mdev: The related media device
- *
- * Return: zero on success or a negative error code.
- */
-static inline __must_check int media_entity_enum_init(
-	struct media_entity_enum *ent_enum, struct media_device *mdev)
-{
-	return __media_entity_enum_init(ent_enum,
-					mdev->entity_internal_idx_max + 1);
-}
-
 /**
  * media_device_init() - Initializes a media device element
  *
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 00f2f0c98e68..e0b2353271b6 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -439,15 +439,15 @@ static inline bool is_media_entity_v4l2_subdev(struct media_entity *entity)
 }
 
 /**
- * __media_entity_enum_init - Initialise an entity enumeration
+ * media_entity_enum_init - Initialise an entity enumeration
  *
  * @ent_enum: Entity enumeration to be initialised
- * @idx_max: Maximum number of entities in the enumeration
+ * @mdev: The related media device
  *
- * Return: Returns zero on success or a negative error code.
+ * Return: zero on success or a negative error code.
  */
-__must_check int __media_entity_enum_init(struct media_entity_enum *ent_enum,
-					  int idx_max);
+__must_check int media_entity_enum_init(struct media_entity_enum *ent_enum,
+					struct media_device *mdev);
 
 /**
  * media_entity_enum_cleanup - Release resources of an entity enumeration
-- 
2.34.1

[PATCH v13 07/34] media: mc: entity: Move media_entity_get_fwnode_pad() out of graph walk section

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The media_entity_get_fwnode_pad() function is unrelated to the graph
traversal code that it is currently bundled with. Move it with the
media_entity_remote_pad() function.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 70 ++++++++++++++++++------------------
 1 file changed, 35 insertions(+), 35 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index d048c3661421..40e302513aa0 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -369,41 +369,6 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
-int media_entity_get_fwnode_pad(struct media_entity *entity,
-				struct fwnode_handle *fwnode,
-				unsigned long direction_flags)
-{
-	struct fwnode_endpoint endpoint;
-	unsigned int i;
-	int ret;
-
-	if (!entity->ops || !entity->ops->get_fwnode_pad) {
-		for (i = 0; i < entity->num_pads; i++) {
-			if (entity->pads[i].flags & direction_flags)
-				return i;
-		}
-
-		return -ENXIO;
-	}
-
-	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
-	if (ret)
-		return ret;
-
-	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
-	if (ret < 0)
-		return ret;
-
-	if (ret >= entity->num_pads)
-		return -ENXIO;
-
-	if (!(entity->pads[ret].flags & direction_flags))
-		return -ENXIO;
-
-	return ret;
-}
-EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
-
 /* -----------------------------------------------------------------------------
  * Pipeline management
  */
@@ -923,6 +888,41 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad)
 }
 EXPORT_SYMBOL_GPL(media_entity_remote_pad);
 
+int media_entity_get_fwnode_pad(struct media_entity *entity,
+				struct fwnode_handle *fwnode,
+				unsigned long direction_flags)
+{
+	struct fwnode_endpoint endpoint;
+	unsigned int i;
+	int ret;
+
+	if (!entity->ops || !entity->ops->get_fwnode_pad) {
+		for (i = 0; i < entity->num_pads; i++) {
+			if (entity->pads[i].flags & direction_flags)
+				return i;
+		}
+
+		return -ENXIO;
+	}
+
+	ret = fwnode_graph_parse_endpoint(fwnode, &endpoint);
+	if (ret)
+		return ret;
+
+	ret = entity->ops->get_fwnode_pad(entity, &endpoint);
+	if (ret < 0)
+		return ret;
+
+	if (ret >= entity->num_pads)
+		return -ENXIO;
+
+	if (!(entity->pads[ret].flags & direction_flags))
+		return -ENXIO;
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
-- 
2.34.1

[PATCH v13 08/34] media: mc: entity: Add media_entity_pipeline() to access the media pipeline

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Replace direct access to the pipe field in drivers with a new helper
function. This will allow easier refactoring of media pipeline handling
in the MC core behind the scenes without affecting drivers.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                   |  6 ++++++
 .../platform/renesas/rcar-vin/rcar-core.c      |  5 ++---
 .../media/platform/renesas/rcar-vin/rcar-dma.c |  2 +-
 drivers/media/platform/ti/omap3isp/isp.c       |  4 +---
 drivers/media/platform/ti/omap3isp/ispvideo.c  |  3 +--
 drivers/media/platform/ti/omap3isp/ispvideo.h  | 11 +++++++++--
 drivers/media/platform/xilinx/xilinx-dma.c     |  3 +--
 drivers/media/platform/xilinx/xilinx-dma.h     |  7 ++++++-
 drivers/staging/media/imx/imx-media-utils.c    |  2 +-
 drivers/staging/media/omap4iss/iss.c           |  4 +---
 drivers/staging/media/omap4iss/iss_video.c     |  3 +--
 drivers/staging/media/omap4iss/iss_video.h     | 11 +++++++++--
 include/media/media-entity.h                   | 18 ++++++++++++++++++
 13 files changed, 57 insertions(+), 22 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 40e302513aa0..b1abaa333d13 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -923,6 +923,12 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
+{
+	return entity->pipe;
+}
+EXPORT_SYMBOL_GPL(media_entity_pipeline);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-core.c b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
index 49bdcfba010b..7b12af3ed8fb 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-core.c
@@ -786,9 +786,8 @@ static int rvin_csi2_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
 	media_device_for_each_entity(entity, &group->mdev)
 		if (media_entity_is_streaming(entity))
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 6644b498929d..924907b71263 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1281,7 +1281,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/isp.c b/drivers/media/platform/ti/omap3isp/isp.c
index 4c937f3f323e..2b5310b07bc3 100644
--- a/drivers/media/platform/ti/omap3isp/isp.c
+++ b/drivers/media/platform/ti/omap3isp/isp.c
@@ -937,10 +937,8 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_isp_pipeline(me);
-	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_entity_remote_pad(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
index 8811d6dd4ee7..44b0d55ee5d8 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.c
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
@@ -1093,8 +1093,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
-	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
+	pipe = to_isp_pipeline(&video->video.entity) ? : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
 	if (ret)
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.h b/drivers/media/platform/ti/omap3isp/ispvideo.h
index a0908670c0cf..1d23df576e6b 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.h
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.h
@@ -99,8 +99,15 @@ struct isp_pipeline {
 	unsigned int external_width;
 };
 
-#define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+static inline struct isp_pipeline *to_isp_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct isp_pipeline, pipe);
+}
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 338c3661d809..72eff5ef626b 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,8 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
-	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
+	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
 	if (ret < 0)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..3ea10f6b0bb9 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,12 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	struct media_pipeline *pipe = media_entity_pipeline(e);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 94bc866ca28c..5d9c6fc6f2e0 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -871,7 +871,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (media_entity_pipeline(entity))
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..77760d913b49 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,10 +548,8 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
-		return 0;
 	pipe = to_iss_pipeline(me);
-	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
+	if (!pipe || pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
 		return 0;
 	pad = media_entity_remote_pad(&pipe->output->pad);
 	return pad->entity == me;
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..67d63a400fa2 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -870,8 +870,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = to_iss_pipeline(&video->video.entity) ? : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..ca2d5edb6261 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -90,8 +90,15 @@ struct iss_pipeline {
 	int external_bpp;
 };
 
-#define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+static inline struct iss_pipeline *to_iss_pipeline(struct media_entity *entity)
+{
+	struct media_pipeline *pipe = media_entity_pipeline(entity);
+
+	if (!pipe)
+		return NULL;
+
+	return container_of(pipe, struct iss_pipeline, pipe);
+}
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index e0b2353271b6..3438954892b7 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -884,6 +884,24 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
 	return entity->pipe;
 }
 
+/**
+ * media_entity_pipeline - Get the media pipeline an entity is part of
+ * @entity: The entity
+ *
+ * This function returns the media pipeline that an entity has been associated
+ * with when constructing the pipeline with media_pipeline_start(). The pointer
+ * remains valid until media_pipeline_stop() is called.
+ *
+ * In general, entities can be part of multiple pipelines, when carrying
+ * multiple streams (either on different pads, or on the same pad using
+ * multiplexed streams). This function is to be used only for entities that
+ * do not support multiple pipelines.
+ *
+ * Return: The media_pipeline the entity is part of, or NULL if the entity is
+ * not part of any pipeline.
+ */
+struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
+
 /**
  * media_entity_get_fwnode_pad - Get pad number from fwnode
  *
-- 
2.34.1

[PATCH v13 09/34] media: v4l2-dev: Add videodev wrappers for media pipelines

[Tomi Valkeinen]

With the upcoming stream related improvements to the pipelines, the
pipelines are moved from media entities to media pads. As the drivers
currently use the pipelines with the entity based model, moving the
pipelines to pads will cause changes to the drivers.

However, most of the uses of media pipelines are related to a video
device (a DMA engine) with a single pad, and thus there's never a need
to support multiple pads in these use cases. We can avoid pushing the
complexities of the pad based model to the drivers by adding video
device wrappers for the pipeline related functions.

This patch adds a number of wrappers to media_pipeline functions, all of
which take a video_device as a parameter (instead of a media_entity),
and verify that there's just one pad.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-dev.c | 61 ++++++++++++++++++++
 include/media/v4l2-dev.h           | 90 ++++++++++++++++++++++++++++++
 2 files changed, 151 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index d00237ee4cae..7f933ff89fd4 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1095,6 +1095,67 @@ void video_unregister_device(struct video_device *vdev)
 }
 EXPORT_SYMBOL(video_unregister_device);
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+__must_check int video_device_pipeline_start(struct video_device *vdev,
+					     struct media_pipeline *pipe)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_start(entity, pipe);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_start);
+
+__must_check int __video_device_pipeline_start(struct video_device *vdev,
+					       struct media_pipeline *pipe)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return __media_pipeline_start(entity, pipe);
+}
+EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
+
+void video_device_pipeline_stop(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return;
+
+	return media_pipeline_stop(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
+
+void __video_device_pipeline_stop(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return;
+
+	return __media_pipeline_stop(entity);
+}
+EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
+
+struct media_pipeline *video_device_pipeline(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (WARN_ON(entity->num_pads != 1))
+		return NULL;
+
+	return media_entity_pipeline(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 /*
  *	Initialise video for linux
  */
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 5cf1edefb822..4ccc24f5918a 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -539,4 +539,94 @@ static inline int video_is_registered(struct video_device *vdev)
 	return test_bit(V4L2_FL_REGISTERED, &vdev->flags);
 }
 
+#if defined(CONFIG_MEDIA_CONTROLLER)
+
+/**
+ * video_device_pipeline_start - Mark a pipeline as streaming
+ * @vdev: Starting video device
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * Mark all entities connected to a given video device through enabled links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every entity in the pipeline and stored in the media_entity pipe
+ * field.
+ *
+ * Calls to this function can be nested, in which case the same number of
+ * video_device_pipeline_stop() calls will be required to stop streaming. The
+ * pipeline pointer must be identical for all nested calls to
+ * video_device_pipeline_start().
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper to media_pipeline_start().
+ */
+__must_check int video_device_pipeline_start(struct video_device *vdev,
+					     struct media_pipeline *pipe);
+
+/**
+ * __video_device_pipeline_start - Mark a pipeline as streaming
+ *
+ * @vdev: Starting video device
+ * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ *
+ * ..note:: This is the non-locking version of video_device_pipeline_start()
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper to __media_pipeline_start().
+ */
+__must_check int __video_device_pipeline_start(struct video_device *vdev,
+					       struct media_pipeline *pipe);
+
+/**
+ * video_device_pipeline_stop - Mark a pipeline as not streaming
+ * @vdev: Starting video device
+ *
+ * Mark all entities connected to a given entity through enabled links, either
+ * directly or indirectly, as not streaming. The media_entity pipe field is
+ * reset to %NULL.
+ *
+ * If multiple calls to media_pipeline_start() have been made, the same
+ * number of calls to this function are required to mark the pipeline as not
+ * streaming.
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper to media_pipeline_stop().
+ */
+void video_device_pipeline_stop(struct video_device *vdev);
+
+/**
+ * __video_device_pipeline_stop - Mark a pipeline as not streaming
+ *
+ * @vdev: Starting video device
+ *
+ * .. note:: This is the non-locking version of media_pipeline_stop()
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper to __media_pipeline_stop().
+ */
+void __video_device_pipeline_stop(struct video_device *vdev);
+
+/**
+ * video_device_pipeline - Get the media pipeline a video device is part of
+ * @vdev: The video device
+ *
+ * This function returns the media pipeline that a video device has been
+ * associated with when constructing the pipeline with
+ * video_device_pipeline_start(). The pointer remains valid until
+ * video_device_pipeline_stop() is called.
+ *
+ * Return: The media_pipeline the video device is part of, or NULL if the video
+ * device is not part of any pipeline.
+ *
+ * The video device must contain a single pad.
+ *
+ * This is a convenience wrapper to media_entity_pipeline().
+ */
+struct media_pipeline *video_device_pipeline(struct video_device *vdev);
+
+#endif /* CONFIG_MEDIA_CONTROLLER */
+
 #endif /* _V4L2_DEV_H */
-- 
2.34.1

[PATCH v13 10/34] media: drivers: use video device pipeline start/stop

[Tomi Valkeinen]

Convert the media drivers to use video device based pipeline start/stop
where possible.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c          |  6 +++---
 drivers/media/platform/qcom/camss/camss-video.c        |  6 +++---
 drivers/media/platform/renesas/rcar-vin/rcar-dma.c     |  6 +++---
 drivers/media/platform/renesas/vsp1/vsp1_video.c       |  6 +++---
 .../media/platform/rockchip/rkisp1/rkisp1-capture.c    | 10 +++++-----
 .../media/platform/samsung/exynos4-is/fimc-capture.c   |  9 ++++-----
 .../media/platform/samsung/exynos4-is/fimc-isp-video.c |  9 ++++-----
 drivers/media/platform/samsung/exynos4-is/fimc-lite.c  |  9 ++++-----
 drivers/media/platform/st/stm32/stm32-dcmi.c           |  6 +++---
 drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c     |  6 +++---
 drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c   |  6 +++---
 drivers/media/platform/ti/cal/cal-video.c              |  6 +++---
 drivers/media/platform/ti/omap3isp/ispvideo.c          |  6 +++---
 drivers/media/platform/xilinx/xilinx-dma.c             |  6 +++---
 drivers/media/test-drivers/vimc/vimc-capture.c         |  7 +++----
 drivers/staging/media/ipu3/ipu3-v4l2.c                 |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c             |  6 +++---
 drivers/staging/media/tegra-video/tegra210.c           |  6 +++---
 18 files changed, 59 insertions(+), 63 deletions(-)

diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index dbdbdb648a0d..b157fcc63b60 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -989,7 +989,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = video_device_pipeline_start(&q->vdev, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1009,7 +1009,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	video_device_pipeline_stop(&q->vdev);
 fail_pipeline:
 	dev_dbg(dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1030,7 +1030,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	video_device_pipeline_stop(&q->vdev);
 	pm_runtime_put(dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index 307bb1dc4589..21f9fb647692 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -493,7 +493,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = video_device_pipeline_start(vdev, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -522,7 +522,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	video_device_pipeline_stop(vdev);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -553,7 +553,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	video_device_pipeline_stop(vdev);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 924907b71263..548067f19576 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1265,7 +1265,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		video_device_pipeline_stop(&vin->vdev);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1282,7 +1282,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __video_device_pipeline_start(&vin->vdev, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1291,7 +1291,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		video_device_pipeline_stop(&vin->vdev);
 
 	return ret;
 }
diff --git a/drivers/media/platform/renesas/vsp1/vsp1_video.c b/drivers/media/platform/renesas/vsp1/vsp1_video.c
index 51219b1b6ea9..62811e8aebe6 100644
--- a/drivers/media/platform/renesas/vsp1/vsp1_video.c
+++ b/drivers/media/platform/renesas/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1046,7 +1046,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1070,7 +1070,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index 0ebd70275e23..f497c5b919eb 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -918,7 +918,7 @@ static void rkisp1_cap_stream_disable(struct rkisp1_capture *cap)
  *
  * Call s_stream(false) in the reverse order from
  * rkisp1_pipeline_stream_enable() and disable the DMA engine.
- * Should be called before media_pipeline_stop()
+ * Should be called before video_device_pipeline_stop()
  */
 static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
 	__must_hold(&cap->rkisp1->stream_lock)
@@ -945,7 +945,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
  * rkisp1_pipeline_stream_enable - enable nodes in the pipeline
  *
  * Enable the DMA Engine and call s_stream(true) through the pipeline.
- * Should be called after media_pipeline_start()
+ * Should be called after video_device_pipeline_start()
  */
 static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
 	__must_hold(&cap->rkisp1->stream_lock)
@@ -1009,7 +1009,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -1023,7 +1023,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = video_device_pipeline_start(&cap->vnode.vdev, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1059,7 +1059,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&cap->vnode.vdev);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
index 7ff4024003f4..e1d43172634c 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		video_device_pipeline_stop(&vc->ve.vdev);
 		vc->streaming = false;
 	}
 
@@ -1176,7 +1176,6 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 {
 	struct fimc_dev *fimc = video_drvdata(file);
 	struct fimc_vid_cap *vc = &fimc->vid_cap;
-	struct media_entity *entity = &vc->ve.vdev.entity;
 	struct fimc_source_info *si = NULL;
 	struct v4l2_subdev *sd;
 	int ret;
@@ -1184,7 +1183,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = video_device_pipeline_start(&vc->ve.vdev, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1217,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&vc->ve.vdev);
 	return ret;
 }
 
@@ -1234,7 +1233,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		video_device_pipeline_stop(&vc->ve.vdev);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
index 83688a7982f7..3bbebaf1e989 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
@@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
 	is_singular_file = v4l2_fh_is_singular_file(file);
 
 	if (is_singular_file && ivc->streaming) {
-		media_pipeline_stop(entity);
+		video_device_pipeline_stop(&ivc->ve.vdev);
 		ivc->streaming = 0;
 	}
 
@@ -490,10 +490,9 @@ static int isp_video_streamon(struct file *file, void *priv,
 {
 	struct fimc_isp *isp = video_drvdata(file);
 	struct exynos_video_entity *ve = &isp->video_capture.ve;
-	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = video_device_pipeline_start(&ve->vdev, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -508,7 +507,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	video_device_pipeline_stop(&ve->vdev);
 	return ret;
 }
 
@@ -523,7 +522,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	video_device_pipeline_stop(&video->ve.vdev);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
index 1a396b7cd9a9..87ce591ccb58 100644
--- a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			video_device_pipeline_stop(&fimc->ve.vdev);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -812,13 +812,12 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 			      enum v4l2_buf_type type)
 {
 	struct fimc_lite *fimc = video_drvdata(file);
-	struct media_entity *entity = &fimc->ve.vdev.entity;
 	int ret;
 
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = video_device_pipeline_start(&fimc->ve.vdev, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -835,7 +834,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	video_device_pipeline_stop(&fimc->ve.vdev);
 	return 0;
 }
 
@@ -849,7 +848,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	video_device_pipeline_stop(&fimc->ve.vdev);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/st/stm32/stm32-dcmi.c b/drivers/media/platform/st/stm32/stm32-dcmi.c
index 09a743cd7004..c17bf075fc49 100644
--- a/drivers/media/platform/st/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/st/stm32/stm32-dcmi.c
@@ -752,7 +752,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_unlocked;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = video_device_pipeline_start(dcmi->vdev, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -866,7 +866,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	video_device_pipeline_stop(dcmi->vdev);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -893,7 +893,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	video_device_pipeline_stop(dcmi->vdev);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 0912a1b6d525..17ad9a3caaa5 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	video_device_pipeline_stop(&csi->vdev);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	video_device_pipeline_stop(&csi->vdev);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 682c26536034..de4c0d47240f 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	video_device_pipeline_stop(&video->vdev);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	video_device_pipeline_stop(&video->vdev);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
index 07ae1a34e6b0..ae29130df819 100644
--- a/drivers/media/platform/ti/cal/cal-video.c
+++ b/drivers/media/platform/ti/cal/cal-video.c
@@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -760,7 +760,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	cal_ctx_unprepare(ctx);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	video_device_pipeline_stop(&ctx->vdev);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -781,7 +781,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	video_device_pipeline_stop(&ctx->vdev);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
index 44b0d55ee5d8..9ab89151c4f4 100644
--- a/drivers/media/platform/ti/omap3isp/ispvideo.c
+++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
@@ -1103,7 +1103,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1160,7 +1160,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1227,7 +1227,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 72eff5ef626b..fcec691ce7fc 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -404,7 +404,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 */
 	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -430,7 +430,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	video_device_pipeline_stop(&dma->video);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -458,7 +458,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	video_device_pipeline_stop(&dma->video);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index d1e2d0739c00..a2296393266c 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -241,13 +241,12 @@ static void vimc_cap_return_all_buffers(struct vimc_cap_device *vcap,
 static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 {
 	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
-	struct media_entity *entity = &vcap->vdev.entity;
 	int ret;
 
 	vcap->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcap->stream.pipe);
+	ret = video_device_pipeline_start(&vcap->vdev, &vcap->stream.pipe);
 	if (ret) {
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -255,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		video_device_pipeline_stop(&vcap->vdev);
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -274,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcap->vdev.entity);
+	video_device_pipeline_stop(&vcap->vdev);
 
 	/* Release all active buffers */
 	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index d1c539cefba8..e06d2012a4ef 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -486,7 +486,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
 	atomic_set(&node->sequence, 0);
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = video_device_pipeline_start(&node->vdev, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -511,7 +511,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	video_device_pipeline_stop(&node->vdev);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 67d63a400fa2..3c7e35b654df 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -886,7 +886,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -977,7 +977,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1031,7 +1031,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	video_device_pipeline_stop(&video->video);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d58370a84737 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = video_device_pipeline_start(&chan->video, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	video_device_pipeline_stop(&chan->video);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	video_device_pipeline_stop(&chan->video);
 }
 
 /*
-- 
2.34.1

[PATCH v13 11/34] media: drivers: use video_device_pipeline()

[Tomi Valkeinen]

Use video_device_pipeline() in the drivers instead of
media_entity_pipeline().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/xilinx/xilinx-dma.c | 4 ++--
 drivers/media/platform/xilinx/xilinx-dma.h | 4 ++--
 2 files changed, 4 insertions(+), 4 deletions(-)

diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index fcec691ce7fc..040d6e7cceeb 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
+	pipe = to_xvip_pipeline(&dma->video) ? : &dma->pipe;
 
 	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
 	if (ret < 0)
@@ -447,7 +447,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 {
 	struct xvip_dma *dma = vb2_get_drv_priv(vq);
-	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video.entity);
+	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video);
 	struct xvip_dma_buffer *buf, *nbuf;
 
 	/* Stop the pipeline. */
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 3ea10f6b0bb9..9c6d4c18d1a9 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -45,9 +45,9 @@ struct xvip_pipeline {
 	struct xvip_dma *output;
 };
 
-static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
+static inline struct xvip_pipeline *to_xvip_pipeline(struct video_device *vdev)
 {
-	struct media_pipeline *pipe = media_entity_pipeline(e);
+	struct media_pipeline *pipe = video_device_pipeline(vdev);
 
 	if (!pipe)
 		return NULL;
-- 
2.34.1

[PATCH v13 12/34] media: mc: entity: add alloc variant of pipeline_start

[Tomi Valkeinen]

Add new variant of media_pipeline_start(), media_pipeline_alloc_start().

media_pipeline_alloc_start() can be used by drivers that do not need to
extend the media_pipeline. The function will either use the pipeline
already associated with the entity, if such exists, or allocate a new
pipeline.

When media_pipeline_stop() is called and the pipeline's use count drops
to zero, the pipeline is automatically freed.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c       | 41 ++++++++++++++++++++++++++++++
 drivers/media/v4l2-core/v4l2-dev.c | 11 ++++++++
 include/media/media-entity.h       | 14 ++++++++++
 include/media/v4l2-dev.h           | 14 ++++++++++
 4 files changed, 80 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b1abaa333d13..d277eed11caf 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -529,6 +529,8 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_cleanup(graph);
 
+	if (pipe->allocated)
+		kfree(pipe);
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
@@ -542,6 +544,45 @@ void media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
+__must_check int media_pipeline_alloc_start(struct media_entity *entity)
+{
+	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pipeline *pipe;
+	int ret;
+	bool new_pipe = false;
+
+	mutex_lock(&mdev->graph_mutex);
+
+	/*
+	 * Is the entity already part of a pipeline? If not, we need to allocate
+	 * a pipe.
+	 */
+	pipe = media_entity_pipeline(entity);
+	if (!pipe) {
+		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
+
+		if (!pipe) {
+			ret = -ENOMEM;
+			goto out;
+		}
+
+		new_pipe = true;
+		pipe->allocated = true;
+	}
+
+	ret = __media_pipeline_start(entity, pipe);
+	if (ret) {
+		if (new_pipe)
+			kfree(pipe);
+	}
+
+out:
+	mutex_unlock(&mdev->graph_mutex);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
+
 /* -----------------------------------------------------------------------------
  * Links management
  */
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 7f933ff89fd4..945bb867a4c1 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1143,6 +1143,17 @@ void __video_device_pipeline_stop(struct video_device *vdev)
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
 
+__must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
+{
+	struct media_entity *entity = &vdev->entity;
+
+	if (entity->num_pads != 1)
+		return -ENODEV;
+
+	return media_pipeline_alloc_start(entity);
+}
+EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
+
 struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 {
 	struct media_entity *entity = &vdev->entity;
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 3438954892b7..3d0d5e3c3333 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -104,6 +104,7 @@ struct media_graph {
  * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
+	bool allocated;
 	int start_count;
 	struct media_graph graph;
 };
@@ -1028,6 +1029,19 @@ void media_pipeline_stop(struct media_entity *entity);
  */
 void __media_pipeline_stop(struct media_entity *entity);
 
+/**
+ * media_pipeline_alloc_start - Mark a pipeline as streaming
+ * @entity: Starting entity
+ *
+ * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
+ * of working on a given pipeline the function will use an existing pipeline if
+ * the entity is already part of a pipeline, or allocate a new pipeline.
+ *
+ * Calls to media_pipeline_alloc_start() must be matched with
+ * media_pipeline_stop().
+ */
+__must_check int media_pipeline_alloc_start(struct media_entity *entity);
+
 /**
  * media_devnode_create() - creates and initializes a device node interface
  *
diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
index 4ccc24f5918a..29eb61244d1f 100644
--- a/include/media/v4l2-dev.h
+++ b/include/media/v4l2-dev.h
@@ -609,6 +609,20 @@ void video_device_pipeline_stop(struct video_device *vdev);
  */
 void __video_device_pipeline_stop(struct video_device *vdev);
 
+/**
+ * video_device_pipeline_alloc_start - Mark a pipeline as streaming
+ * @vdev: Starting video device
+ *
+ * video_device_pipeline_alloc_start() is similar to video_device_pipeline_start()
+ * but instead of working on a given pipeline the function will use an
+ * existing pipeline if the video device is already part of a pipeline, or
+ * allocate a new pipeline.
+ *
+ * Calls to video_device_pipeline_alloc_start() must be matched with
+ * video_device_pipeline_stop().
+ */
+__must_check int video_device_pipeline_alloc_start(struct video_device *vdev);
+
 /**
  * video_device_pipeline - Get the media pipeline a video device is part of
  * @vdev: The video device
-- 
2.34.1

[PATCH v13 13/34] media: drivers: use video_device_pipeline_alloc_start()

[Tomi Valkeinen]

Use video_device_pipeline_alloc_start() instead of manually
allocating/managing the media pipeline storage.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/renesas/rcar-vin/rcar-dma.c | 14 +-------------
 drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c |  2 +-
 .../media/platform/sunxi/sun6i-csi/sun6i_video.c   |  2 +-
 drivers/media/platform/ti/cal/cal-video.c          |  2 +-
 drivers/media/platform/ti/cal/cal.h                |  1 -
 5 files changed, 4 insertions(+), 17 deletions(-)

diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
index 548067f19576..884875600231 100644
--- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
@@ -1244,8 +1244,6 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd,
 
 static int rvin_set_stream(struct rvin_dev *vin, int on)
 {
-	struct media_pipeline *pipe;
-	struct media_device *mdev;
 	struct v4l2_subdev *sd;
 	struct media_pad *pad;
 	int ret;
@@ -1273,17 +1271,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret)
 		return ret;
 
-	/*
-	 * The graph lock needs to be taken to protect concurrent
-	 * starts of multiple VIN instances as they might share
-	 * a common subdevice down the line and then should use
-	 * the same pipe.
-	 */
-	mdev = vin->vdev.entity.graph_obj.mdev;
-	mutex_lock(&mdev->graph_mutex);
-	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
-	ret = __video_device_pipeline_start(&vin->vdev, pipe);
-	mutex_unlock(&mdev->graph_mutex);
+	ret = video_device_pipeline_alloc_start(&vin->vdev);
 	if (ret)
 		return ret;
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 17ad9a3caaa5..a3e826a755fc 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
+	ret = video_device_pipeline_alloc_start(&csi->vdev);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index de4c0d47240f..436922503ece 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
+	ret = video_device_pipeline_alloc_start(&video->vdev);
 	if (ret < 0)
 		goto clear_dma_queue;
 
diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
index ae29130df819..e8122e7ec944 100644
--- a/drivers/media/platform/ti/cal/cal-video.c
+++ b/drivers/media/platform/ti/cal/cal-video.c
@@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
+	ret = video_device_pipeline_alloc_start(&ctx->vdev);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
diff --git a/drivers/media/platform/ti/cal/cal.h b/drivers/media/platform/ti/cal/cal.h
index 61409ddced98..648cec43c3fc 100644
--- a/drivers/media/platform/ti/cal/cal.h
+++ b/drivers/media/platform/ti/cal/cal.h
@@ -174,7 +174,6 @@ struct cal_camerarx {
 	struct device_node	*source_ep_node;
 	struct device_node	*source_node;
 	struct v4l2_subdev	*source;
-	struct media_pipeline	pipe;
 
 	struct v4l2_subdev	subdev;
 	struct media_pad	pads[CAL_CAMERARX_NUM_PADS];
-- 
2.34.1

[PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

[Note: the code is mostly from Laurent but the patch description is from Tomi]

The media_pipeline_start() and media_pipeline_stop() functions use the
media graph walk API to traverse the graph and validate the pipeline.
The graph walk traverses the media graph following links between the
entities.

Also, while the pipeline can't change between the start and stop calls,
the graph is walked again from scratch at stop time, or any time a
driver needs to inspect the pipeline.

With the upcoming multiplexed streams support we will need a bit more
intelligent pipeline construction, as e.g. an entity may be passing
through two independent streams via separate pads, in which case those
pads should not be part of the same pipeline.

This patch essentially rewrites the media_pipeline_start/stop so that
a pipeline is defined as a set of pads instead of entities and the media
graph traversal considers the pad interdependencies when choosing which
links to follow.

Currently all the entity's pads are considered as interdependent. This
means that the behavior with all the current drivers stays the same, but
in the future we can define a more fine-grained pipeline construction.

Additionally the media pipeline's pads are cached at
media_pipeline_start() time, and re-used at media_pipeline_stop() which
avoid the need to re-walk the whole graph as the previous implementation
did.

Also, caching pads in the pipeline can serve in the future as the
foundation to provide a better API than the media graph walk to drivers
to iterate over pads and entities in the pipeline.

Note that the old media_pipeline_start/stop used the media graph walk
API. The new version does not use the media graph walk API, but instead
a new implementation.

There are two reason for not changing the graph walk: it proved to be
rather difficult to change the graph walk to have the features
implemented in this patch, and second, this keeps the backward
compatibility of the graph walk as there are users of the graph walk API

The long term plan is that all the existing code would be converted to
use the new cached pipeline, thus allowing us to remove the graph walk.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst |   7 +-
 drivers/media/mc/mc-entity.c               | 523 ++++++++++++++++++---
 include/media/media-entity.h               |  69 ++-
 3 files changed, 513 insertions(+), 86 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 2a0c0aeec5f2..9c0b09745c0a 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -229,14 +229,13 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads which are part of the pipeline as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
+pipeline through the struct media_pad
 pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index d277eed11caf..df8e82be2bcc 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -225,6 +225,27 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+/*
+ * This function checks the interdependency inside the entity between @pad0
+ * and @pad1. If two pads are interdependent they are part of the same pipeline
+ * and enabling one of the pads means that the other pad will become "locked"
+ * and doesn't allow configuration changes.
+ *
+ * For the time being all pads are considered interdependent.
+ */
+static bool media_entity_has_pad_interdep(struct media_entity *entity,
+					  unsigned int pad0, unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (entity->pads[pad0].flags & entity->pads[pad1].flags &
+	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
+		return false;
+
+	return true;
+}
+
 static struct media_entity *
 media_entity_other(struct media_entity *entity, struct media_link *link)
 {
@@ -373,97 +394,436 @@ EXPORT_SYMBOL_GPL(media_graph_walk_next);
  * Pipeline management
  */
 
+/*
+ * The pipeline traversal stack stores pads that are reached during graph
+ * traversal, with a list of links to be visited to continue the traversal.
+ * When a new pad is reached, an entry is pushed on the top of the stack and
+ * points to the incoming pad and the first link of the entity.
+ *
+ * To find further pads in the pipeline, the traversal algorithm follows
+ * internal pad dependencies in the entity, and then links in the graph. It
+ * does so by iterating over all links of the entity, and following enabled
+ * links that originate from a pad that is internally connected to the incoming
+ * pad, as reported by the media_entity_has_pad_interdep() function.
+ */
+
+/**
+ * struct media_pipeline_walk_entry - Entry in the pipeline traversal stack
+ *
+ * @pad: The media pad being visited
+ * @links: Links left to be visited
+ */
+struct media_pipeline_walk_entry {
+	struct media_pad *pad;
+	struct list_head *links;
+};
+
+/**
+ * struct media_pipeline_walk - State used by the media pipeline traversal
+ *				algorithm
+ *
+ * @mdev: The media device
+ * @stack: Depth-first search stack
+ * @stack.size: Number of allocated entries in @stack.entries
+ * @stack.top: Index of the top stack entry (-1 if the stack is empty)
+ * @stack.entries: Stack entries
+ */
+struct media_pipeline_walk {
+	struct media_device *mdev;
+
+	struct {
+		unsigned int size;
+		int top;
+		struct media_pipeline_walk_entry *entries;
+	} stack;
+};
+
+#define MEDIA_PIPELINE_STACK_GROW_STEP		16
+
+static struct media_pipeline_walk_entry *
+media_pipeline_walk_top(struct media_pipeline_walk *walk)
+{
+	return &walk->stack.entries[walk->stack.top];
+}
+
+static bool media_pipeline_walk_empty(struct media_pipeline_walk *walk)
+{
+	return walk->stack.top == -1;
+}
+
+/* Increase the stack size by MEDIA_PIPELINE_STACK_GROW_STEP elements. */
+static int media_pipeline_walk_resize(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entries;
+	unsigned int new_size;
+
+	/* Safety check, to avoid stack overflows in case of bugs. */
+	if (walk->stack.size >= 256)
+		return -E2BIG;
+
+	new_size = walk->stack.size + MEDIA_PIPELINE_STACK_GROW_STEP;
+
+	entries = krealloc(walk->stack.entries,
+			   new_size * sizeof(*walk->stack.entries),
+			   GFP_KERNEL);
+	if (!entries)
+		return -ENOMEM;
+
+	walk->stack.entries = entries;
+	walk->stack.size = new_size;
+
+	return 0;
+}
+
+/* Push a new entry on the stack. */
+static int media_pipeline_walk_push(struct media_pipeline_walk *walk,
+				    struct media_pad *pad)
+{
+	struct media_pipeline_walk_entry *entry;
+	int ret;
+
+	if (walk->stack.top + 1 >= walk->stack.size) {
+		ret = media_pipeline_walk_resize(walk);
+		if (ret)
+			return ret;
+	}
+
+	walk->stack.top++;
+	entry = media_pipeline_walk_top(walk);
+	entry->pad = pad;
+	entry->links = pad->entity->links.next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: pushed entry %u: '%s':%u\n",
+		walk->stack.top, pad->entity->name, pad->index);
+
+	return 0;
+}
+
+/*
+ * Move the top entry link cursor to the next link. If all links of the entry
+ * have been visited, pop the entry itself.
+ */
+static void media_pipeline_walk_pop(struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry;
+
+	if (WARN_ON(walk->stack.top < 0))
+		return;
+
+	entry = media_pipeline_walk_top(walk);
+
+	if (entry->links->next == &entry->pad->entity->links) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: entry %u has no more links, popping\n",
+			walk->stack.top);
+
+		walk->stack.top--;
+		return;
+	}
+
+	entry->links = entry->links->next;
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: moved entry %u to next link\n",
+		walk->stack.top);
+}
+
+/* Free all memory allocated while walking the pipeline. */
+static void media_pipeline_walk_destroy(struct media_pipeline_walk *walk)
+{
+	kfree(walk->stack.entries);
+}
+
+/* Add a pad to the pipeline and push it to the stack. */
+static int media_pipeline_add_pad(struct media_pipeline *pipe,
+				  struct media_pipeline_walk *walk,
+				  struct media_pad *pad)
+{
+	struct media_pipeline_pad *ppad;
+
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		if (ppad->pad == pad) {
+			dev_dbg(pad->graph_obj.mdev->dev,
+				"media pipeline: already contains pad '%s':%u\n",
+				pad->entity->name, pad->index);
+			return 0;
+		}
+	}
+
+	ppad = kzalloc(sizeof(*ppad), GFP_KERNEL);
+	if (!ppad)
+		return -ENOMEM;
+
+	ppad->pipe = pipe;
+	ppad->pad = pad;
+
+	list_add_tail(&ppad->list, &pipe->pads);
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline: added pad '%s':%u\n",
+		pad->entity->name, pad->index);
+
+	return media_pipeline_walk_push(walk, pad);
+}
+
+/* Explore the next link of the entity at the top of the stack. */
+static int media_pipeline_explore_next_link(struct media_pipeline *pipe,
+					    struct media_pipeline_walk *walk)
+{
+	struct media_pipeline_walk_entry *entry = media_pipeline_walk_top(walk);
+	struct media_pad *pad;
+	struct media_link *link;
+	struct media_pad *local;
+	struct media_pad *remote;
+	int ret;
+
+	pad = entry->pad;
+	link = list_entry(entry->links, typeof(*link), list);
+	media_pipeline_walk_pop(walk);
+
+	dev_dbg(walk->mdev->dev,
+		"media pipeline: exploring link '%s':%u -> '%s':%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	/* Skip links that are not enabled. */
+	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (disabled)\n");
+		return 0;
+	}
+
+	/* Get the local pad and remote pad. */
+	if (link->source->entity == pad->entity) {
+		local = link->source;
+		remote = link->sink;
+	} else {
+		local = link->sink;
+		remote = link->source;
+	}
+
+	/*
+	 * Skip links that originate from a different pad than the incoming pad
+	 * that is not connected internally in the entity to the incoming pad.
+	 */
+	if (pad != local &&
+	    !media_entity_has_pad_interdep(pad->entity, pad->index, local->index)) {
+		dev_dbg(walk->mdev->dev,
+			"media pipeline: skipping link (no route)\n");
+		return 0;
+	}
+
+	/*
+	 * Add the local and remote pads of the link to the pipeline and push
+	 * them to the stack, if they're not already present.
+	 */
+	ret = media_pipeline_add_pad(pipe, walk, local);
+	if (ret)
+		return ret;
+
+	ret = media_pipeline_add_pad(pipe, walk, remote);
+	if (ret)
+		return ret;
+
+	return 0;
+}
+
+static void media_pipeline_cleanup(struct media_pipeline *pipe)
+{
+	while (!list_empty(&pipe->pads)) {
+		struct media_pipeline_pad *ppad;
+
+		ppad = list_first_entry(&pipe->pads, typeof(*ppad), list);
+		list_del(&ppad->list);
+		kfree(ppad);
+	}
+}
+
+static int media_pipeline_populate(struct media_pipeline *pipe,
+				   struct media_pad *pad)
+{
+	struct media_pipeline_walk walk = { };
+	struct media_pipeline_pad *ppad;
+	int ret;
+
+	/*
+	 * Populate the media pipeline by walking the media graph, starting
+	 * from @pad.
+	 */
+	INIT_LIST_HEAD(&pipe->pads);
+	pipe->mdev = pad->graph_obj.mdev;
+
+	walk.mdev = pipe->mdev;
+	walk.stack.top = -1;
+	ret = media_pipeline_add_pad(pipe, &walk, pad);
+	if (ret)
+		goto done;
+
+	/*
+	 * Use a depth-first search algorithm: as long as the stack is not
+	 * empty, explore the next link of the top entry. The
+	 * media_pipeline_explore_next_link() function will either move to the
+	 * next link, pop the entry if fully visited, or add new entries on
+	 * top.
+	 */
+	while (!media_pipeline_walk_empty(&walk)) {
+		ret = media_pipeline_explore_next_link(pipe, &walk);
+		if (ret)
+			goto done;
+	}
+
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"media pipeline populated, found pads:\n");
+
+	list_for_each_entry(ppad, &pipe->pads, list)
+		dev_dbg(pad->graph_obj.mdev->dev, "- '%s':%u\n",
+			ppad->pad->entity->name, ppad->pad->index);
+
+	WARN_ON(walk.stack.top != -1);
+
+	ret = 0;
+
+done:
+	media_pipeline_walk_destroy(&walk);
+
+	if (ret)
+		media_pipeline_cleanup(pipe);
+
+	return ret;
+}
+
 __must_check int __media_pipeline_start(struct media_entity *entity,
 					struct media_pipeline *pipe)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
-	struct media_link *link;
+	struct media_pipeline_pad *err_ppad;
+	struct media_pipeline_pad *ppad;
 	int ret;
 
+	lockdep_assert_held(&mdev->graph_mutex);
+
+	/*
+	 * media_pipeline_start(entity) only makes sense with entities that have
+	 * a single pad.
+	 */
+
+	if (WARN_ON(entity->num_pads != 1))
+		return -EINVAL;
+
+	/*
+	 * If the entity is already part of a pipeline, that pipeline must
+	 * be the same as the pipe given to media_pipeline_start().
+	 */
+	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
+		return -EINVAL;
+
+	/*
+	 * If the pipeline has already been started, it is guaranteed to be
+	 * valid, so just increase the start count.
+	 */
 	if (pipe->start_count) {
 		pipe->start_count++;
 		return 0;
 	}
 
-	ret = media_graph_walk_init(&pipe->graph, mdev);
+	/*
+	 * Populate the pipeline. This populates the media_pipeline pads list
+	 * with media_pipeline_pad instances for each pad found during graph
+	 * walk.
+	 */
+	ret = media_pipeline_populate(pipe, entity->pads);
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&pipe->graph, entity);
+	/*
+	 * Now that all the pads in the pipeline have been gathered, perform
+	 * the validation steps.
+	 */
+
+	list_for_each_entry(ppad, &pipe->pads, list) {
+		struct media_pad *pad = ppad->pad;
+		struct media_entity *entity = pad->entity;
+		bool has_enabled_link = false;
+		bool has_link = false;
+		struct media_link *link;
 
-	while ((entity = media_graph_walk_next(graph))) {
-		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
-		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
+		dev_dbg(mdev->dev, "Validating pad '%s':%u\n", pad->entity->name,
+			pad->index);
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				entity_err->name);
+		/*
+		 * 1. Ensure that the pad doesn't already belong to a different
+		 * pipeline.
+		 */
+		if (pad->pipe) {
+			dev_dbg(mdev->dev, "Failed to start pipeline: pad '%s':%u busy\n",
+				pad->entity->name, pad->index);
 			ret = -EBUSY;
 			goto error;
 		}
 
-		/* Already streaming --- no need to check. */
-		if (entity->pipe)
-			continue;
-
-		entity->pipe = pipe;
-
-		if (!entity->ops || !entity->ops->link_validate)
-			continue;
-
-		bitmap_zero(active, entity->num_pads);
-		bitmap_fill(has_no_links, entity->num_pads);
-
+		/*
+		 * 2. Validate all active links whose sink is the current pad.
+		 * Validation of the source pads is performed in the context of
+		 * the connected sink pad to avoid duplicating checks.
+		 */
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			/* Skip links unrelated to the current pad. */
+			if (link->sink != pad && link->source != pad)
+				continue;
 
-			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			/* Record if the pad has links and enabled links. */
+			if (link->flags & MEDIA_LNK_FL_ENABLED)
+				has_enabled_link = true;
+			has_link = true;
 
 			/*
-			 * Pads that either do not need to connect or
-			 * are connected through an enabled link are
-			 * fine.
+			 * Validate the link if it's enabled and has the
+			 * current pad as its sink.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
-			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+			if (!(link->flags & MEDIA_LNK_FL_ENABLED))
+				continue;
 
-			/*
-			 * Link validation will only take place for
-			 * sink ends of the link that are enabled.
-			 */
-			if (link->sink != pad ||
-			    !(link->flags & MEDIA_LNK_FL_ENABLED))
+			if (link->sink != pad)
+				continue;
+
+			if (!entity->ops || !entity->ops->link_validate)
 				continue;
 
 			ret = entity->ops->link_validate(link);
-			if (ret < 0 && ret != -ENOIOCTLCMD) {
-				dev_dbg(entity->graph_obj.mdev->dev,
-					"link validation failed for '%s':%u -> '%s':%u, error %d\n",
+			if (ret) {
+				dev_dbg(mdev->dev,
+					"Link '%s':%u -> '%s':%u failed validation: %d\n",
 					link->source->entity->name,
 					link->source->index,
-					entity->name, link->sink->index, ret);
+					link->sink->entity->name,
+					link->sink->index, ret);
 				goto error;
 			}
-		}
 
-		/* Either no links or validated links are fine. */
-		bitmap_or(active, active, has_no_links, entity->num_pads);
+			dev_dbg(mdev->dev,
+				"Link '%s':%u -> '%s':%u is valid\n",
+				link->source->entity->name,
+				link->source->index,
+				link->sink->entity->name,
+				link->sink->index);
+		}
 
-		if (!bitmap_full(active, entity->num_pads)) {
+		/*
+		 * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
+		 * ensure that it has either no link or an enabled link.
+		 */
+		if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) && has_link &&
+		    !has_enabled_link) {
+			dev_dbg(mdev->dev,
+				"Pad '%s':%u must be connected by an enabled link\n",
+				pad->entity->name, pad->index);
 			ret = -ENOLINK;
-			dev_dbg(entity->graph_obj.mdev->dev,
-				"'%s':%u must be connected by an enabled link\n",
-				entity->name,
-				(unsigned)find_first_zero_bit(
-					active, entity->num_pads));
 			goto error;
 		}
+
+		/* Validation passed, store the pipe pointer in the pad. */
+		pad->pipe = pipe;
 	}
 
 	pipe->start_count++;
@@ -475,20 +835,15 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		entity_err->pipe = NULL;
-
-		/*
-		 * We haven't started entities further than this so we quit
-		 * here.
-		 */
-		if (entity_err == entity)
+	list_for_each_entry(err_ppad, &pipe->pads, list) {
+		if (err_ppad == ppad)
 			break;
+
+		err_ppad->pad->pipe = NULL;
 	}
 
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 	return ret;
 }
@@ -509,8 +864,8 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline_pad *ppad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -522,12 +877,10 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (--pipe->start_count)
 		return;
 
-	media_graph_walk_start(graph, entity);
-
-	while ((entity = media_graph_walk_next(graph)))
-		entity->pipe = NULL;
+	list_for_each_entry(ppad, &pipe->pads, list)
+		ppad->pad->pipe = NULL;
 
-	media_graph_walk_cleanup(graph);
+	media_pipeline_cleanup(pipe);
 
 	if (pipe->allocated)
 		kfree(pipe);
@@ -837,7 +1190,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -853,12 +1206,11 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
-	    (media_entity_is_streaming(source) ||
-	     media_entity_is_streaming(sink)))
+	    (media_pad_is_streaming(source) || media_pad_is_streaming(sink)))
 		return -EBUSY;
 
 	mdev = source->graph_obj.mdev;
@@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
 
 struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
 {
-	return entity->pipe;
+	struct media_pipeline *pipe;
+	struct media_pad *pad;
+
+	if (entity->num_pads == 0)
+		return NULL;
+
+	pipe = entity->pads->pipe;
+
+	media_entity_for_each_pad(entity, pad) {
+		if (WARN_ON(pad->pipe != pipe))
+			return NULL;
+	}
+
+	return pipe;
 }
 EXPORT_SYMBOL_GPL(media_entity_pipeline);
 
+struct media_pipeline *media_pad_pipeline(struct media_pad *pad)
+{
+	return pad->pipe;
+}
+EXPORT_SYMBOL_GPL(media_pad_pipeline);
+
 static void media_interface_init(struct media_device *mdev,
 				 struct media_interface *intf,
 				 u32 gobj_type,
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 3d0d5e3c3333..107e2d8025bd 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -100,13 +100,33 @@ struct media_graph {
 /**
  * struct media_pipeline - Media pipeline related information
  *
+ * @mdev:		The media device the pipeline is part of
+ * @pads:		List of media_pipeline_pad
  * @start_count:	Media pipeline start - stop count
- * @graph:		Media graph walk during pipeline start / stop
  */
 struct media_pipeline {
 	bool allocated;
+	struct media_device *mdev;
+	struct list_head pads;
 	int start_count;
-	struct media_graph graph;
+};
+
+/**
+ * struct media_pipeline_pad - A pad part of a media pipeline
+ *
+ * @list:		Entry in the media_pad pads list
+ * @pipe:		The media_pipeline that the pad is part of
+ * @pad:		The media pad
+ *
+ * This structure associate a pad with a media pipeline. Instances of
+ * media_pipeline_pad are created by media_pipeline_start() when it builds the
+ * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
+ * removes the entries from the list and deletes them.
+ */
+struct media_pipeline_pad {
+	struct list_head list;
+	struct media_pipeline *pipe;
+	struct media_pad *pad;
 };
 
 /**
@@ -188,6 +208,8 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * @pipe:	Pipeline this pad belongs to. Use media_entity_pipeline() to
+ *		access this field.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -195,6 +217,12 @@ struct media_pad {
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+	struct media_pipeline *pipe;
 };
 
 /**
@@ -271,7 +299,6 @@ enum media_entity_type {
  * @links:	List of data links.
  * @ops:	Entity operations.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -307,8 +334,6 @@ struct media_entity {
 
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
@@ -873,6 +898,18 @@ struct media_link *media_entity_find_link(struct media_pad *source,
  */
 struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
 
+/**
+ * media_pad_is_streaming - Test if a pad is part of a streaming pipeline
+ * @pad: The pad
+ *
+ * Return: True if the pad is part of a pipeline started with the
+ * media_pipeline_start() function, false otherwise.
+ */
+static inline bool media_pad_is_streaming(const struct media_pad *pad)
+{
+	return pad->pipe;
+}
+
 /**
  * media_entity_is_streaming - Test if an entity is part of a streaming pipeline
  * @entity: The entity
@@ -882,7 +919,14 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
  */
 static inline bool media_entity_is_streaming(const struct media_entity *entity)
 {
-	return entity->pipe;
+	struct media_pad *pad;
+
+	media_entity_for_each_pad(entity, pad) {
+		if (media_pad_is_streaming(pad))
+			return true;
+	}
+
+	return false;
 }
 
 /**
@@ -903,6 +947,19 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
  */
 struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
 
+/**
+ * media_pad_pipeline - Get the media pipeline a pad is part of
+ * @pad: The pad
+ *
+ * This function returns the media pipeline that a pad has been associated
+ * with when constructing the pipeline with media_pipeline_start(). The pointer
+ * remains valid until media_pipeline_stop() is called.
+ *
+ * Return: The media_pipeline the pad is part of, or NULL if the pad is
+ * not part of any pipeline.
+ */
+struct media_pipeline *media_pad_pipeline(struct media_pad *pad);
+
 /**
  * media_entity_get_fwnode_pad - Get pad number from fwnode
  *
-- 
2.34.1

[PATCH v13 15/34] media: mc: entity: Add has_pad_interdep entity operation

[Tomi Valkeinen]

Add a new media entity operation, has_pad_interdep. The optional op is
used to discover the pad interdependencies inside an entity during the
pipeline construction.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 10 ++++++++--
 include/media/media-entity.h | 10 ++++++++++
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index df8e82be2bcc..c89dc782840a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -231,7 +231,10 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * and enabling one of the pads means that the other pad will become "locked"
  * and doesn't allow configuration changes.
  *
- * For the time being all pads are considered interdependent.
+ * This function uses the &media_entity_operations.has_pad_interdep() operation
+ * to check the dependency inside the entity between @pad0 and @pad1. If the
+ * has_pad_interdep operation is not implemented, all pads of the entity are
+ * considered to be interdependent.
  */
 static bool media_entity_has_pad_interdep(struct media_entity *entity,
 					  unsigned int pad0, unsigned int pad1)
@@ -243,7 +246,10 @@ static bool media_entity_has_pad_interdep(struct media_entity *entity,
 	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
 		return false;
 
-	return true;
+	if (!entity->ops || !entity->ops->has_pad_interdep)
+		return true;
+
+	return entity->ops->has_pad_interdep(entity, pad0, pad1);
 }
 
 static struct media_entity *
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 107e2d8025bd..beea3f6f12b8 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -236,6 +236,14 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_pad_interdep:	Return whether a two pads inside the entity are
+ *			interdependent. If two pads are interdependent they are
+ *			part of the same pipeline and enabling one of the pads
+ *			means that the other pad will become "locked" and
+ *			doesn't allow configuration changes. pad0 and pad1 are
+ *			guaranteed to not both be sinks or sources.
+ *			Optional: If the operation isn't implemented all pads
+ *			will be considered as interdependent.
  *
  * .. note::
  *
@@ -249,6 +257,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_pad_interdep)(struct media_entity *entity, unsigned int pad0,
+				 unsigned int pad1);
 };
 
 /**
-- 
2.34.1

[PATCH v13 16/34] media: mc: convert pipeline funcs to take media_pad

[Tomi Valkeinen]

Now that the pipeline is stored into pads instead of entities, we can
change the relevant functions to take pads instead of entities.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 40 ++++++++-----------
 .../samsung/s3c-camif/camif-capture.c         |  6 +--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++--
 drivers/media/v4l2-core/v4l2-dev.c            | 12 +++---
 drivers/staging/media/imx/imx-media-utils.c   |  6 +--
 include/media/media-entity.h                  | 28 ++++++-------
 6 files changed, 46 insertions(+), 54 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index c89dc782840a..ce72ee749fe6 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -699,29 +699,21 @@ static int media_pipeline_populate(struct media_pipeline *pipe,
 	return ret;
 }
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	struct media_pipeline_pad *err_ppad;
 	struct media_pipeline_pad *ppad;
 	int ret;
 
 	lockdep_assert_held(&mdev->graph_mutex);
 
-	/*
-	 * media_pipeline_start(entity) only makes sense with entities that have
-	 * a single pad.
-	 */
-
-	if (WARN_ON(entity->num_pads != 1))
-		return -EINVAL;
-
 	/*
 	 * If the entity is already part of a pipeline, that pipeline must
 	 * be the same as the pipe given to media_pipeline_start().
 	 */
-	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
+	if (WARN_ON(pad->pipe && pad->pipe != pipe))
 		return -EINVAL;
 
 	/*
@@ -738,7 +730,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * with media_pipeline_pad instances for each pad found during graph
 	 * walk.
 	 */
-	ret = media_pipeline_populate(pipe, entity->pads);
+	ret = media_pipeline_populate(pipe, pad);
 	if (ret)
 		return ret;
 
@@ -855,22 +847,22 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_pipeline_pad *ppad;
 
 	/*
@@ -893,19 +885,19 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
 
-__must_check int media_pipeline_alloc_start(struct media_entity *entity)
+__must_check int media_pipeline_alloc_start(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	struct media_pipeline *pipe;
 	int ret;
 	bool new_pipe = false;
@@ -916,7 +908,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
 	 * Is the entity already part of a pipeline? If not, we need to allocate
 	 * a pipe.
 	 */
-	pipe = media_entity_pipeline(entity);
+	pipe = media_pad_pipeline(pad);
 	if (!pipe) {
 		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
 
@@ -929,7 +921,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
 		pipe->allocated = true;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	if (ret) {
 		if (new_pipe)
 			kfree(pipe);
diff --git a/drivers/media/platform/samsung/s3c-camif/camif-capture.c b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
index 140854ab4dd8..0189b8a33032 100644
--- a/drivers/media/platform/samsung/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index caefac07af92..877e85a451cb 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
index 945bb867a4c1..397d553177fa 100644
--- a/drivers/media/v4l2-core/v4l2-dev.c
+++ b/drivers/media/v4l2-core/v4l2-dev.c
@@ -1105,7 +1105,7 @@ __must_check int video_device_pipeline_start(struct video_device *vdev,
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return media_pipeline_start(entity, pipe);
+	return media_pipeline_start(&entity->pads[0], pipe);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_start);
 
@@ -1117,7 +1117,7 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return __media_pipeline_start(entity, pipe);
+	return __media_pipeline_start(&entity->pads[0], pipe);
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
 
@@ -1128,7 +1128,7 @@ void video_device_pipeline_stop(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return;
 
-	return media_pipeline_stop(entity);
+	return media_pipeline_stop(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
 
@@ -1139,7 +1139,7 @@ void __video_device_pipeline_stop(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return;
 
-	return __media_pipeline_stop(entity);
+	return __media_pipeline_stop(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
 
@@ -1150,7 +1150,7 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
 	if (entity->num_pads != 1)
 		return -ENODEV;
 
-	return media_pipeline_alloc_start(entity);
+	return media_pipeline_alloc_start(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
 
@@ -1161,7 +1161,7 @@ struct media_pipeline *video_device_pipeline(struct video_device *vdev)
 	if (WARN_ON(entity->num_pads != 1))
 		return NULL;
 
-	return media_entity_pipeline(entity);
+	return media_pad_pipeline(&entity->pads[0]);
 }
 EXPORT_SYMBOL_GPL(video_device_pipeline);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 5d9c6fc6f2e0..3d785b4e8fe7 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -863,16 +863,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
 		if (media_entity_pipeline(entity))
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index beea3f6f12b8..f738d0d81194 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1048,10 +1048,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
+ * Mark all pads connected to a given pad through enabled links, either
  * directly or indirectly, as streaming. The given pipeline object is assigned
  * to every entity in the pipeline and stored in the media_entity pipe field.
  *
@@ -1060,24 +1060,24 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
+ * Mark all pads connected to a given pads through enabled links, either
  * directly or indirectly, as not streaming. The media_entity pipe field is
  * reset to %NULL.
  *
@@ -1085,20 +1085,20 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
  * number of calls to this function are required to mark the pipeline as not
  * streaming.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_pipeline_alloc_start - Mark a pipeline as streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
  * of working on a given pipeline the function will use an existing pipeline if
@@ -1107,7 +1107,7 @@ void __media_pipeline_stop(struct media_entity *entity);
  * Calls to media_pipeline_alloc_start() must be matched with
  * media_pipeline_stop().
  */
-__must_check int media_pipeline_alloc_start(struct media_entity *entity);
+__must_check int media_pipeline_alloc_start(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
-- 
2.34.1

[PATCH v13 17/34] media: add V4L2_SUBDEV_FL_STREAMS

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_STREAMS. It is used to indicate that the
subdev supports the new API with multiplexed streams (routing, stream
configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index dbab00f79527..aad80e5bb342 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -879,6 +879,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed subdev active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads is NULL)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_STREAMS			(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.34.1

[PATCH v13 18/34] media: add V4L2_SUBDEV_CAP_STREAMS

[Tomi Valkeinen]

Add a subdev capability flag to expose to userspace if a subdev supports
multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 4 +++-
 include/uapi/linux/v4l2-subdev.h      | 3 +++
 2 files changed, 6 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5c27bac772ea..90826b956693 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -439,7 +439,9 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 
 		memset(cap->reserved, 0, sizeof(cap->reserved));
 		cap->version = LINUX_VERSION_CODE;
-		cap->capabilities = ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0;
+		cap->capabilities =
+			(ro_subdev ? V4L2_SUBDEV_CAP_RO_SUBDEV : 0) |
+			((sd->flags & V4L2_SUBDEV_FL_STREAMS) ? V4L2_SUBDEV_CAP_STREAMS : 0);
 
 		return 0;
 	}
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..89af27f50a41 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,9 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/* The v4l2 sub-device supports routing and multiplexed streams. */
+#define V4L2_SUBDEV_CAP_STREAMS			0x00000002
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
-- 
2.34.1

[PATCH v13 19/34] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 150 ++++++++++++++++++
 3 files changed, 153 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..a0d9c79e162f
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,150 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the ``VIDIOC_SUBDEV_S_ROUTING`` ioctl, by adding or removing routes and
+setting or clearing flags of the  ``flags`` field of a
+struct :c:type:`v4l2_subdev_route`.
+
+All stream configurations are reset when ``VIDIOC_SUBDEV_S_ROUTING`` is called. This
+means that the userspace mut reconfigure all streams after calling the ioctl
+with e.g. ``VIDIOC_SUBDEV_S_FMT``.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through ``VIDIOC_SUBDEV_G_ROUTING`` and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call ``VIDIOC_SUBDEV_G_ROUTING`` again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.34.1

[PATCH v13 20/34] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 87 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 +++++++
 include/uapi/linux/v4l2-subdev.h      | 51 ++++++++++++++++
 4 files changed, 184 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 21470de62d72..f42f7ffcc247 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/media-device.h> /* for media_set_bus_info() */
@@ -3102,6 +3103,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3365,8 +3381,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 90826b956693..af1f53d99507 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -23,6 +23,16 @@
 #include <media/v4l2-fh.h>
 #include <media/v4l2-event.h>
 
+/*
+ * Maximum stream ID is 63 for now, as we use u64 bitmask to represent a set
+ * of streams.
+ *
+ * Note that V4L2_FRAME_DESC_ENTRY_MAX is related: V4L2_FRAME_DESC_ENTRY_MAX
+ * restricts the total number of streams in a pad, although the stream ID is
+ * not restricted.
+ */
+#define V4L2_SUBDEV_MAX_STREAM_ID 63
+
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
@@ -417,6 +427,10 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 	case VIDIOC_SUBDEV_S_SELECTION:
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING:
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -732,6 +746,78 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg,
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			if (route->sink_stream > V4L2_SUBDEV_MAX_STREAM_ID ||
+			    route->source_stream > V4L2_SUBDEV_MAX_STREAM_ID)
+				return -EINVAL;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -1015,6 +1101,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->_lock);
 
+	kfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index aad80e5bb342..15b299873086 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,12 +695,26 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @_lock: default for 'lock'
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -711,6 +725,7 @@ struct v4l2_subdev_state {
 	struct mutex _lock;
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -763,6 +778,9 @@ struct v4l2_subdev_state {
  *		     this operation as close as possible to stream on time. The
  *		     operation shall fail if the pad index it has been called on
  *		     is not valid or in case of unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -805,6 +823,10 @@ struct v4l2_subdev_pad_ops {
 			      struct v4l2_mbus_frame_desc *fd);
 	int (*get_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 89af27f50a41..77bc0eaf305b 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -191,6 +191,55 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device supports routing and multiplexed streams. */
 #define V4L2_SUBDEV_CAP_STREAMS			0x00000002
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		_BITUL(0)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated by the subdevice (usually a sensor), and thus there is no
+ * sink-side endpoint for the route. The sink_pad and sink_stream fields are
+ * unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		_BITUL(2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @num_routes: the total number of routes in the routes array
+ * @routes: pointer to the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u32 num_routes;
+	__u64 routes;
+	__u32 reserved[6];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -206,6 +255,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.34.1

[PATCH v13 21/34] media: subdev: add v4l2_subdev_has_pad_interdep()

[Tomi Valkeinen]

Add a v4l2_subdev_has_pad_interdep() helper function which can be used
for media_entity_operations.has_pad_interdep op.

It considers two pads interdependent if there is an active route between
pad0 and pad1.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 +++++++++++++++
 2 files changed, 48 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index af1f53d99507..1ce9a7dc0c6e 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1045,6 +1045,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_pad_interdep);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 15b299873086..142089a0d1b7 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1235,6 +1235,23 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_pad_interdep - MC has_pad_interdep implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function is an implementation of the media_entity_operations.has_pad_interdep
+ * operation for subdevs that implement the multiplexed streams API (as
+ * indicated by the V4L2_SUBDEV_FL_STREAMS subdev flag).
+ *
+ * It considers two pads interdependent if there is an active route between pad0
+ * and pad1.
+ */
+bool v4l2_subdev_has_pad_interdep(struct media_entity *entity,
+				  unsigned int pad0, unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.34.1

[PATCH v13 22/34] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 +++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 1ce9a7dc0c6e..f3f872c72180 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1180,6 +1180,34 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+	struct v4l2_subdev_krouting new_routing = { 0 };
+
+	lockdep_assert_held(state->lock);
+
+	if (src->num_routes > 0) {
+		new_routing.routes = kmemdup(src->routes,
+			src->num_routes * sizeof(*src->routes),
+			GFP_KERNEL);
+
+		if (!new_routing.routes)
+			return -ENOMEM;
+	}
+
+	new_routing.num_routes = src->num_routes;
+
+	kfree(dst->routes);
+	*dst = new_routing;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 142089a0d1b7..767e4ebd97fe 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1412,6 +1412,22 @@ v4l2_subdev_lock_and_get_active_state(struct v4l2_subdev *sd)
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    const struct v4l2_subdev_krouting *routing);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 23/34] media: subdev: Add for_each_active_route() macro

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add a for_each_active_route() macro to replace the repeated pattern
of iterating on the active routes of a routing table.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .clang-format                         |  1 +
 drivers/media/v4l2-core/v4l2-subdev.c | 20 ++++++++++++++++++++
 include/media/v4l2-subdev.h           | 13 +++++++++++++
 3 files changed, 34 insertions(+)

diff --git a/.clang-format b/.clang-format
index 9b87ea1fc16e..2629a5051bf8 100644
--- a/.clang-format
+++ b/.clang-format
@@ -190,6 +190,7 @@ ForEachMacros:
   - 'for_each_active_dev_scope'
   - 'for_each_active_drhd_unit'
   - 'for_each_active_iommu'
+  - 'for_each_active_route'
   - 'for_each_aggr_pgid'
   - 'for_each_available_child_of_node'
   - 'for_each_bench'
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index f3f872c72180..da8550598673 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1208,6 +1208,26 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route)
+{
+	if (route)
+		++route;
+	else
+		route = &routing->routes[0];
+
+	for (; route < routing->routes + routing->num_routes; ++route) {
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		return route;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 767e4ebd97fe..092e77ee454d 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1428,6 +1428,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    const struct v4l2_subdev_krouting *routing);
 
+struct v4l2_subdev_route *
+__v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
+				struct v4l2_subdev_route *route);
+
+/**
+ * for_each_active_route - iterate on all active routes of a routing table
+ * @routing: The routing table
+ * @route: The route iterator
+ */
+#define for_each_active_route(routing, route) \
+	for ((route) = NULL;                  \
+	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 24/34] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 173 ++++++++++++++++++
 2 files changed, 181 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index cf3b52bdbfb9..8bf5b3f9acfd 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -587,6 +587,14 @@ before calling v4l2_subdev_init_finalize():
 
 This shares the driver's private mutex between the controls and the states.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_STREAMS subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..5075b1828b32 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,176 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, only a
+single stream can pass through a media link and a media pad. Thus each pad
+contains a format configuration for that single stream. In some cases a subdev
+can do stream processing and split a stream into two or compose two streams
+into one, but the inputs and outputs for the subdev are still a single stream
+per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple
+data streams are transmitted on the same bus, which is represented by a media
+link connecting a transmitter source pad with a sink pad on the receiver. For
+example, a camera sensor can produce two distinct streams, a pixel stream and a
+metadata stream, which are transmitted on the multiplexed data bus, represented
+by a media link which connects the single sensor's source pad with the receiver
+sink pad. The stream-aware receiver will de-multiplex the streams received on
+the its sink pad and allows to route them individually to one of its source
+pads.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, require a routing configuration
+where the link between those two types of drivers contains only a single
+stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a
+receiver and demultiplexer in a SoC). Each media link carries all the enabled
+streams from one end of the link to the other, and subdevices have routing
+tables which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a stream.
+In other words, a particular stream ID must exist on both sides of a media
+link, but another stream ID can be used for the same stream at the other side
+of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that
+setting the routing table will reset all the stream configurations in a media
+entity.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source
+  pad (pad 0) which carries two streams, pixel data stream and metadata
+  stream.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint) instead of just a pad.
+
+A common way to accomplish this is to start from the sensors and propagate the
+configurations along the stream towards the receiver,
+using :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each
+stream endpoint in each subdev.
-- 
2.34.1

[PATCH v13 25/34] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 151 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  79 +++++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 272 insertions(+), 21 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index da8550598673..e752eb3df1a3 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -158,8 +158,19 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+		       u32 which, u32 pad, u32 stream)
 {
+#if defined(CONFIG_MEDIA_CONTROLLER)
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS) {
+		if (!v4l2_subdev_state_get_stream_format(state, pad, stream))
+			return -EINVAL;
+		return 0;
+	}
+#endif
+	if (stream != 0)
+		return -EINVAL;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
@@ -174,7 +185,7 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state(sd, state, format->which, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -201,7 +212,7 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state(sd, state, code->which, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -213,7 +224,7 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state(sd, state, fse->which, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -248,7 +259,7 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state(sd, state, fie->which, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -260,7 +271,7 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state(sd, state, sel->which, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -919,6 +930,59 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 
 #ifdef CONFIG_MEDIA_CONTROLLER
 
+static int
+v4l2_subdev_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+				const struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_stream_configs new_configs = { 0 };
+	struct v4l2_subdev_route *route;
+	u32 format_idx = 0;
+
+	/* Count number of formats needed */
+	for_each_active_route(routing, route) {
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		new_configs.num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (new_configs.num_configs) {
+		new_configs.configs = kvcalloc(new_configs.num_configs,
+						   sizeof(*new_configs.configs),
+						   GFP_KERNEL);
+
+		if (!new_configs.configs)
+			return -ENOMEM;
+	}
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for_each_active_route(routing, route) {
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			new_configs.configs[idx].pad = route->sink_pad;
+			new_configs.configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		new_configs.configs[idx].pad = route->source_pad;
+		new_configs.configs[idx].stream = route->source_stream;
+	}
+
+	kvfree(stream_configs->configs);
+	*stream_configs = new_configs;
+
+	return 0;
+}
+
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
 				      struct fwnode_endpoint *endpoint)
 {
@@ -1093,7 +1157,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 	else
 		state->lock = &state->_lock;
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_STREAMS) && sd->entity.num_pads) {
 		state->pads = kvcalloc(sd->entity.num_pads,
 				       sizeof(*state->pads), GFP_KERNEL);
 		if (!state->pads) {
@@ -1133,6 +1198,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->_lock);
 
 	kfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1187,6 +1253,7 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 	struct v4l2_subdev_krouting *dst = &state->routing;
 	const struct v4l2_subdev_krouting *src = routing;
 	struct v4l2_subdev_krouting new_routing = { 0 };
+	int r;
 
 	lockdep_assert_held(state->lock);
 
@@ -1201,6 +1268,13 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 
 	new_routing.num_routes = src->num_routes;
 
+	r = v4l2_subdev_init_stream_configs(&state->stream_configs,
+					    &new_routing);
+	if (r) {
+		kfree(new_routing.routes);
+		return r;
+	}
+
 	kfree(dst->routes);
 	*dst = new_routing;
 
@@ -1228,6 +1302,69 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_format);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].crop;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_crop);
+
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].compose;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 092e77ee454d..875aa812a632 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -695,6 +695,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -715,6 +746,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state. May be replaced by the user.
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_STREAMS)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -726,6 +758,7 @@ struct v4l2_subdev_state {
 	struct mutex *lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1441,6 +1474,52 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
+				    unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_crop() - Get pointer to a stream crop rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to crop rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_crop(struct v4l2_subdev_state *state,
+				  unsigned int pad, u32 stream);
+
+/**
+ * v4l2_subdev_state_get_stream_compose() - Get pointer to a stream compose
+ *					    rectangle
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to compose rectangle for the given pad + stream in the
+ * subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_rect *
+v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
+				     unsigned int pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 77bc0eaf305b..e077da14e237 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -58,13 +60,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.34.1

[PATCH v13 26/34] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 245 +++++++++++++++++++++++---
 1 file changed, 225 insertions(+), 20 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e752eb3df1a3..a46059cda541 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1062,7 +1063,7 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
-v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+v4l2_subdev_link_validate_get_format(struct media_pad *pad, u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1071,7 +1072,11 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call_state_active(sd, pad, get_fmt, fmt);
+		fmt->stream = stream;
+
+		return v4l2_subdev_call(sd, pad, get_fmt,
+					v4l2_subdev_get_locked_active_state(sd),
+					fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
@@ -1081,31 +1086,231 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_pad *pad,
+					  u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev_route *route;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	struct v4l2_subdev_state *state;
+	int ret;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	subdev = media_entity_to_v4l2_subdev(pad->entity);
+
+	if (!(subdev->flags & V4L2_SUBDEV_FL_STREAMS)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_get_locked_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+	if (!streams)
+		return -ENOMEM;
+
+	num_streams = 0;
+
+	for_each_active_route(routing, route) {
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+		}
+
+		if (route_pad != pad->index)
+			continue;
+
+		/* look for duplicates */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream) {
+				ret = -EINVAL;
+				goto free_streams;
+			}
+		}
+
+		streams[num_streams++] = route_stream;
+	}
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
+
+	return 0;
+
+free_streams:
+	kfree(streams);
+
+	return ret;
+}
+
+static int v4l2_subdev_link_validate_locked(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link->source, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link->sink, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/*
+	 * It is ok to have more source streams than sink streams as extra
+	 * source streams can just be ignored by the receiver, but having extra
+	 * sink streams is an error as streams must have a source.
+	 */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto free_sink;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto free_sink;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+		if (ret != -ENOIOCTLCMD)
+			goto free_sink;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto free_sink;
+	}
+
+free_sink:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *source_sd, *sink_sd;
+	struct v4l2_subdev_state *source_state, *sink_state;
+	int ret;
+
+	sink_sd = media_entity_to_v4l2_subdev(link->sink->entity);
+	source_sd = media_entity_to_v4l2_subdev(link->source->entity);
+
+	sink_state = v4l2_subdev_get_unlocked_active_state(sink_sd);
+	source_state = v4l2_subdev_get_unlocked_active_state(source_sd);
+
+	if (sink_state)
+		v4l2_subdev_lock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_lock_state(source_state);
+
+	ret = v4l2_subdev_link_validate_locked(link);
+
+	if (sink_state)
+		v4l2_subdev_unlock_state(sink_state);
+
+	if (source_state)
+		v4l2_subdev_unlock_state(source_state);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.34.1

[PATCH v13 27/34] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_subdev_routing_find_opposite_end - given a routing table and a pad
+ stream, return the pad + stream on the opposite side of the subdev.

v4l2_subdev_state_get_opposite_stream_format - return a pointer to the
format on the pad + stream on the opposite side from the given pad +
stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 48 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 36 ++++++++++++++++++++
 2 files changed, 84 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a46059cda541..a4ae11c27e24 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1570,6 +1570,54 @@ v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_stream_compose);
 
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			if (other_pad)
+				*other_pad = route->sink_pad;
+			if (other_stream)
+				*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			if (other_pad)
+				*other_pad = route->source_pad;
+			if (other_stream)
+				*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_subdev_routing_find_opposite_end(&state->routing, pad, stream,
+						    &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_subdev_state_get_stream_format(state, other_pad,
+						   other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 875aa812a632..a57a866cdcfe 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1520,6 +1520,42 @@ struct v4l2_rect *
 v4l2_subdev_state_get_stream_compose(struct v4l2_subdev_state *state,
 				     unsigned int pad, u32 stream);
 
+/**
+ * v4l2_subdev_routing_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * @other_pad and/or @other_stream can be NULL if the caller does not need the
+ * value.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_subdev_routing_find_opposite_end(const struct v4l2_subdev_krouting *routing,
+					  u32 pad, u32 stream, u32 *other_pad,
+					  u32 *other_stream);
+
+/**
+ * v4l2_subdev_state_get_opposite_stream_format() - Get pointer to opposite
+ *                                                  stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
+					     u32 pad, u32 stream);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 28/34] media: subdev: add streams to v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add streams support to v4l2_subdev_get_fmt() helper function. Subdev
drivers that do not need to do anything special in their get_fmt op can
use this helper directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 10 +++++++---
 1 file changed, 7 insertions(+), 3 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a4ae11c27e24..76788798e976 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1438,10 +1438,14 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 {
 	struct v4l2_mbus_framefmt *fmt;
 
-	if (format->pad >= sd->entity.num_pads)
-		return -EINVAL;
+	if (sd->flags & V4L2_SUBDEV_FL_STREAMS)
+		fmt = v4l2_subdev_state_get_stream_format(state, format->pad,
+							  format->stream);
+	else if (format->pad < sd->entity.num_pads && format->stream == 0)
+		fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
+	else
+		fmt = NULL;
 
-	fmt = v4l2_subdev_get_pad_format(sd, state, format->pad);
 	if (!fmt)
 		return -EINVAL;
 
-- 
2.34.1

[PATCH v13 29/34] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 76788798e976..92a3142fcf23 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1511,6 +1511,28 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(__v4l2_subdev_next_active_route);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_stream_format(struct v4l2_subdev_state *state,
 				    unsigned int pad, u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a57a866cdcfe..cacc56ee3ee6 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1474,6 +1474,22 @@ __v4l2_subdev_next_active_route(const struct v4l2_subdev_krouting *routing,
 	for ((route) = NULL;                  \
 	     ((route) = __v4l2_subdev_next_active_route((routing), (route)));)
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_subdev_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.34.1

[PATCH v13 30/34] media: subdev: add v4l2_subdev_routing_validate() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a v4l2_subdev_routing_validate() helper for verifying routing for
common cases like only allowing non-overlapping 1-to-1 streams.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 102 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  39 ++++++++++
 2 files changed, 141 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 92a3142fcf23..4fdde9d8c07d 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1644,6 +1644,108 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow)
+{
+	u32 *remote_pads = NULL;
+	unsigned int i, j;
+	int ret = -EINVAL;
+
+	if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+		remote_pads = kcalloc(sd->entity.num_pads, sizeof(*remote_pads),
+				      GFP_KERNEL);
+		if (!remote_pads)
+			return -ENOMEM;
+
+		for (i = 0; i < sd->entity.num_pads; ++i)
+			remote_pads[i] = U32_MAX;
+	}
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		/* Validate the sink and source pad numbers. */
+		if (route->sink_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->sink_pad].flags & MEDIA_PAD_FL_SINK)) {
+			dev_dbg(sd->dev, "route %u sink (%u) is not a sink pad\n",
+				i, route->sink_pad);
+			goto out;
+		}
+
+		if (route->source_pad >= sd->entity.num_pads ||
+		    !(sd->entity.pads[route->source_pad].flags & MEDIA_PAD_FL_SOURCE)) {
+			dev_dbg(sd->dev, "route %u source (%u) is not a source pad\n",
+				i, route->source_pad);
+			goto out;
+		}
+
+		/*
+		 * V4L2_SUBDEV_ROUTING_NO_STREAM_MIX: Streams on the same pad
+		 * may not be routed to streams on different pads.
+		 */
+		if (disallow & V4L2_SUBDEV_ROUTING_NO_STREAM_MIX) {
+			if (remote_pads[route->sink_pad] != U32_MAX &&
+			    remote_pads[route->sink_pad] != route->source_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "sink");
+				goto out;
+			}
+
+			if (remote_pads[route->source_pad] != U32_MAX &&
+			    remote_pads[route->source_pad] != route->sink_pad) {
+				dev_dbg(sd->dev,
+					"route %u attempts to mix %s streams\n",
+					i, "source");
+				goto out;
+			}
+
+			remote_pads[route->sink_pad] = route->source_pad;
+			remote_pads[route->source_pad] = route->sink_pad;
+		}
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_1_TO_N: No two routes can
+			 * originate from the same (sink) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_1_TO_N) &&
+			    route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u originate from same sink (%u/%u)\n",
+					i, j, route->sink_pad,
+					route->sink_stream);
+				goto out;
+			}
+
+			/*
+			 * V4L2_SUBDEV_ROUTING_NO_N_TO_1: No two routes can end
+			 * at the same (source) stream.
+			 */
+			if ((disallow & V4L2_SUBDEV_ROUTING_NO_N_TO_1) &&
+			    route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream) {
+				dev_dbg(sd->dev,
+					"routes %u and %u end at same source (%u/%u)\n",
+					i, j, route->source_pad,
+					route->source_stream);
+				goto out;
+			}
+		}
+	}
+
+	ret = 0;
+
+out:
+	kfree(remote_pads);
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index cacc56ee3ee6..69331adb0aca 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1572,6 +1572,45 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
+ *
+ * @V4L2_SUBDEV_ROUTING_NO_1_TO_N:
+ *	an input stream may not be routed to multiple output streams (stream
+ *	duplication)
+ * @V4L2_SUBDEV_ROUTING_NO_N_TO_1:
+ *	multiple input streams may not be routed to the same output stream
+ *	(stream merging)
+ * @V4L2_SUBDEV_ROUTING_NO_STREAM_MIX:
+ *	streams on the same pad may not be routed to streams on different pads
+ * @V4L2_SUBDEV_ROUTING_ONLY_1_TO_1:
+ *	only non-overlapping 1-to-1 stream routing is allowed (a combination of
+ *	@V4L2_SUBDEV_ROUTING_NO_1_TO_N and @V4L2_SUBDEV_ROUTING_NO_N_TO_1)
+ */
+enum v4l2_subdev_routing_restriction {
+	V4L2_SUBDEV_ROUTING_NO_1_TO_N = BIT(0),
+	V4L2_SUBDEV_ROUTING_NO_N_TO_1 = BIT(1),
+	V4L2_SUBDEV_ROUTING_NO_STREAM_MIX = BIT(2),
+	V4L2_SUBDEV_ROUTING_ONLY_1_TO_1 =
+		V4L2_SUBDEV_ROUTING_NO_1_TO_N |
+		V4L2_SUBDEV_ROUTING_NO_N_TO_1,
+};
+
+/**
+ * v4l2_subdev_routing_validate() - Verify that routes comply with driver
+ *				    constraints
+ * @sd: The subdevice
+ * @routing: Routing to verify
+ * @disallow: Restrictions on routes
+ *
+ * This verifies that the given routing complies with the @disallow constraints.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
+				 const struct v4l2_subdev_krouting *routing,
+				 enum v4l2_subdev_routing_restriction disallow);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 31/34] media: v4l2-subdev: Add v4l2_subdev_state_xlate_streams() helper

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add a helper function to translate streams between two pads of a subdev,
using the subdev's internal routing table.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 26 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 23 +++++++++++++++++++++++
 2 files changed, 49 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 4fdde9d8c07d..7d46c9a5a6c4 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1644,6 +1644,32 @@ v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_state_get_opposite_stream_format);
 
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams)
+{
+	const struct v4l2_subdev_krouting *routing = &state->routing;
+	struct v4l2_subdev_route *route;
+	u64 streams0 = 0;
+	u64 streams1 = 0;
+
+	for_each_active_route(routing, route) {
+		if (route->sink_pad == pad0 && route->source_pad == pad1 &&
+		    (*streams & BIT_ULL(route->sink_stream))) {
+			streams0 |= BIT_ULL(route->sink_stream);
+			streams1 |= BIT_ULL(route->source_stream);
+		}
+		if (route->source_pad == pad0 && route->sink_pad == pad1 &&
+		    (*streams & BIT_ULL(route->source_stream))) {
+			streams0 |= BIT_ULL(route->source_stream);
+			streams1 |= BIT_ULL(route->sink_stream);
+		}
+	}
+
+	*streams = streams0;
+	return streams1;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_state_xlate_streams);
+
 int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 69331adb0aca..86d64a755605 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1572,6 +1572,29 @@ struct v4l2_mbus_framefmt *
 v4l2_subdev_state_get_opposite_stream_format(struct v4l2_subdev_state *state,
 					     u32 pad, u32 stream);
 
+/**
+ * v4l2_subdev_state_xlate_streams() - Translate streams from one pad to another
+ *
+ * @state: Subdevice state
+ * @pad0: The first pad
+ * @pad1: The second pad
+ * @streams: Streams bitmask on the first pad
+ *
+ * Streams on sink pads of a subdev are routed to source pads as expressed in
+ * the subdev state routing table. Stream numbers don't necessarily match on
+ * the sink and source side of a route. This function translates stream numbers
+ * on @pad0, expressed as a bitmask in @streams, to the corresponding streams
+ * on @pad1 using the routing table from the @state. It returns the stream mask
+ * on @pad1, and updates @streams with the streams that have been found in the
+ * routing table.
+ *
+ * @pad0 and @pad1 must be a sink and a source, in any order.
+ *
+ * Return: The bitmask of streams of @pad1 that are routed to @streams on @pad0.
+ */
+u64 v4l2_subdev_state_xlate_streams(const struct v4l2_subdev_state *state,
+				    u32 pad0, u32 pad1, u64 *streams);
+
 /**
  * enum v4l2_subdev_routing_restriction - Subdevice internal routing restrictions
  *
-- 
2.34.1

[PATCH v13 32/34] media: v4l2-subdev: Add subdev .(enable|disable)_streams() operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add two new subdev pad operations, .enable_streams() and
.disable_streams(), to allow control of individual streams per pad. This
is a superset of what the video .s_stream() operation implements.

To help with handling of backward compatibility, add two wrapper
functions around those operations, and require their usage in drivers.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 224 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  85 ++++++++++
 2 files changed, 309 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 7d46c9a5a6c4..258f04968047 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1772,6 +1772,230 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_routing_validate);
 
+static int v4l2_subdev_enable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+					       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * The subdev doesn't implement pad-based stream enable, fall back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if (sd->enabled_streams & streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already enabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Start streaming when the first streams are enabled. */
+	if (!sd->enabled_streams) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 1);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams |= streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .enable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->enable_streams)
+		return v4l2_subdev_enable_streams_fallback(sd, pad,
+							   streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already enabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (cfg->enabled) {
+			dev_dbg(dev, "stream %u already enabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .enable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, enable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as enabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = true;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_enable_streams);
+
+static int v4l2_subdev_disable_streams_fallback(struct v4l2_subdev *sd, u32 pad,
+						u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	unsigned int i;
+	int ret;
+
+	/*
+	 * If the subdev doesn't implement pad-based stream enable, fall  back
+	 * on the .s_stream() operation. This can only be done for subdevs that
+	 * have a single source pad, as sd->enabled_streams is global to the
+	 * subdev.
+	 */
+	if (!(sd->entity.pads[pad].flags & MEDIA_PAD_FL_SOURCE))
+		return -EOPNOTSUPP;
+
+	for (i = 0; i < sd->entity.num_pads; ++i) {
+		if (i != pad && sd->entity.pads[i].flags & MEDIA_PAD_FL_SOURCE)
+			return -EOPNOTSUPP;
+	}
+
+	if ((sd->enabled_streams & streams_mask) != streams_mask) {
+		dev_dbg(dev, "set of streams %#llx already disabled on %s:%u\n",
+			streams_mask, sd->entity.name, pad);
+		return -EALREADY;
+	}
+
+	/* Stop streaming when the last streams are disabled. */
+	if (!(sd->enabled_streams & ~streams_mask)) {
+		ret = v4l2_subdev_call(sd, video, s_stream, 0);
+		if (ret)
+			return ret;
+	}
+
+	sd->enabled_streams &= ~streams_mask;
+
+	return 0;
+}
+
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask)
+{
+	struct device *dev = sd->entity.graph_obj.mdev->dev;
+	struct v4l2_subdev_state *state;
+	u64 found_streams = 0;
+	unsigned int i;
+	int ret;
+
+	/* A few basic sanity checks first. */
+	if (pad >= sd->entity.num_pads)
+		return -EINVAL;
+
+	if (!streams_mask)
+		return 0;
+
+	/* Fallback on .s_stream() if .disable_streams() isn't available. */
+	if (!sd->ops->pad || !sd->ops->pad->disable_streams)
+		return v4l2_subdev_disable_streams_fallback(sd, pad,
+							    streams_mask);
+
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	/*
+	 * Verify that the requested streams exist and that they are not
+	 * already disabled.
+	 */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad != pad || !(streams_mask & BIT_ULL(cfg->stream)))
+			continue;
+
+		found_streams |= BIT_ULL(cfg->stream);
+
+		if (!cfg->enabled) {
+			dev_dbg(dev, "stream %u already disabled on %s:%u\n",
+				cfg->stream, sd->entity.name, pad);
+			ret = -EALREADY;
+			goto done;
+		}
+	}
+
+	if (found_streams != streams_mask) {
+		dev_dbg(dev, "streams 0x%llx not found on %s:%u\n",
+			streams_mask & ~found_streams, sd->entity.name, pad);
+		ret = -EINVAL;
+		goto done;
+	}
+
+	/* Call the .disable_streams() operation. */
+	ret = v4l2_subdev_call(sd, pad, disable_streams, state, pad,
+			       streams_mask);
+	if (ret)
+		goto done;
+
+	/* Mark the streams as disabled. */
+	for (i = 0; i < state->stream_configs.num_configs; ++i) {
+		struct v4l2_subdev_stream_config *cfg =
+			&state->stream_configs.configs[i];
+
+		if (cfg->pad == pad && (streams_mask & BIT_ULL(cfg->stream)))
+			cfg->enabled = false;
+	}
+
+done:
+	v4l2_subdev_unlock_state(state);
+
+	return ret;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 86d64a755605..cf9a6b7714b9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -700,6 +700,7 @@ struct v4l2_subdev_pad_config {
  *
  * @pad: pad number
  * @stream: stream number
+ * @enabled: has the stream been enabled with v4l2_subdev_enable_stream()
  * @fmt: &struct v4l2_mbus_framefmt
  * @crop: &struct v4l2_rect to be used for crop
  * @compose: &struct v4l2_rect to be used for compose
@@ -709,6 +710,7 @@ struct v4l2_subdev_pad_config {
 struct v4l2_subdev_stream_config {
 	u32 pad;
 	u32 stream;
+	bool enabled;
 
 	struct v4l2_mbus_framefmt fmt;
 	struct v4l2_rect crop;
@@ -814,6 +816,18 @@ struct v4l2_subdev_state {
  *
  * @set_routing: enable or disable data connection routes described in the
  *		 subdevice routing table.
+ *
+ * @enable_streams: Enable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_enable_streams() instead.
+ *
+ * @disable_streams: Disable the streams defined in streams_mask on the given
+ *	source pad. Subdevs that implement this operation must use the active
+ *	state management provided by the subdev core (enabled through a call to
+ *	v4l2_subdev_init_finalize() at initialization time). Do not call
+ *	directly, use v4l2_subdev_disable_streams() instead.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -860,6 +874,12 @@ struct v4l2_subdev_pad_ops {
 			   struct v4l2_subdev_state *state,
 			   enum v4l2_subdev_format_whence which,
 			   struct v4l2_subdev_krouting *route);
+	int (*enable_streams)(struct v4l2_subdev *sd,
+			      struct v4l2_subdev_state *state, u32 pad,
+			      u64 streams_mask);
+	int (*disable_streams)(struct v4l2_subdev *sd,
+			       struct v4l2_subdev_state *state, u32 pad,
+			       u64 streams_mask);
 };
 
 /**
@@ -1005,6 +1025,10 @@ struct v4l2_subdev_platform_data {
  * @active_state: Active state for the subdev (NULL for subdevs tracking the
  *		  state internally). Initialized by calling
  *		  v4l2_subdev_init_finalize().
+ * @enabled_streams: Bitmask of enabled streams used by
+ *		     v4l2_subdev_enable_streams() and
+ *		     v4l2_subdev_disable_streams() helper functions for fallback
+ *		     cases.
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -1050,6 +1074,7 @@ struct v4l2_subdev {
 	 * doesn't support it.
 	 */
 	struct v4l2_subdev_state *active_state;
+	u64 enabled_streams;
 };
 
 
@@ -1634,6 +1659,66 @@ int v4l2_subdev_routing_validate(struct v4l2_subdev *sd,
 				 const struct v4l2_subdev_krouting *routing,
 				 enum v4l2_subdev_routing_restriction disallow);
 
+/**
+ * v4l2_subdev_enable_streams() - Enable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to enable
+ *
+ * This function enables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows enabling multiple streams on a pad at
+ * once.
+ *
+ * Enabling a stream that is already enabled isn't allowed. If @streams_mask
+ * contains an already enabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream enable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is already enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
+			       u64 streams_mask);
+
+/**
+ * v4l2_subdev_disable_streams() - Disable streams on a pad
+ * @sd: The subdevice
+ * @pad: The pad
+ * @streams_mask: Bitmask of streams to disable
+ *
+ * This function disables streams on a source @pad of a subdevice. The pad is
+ * identified by its index, while the streams are identified by the
+ * @streams_mask bitmask. This allows disabling multiple streams on a pad at
+ * once.
+ *
+ * Disabling a streams that is not enabled isn't allowed. If @streams_mask
+ * contains a disabled stream, this function returns -EALREADY without
+ * performing any operation.
+ *
+ * Per-stream disable is only available for subdevs that implement the
+ * .enable_streams() and .disable_streams() operations. For other subdevs, this
+ * function implements a best-effort compatibility by calling the .s_stream()
+ * operation, limited to subdevs that have a single source pad.
+ *
+ * Return:
+ * * 0: Success
+ * * -EALREADY: One of the streams in streams_mask is not enabled
+ * * -EINVAL: The pad index is invalid, or doesn't correspond to a source pad
+ * * -EOPNOTSUPP: Falling back to the legacy .s_stream() operation is
+ *   impossible because the subdev has multiple source pads
+ */
+int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
+				u64 streams_mask);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 33/34] media: v4l2-subdev: Add v4l2_subdev_s_stream_helper() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The v4l2_subdev_s_stream_helper() helper can be used by subdevs that
implement the stream-aware .enable_streams() and .disable_streams()
operations to implement .s_stream(). This is limited to subdevs that
have a single source pad.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo@jmondi.org>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 40 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 17 ++++++++++++
 2 files changed, 57 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 258f04968047..cfd7d795abef 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1996,6 +1996,46 @@ int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_disable_streams);
 
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable)
+{
+	struct v4l2_subdev_state *state;
+	struct v4l2_subdev_route *route;
+	struct media_pad *pad;
+	u64 source_mask = 0;
+	int pad_index = -1;
+
+	/*
+	 * Find the source pad. This helper is meant for subdevs that have a
+	 * single source pad, so failures shouldn't happen, but catch them
+	 * loudly nonetheless as they indicate a driver bug.
+	 */
+	media_entity_for_each_pad(&sd->entity, pad) {
+		if (pad->flags & MEDIA_PAD_FL_SOURCE) {
+			pad_index = pad->index;
+			break;
+		}
+	}
+
+	if (WARN_ON(pad_index == -1))
+		return -EINVAL;
+
+	/*
+	 * As there's a single source pad, just collect all the source streams.
+	 */
+	state = v4l2_subdev_lock_and_get_active_state(sd);
+
+	for_each_active_route(&state->routing, route)
+		source_mask |= BIT_ULL(route->source_stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (enable)
+		return v4l2_subdev_enable_streams(sd, pad_index, source_mask);
+	else
+		return v4l2_subdev_disable_streams(sd, pad_index, source_mask);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_s_stream_helper);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index cf9a6b7714b9..bf12c8bf0545 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1719,6 +1719,23 @@ int v4l2_subdev_enable_streams(struct v4l2_subdev *sd, u32 pad,
 int v4l2_subdev_disable_streams(struct v4l2_subdev *sd, u32 pad,
 				u64 streams_mask);
 
+/**
+ * v4l2_subdev_s_stream_helper() - Helper to implement the subdev s_stream
+ *	operation using enable_streams and disable_streams
+ * @sd: The subdevice
+ * @enable: Enable or disable streaming
+ *
+ * Subdevice drivers that implement the streams-aware
+ * &v4l2_subdev_pad_ops.enable_streams and &v4l2_subdev_pad_ops.disable_streams
+ * operations can use this helper to implement the legacy
+ * &v4l2_subdev_video_ops.s_stream operation.
+ *
+ * This helper can only be used by subdevs that have a single source pad.
+ *
+ * Return: 0 on success, or a negative error code otherwise.
+ */
+int v4l2_subdev_s_stream_helper(struct v4l2_subdev *sd, int enable);
+
 #endif /* CONFIG_VIDEO_V4L2_SUBDEV_API */
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
-- 
2.34.1

[PATCH v13 34/34] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index bf12c8bf0545..73933f9133e0 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.34.1

Re: [PATCH v13 01/34] media: Documentation: mc: add definitions for stream and pipeline

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:49PM +0300, Tomi Valkeinen wrote:
> The doc talks about streams and pipelines, but doesn't really define
> them. This is an attempt to define them according to my understanding.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

This is a good start.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  Documentation/driver-api/media/mc-core.rst | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 02481a2513b9..2a0c0aeec5f2 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -213,6 +213,18 @@ Link properties can be modified at runtime by calling
>  Pipelines and media streams
>  ^^^^^^^^^^^^^^^^^^^^^^^^^^^
>  
> +A media stream is a stream of pixels or metadata originating from one or more
> +source devices (such as a sensors) and flowing through media entity pads
> +towards the final sinks. The stream can be modified on the route by the
> +devices (e.g. scaling or pixel format conversions), or it can be split into
> +multiple branches, or multiple branches can be merged.
> +
> +A media pipeline is a set of media streams which are interdependent. This
> +interdependency can be caused by the hardware (e.g. configuration of a second
> +stream cannot be changed if the first stream has been enabled) or by the driver
> +due to the software design. Most commonly a media pipeline consists of a single
> +stream which does not branch.
> +
>  When starting streaming, drivers must notify all entities in the pipeline to
>  prevent link states from being modified during streaming by calling
>  :c:func:`media_pipeline_start()`.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 02/34] media: media-entity.h: add include for min()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:50PM +0300, Tomi Valkeinen wrote:
> ./include/media/media-entity.h:595:34: error: implicit declaration of function ‘min’
> 
> Include kernel.h to get the definition for min().
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/media-entity.h | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index a9a1c0ec5d1c..3e3d06c1ec36 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -15,6 +15,7 @@
>  #include <linux/bug.h>
>  #include <linux/container_of.h>
>  #include <linux/fwnode.h>
> +#include <linux/kernel.h>

This should be minmax.h. With this fixed,

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>  #include <linux/list.h>
>  #include <linux/media.h>
>  #include <linux/types.h>

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 04/34] media: mc: entity: Rename streaming_count -> start_count

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:52PM +0300, Tomi Valkeinen wrote:
> 'streaming_count' is a bit misleading name, as the count is increased
> with media_pipeline_start(). Let's rename it to 'start_count' instead.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/mc/mc-entity.c                            | 8 ++++----
>  drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c | 4 ++--
>  include/media/media-entity.h                            | 4 ++--
>  3 files changed, 8 insertions(+), 8 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 11f5207f73aa..d284f616a456 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -414,8 +414,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	struct media_link *link;
>  	int ret;
>  
> -	if (pipe->streaming_count) {
> -		pipe->streaming_count++;
> +	if (pipe->start_count) {
> +		pipe->start_count++;
>  		return 0;
>  	}
>  
> @@ -498,7 +498,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  		}
>  	}
>  
> -	pipe->streaming_count++;
> +	pipe->start_count++;
>  
>  	return 0;
>  
> @@ -551,7 +551,7 @@ void __media_pipeline_stop(struct media_entity *entity)
>  	if (WARN_ON(!pipe))
>  		return;
>  
> -	if (--pipe->streaming_count)
> +	if (--pipe->start_count)
>  		return;
>  
>  	media_graph_walk_start(graph, entity);
> diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> index fee2aaacb26b..0ebd70275e23 100644
> --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> @@ -931,7 +931,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
>  	 * If the other capture is streaming, isp and sensor nodes shouldn't
>  	 * be disabled, skip them.
>  	 */
> -	if (rkisp1->pipe.streaming_count < 2) {
> +	if (rkisp1->pipe.start_count < 2) {
>  		v4l2_subdev_call(rkisp1->active_sensor->sd, video, s_stream,
>  				 false);
>  		v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, false);
> @@ -964,7 +964,7 @@ static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
>  	 * If the other capture is streaming, isp and sensor nodes are already
>  	 * enabled, skip them.
>  	 */
> -	if (rkisp1->pipe.streaming_count > 1)
> +	if (rkisp1->pipe.start_count > 1)
>  		return 0;
>  
>  	ret = v4l2_subdev_call(&rkisp1->isp.sd, video, s_stream, true);
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 3e3d06c1ec36..68a8ecb2b397 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -100,11 +100,11 @@ struct media_graph {
>  /**
>   * struct media_pipeline - Media pipeline related information
>   *
> - * @streaming_count:	Streaming start count - streaming stop count
> + * @start_count:	Media pipeline start - stop count
>   * @graph:		Media graph walk during pipeline start / stop
>   */
>  struct media_pipeline {
> -	int streaming_count;
> +	int start_count;
>  	struct media_graph graph;
>  };
>  

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 09/34] media: v4l2-dev: Add videodev wrappers for media pipelines

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:57PM +0300, Tomi Valkeinen wrote:
> With the upcoming stream related improvements to the pipelines, the
> pipelines are moved from media entities to media pads. As the drivers
> currently use the pipelines with the entity based model, moving the
> pipelines to pads will cause changes to the drivers.
> 
> However, most of the uses of media pipelines are related to a video
> device (a DMA engine) with a single pad, and thus there's never a need
> to support multiple pads in these use cases. We can avoid pushing the
> complexities of the pad based model to the drivers by adding video
> device wrappers for the pipeline related functions.
> 
> This patch adds a number of wrappers to media_pipeline functions, all of
> which take a video_device as a parameter (instead of a media_entity),
> and verify that there's just one pad.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-dev.c | 61 ++++++++++++++++++++
>  include/media/v4l2-dev.h           | 90 ++++++++++++++++++++++++++++++
>  2 files changed, 151 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index d00237ee4cae..7f933ff89fd4 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -1095,6 +1095,67 @@ void video_unregister_device(struct video_device *vdev)
>  }
>  EXPORT_SYMBOL(video_unregister_device);
>  
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +__must_check int video_device_pipeline_start(struct video_device *vdev,
> +					     struct media_pipeline *pipe)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (entity->num_pads != 1)
> +		return -ENODEV;
> +
> +	return media_pipeline_start(entity, pipe);
> +}
> +EXPORT_SYMBOL_GPL(video_device_pipeline_start);
> +
> +__must_check int __video_device_pipeline_start(struct video_device *vdev,
> +					       struct media_pipeline *pipe)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (entity->num_pads != 1)
> +		return -ENODEV;
> +
> +	return __media_pipeline_start(entity, pipe);
> +}
> +EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
> +
> +void video_device_pipeline_stop(struct video_device *vdev)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (WARN_ON(entity->num_pads != 1))
> +		return;
> +
> +	return media_pipeline_stop(entity);
> +}
> +EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
> +
> +void __video_device_pipeline_stop(struct video_device *vdev)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (WARN_ON(entity->num_pads != 1))
> +		return;
> +
> +	return __media_pipeline_stop(entity);
> +}
> +EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
> +
> +struct media_pipeline *video_device_pipeline(struct video_device *vdev)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (WARN_ON(entity->num_pads != 1))
> +		return NULL;
> +
> +	return media_entity_pipeline(entity);
> +}
> +EXPORT_SYMBOL_GPL(video_device_pipeline);
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
>  /*
>   *	Initialise video for linux
>   */
> diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
> index 5cf1edefb822..4ccc24f5918a 100644
> --- a/include/media/v4l2-dev.h
> +++ b/include/media/v4l2-dev.h
> @@ -539,4 +539,94 @@ static inline int video_is_registered(struct video_device *vdev)
>  	return test_bit(V4L2_FL_REGISTERED, &vdev->flags);
>  }
>  
> +#if defined(CONFIG_MEDIA_CONTROLLER)
> +
> +/**
> + * video_device_pipeline_start - Mark a pipeline as streaming
> + * @vdev: Starting video device
> + * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + *
> + * Mark all entities connected to a given video device through enabled links,
> + * either directly or indirectly, as streaming. The given pipeline object is
> + * assigned to every entity in the pipeline and stored in the media_entity pipe
> + * field.
> + *
> + * Calls to this function can be nested, in which case the same number of
> + * video_device_pipeline_stop() calls will be required to stop streaming. The
> + * pipeline pointer must be identical for all nested calls to
> + * video_device_pipeline_start().
> + *
> + * The video device must contain a single pad.
> + *
> + * This is a convenience wrapper to media_pipeline_start().

s/wrapper to/wrapper around/ maybe. Same below.

> + */
> +__must_check int video_device_pipeline_start(struct video_device *vdev,
> +					     struct media_pipeline *pipe);
> +
> +/**
> + * __video_device_pipeline_start - Mark a pipeline as streaming
> + *

I'd drop the blank line as you don't have one for the previous function.
Same for stop.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> + * @vdev: Starting video device
> + * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + *
> + * ..note:: This is the non-locking version of video_device_pipeline_start()
> + *
> + * The video device must contain a single pad.
> + *
> + * This is a convenience wrapper to __media_pipeline_start().
> + */
> +__must_check int __video_device_pipeline_start(struct video_device *vdev,
> +					       struct media_pipeline *pipe);
> +
> +/**
> + * video_device_pipeline_stop - Mark a pipeline as not streaming
> + * @vdev: Starting video device
> + *
> + * Mark all entities connected to a given entity through enabled links, either
> + * directly or indirectly, as not streaming. The media_entity pipe field is
> + * reset to %NULL.
> + *
> + * If multiple calls to media_pipeline_start() have been made, the same
> + * number of calls to this function are required to mark the pipeline as not
> + * streaming.
> + *
> + * The video device must contain a single pad.
> + *
> + * This is a convenience wrapper to media_pipeline_stop().
> + */
> +void video_device_pipeline_stop(struct video_device *vdev);
> +
> +/**
> + * __video_device_pipeline_stop - Mark a pipeline as not streaming
> + *
> + * @vdev: Starting video device
> + *
> + * .. note:: This is the non-locking version of media_pipeline_stop()
> + *
> + * The video device must contain a single pad.
> + *
> + * This is a convenience wrapper to __media_pipeline_stop().
> + */
> +void __video_device_pipeline_stop(struct video_device *vdev);
> +
> +/**
> + * video_device_pipeline - Get the media pipeline a video device is part of
> + * @vdev: The video device
> + *
> + * This function returns the media pipeline that a video device has been
> + * associated with when constructing the pipeline with
> + * video_device_pipeline_start(). The pointer remains valid until
> + * video_device_pipeline_stop() is called.
> + *
> + * Return: The media_pipeline the video device is part of, or NULL if the video
> + * device is not part of any pipeline.
> + *
> + * The video device must contain a single pad.
> + *
> + * This is a convenience wrapper to media_entity_pipeline().
> + */
> +struct media_pipeline *video_device_pipeline(struct video_device *vdev);
> +
> +#endif /* CONFIG_MEDIA_CONTROLLER */
> +
>  #endif /* _V4L2_DEV_H */

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 10/34] media: drivers: use video device pipeline start/stop

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:58PM +0300, Tomi Valkeinen wrote:
> Convert the media drivers to use video device based pipeline start/stop
> where possible.

Very nice !

Reviewed-by: Laurent Pinchart <laurent.pinchart+renesas@ideasonboard.com>

> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c          |  6 +++---
>  drivers/media/platform/qcom/camss/camss-video.c        |  6 +++---
>  drivers/media/platform/renesas/rcar-vin/rcar-dma.c     |  6 +++---
>  drivers/media/platform/renesas/vsp1/vsp1_video.c       |  6 +++---
>  .../media/platform/rockchip/rkisp1/rkisp1-capture.c    | 10 +++++-----
>  .../media/platform/samsung/exynos4-is/fimc-capture.c   |  9 ++++-----
>  .../media/platform/samsung/exynos4-is/fimc-isp-video.c |  9 ++++-----
>  drivers/media/platform/samsung/exynos4-is/fimc-lite.c  |  9 ++++-----
>  drivers/media/platform/st/stm32/stm32-dcmi.c           |  6 +++---
>  drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c     |  6 +++---
>  drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c   |  6 +++---
>  drivers/media/platform/ti/cal/cal-video.c              |  6 +++---
>  drivers/media/platform/ti/omap3isp/ispvideo.c          |  6 +++---
>  drivers/media/platform/xilinx/xilinx-dma.c             |  6 +++---
>  drivers/media/test-drivers/vimc/vimc-capture.c         |  7 +++----
>  drivers/staging/media/ipu3/ipu3-v4l2.c                 |  6 +++---
>  drivers/staging/media/omap4iss/iss_video.c             |  6 +++---
>  drivers/staging/media/tegra-video/tegra210.c           |  6 +++---
>  18 files changed, 59 insertions(+), 63 deletions(-)
> 
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> index dbdbdb648a0d..b157fcc63b60 100644
> --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> @@ -989,7 +989,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		return r;
>  	}
>  
> -	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
> +	r = video_device_pipeline_start(&q->vdev, &q->pipe);
>  	if (r)
>  		goto fail_pipeline;
>  
> @@ -1009,7 +1009,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  fail_csi2_subdev:
>  	cio2_hw_exit(cio2, q);
>  fail_hw:
> -	media_pipeline_stop(&q->vdev.entity);
> +	video_device_pipeline_stop(&q->vdev);
>  fail_pipeline:
>  	dev_dbg(dev, "failed to start streaming (%d)\n", r);
>  	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
> @@ -1030,7 +1030,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
>  	cio2_hw_exit(cio2, q);
>  	synchronize_irq(cio2->pci_dev->irq);
>  	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
> -	media_pipeline_stop(&q->vdev.entity);
> +	video_device_pipeline_stop(&q->vdev);
>  	pm_runtime_put(dev);
>  	cio2->streaming = false;
>  }
> diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
> index 307bb1dc4589..21f9fb647692 100644
> --- a/drivers/media/platform/qcom/camss/camss-video.c
> +++ b/drivers/media/platform/qcom/camss/camss-video.c
> @@ -493,7 +493,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
>  	struct v4l2_subdev *subdev;
>  	int ret;
>  
> -	ret = media_pipeline_start(&vdev->entity, &video->pipe);
> +	ret = video_device_pipeline_start(vdev, &video->pipe);
>  	if (ret < 0)
>  		return ret;
>  
> @@ -522,7 +522,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
>  	return 0;
>  
>  error:
> -	media_pipeline_stop(&vdev->entity);
> +	video_device_pipeline_stop(vdev);
>  
>  	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
>  
> @@ -553,7 +553,7 @@ static void video_stop_streaming(struct vb2_queue *q)
>  		v4l2_subdev_call(subdev, video, s_stream, 0);
>  	}
>  
> -	media_pipeline_stop(&vdev->entity);
> +	video_device_pipeline_stop(vdev);
>  
>  	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
>  }
> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> index 924907b71263..548067f19576 100644
> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> @@ -1265,7 +1265,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	sd = media_entity_to_v4l2_subdev(pad->entity);
>  
>  	if (!on) {
> -		media_pipeline_stop(&vin->vdev.entity);
> +		video_device_pipeline_stop(&vin->vdev);
>  		return v4l2_subdev_call(sd, video, s_stream, 0);
>  	}
>  
> @@ -1282,7 +1282,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	mdev = vin->vdev.entity.graph_obj.mdev;
>  	mutex_lock(&mdev->graph_mutex);
>  	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
> -	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
> +	ret = __video_device_pipeline_start(&vin->vdev, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	if (ret)
>  		return ret;
> @@ -1291,7 +1291,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	if (ret == -ENOIOCTLCMD)
>  		ret = 0;
>  	if (ret)
> -		media_pipeline_stop(&vin->vdev.entity);
> +		video_device_pipeline_stop(&vin->vdev);
>  
>  	return ret;
>  }
> diff --git a/drivers/media/platform/renesas/vsp1/vsp1_video.c b/drivers/media/platform/renesas/vsp1/vsp1_video.c
> index 51219b1b6ea9..62811e8aebe6 100644
> --- a/drivers/media/platform/renesas/vsp1/vsp1_video.c
> +++ b/drivers/media/platform/renesas/vsp1/vsp1_video.c
> @@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
>  	}
>  	mutex_unlock(&pipe->lock);
>  
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  	vsp1_video_release_buffers(video);
>  	vsp1_video_pipeline_put(pipe);
>  }
> @@ -1046,7 +1046,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  		return PTR_ERR(pipe);
>  	}
>  
> -	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
> +	ret = __video_device_pipeline_start(&video->video, &pipe->pipe);
>  	if (ret < 0) {
>  		mutex_unlock(&mdev->graph_mutex);
>  		goto err_pipe;
> @@ -1070,7 +1070,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	return 0;
>  
>  err_stop:
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  err_pipe:
>  	vsp1_video_pipeline_put(pipe);
>  	return ret;
> diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> index 0ebd70275e23..f497c5b919eb 100644
> --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> @@ -918,7 +918,7 @@ static void rkisp1_cap_stream_disable(struct rkisp1_capture *cap)
>   *
>   * Call s_stream(false) in the reverse order from
>   * rkisp1_pipeline_stream_enable() and disable the DMA engine.
> - * Should be called before media_pipeline_stop()
> + * Should be called before video_device_pipeline_stop()
>   */
>  static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
>  	__must_hold(&cap->rkisp1->stream_lock)
> @@ -945,7 +945,7 @@ static void rkisp1_pipeline_stream_disable(struct rkisp1_capture *cap)
>   * rkisp1_pipeline_stream_enable - enable nodes in the pipeline
>   *
>   * Enable the DMA Engine and call s_stream(true) through the pipeline.
> - * Should be called after media_pipeline_start()
> + * Should be called after video_device_pipeline_start()
>   */
>  static int rkisp1_pipeline_stream_enable(struct rkisp1_capture *cap)
>  	__must_hold(&cap->rkisp1->stream_lock)
> @@ -1009,7 +1009,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
>  
>  	rkisp1_dummy_buf_destroy(cap);
>  
> -	media_pipeline_stop(&node->vdev.entity);
> +	video_device_pipeline_stop(&node->vdev);
>  
>  	mutex_unlock(&cap->rkisp1->stream_lock);
>  }
> @@ -1023,7 +1023,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
>  
>  	mutex_lock(&cap->rkisp1->stream_lock);
>  
> -	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
> +	ret = video_device_pipeline_start(&cap->vnode.vdev, &cap->rkisp1->pipe);
>  	if (ret) {
>  		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
>  		goto err_ret_buffers;
> @@ -1059,7 +1059,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
>  err_destroy_dummy:
>  	rkisp1_dummy_buf_destroy(cap);
>  err_pipeline_stop:
> -	media_pipeline_stop(entity);
> +	video_device_pipeline_stop(&cap->vnode.vdev);
>  err_ret_buffers:
>  	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
>  	mutex_unlock(&cap->rkisp1->stream_lock);
> diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
> index 7ff4024003f4..e1d43172634c 100644
> --- a/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
> +++ b/drivers/media/platform/samsung/exynos4-is/fimc-capture.c
> @@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
>  	mutex_lock(&fimc->lock);
>  
>  	if (close && vc->streaming) {
> -		media_pipeline_stop(&vc->ve.vdev.entity);
> +		video_device_pipeline_stop(&vc->ve.vdev);
>  		vc->streaming = false;
>  	}
>  
> @@ -1176,7 +1176,6 @@ static int fimc_cap_streamon(struct file *file, void *priv,
>  {
>  	struct fimc_dev *fimc = video_drvdata(file);
>  	struct fimc_vid_cap *vc = &fimc->vid_cap;
> -	struct media_entity *entity = &vc->ve.vdev.entity;
>  	struct fimc_source_info *si = NULL;
>  	struct v4l2_subdev *sd;
>  	int ret;
> @@ -1184,7 +1183,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
>  	if (fimc_capture_active(fimc))
>  		return -EBUSY;
>  
> -	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
> +	ret = video_device_pipeline_start(&vc->ve.vdev, &vc->ve.pipe->mp);
>  	if (ret < 0)
>  		return ret;
>  
> @@ -1218,7 +1217,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
>  	}
>  
>  err_p_stop:
> -	media_pipeline_stop(entity);
> +	video_device_pipeline_stop(&vc->ve.vdev);
>  	return ret;
>  }
>  
> @@ -1234,7 +1233,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
>  		return ret;
>  
>  	if (vc->streaming) {
> -		media_pipeline_stop(&vc->ve.vdev.entity);
> +		video_device_pipeline_stop(&vc->ve.vdev);
>  		vc->streaming = false;
>  	}
>  
> diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
> index 83688a7982f7..3bbebaf1e989 100644
> --- a/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
> +++ b/drivers/media/platform/samsung/exynos4-is/fimc-isp-video.c
> @@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
>  	is_singular_file = v4l2_fh_is_singular_file(file);
>  
>  	if (is_singular_file && ivc->streaming) {
> -		media_pipeline_stop(entity);
> +		video_device_pipeline_stop(&ivc->ve.vdev);
>  		ivc->streaming = 0;
>  	}
>  
> @@ -490,10 +490,9 @@ static int isp_video_streamon(struct file *file, void *priv,
>  {
>  	struct fimc_isp *isp = video_drvdata(file);
>  	struct exynos_video_entity *ve = &isp->video_capture.ve;
> -	struct media_entity *me = &ve->vdev.entity;
>  	int ret;
>  
> -	ret = media_pipeline_start(me, &ve->pipe->mp);
> +	ret = video_device_pipeline_start(&ve->vdev, &ve->pipe->mp);
>  	if (ret < 0)
>  		return ret;
>  
> @@ -508,7 +507,7 @@ static int isp_video_streamon(struct file *file, void *priv,
>  	isp->video_capture.streaming = 1;
>  	return 0;
>  p_stop:
> -	media_pipeline_stop(me);
> +	video_device_pipeline_stop(&ve->vdev);
>  	return ret;
>  }
>  
> @@ -523,7 +522,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
>  	if (ret < 0)
>  		return ret;
>  
> -	media_pipeline_stop(&video->ve.vdev.entity);
> +	video_device_pipeline_stop(&video->ve.vdev);
>  	video->streaming = 0;
>  	return 0;
>  }
> diff --git a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
> index 1a396b7cd9a9..87ce591ccb58 100644
> --- a/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
> +++ b/drivers/media/platform/samsung/exynos4-is/fimc-lite.c
> @@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
>  	if (v4l2_fh_is_singular_file(file) &&
>  	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
>  		if (fimc->streaming) {
> -			media_pipeline_stop(entity);
> +			video_device_pipeline_stop(&fimc->ve.vdev);
>  			fimc->streaming = false;
>  		}
>  		fimc_lite_stop_capture(fimc, false);
> @@ -812,13 +812,12 @@ static int fimc_lite_streamon(struct file *file, void *priv,
>  			      enum v4l2_buf_type type)
>  {
>  	struct fimc_lite *fimc = video_drvdata(file);
> -	struct media_entity *entity = &fimc->ve.vdev.entity;
>  	int ret;
>  
>  	if (fimc_lite_active(fimc))
>  		return -EBUSY;
>  
> -	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
> +	ret = video_device_pipeline_start(&fimc->ve.vdev, &fimc->ve.pipe->mp);
>  	if (ret < 0)
>  		return ret;
>  
> @@ -835,7 +834,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
>  	}
>  
>  err_p_stop:
> -	media_pipeline_stop(entity);
> +	video_device_pipeline_stop(&fimc->ve.vdev);
>  	return 0;
>  }
>  
> @@ -849,7 +848,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
>  	if (ret < 0)
>  		return ret;
>  
> -	media_pipeline_stop(&fimc->ve.vdev.entity);
> +	video_device_pipeline_stop(&fimc->ve.vdev);
>  	fimc->streaming = false;
>  	return 0;
>  }
> diff --git a/drivers/media/platform/st/stm32/stm32-dcmi.c b/drivers/media/platform/st/stm32/stm32-dcmi.c
> index 09a743cd7004..c17bf075fc49 100644
> --- a/drivers/media/platform/st/stm32/stm32-dcmi.c
> +++ b/drivers/media/platform/st/stm32/stm32-dcmi.c
> @@ -752,7 +752,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		goto err_unlocked;
>  	}
>  
> -	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
> +	ret = video_device_pipeline_start(dcmi->vdev, &dcmi->pipeline);
>  	if (ret < 0) {
>  		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
>  			__func__, ret);
> @@ -866,7 +866,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	dcmi_pipeline_stop(dcmi);
>  
>  err_media_pipeline_stop:
> -	media_pipeline_stop(&dcmi->vdev->entity);
> +	video_device_pipeline_stop(dcmi->vdev);
>  
>  err_pm_put:
>  	pm_runtime_put(dcmi->dev);
> @@ -893,7 +893,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
>  
>  	dcmi_pipeline_stop(dcmi);
>  
> -	media_pipeline_stop(&dcmi->vdev->entity);
> +	video_device_pipeline_stop(dcmi->vdev);
>  
>  	spin_lock_irq(&dcmi->irqlock);
>  
> diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> index 0912a1b6d525..17ad9a3caaa5 100644
> --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> @@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		goto err_clear_dma_queue;
>  	}
>  
> -	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
> +	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
>  	if (ret < 0)
>  		goto err_free_scratch_buffer;
>  
> @@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	sun4i_csi_capture_stop(csi);
>  
>  err_disable_pipeline:
> -	media_pipeline_stop(&csi->vdev.entity);
> +	video_device_pipeline_stop(&csi->vdev);
>  
>  err_free_scratch_buffer:
>  	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
> @@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
>  	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
>  	spin_unlock_irqrestore(&csi->qlock, flags);
>  
> -	media_pipeline_stop(&csi->vdev.entity);
> +	video_device_pipeline_stop(&csi->vdev);
>  
>  	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
>  			  csi->scratch.paddr);
> diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> index 682c26536034..de4c0d47240f 100644
> --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> @@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
>  
>  	video->sequence = 0;
>  
> -	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
> +	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
>  	if (ret < 0)
>  		goto clear_dma_queue;
>  
> @@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
>  stop_csi_stream:
>  	sun6i_csi_set_stream(video->csi, false);
>  stop_media_pipeline:
> -	media_pipeline_stop(&video->vdev.entity);
> +	video_device_pipeline_stop(&video->vdev);
>  clear_dma_queue:
>  	spin_lock_irqsave(&video->dma_queue_lock, flags);
>  	list_for_each_entry(buf, &video->dma_queue, list)
> @@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
>  
>  	sun6i_csi_set_stream(video->csi, false);
>  
> -	media_pipeline_stop(&video->vdev.entity);
> +	video_device_pipeline_stop(&video->vdev);
>  
>  	/* Release all active buffers */
>  	spin_lock_irqsave(&video->dma_queue_lock, flags);
> diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
> index 07ae1a34e6b0..ae29130df819 100644
> --- a/drivers/media/platform/ti/cal/cal-video.c
> +++ b/drivers/media/platform/ti/cal/cal-video.c
> @@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	dma_addr_t addr;
>  	int ret;
>  
> -	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
> +	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
>  	if (ret < 0) {
>  		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
>  		goto error_release_buffers;
> @@ -760,7 +760,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	cal_ctx_unprepare(ctx);
>  
>  error_pipeline:
> -	media_pipeline_stop(&ctx->vdev.entity);
> +	video_device_pipeline_stop(&ctx->vdev);
>  error_release_buffers:
>  	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
>  
> @@ -781,7 +781,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
>  
>  	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
>  
> -	media_pipeline_stop(&ctx->vdev.entity);
> +	video_device_pipeline_stop(&ctx->vdev);
>  }
>  
>  static const struct vb2_ops cal_video_qops = {
> diff --git a/drivers/media/platform/ti/omap3isp/ispvideo.c b/drivers/media/platform/ti/omap3isp/ispvideo.c
> index 44b0d55ee5d8..9ab89151c4f4 100644
> --- a/drivers/media/platform/ti/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/ti/omap3isp/ispvideo.c
> @@ -1103,7 +1103,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
>  	pipe->max_rate = pipe->l3_ick;
>  
> -	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
> +	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
>  	if (ret < 0)
>  		goto err_pipeline_start;
>  
> @@ -1160,7 +1160,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	return 0;
>  
>  err_check_format:
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  err_pipeline_start:
>  	/* TODO: Implement PM QoS */
>  	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
> @@ -1227,7 +1227,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
>  	video->error = false;
>  
>  	/* TODO: Implement PM QoS */
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  
>  	media_entity_enum_cleanup(&pipe->ent_enum);
>  
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 72eff5ef626b..fcec691ce7fc 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -404,7 +404,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	 */
>  	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
>  
> -	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
> +	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
>  	if (ret < 0)
>  		goto error;
>  
> @@ -430,7 +430,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	return 0;
>  
>  error_stop:
> -	media_pipeline_stop(&dma->video.entity);
> +	video_device_pipeline_stop(&dma->video);
>  
>  error:
>  	/* Give back all queued buffers to videobuf2. */
> @@ -458,7 +458,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
>  
>  	/* Cleanup the pipeline and mark it as being stopped. */
>  	xvip_pipeline_cleanup(pipe);
> -	media_pipeline_stop(&dma->video.entity);
> +	video_device_pipeline_stop(&dma->video);
>  
>  	/* Give back all queued buffers to videobuf2. */
>  	spin_lock_irq(&dma->queued_lock);
> diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
> index d1e2d0739c00..a2296393266c 100644
> --- a/drivers/media/test-drivers/vimc/vimc-capture.c
> +++ b/drivers/media/test-drivers/vimc/vimc-capture.c
> @@ -241,13 +241,12 @@ static void vimc_cap_return_all_buffers(struct vimc_cap_device *vcap,
>  static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
>  {
>  	struct vimc_cap_device *vcap = vb2_get_drv_priv(vq);
> -	struct media_entity *entity = &vcap->vdev.entity;
>  	int ret;
>  
>  	vcap->sequence = 0;
>  
>  	/* Start the media pipeline */
> -	ret = media_pipeline_start(entity, &vcap->stream.pipe);
> +	ret = video_device_pipeline_start(&vcap->vdev, &vcap->stream.pipe);
>  	if (ret) {
>  		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
>  		return ret;
> @@ -255,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
>  
>  	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
>  	if (ret) {
> -		media_pipeline_stop(entity);
> +		video_device_pipeline_stop(&vcap->vdev);
>  		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
>  		return ret;
>  	}
> @@ -274,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
>  	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
>  
>  	/* Stop the media pipeline */
> -	media_pipeline_stop(&vcap->vdev.entity);
> +	video_device_pipeline_stop(&vcap->vdev);
>  
>  	/* Release all active buffers */
>  	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
> index d1c539cefba8..e06d2012a4ef 100644
> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c
> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
> @@ -486,7 +486,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	pipe = node->pipe;
>  	imgu_pipe = &imgu->imgu_pipe[pipe];
>  	atomic_set(&node->sequence, 0);
> -	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
> +	r = video_device_pipeline_start(&node->vdev, &imgu_pipe->pipeline);
>  	if (r < 0)
>  		goto fail_return_bufs;
>  
> @@ -511,7 +511,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	return 0;
>  
>  fail_stop_pipeline:
> -	media_pipeline_stop(&node->vdev.entity);
> +	video_device_pipeline_stop(&node->vdev);
>  fail_return_bufs:
>  	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
>  
> @@ -551,7 +551,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
>  	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
>  	mutex_unlock(&imgu->streaming_lock);
>  
> -	media_pipeline_stop(&node->vdev.entity);
> +	video_device_pipeline_stop(&node->vdev);
>  }
>  
>  /******************** v4l2_ioctl_ops ********************/
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index 67d63a400fa2..3c7e35b654df 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -886,7 +886,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, true);
>  
> -	ret = media_pipeline_start(entity, &pipe->pipe);
> +	ret = video_device_pipeline_start(&video->video, &pipe->pipe);
>  	if (ret < 0)
>  		goto err_media_pipeline_start;
>  
> @@ -977,7 +977,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  err_omap4iss_set_stream:
>  	vb2_streamoff(&vfh->queue, type);
>  err_iss_video_check_format:
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  err_media_pipeline_start:
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, false);
> @@ -1031,7 +1031,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
>  
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, false);
> -	media_pipeline_stop(&video->video.entity);
> +	video_device_pipeline_stop(&video->video);
>  
>  done:
>  	mutex_unlock(&video->stream_lock);
> diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
> index f10a041e3e6c..d58370a84737 100644
> --- a/drivers/staging/media/tegra-video/tegra210.c
> +++ b/drivers/staging/media/tegra-video/tegra210.c
> @@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
>  		       VI_INCR_SYNCPT_NO_STALL);
>  
>  	/* start the pipeline */
> -	ret = media_pipeline_start(&chan->video.entity, pipe);
> +	ret = video_device_pipeline_start(&chan->video, pipe);
>  	if (ret < 0)
>  		goto error_pipeline_start;
>  
> @@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
>  error_kthread_start:
>  	tegra_channel_set_stream(chan, false);
>  error_set_stream:
> -	media_pipeline_stop(&chan->video.entity);
> +	video_device_pipeline_stop(&chan->video);
>  error_pipeline_start:
>  	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
>  	return ret;
> @@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
>  
>  	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
>  	tegra_channel_set_stream(chan, false);
> -	media_pipeline_stop(&chan->video.entity);
> +	video_device_pipeline_stop(&chan->video);
>  }
>  
>  /*

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 11/34] media: drivers: use video_device_pipeline()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:10:59PM +0300, Tomi Valkeinen wrote:
> Use video_device_pipeline() in the drivers instead of
> media_entity_pipeline().
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> ---
>  drivers/media/platform/xilinx/xilinx-dma.c | 4 ++--
>  drivers/media/platform/xilinx/xilinx-dma.h | 4 ++--
>  2 files changed, 4 insertions(+), 4 deletions(-)
> 
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index fcec691ce7fc..040d6e7cceeb 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	 * Use the pipeline object embedded in the first DMA object that starts
>  	 * streaming.
>  	 */
> -	pipe = to_xvip_pipeline(&dma->video.entity) ? : &dma->pipe;
> +	pipe = to_xvip_pipeline(&dma->video) ? : &dma->pipe;
>  
>  	ret = video_device_pipeline_start(&dma->video, &pipe->pipe);
>  	if (ret < 0)
> @@ -447,7 +447,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  static void xvip_dma_stop_streaming(struct vb2_queue *vq)
>  {
>  	struct xvip_dma *dma = vb2_get_drv_priv(vq);
> -	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video.entity);
> +	struct xvip_pipeline *pipe = to_xvip_pipeline(&dma->video);
>  	struct xvip_dma_buffer *buf, *nbuf;
>  
>  	/* Stop the pipeline. */
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
> index 3ea10f6b0bb9..9c6d4c18d1a9 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.h
> +++ b/drivers/media/platform/xilinx/xilinx-dma.h
> @@ -45,9 +45,9 @@ struct xvip_pipeline {
>  	struct xvip_dma *output;
>  };
>  
> -static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
> +static inline struct xvip_pipeline *to_xvip_pipeline(struct video_device *vdev)
>  {
> -	struct media_pipeline *pipe = media_entity_pipeline(e);
> +	struct media_pipeline *pipe = video_device_pipeline(vdev);
>  
>  	if (!pipe)
>  		return NULL;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 12/34] media: mc: entity: add alloc variant of pipeline_start

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:11:00PM +0300, Tomi Valkeinen wrote:
> Add new variant of media_pipeline_start(), media_pipeline_alloc_start().
> 
> media_pipeline_alloc_start() can be used by drivers that do not need to
> extend the media_pipeline. The function will either use the pipeline
> already associated with the entity, if such exists, or allocate a new
> pipeline.
> 
> When media_pipeline_stop() is called and the pipeline's use count drops
> to zero, the pipeline is automatically freed.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c       | 41 ++++++++++++++++++++++++++++++
>  drivers/media/v4l2-core/v4l2-dev.c | 11 ++++++++
>  include/media/media-entity.h       | 14 ++++++++++
>  include/media/v4l2-dev.h           | 14 ++++++++++
>  4 files changed, 80 insertions(+)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index b1abaa333d13..d277eed11caf 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -529,6 +529,8 @@ void __media_pipeline_stop(struct media_entity *entity)
>  
>  	media_graph_walk_cleanup(graph);
>  
> +	if (pipe->allocated)
> +		kfree(pipe);
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_stop);
>  
> @@ -542,6 +544,45 @@ void media_pipeline_stop(struct media_entity *entity)
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_stop);
>  
> +__must_check int media_pipeline_alloc_start(struct media_entity *entity)
> +{
> +	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pipeline *pipe;
> +	int ret;
> +	bool new_pipe = false;
> +
> +	mutex_lock(&mdev->graph_mutex);
> +
> +	/*
> +	 * Is the entity already part of a pipeline? If not, we need to allocate
> +	 * a pipe.
> +	 */
> +	pipe = media_entity_pipeline(entity);
> +	if (!pipe) {
> +		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
> +

You can drop the blank line.

> +		if (!pipe) {
> +			ret = -ENOMEM;
> +			goto out;
> +		}
> +
> +		new_pipe = true;
> +		pipe->allocated = true;
> +	}
> +
> +	ret = __media_pipeline_start(entity, pipe);
> +	if (ret) {
> +		if (new_pipe)
> +			kfree(pipe);

If new_pipe was a media_pipeline pointer, initialized to NULL and set to
pipe when you allocate the pipe, you could write this

	if (ret)
		kfree(new_pipe);

I don't mind much either way.

I have no objection against this patch, but I don't think it aligns well
with what I was imagining as further evolutions of the API. I would like
the pipe to be turned into an object similar to a DRM atomic commit.
There will thus never be a need for drivers to allocate the pipeline, it
will be done by the framework. We can rework (and likely drop) this
function when that happens.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

> +	}
> +
> +out:
> +	mutex_unlock(&mdev->graph_mutex);
> +
> +	return ret;
> +}
> +EXPORT_SYMBOL_GPL(media_pipeline_alloc_start);
> +
>  /* -----------------------------------------------------------------------------
>   * Links management
>   */
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index 7f933ff89fd4..945bb867a4c1 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -1143,6 +1143,17 @@ void __video_device_pipeline_stop(struct video_device *vdev)
>  }
>  EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
>  
> +__must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
> +{
> +	struct media_entity *entity = &vdev->entity;
> +
> +	if (entity->num_pads != 1)
> +		return -ENODEV;
> +
> +	return media_pipeline_alloc_start(entity);
> +}
> +EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
> +
>  struct media_pipeline *video_device_pipeline(struct video_device *vdev)
>  {
>  	struct media_entity *entity = &vdev->entity;
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 3438954892b7..3d0d5e3c3333 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -104,6 +104,7 @@ struct media_graph {
>   * @graph:		Media graph walk during pipeline start / stop
>   */
>  struct media_pipeline {
> +	bool allocated;

This should be documented.

>  	int start_count;
>  	struct media_graph graph;
>  };
> @@ -1028,6 +1029,19 @@ void media_pipeline_stop(struct media_entity *entity);
>   */
>  void __media_pipeline_stop(struct media_entity *entity);
>  
> +/**
> + * media_pipeline_alloc_start - Mark a pipeline as streaming
> + * @entity: Starting entity
> + *
> + * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
> + * of working on a given pipeline the function will use an existing pipeline if
> + * the entity is already part of a pipeline, or allocate a new pipeline.
> + *
> + * Calls to media_pipeline_alloc_start() must be matched with
> + * media_pipeline_stop().
> + */
> +__must_check int media_pipeline_alloc_start(struct media_entity *entity);
> +
>  /**
>   * media_devnode_create() - creates and initializes a device node interface
>   *
> diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
> index 4ccc24f5918a..29eb61244d1f 100644
> --- a/include/media/v4l2-dev.h
> +++ b/include/media/v4l2-dev.h
> @@ -609,6 +609,20 @@ void video_device_pipeline_stop(struct video_device *vdev);
>   */
>  void __video_device_pipeline_stop(struct video_device *vdev);
>  
> +/**
> + * video_device_pipeline_alloc_start - Mark a pipeline as streaming
> + * @vdev: Starting video device
> + *
> + * video_device_pipeline_alloc_start() is similar to video_device_pipeline_start()
> + * but instead of working on a given pipeline the function will use an
> + * existing pipeline if the video device is already part of a pipeline, or
> + * allocate a new pipeline.
> + *
> + * Calls to video_device_pipeline_alloc_start() must be matched with
> + * video_device_pipeline_stop().
> + */
> +__must_check int video_device_pipeline_alloc_start(struct video_device *vdev);
> +
>  /**
>   * video_device_pipeline - Get the media pipeline a video device is part of
>   * @vdev: The video device

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 13/34] media: drivers: use video_device_pipeline_alloc_start()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:11:01PM +0300, Tomi Valkeinen wrote:
> Use video_device_pipeline_alloc_start() instead of manually
> allocating/managing the media pipeline storage.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/renesas/rcar-vin/rcar-dma.c | 14 +-------------
>  drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c |  2 +-
>  .../media/platform/sunxi/sun6i-csi/sun6i_video.c   |  2 +-
>  drivers/media/platform/ti/cal/cal-video.c          |  2 +-
>  drivers/media/platform/ti/cal/cal.h                |  1 -
>  5 files changed, 4 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> index 548067f19576..884875600231 100644
> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> @@ -1244,8 +1244,6 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd,
>  
>  static int rvin_set_stream(struct rvin_dev *vin, int on)
>  {
> -	struct media_pipeline *pipe;
> -	struct media_device *mdev;
>  	struct v4l2_subdev *sd;
>  	struct media_pad *pad;
>  	int ret;
> @@ -1273,17 +1271,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	if (ret)
>  		return ret;
>  
> -	/*
> -	 * The graph lock needs to be taken to protect concurrent
> -	 * starts of multiple VIN instances as they might share
> -	 * a common subdevice down the line and then should use
> -	 * the same pipe.
> -	 */
> -	mdev = vin->vdev.entity.graph_obj.mdev;
> -	mutex_lock(&mdev->graph_mutex);
> -	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
> -	ret = __video_device_pipeline_start(&vin->vdev, pipe);
> -	mutex_unlock(&mdev->graph_mutex);
> +	ret = video_device_pipeline_alloc_start(&vin->vdev);

This doesn't look right, it will use different pipeline instances for
difference video devices, that's a change in behaviour.

>  	if (ret)
>  		return ret;
>  
> diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> index 17ad9a3caaa5..a3e826a755fc 100644
> --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> @@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		goto err_clear_dma_queue;
>  	}
>  
> -	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);

What ? There is a pipe embedded in video_device ? Oh my, I didn't
realize how messed up the V4L2 core had become :-(

> +	ret = video_device_pipeline_alloc_start(&csi->vdev);
>  	if (ret < 0)
>  		goto err_free_scratch_buffer;
>  
> diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> index de4c0d47240f..436922503ece 100644
> --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> @@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
>  
>  	video->sequence = 0;
>  
> -	ret = video_device_pipeline_start(&video->vdev, &video->vdev.pipe);
> +	ret = video_device_pipeline_alloc_start(&video->vdev);
>  	if (ret < 0)
>  		goto clear_dma_queue;
>  
> diff --git a/drivers/media/platform/ti/cal/cal-video.c b/drivers/media/platform/ti/cal/cal-video.c
> index ae29130df819..e8122e7ec944 100644
> --- a/drivers/media/platform/ti/cal/cal-video.c
> +++ b/drivers/media/platform/ti/cal/cal-video.c
> @@ -707,7 +707,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	dma_addr_t addr;
>  	int ret;
>  
> -	ret = video_device_pipeline_start(&ctx->vdev, &ctx->phy->pipe);
> +	ret = video_device_pipeline_alloc_start(&ctx->vdev);
>  	if (ret < 0) {
>  		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
>  		goto error_release_buffers;
> diff --git a/drivers/media/platform/ti/cal/cal.h b/drivers/media/platform/ti/cal/cal.h
> index 61409ddced98..648cec43c3fc 100644
> --- a/drivers/media/platform/ti/cal/cal.h
> +++ b/drivers/media/platform/ti/cal/cal.h
> @@ -174,7 +174,6 @@ struct cal_camerarx {
>  	struct device_node	*source_ep_node;
>  	struct device_node	*source_node;
>  	struct v4l2_subdev	*source;
> -	struct media_pipeline	pipe;
>  
>  	struct v4l2_subdev	subdev;
>  	struct media_pad	pads[CAL_CAMERARX_NUM_PADS];

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> [Note: the code is mostly from Laurent but the patch description is from Tomi]

I'll review the description then :-)

> The media_pipeline_start() and media_pipeline_stop() functions use the
> media graph walk API to traverse the graph and validate the pipeline.
> The graph walk traverses the media graph following links between the
> entities.
> 
> Also, while the pipeline can't change between the start and stop calls,
> the graph is walked again from scratch at stop time, or any time a
> driver needs to inspect the pipeline.
> 
> With the upcoming multiplexed streams support we will need a bit more
> intelligent pipeline construction, as e.g. an entity may be passing
> through two independent streams via separate pads, in which case those

Did you mean "as e.g. two independent streams may be passing through a
single entity via separate pads" ?

> pads should not be part of the same pipeline.
> 
> This patch essentially rewrites the media_pipeline_start/stop so that
> a pipeline is defined as a set of pads instead of entities and the media
> graph traversal considers the pad interdependencies when choosing which
> links to follow.
> 
> Currently all the entity's pads are considered as interdependent. This
> means that the behavior with all the current drivers stays the same, but
> in the future we can define a more fine-grained pipeline construction.
> 
> Additionally the media pipeline's pads are cached at
> media_pipeline_start() time, and re-used at media_pipeline_stop() which
> avoid the need to re-walk the whole graph as the previous implementation
> did.
> 
> Also, caching pads in the pipeline can serve in the future as the
> foundation to provide a better API than the media graph walk to drivers
> to iterate over pads and entities in the pipeline.
> 
> Note that the old media_pipeline_start/stop used the media graph walk
> API. The new version does not use the media graph walk API, but instead
> a new implementation.
> 
> There are two reason for not changing the graph walk: it proved to be
> rather difficult to change the graph walk to have the features
> implemented in this patch, and second, this keeps the backward
> compatibility of the graph walk as there are users of the graph walk API
> 
> The long term plan is that all the existing code would be converted to
> use the new cached pipeline, thus allowing us to remove the graph walk.

Could you mark the graph walk API as deprecated in this patch, or in a
subsequent patch in the series ? I think I did in a previous version,
but I may recall incorrectly.

> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst |   7 +-
>  drivers/media/mc/mc-entity.c               | 523 ++++++++++++++++++---
>  include/media/media-entity.h               |  69 ++-
>  3 files changed, 513 insertions(+), 86 deletions(-)
> 
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 2a0c0aeec5f2..9c0b09745c0a 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -229,14 +229,13 @@ When starting streaming, drivers must notify all entities in the pipeline to
>  prevent link states from being modified during streaming by calling
>  :c:func:`media_pipeline_start()`.
>  
> -The function will mark all entities connected to the given entity through
> -enabled links, either directly or indirectly, as streaming.
> +The function will mark all the pads which are part of the pipeline as streaming.
>  
>  The struct media_pipeline instance pointed to by
> -the pipe argument will be stored in every entity in the pipeline.
> +the pipe argument will be stored in every pad in the pipeline.
>  Drivers should embed the struct media_pipeline
>  in higher-level pipeline structures and can then access the
> -pipeline through the struct media_entity
> +pipeline through the struct media_pad
>  pipe field.
>  
>  Calls to :c:func:`media_pipeline_start()` can be nested.
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index d277eed11caf..df8e82be2bcc 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -225,6 +225,27 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
>   * Graph traversal
>   */
>  
> +/*
> + * This function checks the interdependency inside the entity between @pad0
> + * and @pad1. If two pads are interdependent they are part of the same pipeline
> + * and enabling one of the pads means that the other pad will become "locked"
> + * and doesn't allow configuration changes.
> + *
> + * For the time being all pads are considered interdependent.
> + */
> +static bool media_entity_has_pad_interdep(struct media_entity *entity,

I won't bikeshed this name as it can easily be changed later if needed.
It's good for now.

> +					  unsigned int pad0, unsigned int pad1)
> +{
> +	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
> +		return false;
> +
> +	if (entity->pads[pad0].flags & entity->pads[pad1].flags &
> +	    (MEDIA_PAD_FL_SINK | MEDIA_PAD_FL_SOURCE))
> +		return false;
> +
> +	return true;
> +}
> +
>  static struct media_entity *
>  media_entity_other(struct media_entity *entity, struct media_link *link)
>  {
> @@ -373,97 +394,436 @@ EXPORT_SYMBOL_GPL(media_graph_walk_next);
>   * Pipeline management
>   */
>  
> +/*
> + * The pipeline traversal stack stores pads that are reached during graph
> + * traversal, with a list of links to be visited to continue the traversal.
> + * When a new pad is reached, an entry is pushed on the top of the stack and
> + * points to the incoming pad and the first link of the entity.
> + *
> + * To find further pads in the pipeline, the traversal algorithm follows
> + * internal pad dependencies in the entity, and then links in the graph. It
> + * does so by iterating over all links of the entity, and following enabled
> + * links that originate from a pad that is internally connected to the incoming
> + * pad, as reported by the media_entity_has_pad_interdep() function.
> + */
> +
> +/**
> + * struct media_pipeline_walk_entry - Entry in the pipeline traversal stack
> + *
> + * @pad: The media pad being visited
> + * @links: Links left to be visited
> + */
> +struct media_pipeline_walk_entry {
> +	struct media_pad *pad;
> +	struct list_head *links;
> +};
> +
> +/**
> + * struct media_pipeline_walk - State used by the media pipeline traversal
> + *				algorithm
> + *
> + * @mdev: The media device
> + * @stack: Depth-first search stack
> + * @stack.size: Number of allocated entries in @stack.entries
> + * @stack.top: Index of the top stack entry (-1 if the stack is empty)
> + * @stack.entries: Stack entries
> + */
> +struct media_pipeline_walk {
> +	struct media_device *mdev;
> +
> +	struct {
> +		unsigned int size;
> +		int top;
> +		struct media_pipeline_walk_entry *entries;
> +	} stack;
> +};
> +
> +#define MEDIA_PIPELINE_STACK_GROW_STEP		16
> +
> +static struct media_pipeline_walk_entry *
> +media_pipeline_walk_top(struct media_pipeline_walk *walk)
> +{
> +	return &walk->stack.entries[walk->stack.top];
> +}
> +
> +static bool media_pipeline_walk_empty(struct media_pipeline_walk *walk)
> +{
> +	return walk->stack.top == -1;
> +}
> +
> +/* Increase the stack size by MEDIA_PIPELINE_STACK_GROW_STEP elements. */
> +static int media_pipeline_walk_resize(struct media_pipeline_walk *walk)
> +{
> +	struct media_pipeline_walk_entry *entries;
> +	unsigned int new_size;
> +
> +	/* Safety check, to avoid stack overflows in case of bugs. */
> +	if (walk->stack.size >= 256)
> +		return -E2BIG;
> +
> +	new_size = walk->stack.size + MEDIA_PIPELINE_STACK_GROW_STEP;
> +
> +	entries = krealloc(walk->stack.entries,
> +			   new_size * sizeof(*walk->stack.entries),
> +			   GFP_KERNEL);
> +	if (!entries)
> +		return -ENOMEM;
> +
> +	walk->stack.entries = entries;
> +	walk->stack.size = new_size;
> +
> +	return 0;
> +}
> +
> +/* Push a new entry on the stack. */
> +static int media_pipeline_walk_push(struct media_pipeline_walk *walk,
> +				    struct media_pad *pad)
> +{
> +	struct media_pipeline_walk_entry *entry;
> +	int ret;
> +
> +	if (walk->stack.top + 1 >= walk->stack.size) {
> +		ret = media_pipeline_walk_resize(walk);
> +		if (ret)
> +			return ret;
> +	}
> +
> +	walk->stack.top++;
> +	entry = media_pipeline_walk_top(walk);
> +	entry->pad = pad;
> +	entry->links = pad->entity->links.next;
> +
> +	dev_dbg(walk->mdev->dev,
> +		"media pipeline: pushed entry %u: '%s':%u\n",
> +		walk->stack.top, pad->entity->name, pad->index);
> +
> +	return 0;
> +}
> +
> +/*
> + * Move the top entry link cursor to the next link. If all links of the entry
> + * have been visited, pop the entry itself.
> + */
> +static void media_pipeline_walk_pop(struct media_pipeline_walk *walk)
> +{
> +	struct media_pipeline_walk_entry *entry;
> +
> +	if (WARN_ON(walk->stack.top < 0))
> +		return;
> +
> +	entry = media_pipeline_walk_top(walk);
> +
> +	if (entry->links->next == &entry->pad->entity->links) {
> +		dev_dbg(walk->mdev->dev,
> +			"media pipeline: entry %u has no more links, popping\n",
> +			walk->stack.top);
> +
> +		walk->stack.top--;
> +		return;
> +	}
> +
> +	entry->links = entry->links->next;
> +
> +	dev_dbg(walk->mdev->dev,
> +		"media pipeline: moved entry %u to next link\n",
> +		walk->stack.top);
> +}
> +
> +/* Free all memory allocated while walking the pipeline. */
> +static void media_pipeline_walk_destroy(struct media_pipeline_walk *walk)
> +{
> +	kfree(walk->stack.entries);
> +}
> +
> +/* Add a pad to the pipeline and push it to the stack. */
> +static int media_pipeline_add_pad(struct media_pipeline *pipe,
> +				  struct media_pipeline_walk *walk,
> +				  struct media_pad *pad)
> +{
> +	struct media_pipeline_pad *ppad;
> +
> +	list_for_each_entry(ppad, &pipe->pads, list) {
> +		if (ppad->pad == pad) {
> +			dev_dbg(pad->graph_obj.mdev->dev,
> +				"media pipeline: already contains pad '%s':%u\n",
> +				pad->entity->name, pad->index);
> +			return 0;
> +		}
> +	}
> +
> +	ppad = kzalloc(sizeof(*ppad), GFP_KERNEL);
> +	if (!ppad)
> +		return -ENOMEM;
> +
> +	ppad->pipe = pipe;
> +	ppad->pad = pad;
> +
> +	list_add_tail(&ppad->list, &pipe->pads);
> +
> +	dev_dbg(pad->graph_obj.mdev->dev,
> +		"media pipeline: added pad '%s':%u\n",
> +		pad->entity->name, pad->index);
> +
> +	return media_pipeline_walk_push(walk, pad);
> +}
> +
> +/* Explore the next link of the entity at the top of the stack. */
> +static int media_pipeline_explore_next_link(struct media_pipeline *pipe,
> +					    struct media_pipeline_walk *walk)
> +{
> +	struct media_pipeline_walk_entry *entry = media_pipeline_walk_top(walk);
> +	struct media_pad *pad;
> +	struct media_link *link;
> +	struct media_pad *local;
> +	struct media_pad *remote;
> +	int ret;
> +
> +	pad = entry->pad;
> +	link = list_entry(entry->links, typeof(*link), list);
> +	media_pipeline_walk_pop(walk);
> +
> +	dev_dbg(walk->mdev->dev,
> +		"media pipeline: exploring link '%s':%u -> '%s':%u\n",
> +		link->source->entity->name, link->source->index,
> +		link->sink->entity->name, link->sink->index);
> +
> +	/* Skip links that are not enabled. */
> +	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
> +		dev_dbg(walk->mdev->dev,
> +			"media pipeline: skipping link (disabled)\n");
> +		return 0;
> +	}
> +
> +	/* Get the local pad and remote pad. */
> +	if (link->source->entity == pad->entity) {
> +		local = link->source;
> +		remote = link->sink;
> +	} else {
> +		local = link->sink;
> +		remote = link->source;
> +	}
> +
> +	/*
> +	 * Skip links that originate from a different pad than the incoming pad
> +	 * that is not connected internally in the entity to the incoming pad.
> +	 */
> +	if (pad != local &&
> +	    !media_entity_has_pad_interdep(pad->entity, pad->index, local->index)) {
> +		dev_dbg(walk->mdev->dev,
> +			"media pipeline: skipping link (no route)\n");
> +		return 0;
> +	}
> +
> +	/*
> +	 * Add the local and remote pads of the link to the pipeline and push
> +	 * them to the stack, if they're not already present.
> +	 */
> +	ret = media_pipeline_add_pad(pipe, walk, local);
> +	if (ret)
> +		return ret;
> +
> +	ret = media_pipeline_add_pad(pipe, walk, remote);
> +	if (ret)
> +		return ret;
> +
> +	return 0;
> +}
> +
> +static void media_pipeline_cleanup(struct media_pipeline *pipe)
> +{
> +	while (!list_empty(&pipe->pads)) {
> +		struct media_pipeline_pad *ppad;
> +
> +		ppad = list_first_entry(&pipe->pads, typeof(*ppad), list);
> +		list_del(&ppad->list);
> +		kfree(ppad);
> +	}
> +}
> +
> +static int media_pipeline_populate(struct media_pipeline *pipe,
> +				   struct media_pad *pad)
> +{
> +	struct media_pipeline_walk walk = { };
> +	struct media_pipeline_pad *ppad;
> +	int ret;
> +
> +	/*
> +	 * Populate the media pipeline by walking the media graph, starting
> +	 * from @pad.
> +	 */
> +	INIT_LIST_HEAD(&pipe->pads);
> +	pipe->mdev = pad->graph_obj.mdev;
> +
> +	walk.mdev = pipe->mdev;
> +	walk.stack.top = -1;
> +	ret = media_pipeline_add_pad(pipe, &walk, pad);
> +	if (ret)
> +		goto done;
> +
> +	/*
> +	 * Use a depth-first search algorithm: as long as the stack is not
> +	 * empty, explore the next link of the top entry. The
> +	 * media_pipeline_explore_next_link() function will either move to the
> +	 * next link, pop the entry if fully visited, or add new entries on
> +	 * top.
> +	 */
> +	while (!media_pipeline_walk_empty(&walk)) {
> +		ret = media_pipeline_explore_next_link(pipe, &walk);
> +		if (ret)
> +			goto done;
> +	}
> +
> +	dev_dbg(pad->graph_obj.mdev->dev,
> +		"media pipeline populated, found pads:\n");
> +
> +	list_for_each_entry(ppad, &pipe->pads, list)
> +		dev_dbg(pad->graph_obj.mdev->dev, "- '%s':%u\n",
> +			ppad->pad->entity->name, ppad->pad->index);
> +
> +	WARN_ON(walk.stack.top != -1);
> +
> +	ret = 0;
> +
> +done:
> +	media_pipeline_walk_destroy(&walk);
> +
> +	if (ret)
> +		media_pipeline_cleanup(pipe);
> +
> +	return ret;
> +}
> +
>  __must_check int __media_pipeline_start(struct media_entity *entity,
>  					struct media_pipeline *pipe)
>  {
>  	struct media_device *mdev = entity->graph_obj.mdev;
> -	struct media_graph *graph = &pipe->graph;
> -	struct media_entity *entity_err = entity;
> -	struct media_link *link;
> +	struct media_pipeline_pad *err_ppad;
> +	struct media_pipeline_pad *ppad;
>  	int ret;
>  
> +	lockdep_assert_held(&mdev->graph_mutex);
> +
> +	/*
> +	 * media_pipeline_start(entity) only makes sense with entities that have
> +	 * a single pad.
> +	 */
> +
> +	if (WARN_ON(entity->num_pads != 1))
> +		return -EINVAL;
> +
> +	/*
> +	 * If the entity is already part of a pipeline, that pipeline must
> +	 * be the same as the pipe given to media_pipeline_start().
> +	 */
> +	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
> +		return -EINVAL;
> +
> +	/*
> +	 * If the pipeline has already been started, it is guaranteed to be
> +	 * valid, so just increase the start count.
> +	 */
>  	if (pipe->start_count) {
>  		pipe->start_count++;
>  		return 0;
>  	}
>  
> -	ret = media_graph_walk_init(&pipe->graph, mdev);
> +	/*
> +	 * Populate the pipeline. This populates the media_pipeline pads list
> +	 * with media_pipeline_pad instances for each pad found during graph
> +	 * walk.
> +	 */
> +	ret = media_pipeline_populate(pipe, entity->pads);
>  	if (ret)
>  		return ret;
>  
> -	media_graph_walk_start(&pipe->graph, entity);
> +	/*
> +	 * Now that all the pads in the pipeline have been gathered, perform
> +	 * the validation steps.
> +	 */
> +
> +	list_for_each_entry(ppad, &pipe->pads, list) {
> +		struct media_pad *pad = ppad->pad;
> +		struct media_entity *entity = pad->entity;
> +		bool has_enabled_link = false;
> +		bool has_link = false;
> +		struct media_link *link;
>  
> -	while ((entity = media_graph_walk_next(graph))) {
> -		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
> -		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
> +		dev_dbg(mdev->dev, "Validating pad '%s':%u\n", pad->entity->name,
> +			pad->index);
>  
> -		if (entity->pipe && entity->pipe != pipe) {
> -			pr_err("Pipe active for %s. Can't start for %s\n",
> -				entity->name,
> -				entity_err->name);
> +		/*
> +		 * 1. Ensure that the pad doesn't already belong to a different
> +		 * pipeline.
> +		 */
> +		if (pad->pipe) {
> +			dev_dbg(mdev->dev, "Failed to start pipeline: pad '%s':%u busy\n",
> +				pad->entity->name, pad->index);
>  			ret = -EBUSY;
>  			goto error;
>  		}
>  
> -		/* Already streaming --- no need to check. */
> -		if (entity->pipe)
> -			continue;
> -
> -		entity->pipe = pipe;
> -
> -		if (!entity->ops || !entity->ops->link_validate)
> -			continue;
> -
> -		bitmap_zero(active, entity->num_pads);
> -		bitmap_fill(has_no_links, entity->num_pads);
> -
> +		/*
> +		 * 2. Validate all active links whose sink is the current pad.
> +		 * Validation of the source pads is performed in the context of
> +		 * the connected sink pad to avoid duplicating checks.
> +		 */
>  		list_for_each_entry(link, &entity->links, list) {
> -			struct media_pad *pad = link->sink->entity == entity
> -						? link->sink : link->source;
> +			/* Skip links unrelated to the current pad. */
> +			if (link->sink != pad && link->source != pad)
> +				continue;
>  
> -			/* Mark that a pad is connected by a link. */
> -			bitmap_clear(has_no_links, pad->index, 1);
> +			/* Record if the pad has links and enabled links. */
> +			if (link->flags & MEDIA_LNK_FL_ENABLED)
> +				has_enabled_link = true;
> +			has_link = true;
>  
>  			/*
> -			 * Pads that either do not need to connect or
> -			 * are connected through an enabled link are
> -			 * fine.
> +			 * Validate the link if it's enabled and has the
> +			 * current pad as its sink.
>  			 */
> -			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
> -			    link->flags & MEDIA_LNK_FL_ENABLED)
> -				bitmap_set(active, pad->index, 1);
> +			if (!(link->flags & MEDIA_LNK_FL_ENABLED))
> +				continue;
>  
> -			/*
> -			 * Link validation will only take place for
> -			 * sink ends of the link that are enabled.
> -			 */
> -			if (link->sink != pad ||
> -			    !(link->flags & MEDIA_LNK_FL_ENABLED))
> +			if (link->sink != pad)
> +				continue;
> +
> +			if (!entity->ops || !entity->ops->link_validate)
>  				continue;
>  
>  			ret = entity->ops->link_validate(link);
> -			if (ret < 0 && ret != -ENOIOCTLCMD) {
> -				dev_dbg(entity->graph_obj.mdev->dev,
> -					"link validation failed for '%s':%u -> '%s':%u, error %d\n",
> +			if (ret) {
> +				dev_dbg(mdev->dev,
> +					"Link '%s':%u -> '%s':%u failed validation: %d\n",
>  					link->source->entity->name,
>  					link->source->index,
> -					entity->name, link->sink->index, ret);
> +					link->sink->entity->name,
> +					link->sink->index, ret);
>  				goto error;
>  			}
> -		}
>  
> -		/* Either no links or validated links are fine. */
> -		bitmap_or(active, active, has_no_links, entity->num_pads);
> +			dev_dbg(mdev->dev,
> +				"Link '%s':%u -> '%s':%u is valid\n",
> +				link->source->entity->name,
> +				link->source->index,
> +				link->sink->entity->name,
> +				link->sink->index);
> +		}
>  
> -		if (!bitmap_full(active, entity->num_pads)) {
> +		/*
> +		 * 3. If the pad has the MEDIA_PAD_FL_MUST_CONNECT flag set,
> +		 * ensure that it has either no link or an enabled link.
> +		 */
> +		if ((pad->flags & MEDIA_PAD_FL_MUST_CONNECT) && has_link &&
> +		    !has_enabled_link) {
> +			dev_dbg(mdev->dev,
> +				"Pad '%s':%u must be connected by an enabled link\n",
> +				pad->entity->name, pad->index);
>  			ret = -ENOLINK;
> -			dev_dbg(entity->graph_obj.mdev->dev,
> -				"'%s':%u must be connected by an enabled link\n",
> -				entity->name,
> -				(unsigned)find_first_zero_bit(
> -					active, entity->num_pads));
>  			goto error;
>  		}
> +
> +		/* Validation passed, store the pipe pointer in the pad. */
> +		pad->pipe = pipe;
>  	}
>  
>  	pipe->start_count++;
> @@ -475,20 +835,15 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	 * Link validation on graph failed. We revert what we did and
>  	 * return the error.
>  	 */
> -	media_graph_walk_start(graph, entity_err);
>  
> -	while ((entity_err = media_graph_walk_next(graph))) {
> -		entity_err->pipe = NULL;
> -
> -		/*
> -		 * We haven't started entities further than this so we quit
> -		 * here.
> -		 */
> -		if (entity_err == entity)
> +	list_for_each_entry(err_ppad, &pipe->pads, list) {
> +		if (err_ppad == ppad)
>  			break;
> +
> +		err_ppad->pad->pipe = NULL;
>  	}
>  
> -	media_graph_walk_cleanup(graph);
> +	media_pipeline_cleanup(pipe);
>  
>  	return ret;
>  }
> @@ -509,8 +864,8 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
>  
>  void __media_pipeline_stop(struct media_entity *entity)
>  {
> -	struct media_graph *graph = &entity->pipe->graph;
> -	struct media_pipeline *pipe = entity->pipe;
> +	struct media_pipeline *pipe = entity->pads->pipe;
> +	struct media_pipeline_pad *ppad;
>  
>  	/*
>  	 * If the following check fails, the driver has performed an
> @@ -522,12 +877,10 @@ void __media_pipeline_stop(struct media_entity *entity)
>  	if (--pipe->start_count)
>  		return;
>  
> -	media_graph_walk_start(graph, entity);
> -
> -	while ((entity = media_graph_walk_next(graph)))
> -		entity->pipe = NULL;
> +	list_for_each_entry(ppad, &pipe->pads, list)
> +		ppad->pad->pipe = NULL;
>  
> -	media_graph_walk_cleanup(graph);
> +	media_pipeline_cleanup(pipe);
>  
>  	if (pipe->allocated)
>  		kfree(pipe);
> @@ -837,7 +1190,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  {
>  	const u32 mask = MEDIA_LNK_FL_ENABLED;
>  	struct media_device *mdev;
> -	struct media_entity *source, *sink;
> +	struct media_pad *source, *sink;
>  	int ret = -EBUSY;
>  
>  	if (link == NULL)
> @@ -853,12 +1206,11 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  	if (link->flags == flags)
>  		return 0;
>  
> -	source = link->source->entity;
> -	sink = link->sink->entity;
> +	source = link->source;
> +	sink = link->sink;
>  
>  	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
> -	    (media_entity_is_streaming(source) ||
> -	     media_entity_is_streaming(sink)))
> +	    (media_pad_is_streaming(source) || media_pad_is_streaming(sink)))
>  		return -EBUSY;
>  
>  	mdev = source->graph_obj.mdev;
> @@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>  
>  struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>  {
> -	return entity->pipe;
> +	struct media_pipeline *pipe;
> +	struct media_pad *pad;
> +
> +	if (entity->num_pads == 0)
> +		return NULL;
> +
> +	pipe = entity->pads->pipe;
> +
> +	media_entity_for_each_pad(entity, pad) {
> +		if (WARN_ON(pad->pipe != pipe))
> +			return NULL;
> +	}

This wasn't done in v12, could you explain the rationale ? The
documentation of media_entity_pipeline() should probably be updated too.

> +
> +	return pipe;
>  }
>  EXPORT_SYMBOL_GPL(media_entity_pipeline);
>  
> +struct media_pipeline *media_pad_pipeline(struct media_pad *pad)
> +{
> +	return pad->pipe;
> +}
> +EXPORT_SYMBOL_GPL(media_pad_pipeline);
> +
>  static void media_interface_init(struct media_device *mdev,
>  				 struct media_interface *intf,
>  				 u32 gobj_type,
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 3d0d5e3c3333..107e2d8025bd 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -100,13 +100,33 @@ struct media_graph {
>  /**
>   * struct media_pipeline - Media pipeline related information
>   *
> + * @mdev:		The media device the pipeline is part of
> + * @pads:		List of media_pipeline_pad
>   * @start_count:	Media pipeline start - stop count
> - * @graph:		Media graph walk during pipeline start / stop
>   */
>  struct media_pipeline {
>  	bool allocated;
> +	struct media_device *mdev;
> +	struct list_head pads;
>  	int start_count;
> -	struct media_graph graph;
> +};
> +
> +/**
> + * struct media_pipeline_pad - A pad part of a media pipeline
> + *
> + * @list:		Entry in the media_pad pads list
> + * @pipe:		The media_pipeline that the pad is part of
> + * @pad:		The media pad
> + *
> + * This structure associate a pad with a media pipeline. Instances of
> + * media_pipeline_pad are created by media_pipeline_start() when it builds the
> + * pipeline, and stored in the &media_pad.pads list. media_pipeline_stop()
> + * removes the entries from the list and deletes them.
> + */
> +struct media_pipeline_pad {
> +	struct list_head list;
> +	struct media_pipeline *pipe;
> +	struct media_pad *pad;
>  };
>  
>  /**
> @@ -188,6 +208,8 @@ enum media_pad_signal_type {
>   * @flags:	Pad flags, as defined in
>   *		:ref:`include/uapi/linux/media.h <media_header>`
>   *		(seek for ``MEDIA_PAD_FL_*``)
> + * @pipe:	Pipeline this pad belongs to. Use media_entity_pipeline() to
> + *		access this field.
>   */
>  struct media_pad {
>  	struct media_gobj graph_obj;	/* must be first field in struct */
> @@ -195,6 +217,12 @@ struct media_pad {
>  	u16 index;
>  	enum media_pad_signal_type sig_type;
>  	unsigned long flags;
> +
> +	/*
> +	 * The fields below are private, and should only be accessed via
> +	 * appropriate functions.
> +	 */
> +	struct media_pipeline *pipe;
>  };
>  
>  /**
> @@ -271,7 +299,6 @@ enum media_entity_type {
>   * @links:	List of data links.
>   * @ops:	Entity operations.
>   * @use_count:	Use count for the entity.
> - * @pipe:	Pipeline this entity belongs to.
>   * @info:	Union with devnode information.  Kept just for backward
>   *		compatibility.
>   * @info.dev:	Contains device major and minor info.
> @@ -307,8 +334,6 @@ struct media_entity {
>  
>  	int use_count;
>  
> -	struct media_pipeline *pipe;
> -
>  	union {
>  		struct {
>  			u32 major;
> @@ -873,6 +898,18 @@ struct media_link *media_entity_find_link(struct media_pad *source,
>   */
>  struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
>  
> +/**
> + * media_pad_is_streaming - Test if a pad is part of a streaming pipeline
> + * @pad: The pad
> + *
> + * Return: True if the pad is part of a pipeline started with the
> + * media_pipeline_start() function, false otherwise.
> + */
> +static inline bool media_pad_is_streaming(const struct media_pad *pad)
> +{
> +	return pad->pipe;
> +}
> +
>  /**
>   * media_entity_is_streaming - Test if an entity is part of a streaming pipeline
>   * @entity: The entity
> @@ -882,7 +919,14 @@ struct media_pad *media_entity_remote_pad(const struct media_pad *pad);
>   */
>  static inline bool media_entity_is_streaming(const struct media_entity *entity)
>  {
> -	return entity->pipe;
> +	struct media_pad *pad;
> +
> +	media_entity_for_each_pad(entity, pad) {
> +		if (media_pad_is_streaming(pad))
> +			return true;
> +	}
> +
> +	return false;
>  }
>  
>  /**
> @@ -903,6 +947,19 @@ static inline bool media_entity_is_streaming(const struct media_entity *entity)
>   */
>  struct media_pipeline *media_entity_pipeline(struct media_entity *entity);
>  
> +/**
> + * media_pad_pipeline - Get the media pipeline a pad is part of
> + * @pad: The pad
> + *
> + * This function returns the media pipeline that a pad has been associated
> + * with when constructing the pipeline with media_pipeline_start(). The pointer
> + * remains valid until media_pipeline_stop() is called.
> + *
> + * Return: The media_pipeline the pad is part of, or NULL if the pad is
> + * not part of any pipeline.
> + */
> +struct media_pipeline *media_pad_pipeline(struct media_pad *pad);
> +
>  /**
>   * media_entity_get_fwnode_pad - Get pad number from fwnode
>   *

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 16/34] media: mc: convert pipeline funcs to take media_pad

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Wed, Aug 10, 2022 at 03:11:04PM +0300, Tomi Valkeinen wrote:
> Now that the pipeline is stored into pads instead of entities, we can
> change the relevant functions to take pads instead of entities.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c                  | 40 ++++++++-----------
>  .../samsung/s3c-camif/camif-capture.c         |  6 +--
>  drivers/media/usb/au0828/au0828-core.c        |  8 ++--
>  drivers/media/v4l2-core/v4l2-dev.c            | 12 +++---
>  drivers/staging/media/imx/imx-media-utils.c   |  6 +--
>  include/media/media-entity.h                  | 28 ++++++-------
>  6 files changed, 46 insertions(+), 54 deletions(-)
> 
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index c89dc782840a..ce72ee749fe6 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -699,29 +699,21 @@ static int media_pipeline_populate(struct media_pipeline *pipe,
>  	return ret;
>  }
>  
> -__must_check int __media_pipeline_start(struct media_entity *entity,
> +__must_check int __media_pipeline_start(struct media_pad *pad,
>  					struct media_pipeline *pipe)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>  	struct media_pipeline_pad *err_ppad;
>  	struct media_pipeline_pad *ppad;
>  	int ret;
>  
>  	lockdep_assert_held(&mdev->graph_mutex);
>  
> -	/*
> -	 * media_pipeline_start(entity) only makes sense with entities that have
> -	 * a single pad.
> -	 */
> -
> -	if (WARN_ON(entity->num_pads != 1))
> -		return -EINVAL;
> -
>  	/*
>  	 * If the entity is already part of a pipeline, that pipeline must
>  	 * be the same as the pipe given to media_pipeline_start().
>  	 */
> -	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
> +	if (WARN_ON(pad->pipe && pad->pipe != pipe))
>  		return -EINVAL;
>  
>  	/*
> @@ -738,7 +730,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	 * with media_pipeline_pad instances for each pad found during graph
>  	 * walk.
>  	 */
> -	ret = media_pipeline_populate(pipe, entity->pads);
> +	ret = media_pipeline_populate(pipe, pad);
>  	if (ret)
>  		return ret;
>  
> @@ -855,22 +847,22 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_start);
>  
> -__must_check int media_pipeline_start(struct media_entity *entity,
> +__must_check int media_pipeline_start(struct media_pad *pad,
>  				      struct media_pipeline *pipe)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>  	int ret;
>  
>  	mutex_lock(&mdev->graph_mutex);
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(pad, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_start);
>  
> -void __media_pipeline_stop(struct media_entity *entity)
> +void __media_pipeline_stop(struct media_pad *pad)
>  {
> -	struct media_pipeline *pipe = entity->pads->pipe;
> +	struct media_pipeline *pipe = pad->pipe;
>  	struct media_pipeline_pad *ppad;
>  
>  	/*
> @@ -893,19 +885,19 @@ void __media_pipeline_stop(struct media_entity *entity)
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_stop);
>  
> -void media_pipeline_stop(struct media_entity *entity)
> +void media_pipeline_stop(struct media_pad *pad)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>  
>  	mutex_lock(&mdev->graph_mutex);
> -	__media_pipeline_stop(entity);
> +	__media_pipeline_stop(pad);
>  	mutex_unlock(&mdev->graph_mutex);
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_stop);
>  
> -__must_check int media_pipeline_alloc_start(struct media_entity *entity)
> +__must_check int media_pipeline_alloc_start(struct media_pad *pad)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>  	struct media_pipeline *pipe;
>  	int ret;
>  	bool new_pipe = false;
> @@ -916,7 +908,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
>  	 * Is the entity already part of a pipeline? If not, we need to allocate
>  	 * a pipe.
>  	 */
> -	pipe = media_entity_pipeline(entity);
> +	pipe = media_pad_pipeline(pad);
>  	if (!pipe) {
>  		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
>  
> @@ -929,7 +921,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
>  		pipe->allocated = true;
>  	}
>  
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(pad, pipe);
>  	if (ret) {
>  		if (new_pipe)
>  			kfree(pipe);
> diff --git a/drivers/media/platform/samsung/s3c-camif/camif-capture.c b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
> index 140854ab4dd8..0189b8a33032 100644
> --- a/drivers/media/platform/samsung/s3c-camif/camif-capture.c
> +++ b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
> @@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
>  	if (s3c_vp_active(vp))
>  		return 0;
>  
> -	ret = media_pipeline_start(sensor, camif->m_pipeline);
> +	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
>  	if (ret < 0)
>  		return ret;
>  
>  	ret = camif_pipeline_validate(camif);
>  	if (ret < 0) {
> -		media_pipeline_stop(sensor);
> +		media_pipeline_stop(sensor->pads);
>  		return ret;
>  	}
>  
> @@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
>  
>  	ret = vb2_streamoff(&vp->vb_queue, type);
>  	if (ret == 0)
> -		media_pipeline_stop(&camif->sensor.sd->entity);
> +		media_pipeline_stop(camif->sensor.sd->entity.pads);
>  	return ret;
>  }
>  
> diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
> index caefac07af92..877e85a451cb 100644
> --- a/drivers/media/usb/au0828/au0828-core.c
> +++ b/drivers/media/usb/au0828/au0828-core.c
> @@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
>  		goto end;
>  	}
>  
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(entity->pads, pipe);
>  	if (ret) {
>  		pr_err("Start Pipeline: %s->%s Error %d\n",
>  			source->name, entity->name, ret);
> @@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
>  				return;
>  
>  			/* stop pipeline */
> -			__media_pipeline_stop(dev->active_link_owner);
> +			__media_pipeline_stop(dev->active_link_owner->pads);
>  			pr_debug("Pipeline stop for %s\n",
>  				dev->active_link_owner->name);
>  
>  			ret = __media_pipeline_start(
> -					dev->active_link_user,
> +					dev->active_link_user->pads,
>  					dev->active_link_user_pipe);
>  			if (ret) {
>  				pr_err("Start Pipeline: %s->%s %d\n",
> @@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
>  			return;
>  
>  		/* stop pipeline */
> -		__media_pipeline_stop(dev->active_link_owner);
> +		__media_pipeline_stop(dev->active_link_owner->pads);
>  		pr_debug("Pipeline stop for %s\n",
>  			dev->active_link_owner->name);
>  
> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
> index 945bb867a4c1..397d553177fa 100644
> --- a/drivers/media/v4l2-core/v4l2-dev.c
> +++ b/drivers/media/v4l2-core/v4l2-dev.c
> @@ -1105,7 +1105,7 @@ __must_check int video_device_pipeline_start(struct video_device *vdev,
>  	if (entity->num_pads != 1)
>  		return -ENODEV;
>  
> -	return media_pipeline_start(entity, pipe);
> +	return media_pipeline_start(&entity->pads[0], pipe);
>  }
>  EXPORT_SYMBOL_GPL(video_device_pipeline_start);
>  
> @@ -1117,7 +1117,7 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
>  	if (entity->num_pads != 1)
>  		return -ENODEV;
>  
> -	return __media_pipeline_start(entity, pipe);
> +	return __media_pipeline_start(&entity->pads[0], pipe);
>  }
>  EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
>  
> @@ -1128,7 +1128,7 @@ void video_device_pipeline_stop(struct video_device *vdev)
>  	if (WARN_ON(entity->num_pads != 1))
>  		return;
>  
> -	return media_pipeline_stop(entity);
> +	return media_pipeline_stop(&entity->pads[0]);
>  }
>  EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
>  
> @@ -1139,7 +1139,7 @@ void __video_device_pipeline_stop(struct video_device *vdev)
>  	if (WARN_ON(entity->num_pads != 1))
>  		return;
>  
> -	return __media_pipeline_stop(entity);
> +	return __media_pipeline_stop(&entity->pads[0]);
>  }
>  EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
>  
> @@ -1150,7 +1150,7 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
>  	if (entity->num_pads != 1)
>  		return -ENODEV;
>  
> -	return media_pipeline_alloc_start(entity);
> +	return media_pipeline_alloc_start(&entity->pads[0]);
>  }
>  EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
>  
> @@ -1161,7 +1161,7 @@ struct media_pipeline *video_device_pipeline(struct video_device *vdev)
>  	if (WARN_ON(entity->num_pads != 1))
>  		return NULL;
>  
> -	return media_entity_pipeline(entity);
> +	return media_pad_pipeline(&entity->pads[0]);
>  }
>  EXPORT_SYMBOL_GPL(video_device_pipeline);
>  
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 5d9c6fc6f2e0..3d785b4e8fe7 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -863,16 +863,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  	mutex_lock(&imxmd->md.graph_mutex);
>  
>  	if (on) {
> -		ret = __media_pipeline_start(entity, &imxmd->pipe);
> +		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
>  		if (ret)
>  			goto out;
>  		ret = v4l2_subdev_call(sd, video, s_stream, 1);
>  		if (ret)
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
>  		if (media_entity_pipeline(entity))
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	}
>  
>  out:
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index beea3f6f12b8..f738d0d81194 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -1048,10 +1048,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
>  
>  /**
>   * media_pipeline_start - Mark a pipeline as streaming
> - * @entity: Starting entity
> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + * @pad: Starting pad
> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>   *
> - * Mark all entities connected to a given entity through enabled links, either
> + * Mark all pads connected to a given pad through enabled links, either
>   * directly or indirectly, as streaming. The given pipeline object is assigned
>   * to every entity in the pipeline and stored in the media_entity pipe field.

s/entity/pad/

>   *
> @@ -1060,24 +1060,24 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
>   * pipeline pointer must be identical for all nested calls to
>   * media_pipeline_start().
>   */
> -__must_check int media_pipeline_start(struct media_entity *entity,
> +__must_check int media_pipeline_start(struct media_pad *pad,
>  				      struct media_pipeline *pipe);
>  /**
>   * __media_pipeline_start - Mark a pipeline as streaming
>   *
> - * @entity: Starting entity
> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + * @pad: Starting pad
> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>   *
>   * ..note:: This is the non-locking version of media_pipeline_start()
>   */
> -__must_check int __media_pipeline_start(struct media_entity *entity,
> +__must_check int __media_pipeline_start(struct media_pad *pad,
>  					struct media_pipeline *pipe);
>  
>  /**
>   * media_pipeline_stop - Mark a pipeline as not streaming
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
> - * Mark all entities connected to a given entity through enabled links, either
> + * Mark all pads connected to a given pads through enabled links, either
>   * directly or indirectly, as not streaming. The media_entity pipe field is

This needs to be updated too. Please check the other functions to verify
there's no leftover.

Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

>   * reset to %NULL.
>   *
> @@ -1085,20 +1085,20 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>   * number of calls to this function are required to mark the pipeline as not
>   * streaming.
>   */
> -void media_pipeline_stop(struct media_entity *entity);
> +void media_pipeline_stop(struct media_pad *pad);
>  
>  /**
>   * __media_pipeline_stop - Mark a pipeline as not streaming
>   *
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
>   * .. note:: This is the non-locking version of media_pipeline_stop()
>   */
> -void __media_pipeline_stop(struct media_entity *entity);
> +void __media_pipeline_stop(struct media_pad *pad);
>  
>  /**
>   * media_pipeline_alloc_start - Mark a pipeline as streaming
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
>   * media_pipeline_alloc_start() is similar to media_pipeline_start() but instead
>   * of working on a given pipeline the function will use an existing pipeline if
> @@ -1107,7 +1107,7 @@ void __media_pipeline_stop(struct media_entity *entity);
>   * Calls to media_pipeline_alloc_start() must be matched with
>   * media_pipeline_stop().
>   */
> -__must_check int media_pipeline_alloc_start(struct media_entity *entity);
> +__must_check int media_pipeline_alloc_start(struct media_pad *pad);
>  
>  /**
>   * media_devnode_create() - creates and initializes a device node interface

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 09/34] media: v4l2-dev: Add videodev wrappers for media pipelines

[Tomi Valkeinen]

On 29/08/2022 19:34, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:10:57PM +0300, Tomi Valkeinen wrote:
>> With the upcoming stream related improvements to the pipelines, the
>> pipelines are moved from media entities to media pads. As the drivers
>> currently use the pipelines with the entity based model, moving the
>> pipelines to pads will cause changes to the drivers.
>>
>> However, most of the uses of media pipelines are related to a video
>> device (a DMA engine) with a single pad, and thus there's never a need
>> to support multiple pads in these use cases. We can avoid pushing the
>> complexities of the pad based model to the drivers by adding video
>> device wrappers for the pipeline related functions.
>>
>> This patch adds a number of wrappers to media_pipeline functions, all of
>> which take a video_device as a parameter (instead of a media_entity),
>> and verify that there's just one pad.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-dev.c | 61 ++++++++++++++++++++
>>   include/media/v4l2-dev.h           | 90 ++++++++++++++++++++++++++++++
>>   2 files changed, 151 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index d00237ee4cae..7f933ff89fd4 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -1095,6 +1095,67 @@ void video_unregister_device(struct video_device *vdev)
>>   }
>>   EXPORT_SYMBOL(video_unregister_device);
>>   
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +__must_check int video_device_pipeline_start(struct video_device *vdev,
>> +					     struct media_pipeline *pipe)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +
>> +	if (entity->num_pads != 1)
>> +		return -ENODEV;
>> +
>> +	return media_pipeline_start(entity, pipe);
>> +}
>> +EXPORT_SYMBOL_GPL(video_device_pipeline_start);
>> +
>> +__must_check int __video_device_pipeline_start(struct video_device *vdev,
>> +					       struct media_pipeline *pipe)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +
>> +	if (entity->num_pads != 1)
>> +		return -ENODEV;
>> +
>> +	return __media_pipeline_start(entity, pipe);
>> +}
>> +EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
>> +
>> +void video_device_pipeline_stop(struct video_device *vdev)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +
>> +	if (WARN_ON(entity->num_pads != 1))
>> +		return;
>> +
>> +	return media_pipeline_stop(entity);
>> +}
>> +EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
>> +
>> +void __video_device_pipeline_stop(struct video_device *vdev)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +
>> +	if (WARN_ON(entity->num_pads != 1))
>> +		return;
>> +
>> +	return __media_pipeline_stop(entity);
>> +}
>> +EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
>> +
>> +struct media_pipeline *video_device_pipeline(struct video_device *vdev)
>> +{
>> +	struct media_entity *entity = &vdev->entity;
>> +
>> +	if (WARN_ON(entity->num_pads != 1))
>> +		return NULL;
>> +
>> +	return media_entity_pipeline(entity);
>> +}
>> +EXPORT_SYMBOL_GPL(video_device_pipeline);
>> +
>> +#endif /* CONFIG_MEDIA_CONTROLLER */
>> +
>>   /*
>>    *	Initialise video for linux
>>    */
>> diff --git a/include/media/v4l2-dev.h b/include/media/v4l2-dev.h
>> index 5cf1edefb822..4ccc24f5918a 100644
>> --- a/include/media/v4l2-dev.h
>> +++ b/include/media/v4l2-dev.h
>> @@ -539,4 +539,94 @@ static inline int video_is_registered(struct video_device *vdev)
>>   	return test_bit(V4L2_FL_REGISTERED, &vdev->flags);
>>   }
>>   
>> +#if defined(CONFIG_MEDIA_CONTROLLER)
>> +
>> +/**
>> + * video_device_pipeline_start - Mark a pipeline as streaming
>> + * @vdev: Starting video device
>> + * @pipe: Media pipeline to be assigned to all entities in the pipeline.
>> + *
>> + * Mark all entities connected to a given video device through enabled links,
>> + * either directly or indirectly, as streaming. The given pipeline object is
>> + * assigned to every entity in the pipeline and stored in the media_entity pipe
>> + * field.
>> + *
>> + * Calls to this function can be nested, in which case the same number of
>> + * video_device_pipeline_stop() calls will be required to stop streaming. The
>> + * pipeline pointer must be identical for all nested calls to
>> + * video_device_pipeline_start().
>> + *
>> + * The video device must contain a single pad.
>> + *
>> + * This is a convenience wrapper to media_pipeline_start().
> 
> s/wrapper to/wrapper around/ maybe. Same below.

Sounds good to me.

>> + */
>> +__must_check int video_device_pipeline_start(struct video_device *vdev,
>> +					     struct media_pipeline *pipe);
>> +
>> +/**
>> + * __video_device_pipeline_start - Mark a pipeline as streaming
>> + *
> 
> I'd drop the blank line as you don't have one for the previous function.
> Same for stop.

Yep.

> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

Re: [PATCH v13 12/34] media: mc: entity: add alloc variant of pipeline_start

[Tomi Valkeinen]

On 29/08/2022 19:57, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:11:00PM +0300, Tomi Valkeinen wrote:
>> Add new variant of media_pipeline_start(), media_pipeline_alloc_start().
>>
>> media_pipeline_alloc_start() can be used by drivers that do not need to
>> extend the media_pipeline. The function will either use the pipeline
>> already associated with the entity, if such exists, or allocate a new
>> pipeline.
>>
>> When media_pipeline_stop() is called and the pipeline's use count drops
>> to zero, the pipeline is automatically freed.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c       | 41 ++++++++++++++++++++++++++++++
>>   drivers/media/v4l2-core/v4l2-dev.c | 11 ++++++++
>>   include/media/media-entity.h       | 14 ++++++++++
>>   include/media/v4l2-dev.h           | 14 ++++++++++
>>   4 files changed, 80 insertions(+)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index b1abaa333d13..d277eed11caf 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -529,6 +529,8 @@ void __media_pipeline_stop(struct media_entity *entity)
>>   
>>   	media_graph_walk_cleanup(graph);
>>   
>> +	if (pipe->allocated)
>> +		kfree(pipe);
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_stop);
>>   
>> @@ -542,6 +544,45 @@ void media_pipeline_stop(struct media_entity *entity)
>>   }
>>   EXPORT_SYMBOL_GPL(media_pipeline_stop);
>>   
>> +__must_check int media_pipeline_alloc_start(struct media_entity *entity)
>> +{
>> +	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_pipeline *pipe;
>> +	int ret;
>> +	bool new_pipe = false;
>> +
>> +	mutex_lock(&mdev->graph_mutex);
>> +
>> +	/*
>> +	 * Is the entity already part of a pipeline? If not, we need to allocate
>> +	 * a pipe.
>> +	 */
>> +	pipe = media_entity_pipeline(entity);
>> +	if (!pipe) {
>> +		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
>> +
> 
> You can drop the blank line.

Ok.

>> +		if (!pipe) {
>> +			ret = -ENOMEM;
>> +			goto out;
>> +		}
>> +
>> +		new_pipe = true;
>> +		pipe->allocated = true;
>> +	}
>> +
>> +	ret = __media_pipeline_start(entity, pipe);
>> +	if (ret) {
>> +		if (new_pipe)
>> +			kfree(pipe);
> 
> If new_pipe was a media_pipeline pointer, initialized to NULL and set to
> pipe when you allocate the pipe, you could write this
> 
> 	if (ret)
> 		kfree(new_pipe);

Yes, that's slightly neater.

> I don't mind much either way.
> 
> I have no objection against this patch, but I don't think it aligns well
> with what I was imagining as further evolutions of the API. I would like
> the pipe to be turned into an object similar to a DRM atomic commit.
> There will thus never be a need for drivers to allocate the pipeline, it
> will be done by the framework. We can rework (and likely drop) this
> function when that happens.

True, but isn't this a bit in the right direction? This goes a bit 
towards automating the pipeline management. Having the drivers use this 
instead of doing the management themselves should make it easier to 
eventually move to a framework managed model.

> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Thanks!

  Tomi

Re: [PATCH v13 13/34] media: drivers: use video_device_pipeline_alloc_start()

[Tomi Valkeinen]

On 29/08/2022 20:00, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:11:01PM +0300, Tomi Valkeinen wrote:
>> Use video_device_pipeline_alloc_start() instead of manually
>> allocating/managing the media pipeline storage.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/platform/renesas/rcar-vin/rcar-dma.c | 14 +-------------
>>   drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c |  2 +-
>>   .../media/platform/sunxi/sun6i-csi/sun6i_video.c   |  2 +-
>>   drivers/media/platform/ti/cal/cal-video.c          |  2 +-
>>   drivers/media/platform/ti/cal/cal.h                |  1 -
>>   5 files changed, 4 insertions(+), 17 deletions(-)
>>
>> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
>> index 548067f19576..884875600231 100644
>> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
>> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
>> @@ -1244,8 +1244,6 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd,
>>   
>>   static int rvin_set_stream(struct rvin_dev *vin, int on)
>>   {
>> -	struct media_pipeline *pipe;
>> -	struct media_device *mdev;
>>   	struct v4l2_subdev *sd;
>>   	struct media_pad *pad;
>>   	int ret;
>> @@ -1273,17 +1271,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>>   	if (ret)
>>   		return ret;
>>   
>> -	/*
>> -	 * The graph lock needs to be taken to protect concurrent
>> -	 * starts of multiple VIN instances as they might share
>> -	 * a common subdevice down the line and then should use
>> -	 * the same pipe.
>> -	 */
>> -	mdev = vin->vdev.entity.graph_obj.mdev;
>> -	mutex_lock(&mdev->graph_mutex);
>> -	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
>> -	ret = __video_device_pipeline_start(&vin->vdev, pipe);
>> -	mutex_unlock(&mdev->graph_mutex);
>> +	ret = video_device_pipeline_alloc_start(&vin->vdev);
> 
> This doesn't look right, it will use different pipeline instances for
> difference video devices, that's a change in behaviour.

I hope not, but it's a bit difficult to be sure without having the board 
and testing.

Afaics, the previous code uses the existing pipeline if such exists in 
the first subdev (i.e. if media_pipeline_start has been called earlier 
for another vdev), or if not, uses pipeline specific to the vdev.

This behavior should match the new one, although the operation 
underneath is slightly different.

It's perhaps the name, video_device_pipeline_alloc_start, that confuses. 
It doesn't indicate that it possibly uses an existing pipeline. But I 
don't have any good idea for a better name. 
video_device_pipeline_find_or_alloc_start? =) Well, it's not really 
"finding" anything either...

>>   	if (ret)
>>   		return ret;
>>   
>> diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
>> index 17ad9a3caaa5..a3e826a755fc 100644
>> --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
>> +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
>> @@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>>   		goto err_clear_dma_queue;
>>   	}
>>   
>> -	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
> 
> What ? There is a pipe embedded in video_device ? Oh my, I didn't
> realize how messed up the V4L2 core had become :-(

That's what the current Renesas driver earlier in this patch also uses. 
Didn't you write that code? =)

  Tomi

Re: [PATCH v13 12/34] media: mc: entity: add alloc variant of pipeline_start

[Laurent Pinchart]

Hi Tomi,

On Tue, Aug 30, 2022 at 02:08:53PM +0300, Tomi Valkeinen wrote:
> On 29/08/2022 19:57, Laurent Pinchart wrote:
> > On Wed, Aug 10, 2022 at 03:11:00PM +0300, Tomi Valkeinen wrote:
> >> Add new variant of media_pipeline_start(), media_pipeline_alloc_start().
> >>
> >> media_pipeline_alloc_start() can be used by drivers that do not need to
> >> extend the media_pipeline. The function will either use the pipeline
> >> already associated with the entity, if such exists, or allocate a new
> >> pipeline.
> >>
> >> When media_pipeline_stop() is called and the pipeline's use count drops
> >> to zero, the pipeline is automatically freed.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/mc/mc-entity.c       | 41 ++++++++++++++++++++++++++++++
> >>   drivers/media/v4l2-core/v4l2-dev.c | 11 ++++++++
> >>   include/media/media-entity.h       | 14 ++++++++++
> >>   include/media/v4l2-dev.h           | 14 ++++++++++
> >>   4 files changed, 80 insertions(+)
> >>
> >> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> >> index b1abaa333d13..d277eed11caf 100644
> >> --- a/drivers/media/mc/mc-entity.c
> >> +++ b/drivers/media/mc/mc-entity.c
> >> @@ -529,6 +529,8 @@ void __media_pipeline_stop(struct media_entity *entity)
> >>   
> >>   	media_graph_walk_cleanup(graph);
> >>   
> >> +	if (pipe->allocated)
> >> +		kfree(pipe);
> >>   }
> >>   EXPORT_SYMBOL_GPL(__media_pipeline_stop);
> >>   
> >> @@ -542,6 +544,45 @@ void media_pipeline_stop(struct media_entity *entity)
> >>   }
> >>   EXPORT_SYMBOL_GPL(media_pipeline_stop);
> >>   
> >> +__must_check int media_pipeline_alloc_start(struct media_entity *entity)
> >> +{
> >> +	struct media_device *mdev = entity->graph_obj.mdev;
> >> +	struct media_pipeline *pipe;
> >> +	int ret;
> >> +	bool new_pipe = false;
> >> +
> >> +	mutex_lock(&mdev->graph_mutex);
> >> +
> >> +	/*
> >> +	 * Is the entity already part of a pipeline? If not, we need to allocate
> >> +	 * a pipe.
> >> +	 */
> >> +	pipe = media_entity_pipeline(entity);
> >> +	if (!pipe) {
> >> +		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
> >> +
> > 
> > You can drop the blank line.
> 
> Ok.
> 
> >> +		if (!pipe) {
> >> +			ret = -ENOMEM;
> >> +			goto out;
> >> +		}
> >> +
> >> +		new_pipe = true;
> >> +		pipe->allocated = true;
> >> +	}
> >> +
> >> +	ret = __media_pipeline_start(entity, pipe);
> >> +	if (ret) {
> >> +		if (new_pipe)
> >> +			kfree(pipe);
> > 
> > If new_pipe was a media_pipeline pointer, initialized to NULL and set to
> > pipe when you allocate the pipe, you could write this
> > 
> > 	if (ret)
> > 		kfree(new_pipe);
> 
> Yes, that's slightly neater.
> 
> > I don't mind much either way.
> > 
> > I have no objection against this patch, but I don't think it aligns well
> > with what I was imagining as further evolutions of the API. I would like
> > the pipe to be turned into an object similar to a DRM atomic commit.
> > There will thus never be a need for drivers to allocate the pipeline, it
> > will be done by the framework. We can rework (and likely drop) this
> > function when that happens.
> 
> True, but isn't this a bit in the right direction? This goes a bit 
> towards automating the pipeline management. Having the drivers use this 
> instead of doing the management themselves should make it easier to 
> eventually move to a framework managed model.

It's hard for me at this point to tell how much this would help, I don't
have enough visibility. I think it doesn't go in the wrong direction, so
it's OK :-)

> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

On 29/08/2022 20:18, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> [Note: the code is mostly from Laurent but the patch description is from Tomi]
> 
> I'll review the description then :-)
> 
>> The media_pipeline_start() and media_pipeline_stop() functions use the
>> media graph walk API to traverse the graph and validate the pipeline.
>> The graph walk traverses the media graph following links between the
>> entities.
>>
>> Also, while the pipeline can't change between the start and stop calls,
>> the graph is walked again from scratch at stop time, or any time a
>> driver needs to inspect the pipeline.
>>
>> With the upcoming multiplexed streams support we will need a bit more
>> intelligent pipeline construction, as e.g. an entity may be passing
>> through two independent streams via separate pads, in which case those
> 
> Did you mean "as e.g. two independent streams may be passing through a
> single entity via separate pads" ?

Yes. That's a bit more understandable.

>> pads should not be part of the same pipeline.
>>
>> This patch essentially rewrites the media_pipeline_start/stop so that
>> a pipeline is defined as a set of pads instead of entities and the media
>> graph traversal considers the pad interdependencies when choosing which
>> links to follow.
>>
>> Currently all the entity's pads are considered as interdependent. This
>> means that the behavior with all the current drivers stays the same, but
>> in the future we can define a more fine-grained pipeline construction.
>>
>> Additionally the media pipeline's pads are cached at
>> media_pipeline_start() time, and re-used at media_pipeline_stop() which
>> avoid the need to re-walk the whole graph as the previous implementation
>> did.
>>
>> Also, caching pads in the pipeline can serve in the future as the
>> foundation to provide a better API than the media graph walk to drivers
>> to iterate over pads and entities in the pipeline.
>>
>> Note that the old media_pipeline_start/stop used the media graph walk
>> API. The new version does not use the media graph walk API, but instead
>> a new implementation.
>>
>> There are two reason for not changing the graph walk: it proved to be
>> rather difficult to change the graph walk to have the features
>> implemented in this patch, and second, this keeps the backward
>> compatibility of the graph walk as there are users of the graph walk API
>>
>> The long term plan is that all the existing code would be converted to
>> use the new cached pipeline, thus allowing us to remove the graph walk.
> 
> Could you mark the graph walk API as deprecated in this patch, or in a
> subsequent patch in the series ? I think I did in a previous version,
> but I may recall incorrectly.

I don't remember removing deprecation comments, but who knows...

>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index d277eed11caf..df8e82be2bcc 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -225,6 +225,27 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
>>    * Graph traversal
>>    */
>>   
>> +/*
>> + * This function checks the interdependency inside the entity between @pad0
>> + * and @pad1. If two pads are interdependent they are part of the same pipeline
>> + * and enabling one of the pads means that the other pad will become "locked"
>> + * and doesn't allow configuration changes.
>> + *
>> + * For the time being all pads are considered interdependent.
>> + */
>> +static bool media_entity_has_pad_interdep(struct media_entity *entity,
> 
> I won't bikeshed this name as it can easily be changed later if needed.

Thank you ;).

>> @@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>   
>>   struct media_pipeline *media_entity_pipeline(struct media_entity *entity)
>>   {
>> -	return entity->pipe;
>> +	struct media_pipeline *pipe;
>> +	struct media_pad *pad;
>> +
>> +	if (entity->num_pads == 0)
>> +		return NULL;
>> +
>> +	pipe = entity->pads->pipe;
>> +
>> +	media_entity_for_each_pad(entity, pad) {
>> +		if (WARN_ON(pad->pipe != pipe))
>> +			return NULL;
>> +	}
> 
> This wasn't done in v12, could you explain the rationale ? The
> documentation of media_entity_pipeline() should probably be updated too.

This is just a sanity check.

The doc says "This function is to be used only for entities that do not 
support multiple pipelines". So the pipeline must be the same for all 
pads, and if that's not the case, something is horribly wrong.

However, now that I think about it, can we have NULL pipes in some pads, 
if those pads are e.g. not linked... Then I think the check should be 
(pipe && pad->pipe && pipe != pad->pipe).

But if that's the case, does this even work... The code expects the 
first pad to contain the pipeline for the entity. If the first pad's 
pipe can be NULL, but, say, second and fourth pads contain the pipeline 
(as those pads are actually used), then this is quite broken.

  Tomi

Re: [PATCH v13 13/34] media: drivers: use video_device_pipeline_alloc_start()

[Laurent Pinchart]

Hi Tomi,

On Tue, Aug 30, 2022 at 02:20:21PM +0300, Tomi Valkeinen wrote:
> On 29/08/2022 20:00, Laurent Pinchart wrote:
> > On Wed, Aug 10, 2022 at 03:11:01PM +0300, Tomi Valkeinen wrote:
> >> Use video_device_pipeline_alloc_start() instead of manually
> >> allocating/managing the media pipeline storage.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/platform/renesas/rcar-vin/rcar-dma.c | 14 +-------------
> >>   drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c |  2 +-
> >>   .../media/platform/sunxi/sun6i-csi/sun6i_video.c   |  2 +-
> >>   drivers/media/platform/ti/cal/cal-video.c          |  2 +-
> >>   drivers/media/platform/ti/cal/cal.h                |  1 -
> >>   5 files changed, 4 insertions(+), 17 deletions(-)
> >>
> >> diff --git a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> >> index 548067f19576..884875600231 100644
> >> --- a/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> >> +++ b/drivers/media/platform/renesas/rcar-vin/rcar-dma.c
> >> @@ -1244,8 +1244,6 @@ static int rvin_mc_validate_format(struct rvin_dev *vin, struct v4l2_subdev *sd,
> >>   
> >>   static int rvin_set_stream(struct rvin_dev *vin, int on)
> >>   {
> >> -	struct media_pipeline *pipe;
> >> -	struct media_device *mdev;
> >>   	struct v4l2_subdev *sd;
> >>   	struct media_pad *pad;
> >>   	int ret;
> >> @@ -1273,17 +1271,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
> >>   	if (ret)
> >>   		return ret;
> >>   
> >> -	/*
> >> -	 * The graph lock needs to be taken to protect concurrent
> >> -	 * starts of multiple VIN instances as they might share
> >> -	 * a common subdevice down the line and then should use
> >> -	 * the same pipe.
> >> -	 */
> >> -	mdev = vin->vdev.entity.graph_obj.mdev;
> >> -	mutex_lock(&mdev->graph_mutex);
> >> -	pipe = media_entity_pipeline(&sd->entity) ? : &vin->vdev.pipe;
> >> -	ret = __video_device_pipeline_start(&vin->vdev, pipe);
> >> -	mutex_unlock(&mdev->graph_mutex);
> >> +	ret = video_device_pipeline_alloc_start(&vin->vdev);
> > 
> > This doesn't look right, it will use different pipeline instances for
> > difference video devices, that's a change in behaviour.
> 
> I hope not, but it's a bit difficult to be sure without having the board 
> and testing.
> 
> Afaics, the previous code uses the existing pipeline if such exists in 
> the first subdev (i.e. if media_pipeline_start has been called earlier 
> for another vdev), or if not, uses pipeline specific to the vdev.
> 
> This behavior should match the new one, although the operation 
> underneath is slightly different.

I think you're right. I'd probably feel better if we could test it :-)

> It's perhaps the name, video_device_pipeline_alloc_start, that confuses. 
> It doesn't indicate that it possibly uses an existing pipeline. But I 
> don't have any good idea for a better name. 
> video_device_pipeline_find_or_alloc_start? =) Well, it's not really 
> "finding" anything either...

Yes, I think the new threw me off, but I can't really think of a better
one right now.

> >>   	if (ret)
> >>   		return ret;
> >>   
> >> diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> >> index 17ad9a3caaa5..a3e826a755fc 100644
> >> --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> >> +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> >> @@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
> >>   		goto err_clear_dma_queue;
> >>   	}
> >>   
> >> -	ret = video_device_pipeline_start(&csi->vdev, &csi->vdev.pipe);
> > 
> > What ? There is a pipe embedded in video_device ? Oh my, I didn't
> > realize how messed up the V4L2 core had become :-(
> 
> That's what the current Renesas driver earlier in this patch also uses. 
> Didn't you write that code? =)

As a matter of fact, no, I didn't :-)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 16/34] media: mc: convert pipeline funcs to take media_pad

[Tomi Valkeinen]

On 29/08/2022 20:21, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:11:04PM +0300, Tomi Valkeinen wrote:
>> Now that the pipeline is stored into pads instead of entities, we can
>> change the relevant functions to take pads instead of entities.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c                  | 40 ++++++++-----------
>>   .../samsung/s3c-camif/camif-capture.c         |  6 +--
>>   drivers/media/usb/au0828/au0828-core.c        |  8 ++--
>>   drivers/media/v4l2-core/v4l2-dev.c            | 12 +++---
>>   drivers/staging/media/imx/imx-media-utils.c   |  6 +--
>>   include/media/media-entity.h                  | 28 ++++++-------
>>   6 files changed, 46 insertions(+), 54 deletions(-)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index c89dc782840a..ce72ee749fe6 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -699,29 +699,21 @@ static int media_pipeline_populate(struct media_pipeline *pipe,
>>   	return ret;
>>   }
>>   
>> -__must_check int __media_pipeline_start(struct media_entity *entity,
>> +__must_check int __media_pipeline_start(struct media_pad *pad,
>>   					struct media_pipeline *pipe)
>>   {
>> -	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>>   	struct media_pipeline_pad *err_ppad;
>>   	struct media_pipeline_pad *ppad;
>>   	int ret;
>>   
>>   	lockdep_assert_held(&mdev->graph_mutex);
>>   
>> -	/*
>> -	 * media_pipeline_start(entity) only makes sense with entities that have
>> -	 * a single pad.
>> -	 */
>> -
>> -	if (WARN_ON(entity->num_pads != 1))
>> -		return -EINVAL;
>> -
>>   	/*
>>   	 * If the entity is already part of a pipeline, that pipeline must
>>   	 * be the same as the pipe given to media_pipeline_start().
>>   	 */
>> -	if (WARN_ON(entity->pads->pipe && entity->pads->pipe != pipe))
>> +	if (WARN_ON(pad->pipe && pad->pipe != pipe))
>>   		return -EINVAL;
>>   
>>   	/*
>> @@ -738,7 +730,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>>   	 * with media_pipeline_pad instances for each pad found during graph
>>   	 * walk.
>>   	 */
>> -	ret = media_pipeline_populate(pipe, entity->pads);
>> +	ret = media_pipeline_populate(pipe, pad);
>>   	if (ret)
>>   		return ret;
>>   
>> @@ -855,22 +847,22 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_start);
>>   
>> -__must_check int media_pipeline_start(struct media_entity *entity,
>> +__must_check int media_pipeline_start(struct media_pad *pad,
>>   				      struct media_pipeline *pipe)
>>   {
>> -	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>>   	int ret;
>>   
>>   	mutex_lock(&mdev->graph_mutex);
>> -	ret = __media_pipeline_start(entity, pipe);
>> +	ret = __media_pipeline_start(pad, pipe);
>>   	mutex_unlock(&mdev->graph_mutex);
>>   	return ret;
>>   }
>>   EXPORT_SYMBOL_GPL(media_pipeline_start);
>>   
>> -void __media_pipeline_stop(struct media_entity *entity)
>> +void __media_pipeline_stop(struct media_pad *pad)
>>   {
>> -	struct media_pipeline *pipe = entity->pads->pipe;
>> +	struct media_pipeline *pipe = pad->pipe;
>>   	struct media_pipeline_pad *ppad;
>>   
>>   	/*
>> @@ -893,19 +885,19 @@ void __media_pipeline_stop(struct media_entity *entity)
>>   }
>>   EXPORT_SYMBOL_GPL(__media_pipeline_stop);
>>   
>> -void media_pipeline_stop(struct media_entity *entity)
>> +void media_pipeline_stop(struct media_pad *pad)
>>   {
>> -	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>>   
>>   	mutex_lock(&mdev->graph_mutex);
>> -	__media_pipeline_stop(entity);
>> +	__media_pipeline_stop(pad);
>>   	mutex_unlock(&mdev->graph_mutex);
>>   }
>>   EXPORT_SYMBOL_GPL(media_pipeline_stop);
>>   
>> -__must_check int media_pipeline_alloc_start(struct media_entity *entity)
>> +__must_check int media_pipeline_alloc_start(struct media_pad *pad)
>>   {
>> -	struct media_device *mdev = entity->graph_obj.mdev;
>> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>>   	struct media_pipeline *pipe;
>>   	int ret;
>>   	bool new_pipe = false;
>> @@ -916,7 +908,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
>>   	 * Is the entity already part of a pipeline? If not, we need to allocate
>>   	 * a pipe.
>>   	 */
>> -	pipe = media_entity_pipeline(entity);
>> +	pipe = media_pad_pipeline(pad);
>>   	if (!pipe) {
>>   		pipe = kzalloc(sizeof(*pipe), GFP_KERNEL);
>>   
>> @@ -929,7 +921,7 @@ __must_check int media_pipeline_alloc_start(struct media_entity *entity)
>>   		pipe->allocated = true;
>>   	}
>>   
>> -	ret = __media_pipeline_start(entity, pipe);
>> +	ret = __media_pipeline_start(pad, pipe);
>>   	if (ret) {
>>   		if (new_pipe)
>>   			kfree(pipe);
>> diff --git a/drivers/media/platform/samsung/s3c-camif/camif-capture.c b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
>> index 140854ab4dd8..0189b8a33032 100644
>> --- a/drivers/media/platform/samsung/s3c-camif/camif-capture.c
>> +++ b/drivers/media/platform/samsung/s3c-camif/camif-capture.c
>> @@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
>>   	if (s3c_vp_active(vp))
>>   		return 0;
>>   
>> -	ret = media_pipeline_start(sensor, camif->m_pipeline);
>> +	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
>>   	if (ret < 0)
>>   		return ret;
>>   
>>   	ret = camif_pipeline_validate(camif);
>>   	if (ret < 0) {
>> -		media_pipeline_stop(sensor);
>> +		media_pipeline_stop(sensor->pads);
>>   		return ret;
>>   	}
>>   
>> @@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
>>   
>>   	ret = vb2_streamoff(&vp->vb_queue, type);
>>   	if (ret == 0)
>> -		media_pipeline_stop(&camif->sensor.sd->entity);
>> +		media_pipeline_stop(camif->sensor.sd->entity.pads);
>>   	return ret;
>>   }
>>   
>> diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
>> index caefac07af92..877e85a451cb 100644
>> --- a/drivers/media/usb/au0828/au0828-core.c
>> +++ b/drivers/media/usb/au0828/au0828-core.c
>> @@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
>>   		goto end;
>>   	}
>>   
>> -	ret = __media_pipeline_start(entity, pipe);
>> +	ret = __media_pipeline_start(entity->pads, pipe);
>>   	if (ret) {
>>   		pr_err("Start Pipeline: %s->%s Error %d\n",
>>   			source->name, entity->name, ret);
>> @@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
>>   				return;
>>   
>>   			/* stop pipeline */
>> -			__media_pipeline_stop(dev->active_link_owner);
>> +			__media_pipeline_stop(dev->active_link_owner->pads);
>>   			pr_debug("Pipeline stop for %s\n",
>>   				dev->active_link_owner->name);
>>   
>>   			ret = __media_pipeline_start(
>> -					dev->active_link_user,
>> +					dev->active_link_user->pads,
>>   					dev->active_link_user_pipe);
>>   			if (ret) {
>>   				pr_err("Start Pipeline: %s->%s %d\n",
>> @@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
>>   			return;
>>   
>>   		/* stop pipeline */
>> -		__media_pipeline_stop(dev->active_link_owner);
>> +		__media_pipeline_stop(dev->active_link_owner->pads);
>>   		pr_debug("Pipeline stop for %s\n",
>>   			dev->active_link_owner->name);
>>   
>> diff --git a/drivers/media/v4l2-core/v4l2-dev.c b/drivers/media/v4l2-core/v4l2-dev.c
>> index 945bb867a4c1..397d553177fa 100644
>> --- a/drivers/media/v4l2-core/v4l2-dev.c
>> +++ b/drivers/media/v4l2-core/v4l2-dev.c
>> @@ -1105,7 +1105,7 @@ __must_check int video_device_pipeline_start(struct video_device *vdev,
>>   	if (entity->num_pads != 1)
>>   		return -ENODEV;
>>   
>> -	return media_pipeline_start(entity, pipe);
>> +	return media_pipeline_start(&entity->pads[0], pipe);
>>   }
>>   EXPORT_SYMBOL_GPL(video_device_pipeline_start);
>>   
>> @@ -1117,7 +1117,7 @@ __must_check int __video_device_pipeline_start(struct video_device *vdev,
>>   	if (entity->num_pads != 1)
>>   		return -ENODEV;
>>   
>> -	return __media_pipeline_start(entity, pipe);
>> +	return __media_pipeline_start(&entity->pads[0], pipe);
>>   }
>>   EXPORT_SYMBOL_GPL(__video_device_pipeline_start);
>>   
>> @@ -1128,7 +1128,7 @@ void video_device_pipeline_stop(struct video_device *vdev)
>>   	if (WARN_ON(entity->num_pads != 1))
>>   		return;
>>   
>> -	return media_pipeline_stop(entity);
>> +	return media_pipeline_stop(&entity->pads[0]);
>>   }
>>   EXPORT_SYMBOL_GPL(video_device_pipeline_stop);
>>   
>> @@ -1139,7 +1139,7 @@ void __video_device_pipeline_stop(struct video_device *vdev)
>>   	if (WARN_ON(entity->num_pads != 1))
>>   		return;
>>   
>> -	return __media_pipeline_stop(entity);
>> +	return __media_pipeline_stop(&entity->pads[0]);
>>   }
>>   EXPORT_SYMBOL_GPL(__video_device_pipeline_stop);
>>   
>> @@ -1150,7 +1150,7 @@ __must_check int video_device_pipeline_alloc_start(struct video_device *vdev)
>>   	if (entity->num_pads != 1)
>>   		return -ENODEV;
>>   
>> -	return media_pipeline_alloc_start(entity);
>> +	return media_pipeline_alloc_start(&entity->pads[0]);
>>   }
>>   EXPORT_SYMBOL_GPL(video_device_pipeline_alloc_start);
>>   
>> @@ -1161,7 +1161,7 @@ struct media_pipeline *video_device_pipeline(struct video_device *vdev)
>>   	if (WARN_ON(entity->num_pads != 1))
>>   		return NULL;
>>   
>> -	return media_entity_pipeline(entity);
>> +	return media_pad_pipeline(&entity->pads[0]);
>>   }
>>   EXPORT_SYMBOL_GPL(video_device_pipeline);
>>   
>> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
>> index 5d9c6fc6f2e0..3d785b4e8fe7 100644
>> --- a/drivers/staging/media/imx/imx-media-utils.c
>> +++ b/drivers/staging/media/imx/imx-media-utils.c
>> @@ -863,16 +863,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>>   	mutex_lock(&imxmd->md.graph_mutex);
>>   
>>   	if (on) {
>> -		ret = __media_pipeline_start(entity, &imxmd->pipe);
>> +		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
>>   		if (ret)
>>   			goto out;
>>   		ret = v4l2_subdev_call(sd, video, s_stream, 1);
>>   		if (ret)
>> -			__media_pipeline_stop(entity);
>> +			__media_pipeline_stop(entity->pads);
>>   	} else {
>>   		v4l2_subdev_call(sd, video, s_stream, 0);
>>   		if (media_entity_pipeline(entity))
>> -			__media_pipeline_stop(entity);
>> +			__media_pipeline_stop(entity->pads);
>>   	}
>>   
>>   out:
>> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
>> index beea3f6f12b8..f738d0d81194 100644
>> --- a/include/media/media-entity.h
>> +++ b/include/media/media-entity.h
>> @@ -1048,10 +1048,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
>>   
>>   /**
>>    * media_pipeline_start - Mark a pipeline as streaming
>> - * @entity: Starting entity
>> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
>> + * @pad: Starting pad
>> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>>    *
>> - * Mark all entities connected to a given entity through enabled links, either
>> + * Mark all pads connected to a given pad through enabled links, either
>>    * directly or indirectly, as streaming. The given pipeline object is assigned
>>    * to every entity in the pipeline and stored in the media_entity pipe field.
> 
> s/entity/pad/
> 
>>    *
>> @@ -1060,24 +1060,24 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph);
>>    * pipeline pointer must be identical for all nested calls to
>>    * media_pipeline_start().
>>    */
>> -__must_check int media_pipeline_start(struct media_entity *entity,
>> +__must_check int media_pipeline_start(struct media_pad *pad,
>>   				      struct media_pipeline *pipe);
>>   /**
>>    * __media_pipeline_start - Mark a pipeline as streaming
>>    *
>> - * @entity: Starting entity
>> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
>> + * @pad: Starting pad
>> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>>    *
>>    * ..note:: This is the non-locking version of media_pipeline_start()
>>    */
>> -__must_check int __media_pipeline_start(struct media_entity *entity,
>> +__must_check int __media_pipeline_start(struct media_pad *pad,
>>   					struct media_pipeline *pipe);
>>   
>>   /**
>>    * media_pipeline_stop - Mark a pipeline as not streaming
>> - * @entity: Starting entity
>> + * @pad: Starting pad
>>    *
>> - * Mark all entities connected to a given entity through enabled links, either
>> + * Mark all pads connected to a given pads through enabled links, either
>>    * directly or indirectly, as not streaming. The media_entity pipe field is
> 
> This needs to be updated too. Please check the other functions to verify
> there's no leftover.

Thanks. I found a bit more, also in the vdev versions. I hope I caught 
them all...

  Tomi

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

On 30/08/2022 14:44, Tomi Valkeinen wrote:

>>> @@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>>   struct media_pipeline *media_entity_pipeline(struct media_entity 
>>> *entity)
>>>   {
>>> -    return entity->pipe;
>>> +    struct media_pipeline *pipe;
>>> +    struct media_pad *pad;
>>> +
>>> +    if (entity->num_pads == 0)
>>> +        return NULL;
>>> +
>>> +    pipe = entity->pads->pipe;
>>> +
>>> +    media_entity_for_each_pad(entity, pad) {
>>> +        if (WARN_ON(pad->pipe != pipe))
>>> +            return NULL;
>>> +    }
>>
>> This wasn't done in v12, could you explain the rationale ? The
>> documentation of media_entity_pipeline() should probably be updated too.
> 
> This is just a sanity check.
> 
> The doc says "This function is to be used only for entities that do not 
> support multiple pipelines". So the pipeline must be the same for all 
> pads, and if that's not the case, something is horribly wrong.
> 
> However, now that I think about it, can we have NULL pipes in some pads, 
> if those pads are e.g. not linked... Then I think the check should be 
> (pipe && pad->pipe && pipe != pad->pipe).
> 
> But if that's the case, does this even work... The code expects the 
> first pad to contain the pipeline for the entity. If the first pad's 
> pipe can be NULL, but, say, second and fourth pads contain the pipeline 
> (as those pads are actually used), then this is quite broken.

Looks like my concern has some grounds. While I don't have a perfect 
setup to test this, I think my test with UB960 shows that unlinked pads 
have NULL pipeline. So if we have an entity with two sink pads (0 and 1) 
and two source pads (2 and 3), and the pipeline goes through pads 1 and 
3, this function will always return NULL.

I'm not sure how to fix this. One option would be to make this function 
available only for entities with 1 or 2 (a sink and a source) pads. But 
I have no idea if that covers the current users.

Or should this function iterate through the pads and return the first 
non-null pipe?

Looks like there are only a few users for this. The one in 
imx-media-utils.c can be dropped easily. omap4iss and omap3isp uses are 
not clear to me yet.

  Tomi

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

On 29/08/2022 20:18, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> [Note: the code is mostly from Laurent but the patch description is from Tomi]
> 
> I'll review the description then :-)
> 
>> The media_pipeline_start() and media_pipeline_stop() functions use the
>> media graph walk API to traverse the graph and validate the pipeline.
>> The graph walk traverses the media graph following links between the
>> entities.
>>
>> Also, while the pipeline can't change between the start and stop calls,
>> the graph is walked again from scratch at stop time, or any time a
>> driver needs to inspect the pipeline.
>>
>> With the upcoming multiplexed streams support we will need a bit more
>> intelligent pipeline construction, as e.g. an entity may be passing
>> through two independent streams via separate pads, in which case those
> 
> Did you mean "as e.g. two independent streams may be passing through a
> single entity via separate pads" ?
> 
>> pads should not be part of the same pipeline.
>>
>> This patch essentially rewrites the media_pipeline_start/stop so that
>> a pipeline is defined as a set of pads instead of entities and the media
>> graph traversal considers the pad interdependencies when choosing which
>> links to follow.
>>
>> Currently all the entity's pads are considered as interdependent. This
>> means that the behavior with all the current drivers stays the same, but
>> in the future we can define a more fine-grained pipeline construction.
>>
>> Additionally the media pipeline's pads are cached at
>> media_pipeline_start() time, and re-used at media_pipeline_stop() which
>> avoid the need to re-walk the whole graph as the previous implementation
>> did.
>>
>> Also, caching pads in the pipeline can serve in the future as the
>> foundation to provide a better API than the media graph walk to drivers
>> to iterate over pads and entities in the pipeline.
>>
>> Note that the old media_pipeline_start/stop used the media graph walk
>> API. The new version does not use the media graph walk API, but instead
>> a new implementation.
>>
>> There are two reason for not changing the graph walk: it proved to be
>> rather difficult to change the graph walk to have the features
>> implemented in this patch, and second, this keeps the backward
>> compatibility of the graph walk as there are users of the graph walk API
>>
>> The long term plan is that all the existing code would be converted to
>> use the new cached pipeline, thus allowing us to remove the graph walk.
> 
> Could you mark the graph walk API as deprecated in this patch, or in a
> subsequent patch in the series ? I think I did in a previous version,
> but I may recall incorrectly.

I didn't mark the graph walk API as deprecated in v14 series. We can do 
it on top, but I'm not sure if it's a valid thing to say yet. Have we 
tried converting any graph walk uses to the new pipeline code? We could 
well have code missing that prevents the conversion.

  Tomi

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Laurent Pinchart]

Hi Tomi,

On Wed, Aug 31, 2022 at 05:21:58PM +0300, Tomi Valkeinen wrote:
> On 29/08/2022 20:18, Laurent Pinchart wrote:
> > On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
> >> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >>
> >> [Note: the code is mostly from Laurent but the patch description is from Tomi]
> > 
> > I'll review the description then :-)
> > 
> >> The media_pipeline_start() and media_pipeline_stop() functions use the
> >> media graph walk API to traverse the graph and validate the pipeline.
> >> The graph walk traverses the media graph following links between the
> >> entities.
> >>
> >> Also, while the pipeline can't change between the start and stop calls,
> >> the graph is walked again from scratch at stop time, or any time a
> >> driver needs to inspect the pipeline.
> >>
> >> With the upcoming multiplexed streams support we will need a bit more
> >> intelligent pipeline construction, as e.g. an entity may be passing
> >> through two independent streams via separate pads, in which case those
> > 
> > Did you mean "as e.g. two independent streams may be passing through a
> > single entity via separate pads" ?
> > 
> >> pads should not be part of the same pipeline.
> >>
> >> This patch essentially rewrites the media_pipeline_start/stop so that
> >> a pipeline is defined as a set of pads instead of entities and the media
> >> graph traversal considers the pad interdependencies when choosing which
> >> links to follow.
> >>
> >> Currently all the entity's pads are considered as interdependent. This
> >> means that the behavior with all the current drivers stays the same, but
> >> in the future we can define a more fine-grained pipeline construction.
> >>
> >> Additionally the media pipeline's pads are cached at
> >> media_pipeline_start() time, and re-used at media_pipeline_stop() which
> >> avoid the need to re-walk the whole graph as the previous implementation
> >> did.
> >>
> >> Also, caching pads in the pipeline can serve in the future as the
> >> foundation to provide a better API than the media graph walk to drivers
> >> to iterate over pads and entities in the pipeline.
> >>
> >> Note that the old media_pipeline_start/stop used the media graph walk
> >> API. The new version does not use the media graph walk API, but instead
> >> a new implementation.
> >>
> >> There are two reason for not changing the graph walk: it proved to be
> >> rather difficult to change the graph walk to have the features
> >> implemented in this patch, and second, this keeps the backward
> >> compatibility of the graph walk as there are users of the graph walk API
> >>
> >> The long term plan is that all the existing code would be converted to
> >> use the new cached pipeline, thus allowing us to remove the graph walk.
> > 
> > Could you mark the graph walk API as deprecated in this patch, or in a
> > subsequent patch in the series ? I think I did in a previous version,
> > but I may recall incorrectly.
> 
> I didn't mark the graph walk API as deprecated in v14 series. We can do 
> it on top, but I'm not sure if it's a valid thing to say yet. Have we 
> tried converting any graph walk uses to the new pipeline code? We could 
> well have code missing that prevents the conversion.

I'd like to avoid new users of the graph walk API. This requires two
patches that you haven't included or squashed in v14 though, namely

20a31d49bd75 media: mc: entity: Add entity iterator for media_pipeline
50659eb74afc media: mc: entity: Add pad iterator for media_pipeline

that you can find at

git://linuxtv.org/pinchartl/media.git streams/v6.0/v11

Should I submit that on top of v14 for inclusion in v6.1 too ?

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Laurent Pinchart]

Hi Tomi,

On Tue, Aug 30, 2022 at 03:40:30PM +0300, Tomi Valkeinen wrote:
> On 30/08/2022 14:44, Tomi Valkeinen wrote:
> 
> >>> @@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> >>>   struct media_pipeline *media_entity_pipeline(struct media_entity 
> >>> *entity)
> >>>   {
> >>> -    return entity->pipe;
> >>> +    struct media_pipeline *pipe;
> >>> +    struct media_pad *pad;
> >>> +
> >>> +    if (entity->num_pads == 0)
> >>> +        return NULL;
> >>> +
> >>> +    pipe = entity->pads->pipe;
> >>> +
> >>> +    media_entity_for_each_pad(entity, pad) {
> >>> +        if (WARN_ON(pad->pipe != pipe))
> >>> +            return NULL;
> >>> +    }
> >>
> >> This wasn't done in v12, could you explain the rationale ? The
> >> documentation of media_entity_pipeline() should probably be updated too.
> > 
> > This is just a sanity check.
> > 
> > The doc says "This function is to be used only for entities that do not 
> > support multiple pipelines". So the pipeline must be the same for all 
> > pads, and if that's not the case, something is horribly wrong.
> > 
> > However, now that I think about it, can we have NULL pipes in some pads, 
> > if those pads are e.g. not linked... Then I think the check should be 
> > (pipe && pad->pipe && pipe != pad->pipe).
> > 
> > But if that's the case, does this even work... The code expects the 
> > first pad to contain the pipeline for the entity. If the first pad's 
> > pipe can be NULL, but, say, second and fourth pads contain the pipeline 
> > (as those pads are actually used), then this is quite broken.
> 
> Looks like my concern has some grounds. While I don't have a perfect 
> setup to test this, I think my test with UB960 shows that unlinked pads 
> have NULL pipeline. So if we have an entity with two sink pads (0 and 1) 
> and two source pads (2 and 3), and the pipeline goes through pads 1 and 
> 3, this function will always return NULL.
> 
> I'm not sure how to fix this. One option would be to make this function 
> available only for entities with 1 or 2 (a sink and a source) pads. But 
> I have no idea if that covers the current users.

I fear that could introduce regressions silently.

> Or should this function iterate through the pads and return the first 
> non-null pipe?

Another option is to make sure all pads are included in the pipeline,
even if they're not linked. Have you checked why that's not the case ?

> Looks like there are only a few users for this. The one in 
> imx-media-utils.c can be dropped easily. omap4iss and omap3isp uses are 
> not clear to me yet.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

On 06/09/2022 11:44, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Wed, Aug 31, 2022 at 05:21:58PM +0300, Tomi Valkeinen wrote:
>> On 29/08/2022 20:18, Laurent Pinchart wrote:
>>> On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
>>>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>>>
>>>> [Note: the code is mostly from Laurent but the patch description is from Tomi]
>>>
>>> I'll review the description then :-)
>>>
>>>> The media_pipeline_start() and media_pipeline_stop() functions use the
>>>> media graph walk API to traverse the graph and validate the pipeline.
>>>> The graph walk traverses the media graph following links between the
>>>> entities.
>>>>
>>>> Also, while the pipeline can't change between the start and stop calls,
>>>> the graph is walked again from scratch at stop time, or any time a
>>>> driver needs to inspect the pipeline.
>>>>
>>>> With the upcoming multiplexed streams support we will need a bit more
>>>> intelligent pipeline construction, as e.g. an entity may be passing
>>>> through two independent streams via separate pads, in which case those
>>>
>>> Did you mean "as e.g. two independent streams may be passing through a
>>> single entity via separate pads" ?
>>>
>>>> pads should not be part of the same pipeline.
>>>>
>>>> This patch essentially rewrites the media_pipeline_start/stop so that
>>>> a pipeline is defined as a set of pads instead of entities and the media
>>>> graph traversal considers the pad interdependencies when choosing which
>>>> links to follow.
>>>>
>>>> Currently all the entity's pads are considered as interdependent. This
>>>> means that the behavior with all the current drivers stays the same, but
>>>> in the future we can define a more fine-grained pipeline construction.
>>>>
>>>> Additionally the media pipeline's pads are cached at
>>>> media_pipeline_start() time, and re-used at media_pipeline_stop() which
>>>> avoid the need to re-walk the whole graph as the previous implementation
>>>> did.
>>>>
>>>> Also, caching pads in the pipeline can serve in the future as the
>>>> foundation to provide a better API than the media graph walk to drivers
>>>> to iterate over pads and entities in the pipeline.
>>>>
>>>> Note that the old media_pipeline_start/stop used the media graph walk
>>>> API. The new version does not use the media graph walk API, but instead
>>>> a new implementation.
>>>>
>>>> There are two reason for not changing the graph walk: it proved to be
>>>> rather difficult to change the graph walk to have the features
>>>> implemented in this patch, and second, this keeps the backward
>>>> compatibility of the graph walk as there are users of the graph walk API
>>>>
>>>> The long term plan is that all the existing code would be converted to
>>>> use the new cached pipeline, thus allowing us to remove the graph walk.
>>>
>>> Could you mark the graph walk API as deprecated in this patch, or in a
>>> subsequent patch in the series ? I think I did in a previous version,
>>> but I may recall incorrectly.
>>
>> I didn't mark the graph walk API as deprecated in v14 series. We can do
>> it on top, but I'm not sure if it's a valid thing to say yet. Have we
>> tried converting any graph walk uses to the new pipeline code? We could
>> well have code missing that prevents the conversion.
> 
> I'd like to avoid new users of the graph walk API. This requires two
> patches that you haven't included or squashed in v14 though, namely
> 
> 20a31d49bd75 media: mc: entity: Add entity iterator for media_pipeline
> 50659eb74afc media: mc: entity: Add pad iterator for media_pipeline
> 
> that you can find at
> 
> git://linuxtv.org/pinchartl/media.git streams/v6.0/v11
> 
> Should I submit that on top of v14 for inclusion in v6.1 too ?

Both look fine to me with a quick glance, and I'm ok with adding these 
on top of v14 (or I can pick them for v15 if there's one).

I don't think we have any code that uses them, have you tested them?

  Tomi

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Tomi Valkeinen]

On 06/09/2022 11:48, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Tue, Aug 30, 2022 at 03:40:30PM +0300, Tomi Valkeinen wrote:
>> On 30/08/2022 14:44, Tomi Valkeinen wrote:
>>
>>>>> @@ -966,10 +1318,29 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>>>>>    struct media_pipeline *media_entity_pipeline(struct media_entity
>>>>> *entity)
>>>>>    {
>>>>> -    return entity->pipe;
>>>>> +    struct media_pipeline *pipe;
>>>>> +    struct media_pad *pad;
>>>>> +
>>>>> +    if (entity->num_pads == 0)
>>>>> +        return NULL;
>>>>> +
>>>>> +    pipe = entity->pads->pipe;
>>>>> +
>>>>> +    media_entity_for_each_pad(entity, pad) {
>>>>> +        if (WARN_ON(pad->pipe != pipe))
>>>>> +            return NULL;
>>>>> +    }
>>>>
>>>> This wasn't done in v12, could you explain the rationale ? The
>>>> documentation of media_entity_pipeline() should probably be updated too.
>>>
>>> This is just a sanity check.
>>>
>>> The doc says "This function is to be used only for entities that do not
>>> support multiple pipelines". So the pipeline must be the same for all
>>> pads, and if that's not the case, something is horribly wrong.
>>>
>>> However, now that I think about it, can we have NULL pipes in some pads,
>>> if those pads are e.g. not linked... Then I think the check should be
>>> (pipe && pad->pipe && pipe != pad->pipe).
>>>
>>> But if that's the case, does this even work... The code expects the
>>> first pad to contain the pipeline for the entity. If the first pad's
>>> pipe can be NULL, but, say, second and fourth pads contain the pipeline
>>> (as those pads are actually used), then this is quite broken.
>>
>> Looks like my concern has some grounds. While I don't have a perfect
>> setup to test this, I think my test with UB960 shows that unlinked pads
>> have NULL pipeline. So if we have an entity with two sink pads (0 and 1)
>> and two source pads (2 and 3), and the pipeline goes through pads 1 and
>> 3, this function will always return NULL.
>>
>> I'm not sure how to fix this. One option would be to make this function
>> available only for entities with 1 or 2 (a sink and a source) pads. But
>> I have no idea if that covers the current users.
> 
> I fear that could introduce regressions silently.
> 
>> Or should this function iterate through the pads and return the first
>> non-null pipe?
> 
> Another option is to make sure all pads are included in the pipeline,
> even if they're not linked. Have you checked why that's not the case ?

That doesn't sound correct to me, especially if the only reason to 
create the pipeline in such a way would be to accommodate two old drivers.

>> Looks like there are only a few users for this. The one in
>> imx-media-utils.c can be dropped easily. omap4iss and omap3isp uses are
>> not clear to me yet.

Only omap4iss and omap3isdp use this function now. I wonder if we should 
just drop the function and add a private functions for both drivers. The 
question remains, of course, what the function should do. I think 
returning the first non-NULL pipe should work (which is what the func 
does in v14).

  Tomi

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Sakari Ailus]

On Tue, Sep 06, 2022 at 12:19:19PM +0300, Tomi Valkeinen wrote:
> On 06/09/2022 11:44, Laurent Pinchart wrote:
> > Hi Tomi,
> > 
> > On Wed, Aug 31, 2022 at 05:21:58PM +0300, Tomi Valkeinen wrote:
> > > On 29/08/2022 20:18, Laurent Pinchart wrote:
> > > > On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
> > > > > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > > 
> > > > > [Note: the code is mostly from Laurent but the patch description is from Tomi]
> > > > 
> > > > I'll review the description then :-)
> > > > 
> > > > > The media_pipeline_start() and media_pipeline_stop() functions use the
> > > > > media graph walk API to traverse the graph and validate the pipeline.
> > > > > The graph walk traverses the media graph following links between the
> > > > > entities.
> > > > > 
> > > > > Also, while the pipeline can't change between the start and stop calls,
> > > > > the graph is walked again from scratch at stop time, or any time a
> > > > > driver needs to inspect the pipeline.
> > > > > 
> > > > > With the upcoming multiplexed streams support we will need a bit more
> > > > > intelligent pipeline construction, as e.g. an entity may be passing
> > > > > through two independent streams via separate pads, in which case those
> > > > 
> > > > Did you mean "as e.g. two independent streams may be passing through a
> > > > single entity via separate pads" ?
> > > > 
> > > > > pads should not be part of the same pipeline.
> > > > > 
> > > > > This patch essentially rewrites the media_pipeline_start/stop so that
> > > > > a pipeline is defined as a set of pads instead of entities and the media
> > > > > graph traversal considers the pad interdependencies when choosing which
> > > > > links to follow.
> > > > > 
> > > > > Currently all the entity's pads are considered as interdependent. This
> > > > > means that the behavior with all the current drivers stays the same, but
> > > > > in the future we can define a more fine-grained pipeline construction.
> > > > > 
> > > > > Additionally the media pipeline's pads are cached at
> > > > > media_pipeline_start() time, and re-used at media_pipeline_stop() which
> > > > > avoid the need to re-walk the whole graph as the previous implementation
> > > > > did.
> > > > > 
> > > > > Also, caching pads in the pipeline can serve in the future as the
> > > > > foundation to provide a better API than the media graph walk to drivers
> > > > > to iterate over pads and entities in the pipeline.
> > > > > 
> > > > > Note that the old media_pipeline_start/stop used the media graph walk
> > > > > API. The new version does not use the media graph walk API, but instead
> > > > > a new implementation.
> > > > > 
> > > > > There are two reason for not changing the graph walk: it proved to be
> > > > > rather difficult to change the graph walk to have the features
> > > > > implemented in this patch, and second, this keeps the backward
> > > > > compatibility of the graph walk as there are users of the graph walk API
> > > > > 
> > > > > The long term plan is that all the existing code would be converted to
> > > > > use the new cached pipeline, thus allowing us to remove the graph walk.
> > > > 
> > > > Could you mark the graph walk API as deprecated in this patch, or in a
> > > > subsequent patch in the series ? I think I did in a previous version,
> > > > but I may recall incorrectly.
> > > 
> > > I didn't mark the graph walk API as deprecated in v14 series. We can do
> > > it on top, but I'm not sure if it's a valid thing to say yet. Have we
> > > tried converting any graph walk uses to the new pipeline code? We could
> > > well have code missing that prevents the conversion.
> > 
> > I'd like to avoid new users of the graph walk API. This requires two
> > patches that you haven't included or squashed in v14 though, namely
> > 
> > 20a31d49bd75 media: mc: entity: Add entity iterator for media_pipeline
> > 50659eb74afc media: mc: entity: Add pad iterator for media_pipeline
> > 
> > that you can find at
> > 
> > git://linuxtv.org/pinchartl/media.git streams/v6.0/v11
> > 
> > Should I submit that on top of v14 for inclusion in v6.1 too ?
> 
> Both look fine to me with a quick glance, and I'm ok with adding these on
> top of v14 (or I can pick them for v15 if there's one).
> 
> I don't think we have any code that uses them, have you tested them?

They're from Laurent, why would you test them? :-)

I've taken the 16 first patches (v14) to my tree, will send a PR shortly.

-- 
Sakari Ailus

Re: [PATCH v13 14/34] media: mc: entity: Rewrite media_pipeline_start()

[Laurent Pinchart]

On Tue, Sep 06, 2022 at 12:58:45PM +0000, Sakari Ailus wrote:
> On Tue, Sep 06, 2022 at 12:19:19PM +0300, Tomi Valkeinen wrote:
> > On 06/09/2022 11:44, Laurent Pinchart wrote:
> > > On Wed, Aug 31, 2022 at 05:21:58PM +0300, Tomi Valkeinen wrote:
> > > > On 29/08/2022 20:18, Laurent Pinchart wrote:
> > > > > On Wed, Aug 10, 2022 at 03:11:02PM +0300, Tomi Valkeinen wrote:
> > > > > > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > > > 
> > > > > > [Note: the code is mostly from Laurent but the patch description is from Tomi]
> > > > > 
> > > > > I'll review the description then :-)
> > > > > 
> > > > > > The media_pipeline_start() and media_pipeline_stop() functions use the
> > > > > > media graph walk API to traverse the graph and validate the pipeline.
> > > > > > The graph walk traverses the media graph following links between the
> > > > > > entities.
> > > > > > 
> > > > > > Also, while the pipeline can't change between the start and stop calls,
> > > > > > the graph is walked again from scratch at stop time, or any time a
> > > > > > driver needs to inspect the pipeline.
> > > > > > 
> > > > > > With the upcoming multiplexed streams support we will need a bit more
> > > > > > intelligent pipeline construction, as e.g. an entity may be passing
> > > > > > through two independent streams via separate pads, in which case those
> > > > > 
> > > > > Did you mean "as e.g. two independent streams may be passing through a
> > > > > single entity via separate pads" ?
> > > > > 
> > > > > > pads should not be part of the same pipeline.
> > > > > > 
> > > > > > This patch essentially rewrites the media_pipeline_start/stop so that
> > > > > > a pipeline is defined as a set of pads instead of entities and the media
> > > > > > graph traversal considers the pad interdependencies when choosing which
> > > > > > links to follow.
> > > > > > 
> > > > > > Currently all the entity's pads are considered as interdependent. This
> > > > > > means that the behavior with all the current drivers stays the same, but
> > > > > > in the future we can define a more fine-grained pipeline construction.
> > > > > > 
> > > > > > Additionally the media pipeline's pads are cached at
> > > > > > media_pipeline_start() time, and re-used at media_pipeline_stop() which
> > > > > > avoid the need to re-walk the whole graph as the previous implementation
> > > > > > did.
> > > > > > 
> > > > > > Also, caching pads in the pipeline can serve in the future as the
> > > > > > foundation to provide a better API than the media graph walk to drivers
> > > > > > to iterate over pads and entities in the pipeline.
> > > > > > 
> > > > > > Note that the old media_pipeline_start/stop used the media graph walk
> > > > > > API. The new version does not use the media graph walk API, but instead
> > > > > > a new implementation.
> > > > > > 
> > > > > > There are two reason for not changing the graph walk: it proved to be
> > > > > > rather difficult to change the graph walk to have the features
> > > > > > implemented in this patch, and second, this keeps the backward
> > > > > > compatibility of the graph walk as there are users of the graph walk API
> > > > > > 
> > > > > > The long term plan is that all the existing code would be converted to
> > > > > > use the new cached pipeline, thus allowing us to remove the graph walk.
> > > > > 
> > > > > Could you mark the graph walk API as deprecated in this patch, or in a
> > > > > subsequent patch in the series ? I think I did in a previous version,
> > > > > but I may recall incorrectly.
> > > > 
> > > > I didn't mark the graph walk API as deprecated in v14 series. We can do
> > > > it on top, but I'm not sure if it's a valid thing to say yet. Have we
> > > > tried converting any graph walk uses to the new pipeline code? We could
> > > > well have code missing that prevents the conversion.
> > > 
> > > I'd like to avoid new users of the graph walk API. This requires two
> > > patches that you haven't included or squashed in v14 though, namely
> > > 
> > > 20a31d49bd75 media: mc: entity: Add entity iterator for media_pipeline
> > > 50659eb74afc media: mc: entity: Add pad iterator for media_pipeline
> > > 
> > > that you can find at
> > > 
> > > git://linuxtv.org/pinchartl/media.git streams/v6.0/v11
> > > 
> > > Should I submit that on top of v14 for inclusion in v6.1 too ?
> > 
> > Both look fine to me with a quick glance, and I'm ok with adding these on
> > top of v14 (or I can pick them for v15 if there's one).
> > 
> > I don't think we have any code that uses them, have you tested them?
> 
> They're from Laurent, why would you test them? :-)

http://events17.linuxfoundation.org/sites/events/files/slides/20170601-ossj.pdf

> I've taken the 16 first patches (v14) to my tree, will send a PR shortly.

-- 
Regards,

Laurent Pinchart