linux-media.vger.kernel.org archive mirror
 help / color / mirror / Atom feed
* [PATCH v7 00/27] v4l: subdev internal routing and streams
@ 2021-05-24 10:43 Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk Tomi Valkeinen
                   ` (28 more replies)
  0 siblings, 29 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Hi,

This is v7 of the series, the previous one is:

https://lore.kernel.org/linux-media/20210427124523.990938-1-tomi.valkeinen@ideasonboard.com/

In this version I have changed the approach to multiplexed streams, and
I went with the approach described in the RFC I sent:

https://lore.kernel.org/linux-media/20210507123558.146948-1-tomi.valkeinen@ideasonboard.com/

The main change is that in this series each pad+stream combination can
have its own configuration, versus only pad having its own
configuration. In other words, a pad with 4 streams will contain 4
configurations.

The patches up to and including "v4l: Add stream to frame descriptor"
are the same as previously, except changes done according to the review
comments. After that, the new pad+stream approach is implemented.

This series is based on the subdev-wide state change:

https://lore.kernel.org/linux-media/20210519091558.562318-1-tomi.valkeinen@ideasonboard.com/

 Tomi

Jacopo Mondi (2):
  media: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation

Laurent Pinchart (4):
  media: entity: Add has_route entity operation
  media: entity: Add media_entity_has_route() function
  media: entity: Use routing information during graph traversal
  v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations

Sakari Ailus (13):
  media: entity: Use pad as a starting point for graph walk
  media: entity: Use pads instead of entities in the media graph walk
    stack
  media: entity: Walk the graph based on pads
  v4l: mc: Start walk from a specific pad in use count calculation
  media: entity: Move the pipeline from entity to pads
  media: entity: Use pad as the starting point for a pipeline
  media: entity: Skip link validation for pads to which there is no
    route
  media: entity: Add an iterator helper for connected pads
  media: entity: Add only connected pads to the pipeline
  media: entity: Add debug information in graph walk route check
  v4l: Add bus type to frame descriptors
  v4l: Add CSI-2 bus configuration to frame descriptors
  v4l: Add stream to frame descriptor

Tomi Valkeinen (8):
  v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
  v4l: subdev: routing kernel helper functions
  v4l: subdev: add stream based configuration
  v4l: subdev: add 'stream' to subdev ioctls
  v4l: subdev: use streams in v4l2_subdev_link_validate()
  v4l: subdev: add routing & stream config to v4l2_subdev_state
  v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED
  v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

 Documentation/driver-api/media/mc-core.rst    |  15 +-
 .../userspace-api/media/v4l/dev-subdev.rst    | 128 ++++++
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     | 142 +++++++
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/mc/mc-device.c                  |  13 +-
 drivers/media/mc/mc-entity.c                  | 257 +++++++-----
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
 .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
 .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
 drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
 drivers/media/platform/exynos4-is/media-dev.c |  20 +-
 drivers/media/platform/omap3isp/isp.c         |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
 drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
 .../media/platform/qcom/camss/camss-video.c   |   6 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
 drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
 .../media/platform/s3c-camif/camif-capture.c  |   6 +-
 drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
 drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
 .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
 drivers/media/usb/au0828/au0828-core.c        |   8 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 396 +++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |   8 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
 drivers/staging/media/omap4iss/iss.c          |   2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  40 +-
 drivers/staging/media/omap4iss/iss_video.h    |   2 +-
 drivers/staging/media/tegra-video/tegra210.c  |   6 +-
 include/media/media-entity.h                  | 142 +++++--
 include/media/v4l2-subdev.h                   | 204 ++++++++-
 include/uapi/linux/v4l2-subdev.h              |  76 +++-
 48 files changed, 1410 insertions(+), 334 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.25.1


^ permalink raw reply	[flat|nested] 60+ messages in thread

* [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-07-08 10:45   ` Jacopo Mondi
  2021-05-24 10:43 ` [PATCH v7 02/27] media: entity: Use pads instead of entities in the media graph walk stack Tomi Valkeinen
                   ` (27 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the upcoming use of the recently added has_route() media entity op, all
the pads in an entity will no longer be considered interconnected. This has
an effect where the media graph is traversed: the starting pad does make a
difference.

Prepare for this change by using pad instead of the entity as an argument
for the graph walk operations. The actual graph traversal algorithm change
is in further patches.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  2 +-
 drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
 drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
 include/media/media-entity.h                  | 10 ++++------
 9 files changed, 23 insertions(+), 26 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 57b5bbba944e..ba0aee982124 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
 :c:func:`media_graph_walk_start()`
 
 The graph structure, provided by the caller, is initialized to start graph
-traversal at the given entity.
+traversal at the given pad in an entity.
 
 Drivers can then retrieve the next entity by calling
 :c:func:`media_graph_walk_next()`
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 678b99771cfa..b18502b61396 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
 
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity)
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 {
 	media_entity_enum_zero(&graph->ent_enum);
-	media_entity_enum_set(&graph->ent_enum, entity);
+	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
 	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, entity);
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"begin graph walk at '%s'\n", entity->name);
+	stack_push(graph, pad->entity);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
@@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity);
+	media_graph_walk_start(&pipe->graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
@@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
@@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index e025178db06c..5aaecdf92c53 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity))
@@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity_err))
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 8811d6dd4ee7..3c1485d59404 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 044eb5778820..61e4fbaba7b7 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 2a56201cb853..d64c3bee8b95 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -190,7 +190,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index b01474717dca..d215fe31b9a2 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
 {
 	int use = 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(entity))
@@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while (!ret && (entity = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(entity))
@@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first);
+	media_graph_walk_start(graph, first->pads);
 
 	while ((first = media_graph_walk_next(graph))
 	       && first != entity)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..760cd0ab1feb 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		if (entity == &video->video.entity)
@@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 	while ((entity = media_graph_walk_next(&graph)))
 		media_entity_enum_set(&pipe->ent_enum, entity);
 	mutex_unlock(&mdev->graph_mutex);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 09737b47881f..b9bfcf34eb0a 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
 void media_graph_walk_cleanup(struct media_graph *graph);
 
 /**
- * media_graph_walk_start - Start walking the media graph at a
- *	given entity
+ * media_graph_walk_start - Start walking the media graph at a given pad
  *
  * @graph: Media graph structure that will be used to walk the graph
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * Before using this function, media_graph_walk_init() must be
  * used to allocate resources used for walking the graph. This
  * function initializes the graph traversal structure to walk the
- * entities graph starting at the given entity. The traversal
+ * entities graph starting at the given pad. The traversal
  * structure must not be modified by the caller during graph
  * traversal. After the graph walk, the resources must be released
  * using media_graph_walk_cleanup().
  */
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity);
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
 
 /**
  * media_graph_walk_next - Get the next entity in the graph
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 02/27] media: entity: Use pads instead of entities in the media graph walk stack
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 03/27] media: entity: Walk the graph based on pads Tomi Valkeinen
                   ` (26 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Change the media graph walk stack structure to use media pads instead of
using media entities. In addition to the entity, the pad contains the
information which pad in the entity are being dealt with.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 55 ++++++++++++++++++------------------
 include/media/media-entity.h |  8 +++---
 2 files changed, 31 insertions(+), 32 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b18502b61396..0d58634b6fa8 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -228,40 +228,39 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
-static struct media_entity *
-media_entity_other(struct media_entity *entity, struct media_link *link)
+static struct media_pad *
+media_pad_other(struct media_pad *pad, struct media_link *link)
 {
-	if (link->source->entity == entity)
-		return link->sink->entity;
+	if (link->source == pad)
+		return link->sink;
 	else
-		return link->source->entity;
+		return link->source;
 }
 
 /* push an entity to traversal stack */
-static void stack_push(struct media_graph *graph,
-		       struct media_entity *entity)
+static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
 	if (graph->top == MEDIA_ENTITY_ENUM_MAX_DEPTH - 1) {
 		WARN_ON(1);
 		return;
 	}
 	graph->top++;
-	graph->stack[graph->top].link = entity->links.next;
-	graph->stack[graph->top].entity = entity;
+	graph->stack[graph->top].link = pad->entity->links.next;
+	graph->stack[graph->top].pad = pad;
 }
 
-static struct media_entity *stack_pop(struct media_graph *graph)
+static struct media_pad *stack_pop(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
-	entity = graph->stack[graph->top].entity;
+	pad = graph->stack[graph->top].pad;
 	graph->top--;
 
-	return entity;
+	return pad;
 }
 
 #define link_top(en)	((en)->stack[(en)->top].link)
-#define stack_top(en)	((en)->stack[(en)->top].entity)
+#define stack_top(en)	((en)->stack[(en)->top].pad)
 
 /**
  * media_graph_walk_init - Allocate resources for graph walk
@@ -297,8 +296,8 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
-	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, pad->entity);
+	graph->stack[graph->top].pad = NULL;
+	stack_push(graph, pad);
 	dev_dbg(pad->graph_obj.mdev->dev,
 		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
@@ -306,16 +305,16 @@ EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
 static void media_graph_walk_iter(struct media_graph *graph)
 {
-	struct media_entity *entity = stack_top(graph);
+	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_entity *next;
+	struct media_pad *next;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
 	/* The link is not enabled so we do not follow. */
 	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping disabled link '%s':%u -> '%s':%u\n",
 			link->source->entity->name, link->source->index,
 			link->sink->entity->name, link->sink->index);
@@ -323,23 +322,23 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	}
 
 	/* Get the entity in the other end of the link . */
-	next = media_entity_other(entity, link);
+	next = media_pad_other(pad, link);
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->name);
+			next->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
 	stack_push(graph, next);
-	dev_dbg(entity->graph_obj.mdev->dev, "walk: pushing '%s' on stack\n",
-		next->name);
-	lockdep_assert_held(&entity->graph_obj.mdev->graph_mutex);
+	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		next->entity->name, next->index);
+	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
 struct media_entity *media_graph_walk_next(struct media_graph *graph)
@@ -354,10 +353,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	 * top of the stack until no more entities on the level can be
 	 * found.
 	 */
-	while (link_top(graph) != &stack_top(graph)->links)
+	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph);
+	entity = stack_pop(graph)->entity;
 	dev_dbg(entity->graph_obj.mdev->dev,
 		"walk: returning entity '%s'\n", entity->name);
 
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b9bfcf34eb0a..5b55d6179e13 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -78,16 +78,16 @@ struct media_entity_enum {
  * struct media_graph - Media graph traversal state
  *
  * @stack:		Graph traversal stack; the stack contains information
- *			on the path the media entities to be walked and the
- *			links through which they were reached.
- * @stack.entity:	pointer to &struct media_entity at the graph.
+ *			on the media pads to be walked and the links through
+ *			which they were reached.
+ * @stack.pad:		pointer to &struct media_pad at the graph.
  * @stack.link:		pointer to &struct list_head.
  * @ent_enum:		Visited entities
  * @top:		The top of the stack
  */
 struct media_graph {
 	struct {
-		struct media_entity *entity;
+		struct media_pad *pad;
 		struct list_head *link;
 	} stack[MEDIA_ENTITY_ENUM_MAX_DEPTH];
 
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 03/27] media: entity: Walk the graph based on pads
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 02/27] media: entity: Use pads instead of entities in the media graph walk stack Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-07-08 10:48   ` Jacopo Mondi
  2021-05-24 10:43 ` [PATCH v7 04/27] v4l: mc: Start walk from a specific pad in use count calculation Tomi Valkeinen
                   ` (25 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Instead of iterating over graph entities during the walk, iterate the pads
through which the entity was first reached. This is required in order to
make the entity pipeline pad-based rather than entity based.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  7 ++-
 drivers/media/mc/mc-entity.c                  | 51 +++++++++++--------
 drivers/media/platform/exynos4-is/media-dev.c | 20 ++++----
 drivers/media/platform/omap3isp/ispvideo.c    | 17 ++++---
 drivers/media/platform/vsp1/vsp1_video.c      | 12 ++---
 drivers/media/platform/xilinx/xilinx-dma.c    | 12 ++---
 drivers/media/v4l2-core/v4l2-mc.c             | 24 ++++-----
 drivers/staging/media/omap4iss/iss_video.c    | 36 ++++++-------
 include/media/media-entity.h                  |  7 +--
 9 files changed, 101 insertions(+), 85 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index ba0aee982124..8a13640bed56 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -169,8 +169,11 @@ Drivers initiate a graph traversal by calling
 The graph structure, provided by the caller, is initialized to start graph
 traversal at the given pad in an entity.
 
-Drivers can then retrieve the next entity by calling
-:c:func:`media_graph_walk_next()`
+Drivers can then retrieve the next pad by calling
+:c:func:`media_graph_walk_next()`. Only the pad through which the entity
+is first reached is returned. If the caller is interested in knowing which
+further pads would be connected, the :c:func:`media_entity_has_route()`
+function can be used for that.
 
 When the graph traversal is complete the function will return ``NULL``.
 
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 0d58634b6fa8..32045671eaff 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -341,9 +341,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
-struct media_entity *media_graph_walk_next(struct media_graph *graph)
+struct media_pad *media_graph_walk_next(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
 	if (stack_top(graph) == NULL)
 		return NULL;
@@ -356,11 +356,11 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph)->entity;
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"walk: returning entity '%s'\n", entity->name);
+	pad = stack_pop(graph);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"walk: returning pad '%s':%u\n", pad->entity->name, pad->index);
 
-	return entity;
+	return pad;
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
@@ -408,7 +408,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad = entity->pads;
+	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
 
@@ -418,9 +419,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity->pads);
+	media_graph_walk_start(&pipe->graph, pad);
+
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
 
-	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
@@ -429,7 +432,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		if (entity->pipe && entity->pipe != pipe) {
 			pr_err("Pipe active for %s. Can't start for %s\n",
 				entity->name,
-				entity_err->name);
+				pad_err->entity->name);
 			ret = -EBUSY;
 			goto error;
 		}
@@ -447,26 +450,27 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		bitmap_fill(has_no_links, entity->num_pads);
 
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			struct media_pad *other_pad =
+				link->sink->entity == entity ?
+				link->sink : link->source;
 
 			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			bitmap_clear(has_no_links, other_pad->index, 1);
 
 			/*
 			 * Pads that either do not need to connect or
 			 * are connected through an enabled link are
 			 * fine.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
+			if (!(other_pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
 			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+				bitmap_set(active, other_pad->index, 1);
 
 			/*
 			 * Link validation will only take place for
 			 * sink ends of the link that are enabled.
 			 */
-			if (link->sink != pad ||
+			if (link->sink != other_pad ||
 			    !(link->flags & MEDIA_LNK_FL_ENABLED))
 				continue;
 
@@ -502,9 +506,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
+
+	while ((pad_err = media_graph_walk_next(graph))) {
+		struct media_entity *entity_err = pad_err->entity;
 
-	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
 			entity_err->stream_count--;
@@ -516,7 +522,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
@@ -543,8 +549,9 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
 	struct media_pipeline *pipe = entity->pipe;
+	struct media_graph *graph = &pipe->graph;
+	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -555,7 +562,9 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
+
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
 			entity->stream_count--;
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index 5aaecdf92c53..8938a6367703 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1164,7 +1164,7 @@ static int __fimc_md_modify_pipeline(struct media_entity *entity, bool enable)
 static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 				      struct media_graph *graph)
 {
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad, *pad_err = entity->pads;
 	int ret;
 
 	/*
@@ -1173,13 +1173,13 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity))
+	while ((pad = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		ret  = __fimc_md_modify_pipeline(entity, enable);
+		ret  = __fimc_md_modify_pipeline(pad->entity, enable);
 
 		if (ret < 0)
 			goto err;
@@ -1188,15 +1188,15 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity_err))
+	while ((pad_err = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad_err->entity))
 			continue;
 
-		__fimc_md_modify_pipeline(entity_err, !enable);
+		__fimc_md_modify_pipeline(pad_err->entity, !enable);
 
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 3c1485d59404..5c1cbb1a9003 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -222,8 +222,8 @@ static int isp_video_get_graph_data(struct isp_video *video,
 				    struct isp_pipeline *pipe)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct isp_video *far_end = NULL;
 	int ret;
 
@@ -234,23 +234,24 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
 
-		media_entity_enum_set(&pipe->ent_enum, entity);
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 
 		if (far_end != NULL)
 			continue;
 
-		if (entity == &video->video.entity)
+		if (pad == video->video.entity.pads)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		__video = to_isp_video(media_entity_to_video_device(entity));
+		__video = to_isp_video(media_entity_to_video_device(
+					       pad->entity));
 		if (__video->type != video->type)
 			far_end = __video;
 	}
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 61e4fbaba7b7..f2c36f2fdf53 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -559,8 +559,8 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 				     struct vsp1_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	unsigned int i;
 	int ret;
 
@@ -569,17 +569,17 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
 		struct vsp1_rwpf *rwpf;
 		struct vsp1_entity *e;
 
-		if (!is_media_entity_v4l2_subdev(entity))
+		if (!is_media_entity_v4l2_subdev(pad->entity))
 			continue;
 
-		subdev = media_entity_to_v4l2_subdev(entity);
+		subdev = media_entity_to_v4l2_subdev(pad->entity);
 		e = to_vsp1_entity(subdev);
 		list_add_tail(&e->list_pipe, &pipe->entities);
 		e->pipe = pipe;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index d64c3bee8b95..8df3c43aecbe 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -175,8 +175,8 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 				  struct xvip_dma *start)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &start->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = start->video.entity.pads;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	unsigned int num_inputs = 0;
 	unsigned int num_outputs = 0;
 	int ret;
@@ -190,15 +190,15 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
 
-		if (entity->function != MEDIA_ENT_F_IO_V4L)
+		if (pad->entity->function != MEDIA_ENT_F_IO_V4L)
 			continue;
 
-		dma = to_xvip_dma(media_entity_to_video_device(entity));
+		dma = to_xvip_dma(media_entity_to_video_device(pad->entity));
 
 		if (dma->pad.flags & MEDIA_PAD_FL_SINK) {
 			pipe->output = dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index d215fe31b9a2..cbeb580c6754 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -434,13 +434,14 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 static int pipeline_pm_use_count(struct media_entity *entity,
 	struct media_graph *graph)
 {
+	struct media_pad *pad;
 	int use = 0;
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (is_media_entity_v4l2_video_device(entity))
-			use += entity->use_count;
+	while ((pad = media_graph_walk_next(graph))) {
+		if (is_media_entity_v4l2_video_device(pad->entity))
+			use += pad->entity->use_count;
 	}
 
 	return use;
@@ -493,7 +494,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 static int pipeline_pm_power(struct media_entity *entity, int change,
 	struct media_graph *graph)
 {
-	struct media_entity *first = entity;
+	struct media_pad *tmp_pad, *pad;
 	int ret = 0;
 
 	if (!change)
@@ -501,19 +502,18 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while (!ret && (entity = media_graph_walk_next(graph)))
-		if (is_media_entity_v4l2_subdev(entity))
-			ret = pipeline_pm_power_one(entity, change);
+	while (!ret && (pad = media_graph_walk_next(graph)))
+		if (is_media_entity_v4l2_subdev(pad->entity))
+			ret = pipeline_pm_power_one(pad->entity, change);
 
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first->pads);
+	media_graph_walk_start(graph, entity->pads);
 
-	while ((first = media_graph_walk_next(graph))
-	       && first != entity)
-		if (is_media_entity_v4l2_subdev(first))
-			pipeline_pm_power_one(first, -change);
+	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
+		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
+			pipeline_pm_power_one(tmp_pad->entity, -change);
 
 	return ret;
 }
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 760cd0ab1feb..8c25ad73a81e 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -206,8 +206,8 @@ static struct iss_video *
 iss_video_far_end(struct iss_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct iss_video *far_end = NULL;
 
 	mutex_lock(&mdev->graph_mutex);
@@ -217,16 +217,17 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
-		if (entity == &video->video.entity)
+	while ((pad = media_graph_walk_next(&graph))) {
+		if (pad->entity == &video->video.entity)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		far_end = to_iss_video(media_entity_to_video_device(entity));
+		far_end = to_iss_video(media_entity_to_video_device(
+						pad->entity));
 		if (far_end->type != video->type)
 			break;
 
@@ -853,8 +854,8 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	struct iss_video_fh *vfh = to_iss_video_fh(fh);
 	struct iss_video *video = video_drvdata(file);
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	enum iss_pipeline_state state;
 	struct iss_pipeline *pipe;
 	struct iss_video *far_end;
@@ -870,31 +871,32 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = pad->entity->pipe
+	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
 
-	ret = media_entity_enum_init(&pipe->ent_enum, entity->graph_obj.mdev);
+	ret = media_entity_enum_init(&pipe->ent_enum,
+				     pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
-	ret = media_graph_walk_init(&graph, entity->graph_obj.mdev);
+	ret = media_graph_walk_init(&graph, pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = media_pipeline_start(pad->entity, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity->pads);
-	while ((entity = media_graph_walk_next(&graph)))
-		media_entity_enum_set(&pipe->ent_enum, entity);
+	media_graph_walk_start(&graph, pad);
+	while ((pad = media_graph_walk_next(&graph)))
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 	mutex_unlock(&mdev->graph_mutex);
 
 	/*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5b55d6179e13..926fd201eae3 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -926,10 +926,11 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
  * The graph structure must have been previously initialized with a call to
  * media_graph_walk_start().
  *
- * Return: returns the next entity in the graph or %NULL if the whole graph
- * have been traversed.
+ * Return: returns the next entity in the graph, identified by the pad through
+ * which it has been reached. If the whole graph has been traversed, return
+ * %NULL.
  */
-struct media_entity *media_graph_walk_next(struct media_graph *graph);
+struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 04/27] v4l: mc: Start walk from a specific pad in use count calculation
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (2 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 03/27] media: entity: Walk the graph based on pads Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 05/27] media: entity: Add iterator helper for entity pads Tomi Valkeinen
                   ` (24 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the addition of the recent has_route() media entity op, the pads of a
media entity are no longer all interconnected. This has to be taken into
account in power management.

Prepare for the addition of a helper function supporting S_ROUTING.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-mc.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index cbeb580c6754..35d18ed89fa5 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -427,17 +427,16 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 
 /*
  * pipeline_pm_use_count - Count the number of users of a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  *
  * Return the total number of users of all video device nodes in the pipeline.
  */
-static int pipeline_pm_use_count(struct media_entity *entity,
-	struct media_graph *graph)
+static int pipeline_pm_use_count(struct media_pad *pad,
+				 struct media_graph *graph)
 {
-	struct media_pad *pad;
 	int use = 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(pad->entity))
@@ -483,7 +482,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 
 /*
  * pipeline_pm_power - Apply power change to all entities in a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  * @change: Use count change
  *
  * Walk the pipeline to update the use count and the power state of all non-node
@@ -491,16 +490,16 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
  *
  * Return 0 on success or a negative error code on failure.
  */
-static int pipeline_pm_power(struct media_entity *entity, int change,
-	struct media_graph *graph)
+static int pipeline_pm_power(struct media_pad *pad, int change,
+			     struct media_graph *graph)
 {
-	struct media_pad *tmp_pad, *pad;
+	struct media_pad *tmp_pad, *first = pad;
 	int ret = 0;
 
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while (!ret && (pad = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(pad->entity))
@@ -509,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, first);
 
 	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
 		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
@@ -531,7 +530,7 @@ static int v4l2_pipeline_pm_use(struct media_entity *entity, unsigned int use)
 	WARN_ON(entity->use_count < 0);
 
 	/* Apply power change to connected non-nodes. */
-	ret = pipeline_pm_power(entity, change, &mdev->pm_count_walk);
+	ret = pipeline_pm_power(entity->pads, change, &mdev->pm_count_walk);
 	if (ret < 0)
 		entity->use_count -= change;
 
@@ -557,8 +556,8 @@ int v4l2_pipeline_link_notify(struct media_link *link, u32 flags,
 			      unsigned int notification)
 {
 	struct media_graph *graph = &link->graph_obj.mdev->pm_count_walk;
-	struct media_entity *source = link->source->entity;
-	struct media_entity *sink = link->sink->entity;
+	struct media_pad *source = link->source;
+	struct media_pad *sink = link->sink;
 	int source_use;
 	int sink_use;
 	int ret = 0;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 05/27] media: entity: Add iterator helper for entity pads
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (3 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 04/27] v4l: mc: Start walk from a specific pad in use count calculation Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads Tomi Valkeinen
                   ` (23 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index 9e56d2ad6b94..704ef1360eba 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -617,7 +617,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -646,9 +646,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 32045671eaff..40ae9b6bac47 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -198,7 +198,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -209,12 +210,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					&iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 926fd201eae3..5f6eed24e63f 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1107,3 +1107,15 @@ void media_remove_intf_links(struct media_interface *intf);
 	 (entity)->ops->operation((entity) , ##args) : -ENOIOCTLCMD)
 
 #endif
+
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (4 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 05/27] media: entity: Add iterator helper for entity pads Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-07-08 13:11   ` Jacopo Mondi
  2021-05-24 10:43 ` [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline Tomi Valkeinen
                   ` (22 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

This moves the pipe and stream_count fields from struct media_entity to
struct media_pad. Effectively streams become pad-specific rather than
being entity specific, allowing several independent streams to traverse a
single entity and an entity to be part of several streams.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Update documentation to use 'pads'
- Use the media pad iterator in media_entity.c
- Update rcar-dma.c to use the new per-pad stream count
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix cleanup in the error path of __media_pipeline_start()
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
 drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
 drivers/media/platform/omap3isp/isp.c         |  2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
 drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
 drivers/staging/media/imx/imx-media-utils.c   |  2 +-
 drivers/staging/media/omap4iss/iss.c          |  2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  2 +-
 drivers/staging/media/omap4iss/iss_video.h    |  2 +-
 include/media/media-entity.h                  | 21 +++---
 15 files changed, 73 insertions(+), 56 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 40ae9b6bac47..ea1cf7f63ae8 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -424,24 +424,28 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
+		bool skip_validation = pad->pipe != NULL;
+		struct media_pad *iter;
 
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
-		entity->stream_count++;
-
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				pad_err->entity->name);
-			ret = -EBUSY;
-			goto error;
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->pipe && iter->pipe != pipe) {
+				pr_err("Pipe active for %s. Can't start for %s\n",
+				       entity->name, iter->entity->name);
+				ret = -EBUSY;
+			} else {
+				iter->pipe = pipe;
+			}
+			iter->stream_count++;
 		}
 
-		entity->pipe = pipe;
+		if (ret)
+			goto error;
 
-		/* Already streaming --- no need to check. */
-		if (entity->stream_count > 1)
+		/* Already part of the pipeline, skip validation. */
+		if (skip_validation)
 			continue;
 
 		if (!entity->ops || !entity->ops->link_validate)
@@ -510,20 +514,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity_err = pad_err->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
-			entity_err->stream_count--;
-			if (entity_err->stream_count == 0)
-				entity_err->pipe = NULL;
+		struct media_entity *entity = pad_err->entity;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				--iter->stream_count;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 
 		/*
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (pad_err == pad)
+		if (pad_err->entity == pad->entity)
 			break;
 	}
 
@@ -550,7 +557,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
 	struct media_graph *graph = &pipe->graph;
 	struct media_pad *pad;
 
@@ -565,12 +572,15 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
-			entity->stream_count--;
-			if (entity->stream_count == 0)
-				entity->pipe = NULL;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				iter->stream_count--;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 	}
 
@@ -840,7 +850,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -856,8 +866,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
 	    (source->stream_count || sink->stream_count))
diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
index 855235bea46d..80274e29ccc5 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp.c
@@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
 			}
 		}
 	} else {
-		if (sd->entity.stream_count == 0) {
+		if (sd->entity.pads->stream_count == 0) {
 			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
 				struct v4l2_subdev_format format = *fmt;
 
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index aaa3af0493ce..67bfb1ad2ba2 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
 	mutex_lock(&fimc->lock);
 
 	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
-	    sd->entity.stream_count > 0) ||
+	    sd->entity.pads->stream_count > 0) ||
 	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
 	    vb2_is_busy(&fimc->vb_queue))) {
 		mutex_unlock(&fimc->lock);
diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
index 53025c8c7531..5d6021ac708b 100644
--- a/drivers/media/platform/omap3isp/isp.c
+++ b/drivers/media/platform/omap3isp/isp.c
@@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_isp_pipeline(me);
 	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 5c1cbb1a9003..a8438040c4aa 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
+	pipe = video->video.entity.pads->pipe
 	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
index a0908670c0cf..4c9c5b719ec5 100644
--- a/drivers/media/platform/omap3isp/ispvideo.h
+++ b/drivers/media/platform/omap3isp/ispvideo.h
@@ -100,7 +100,7 @@ struct isp_pipeline {
 };
 
 #define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
index cb3025992817..07ec008aacc4 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
-	media_device_for_each_entity(entity, &group->mdev)
-		if (entity->stream_count)
-			return -EBUSY;
+	media_device_for_each_entity(entity, &group->mdev) {
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->stream_count)
+				return -EBUSY;
+		}
+	}
 
 	mutex_lock(&group->lock);
 
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index f30dafbdf61c..7994262c9b63 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 8df3c43aecbe..7fa0467dddde 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -403,7 +403,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
+	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..69ced71a5696 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,7 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 6f90acf9c725..535da4dda3c6 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (entity->pads->pipe)
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..4c6f25aa8b57 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_iss_pipeline(me);
 	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 8c25ad73a81e..b74f7891711d 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = pad->entity->pipe
+	pipe = pad->pipe
 	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..9b8ec27bf87d 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -91,7 +91,7 @@ struct iss_pipeline {
 };
 
 #define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5f6eed24e63f..c9d97c902d05 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -180,15 +180,24 @@ enum media_pad_signal_type {
  *
  * @graph_obj:	Embedded structure containing the media object common data
  * @entity:	Entity this pad belongs to
+ * @pipe:	Pipeline this pad belongs to
+ * @stream_count: Stream count for the pad
  * @index:	Pad index in the entity pads array, numbered from 0 to n
  * @sig_type:	Type of the signal inside a media pad
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * .. note::
+ *
+ *    @stream_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
 	struct media_entity *entity;
+	struct media_pipeline *pipe;
+	int stream_count;
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
@@ -267,9 +276,7 @@ enum media_entity_type {
  * @pads:	Pads array with the size defined by @num_pads.
  * @links:	List of data links.
  * @ops:	Entity operations.
- * @stream_count: Stream count for the entity.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -282,10 +289,9 @@ enum media_entity_type {
  *
  * .. note::
  *
- *    @stream_count and @use_count reference counts must never be
- *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
- *    check can be used to detect reference count bugs that would make them
- *    negative.
+ *    @use_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_entity {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -304,11 +310,8 @@ struct media_entity {
 
 	const struct media_entity_operations *ops;
 
-	int stream_count;
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (5 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-07-08 12:36   ` Jacopo Mondi
  2021-05-24 10:43 ` [PATCH v7 08/27] media: entity: Add has_route entity operation Tomi Valkeinen
                   ` (21 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The pipeline has been moved from the entity to the pads; reflect this in
the media pipeline function API.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  6 ++--
 drivers/media/mc/mc-entity.c                  | 24 ++++++-------
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
 .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
 .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
 drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
 drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
 .../media/platform/qcom/camss/camss-video.c   |  6 ++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
 .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
 .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
 drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
 drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
 drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
 drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
 .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++---
 drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
 drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
 drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
 drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
 include/media/media-entity.h                  | 34 +++++++++----------
 24 files changed, 98 insertions(+), 100 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 8a13640bed56..69a64279a61f 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,11 +213,11 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all entities connected to the given pad through
+enabled routes and links, either directly or indirectly, as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
 pipeline through the struct media_entity
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index ea1cf7f63ae8..e6451903359c 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
  * Pipeline management
  */
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad = entity->pads;
 	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
@@ -542,24 +541,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -568,7 +566,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
@@ -590,12 +588,12 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index c2644efe4dbd..e0e1d8da70af 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -982,7 +982,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = media_pipeline_start(q->vdev.entity.pads, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1002,7 +1002,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 fail_pipeline:
 	dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1023,7 +1023,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 	pm_runtime_put(&cio2->pci_dev->dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index 7ff4024003f4..eaac0ac4e406 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
@@ -1184,7 +1184,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1218,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return ret;
 }
 
@@ -1234,7 +1234,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 8d9dc597deaa..ad8195e0b8f2 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -309,7 +309,7 @@ static int isp_video_release(struct file *file)
 	mutex_lock(&isp->video_lock);
 
 	if (v4l2_fh_is_singular_file(file) && ivc->streaming) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		ivc->streaming = 0;
 	}
 
@@ -490,7 +490,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = media_pipeline_start(me->pads, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -505,7 +505,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	media_pipeline_stop(me->pads);
 	return ret;
 }
 
@@ -520,7 +520,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	media_pipeline_stop(video->ve.vdev.entity.pads);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index 67bfb1ad2ba2..a979600ff6e6 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			media_pipeline_stop(entity->pads);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -822,7 +822,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -839,7 +839,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return 0;
 }
 
@@ -853,7 +853,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	media_pipeline_stop(fimc->ve.vdev.entity.pads);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a8438040c4aa..87334477f223 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1105,7 +1105,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1162,7 +1162,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1229,7 +1229,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index f282275af626..5cd494f17589 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -491,7 +491,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = media_pipeline_start(vdev->entity.pads, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -520,7 +520,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index 7994262c9b63..2d930e53c1e2 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1215,7 +1215,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1232,7 +1232,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __media_pipeline_start(vin->vdev.entity.pads, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1241,7 +1241,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 
 	return ret;
 }
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index 5f6c9d1623e4..c2a11fcc9709 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -979,7 +979,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -993,7 +993,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = media_pipeline_start(entity->pads, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1030,7 +1030,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 3a8d0b05c117..77ad4d3d0d48 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c
index 6f163e301eb1..14a86f3826b8 100644
--- a/drivers/media/platform/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/stm32/stm32-dcmi.c
@@ -730,7 +730,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_pm_put;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = media_pipeline_start(dcmi->vdev->entity.pads, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -844,7 +844,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -870,7 +870,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 2c39cd7f2862..be0defdf74f1 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = media_pipeline_start(csi->vdev.entity.pads, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 3181d0781b61..537057a75eaa 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = media_pipeline_start(video->vdev.entity.pads, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti-vpe/cal-video.c b/drivers/media/platform/ti-vpe/cal-video.c
index 7b7436a355ee..ca75b54311a8 100644
--- a/drivers/media/platform/ti-vpe/cal-video.c
+++ b/drivers/media/platform/ti-vpe/cal-video.c
@@ -675,7 +675,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = media_pipeline_start(ctx->vdev.entity.pads, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -719,7 +719,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pm_runtime_put_sync(ctx->cal->dev);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -738,7 +738,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index f2c36f2fdf53..978f820b0f34 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1048,7 +1048,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1072,7 +1072,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 7fa0467dddde..07074eda5f70 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -406,7 +406,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(dma->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -432,7 +432,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -460,7 +460,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index 5e9fd902cd37..10724b0a868c 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -246,7 +246,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 	vcap->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcap->stream.pipe);
+	ret = media_pipeline_start(entity->pads, &vcap->stream.pipe);
 	if (ret) {
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -254,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -273,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcap->vdev.entity);
+	media_pipeline_stop(vcap->vdev.entity.pads);
 
 	/* Release all active buffers */
 	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index a8a72d5fbd12..93dd7bb0ece0 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 535da4dda3c6..74218af45551 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
 		if (entity->pads->pipe)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index 38a240764509..db5867da3f11 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -485,7 +485,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = media_pipeline_start(node->vdev.entity.pads, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -510,7 +510,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -550,7 +550,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index b74f7891711d..20fac40581c6 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -889,7 +889,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(pad->entity, &pipe->pipe);
+	ret = media_pipeline_start(pad, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -980,7 +980,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1034,7 +1034,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d2d7dd0e8624 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = media_pipeline_start(chan->video.entity.pads, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 }
 
 /*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index c9d97c902d05..516d73a2941e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -937,53 +937,53 @@ struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as streaming. The given pipeline object is assigned
- * to every entity in the pipeline and stored in the media_entity pipe field.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every pad in the pipeline and stored in the media_pad pipe
+ * field.
  *
  * Calls to this function can be nested, in which case the same number of
  * media_pipeline_stop() calls will be required to stop streaming. The
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as not streaming. The media_entity pipe field is
- * reset to %NULL.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as not streaming.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
  * number of calls to this function are required to mark the pipeline as not
- * streaming.
+ * streaming and reset the media_pad pipe field to %NULL.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 08/27] media: entity: Add has_route entity operation
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (6 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-07-08 12:43   ` Jacopo Mondi
  2021-05-24 10:43 ` [PATCH v7 09/27] media: entity: Add media_entity_has_route() function Tomi Valkeinen
                   ` (20 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The optional operation can be used by entities to report whether two
pads are internally connected.

While at there, fix a Sphinx compiler warning in a comment block a few
lines above.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/media-entity.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 516d73a2941e..ad4020b2df65 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -187,6 +187,7 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ *
  * .. note::
  *
  *    @stream_count reference count must never be negative, but is a signed
@@ -214,6 +215,10 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_route:		Return whether a route exists inside the entity between
+ *			two given pads. Pads are passed to the operation ordered
+ *			by index. Optional: If the operation isn't implemented
+ *			all pads will be considered as connected.
  *
  * .. note::
  *
@@ -227,6 +232,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
+			  unsigned int pad1);
 };
 
 /**
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 09/27] media: entity: Add media_entity_has_route() function
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (7 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 08/27] media: entity: Add has_route entity operation Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 10/27] media: entity: Use routing information during graph traversal Tomi Valkeinen
                   ` (19 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

This is a wrapper around the media entity has_route operation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
 include/media/media-entity.h | 17 +++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e6451903359c..db62f852bd67 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (pad0 == pad1)
+		return true;
+
+	if (!entity->ops || !entity->ops->has_route)
+		return true;
+
+	if (entity->pads[pad1].index < entity->pads[pad0].index)
+		swap(pad0, pad1);
+
+	return entity->ops->has_route(entity, pad0, pad1);
+}
+EXPORT_SYMBOL_GPL(media_entity_has_route);
+
 static struct media_pad *
 media_pad_other(struct media_pad *pad, struct media_link *link)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index ad4020b2df65..b3069eef7fdb 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -904,6 +904,23 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 __must_check int media_graph_walk_init(
 	struct media_graph *graph, struct media_device *mdev);
 
+/**
+ * media_entity_has_route - Check if two entity pads are connected internally
+ *
+ * @entity: The entity
+ * @pad0: The first pad index
+ * @pad1: The second pad index
+ *
+ * This function can be used to check whether two pads of an entity are
+ * connected internally in the entity.
+ *
+ * The caller must hold entity->graph_obj.mdev->mutex.
+ *
+ * Return: true if the pads are connected internally and false otherwise.
+ */
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1);
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 10/27] media: entity: Use routing information during graph traversal
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (8 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 09/27] media: entity: Add media_entity_has_route() function Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 11/27] media: entity: Skip link validation for pads to which there is no route Tomi Valkeinen
                   ` (18 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Take internal routing information as reported by the entity has_route
operation into account during graph traversal to avoid following
unrelated links.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 46 ++++++++++++++++++++++--------------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index db62f852bd67..6a270eeae280 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,15 +248,6 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
-static struct media_pad *
-media_pad_other(struct media_pad *pad, struct media_link *link)
-{
-	if (link->source == pad)
-		return link->sink;
-	else
-		return link->source;
-}
-
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
@@ -327,7 +318,8 @@ static void media_graph_walk_iter(struct media_graph *graph)
 {
 	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_pad *next;
+	struct media_pad *remote;
+	struct media_pad *local;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
@@ -341,24 +333,42 @@ static void media_graph_walk_iter(struct media_graph *graph)
 		return;
 	}
 
-	/* Get the entity in the other end of the link . */
-	next = media_pad_other(pad, link);
+	/*
+	 * Get the local pad, the remote pad and the entity at the other
+	 * end of the link.
+	 */
+	if (link->source->entity == pad->entity) {
+		remote = link->sink;
+		local = link->source;
+	} else {
+		remote = link->source;
+		local = link->sink;
+	}
+
+	/*
+	 * Are the local pad and the pad we came from connected
+	 * internally in the entity ?
+	 */
+	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
+		link_top(graph) = link_top(graph)->next;
+		return;
+	}
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, remote->entity)) {
 		link_top(graph) = link_top(graph)->next;
 		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->entity->name);
+			remote->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
-	stack_push(graph, next);
-	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
-		next->entity->name, next->index);
-	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
+	stack_push(graph, remote);
+	dev_dbg(remote->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		remote->entity->name, remote->index);
+	lockdep_assert_held(&remote->graph_obj.mdev->graph_mutex);
 }
 
 struct media_pad *media_graph_walk_next(struct media_graph *graph)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 11/27] media: entity: Skip link validation for pads to which there is no route
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (9 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 10/27] media: entity: Use routing information during graph traversal Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 12/27] media: entity: Add an iterator helper for connected pads Tomi Valkeinen
                   ` (17 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Links are validated along the pipeline which is about to start streaming.
Not all the pads in entities that are traversed along that pipeline are
part of the pipeline, however. Skip the link validation for such pads.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 6a270eeae280..2ff0ae4c13c1 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -487,6 +487,11 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 				link->sink->entity == entity ?
 				link->sink : link->source;
 
+			/* Ignore pads to which there is no route. */
+			if (!media_entity_has_route(entity, pad->index,
+						    other_pad->index))
+				continue;
+
 			/* Mark that a pad is connected by a link. */
 			bitmap_clear(has_no_links, other_pad->index, 1);
 
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 12/27] media: entity: Add an iterator helper for connected pads
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (10 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 11/27] media: entity: Skip link validation for pads to which there is no route Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 13/27] media: entity: Add only connected pads to the pipeline Tomi Valkeinen
                   ` (16 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add a helper macro for iterating over pads that are connected through
enabled routes. This can be used to find all the connected pads within an
entity, for instance starting from the pad which has been obtained during
the graph walk.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make __media_entity_next_routed_pad() return NULL and adjust the
  iterator to handle that
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 14 ++++++++++++++
 include/media/media-entity.h | 26 ++++++++++++++++++++++++++
 2 files changed, 40 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 2ff0ae4c13c1..71dd14dd6c4d 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,6 +248,20 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter)
+{
+	struct media_entity *entity = root->entity;
+
+	for (; iter < &entity->pads[entity->num_pads]; iter++) {
+		if (media_entity_has_route(entity, root->index, iter->index))
+			return iter;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__media_entity_next_routed_pad);
+
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b3069eef7fdb..cd1750e495df 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -921,6 +921,32 @@ __must_check int media_graph_walk_init(
 bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 			    unsigned int pad1);
 
+/**
+ * __media_entity_next_routed_pad - Get next pad connected to @root
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Get next pad which has a route to @root.
+ */
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter);
+
+/**
+ * media_entity_for_each_routed_pad - Iterate over entity pads connected by routes
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Iterate over all pads of an entity which have an internal route to @root pad.
+ * The iteration will include the @root pad itself.
+ */
+#define media_entity_for_each_routed_pad(root, iter)                           \
+	for (iter = __media_entity_next_routed_pad(root,                       \
+						   (root)->entity->pads);      \
+	     iter != NULL;                                                     \
+	     iter = __media_entity_next_routed_pad(root, iter + 1))
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 13/27] media: entity: Add only connected pads to the pipeline
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (11 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 12/27] media: entity: Add an iterator helper for connected pads Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 14/27] media: entity: Add debug information in graph walk route check Tomi Valkeinen
                   ` (15 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

A single entity may contain multiple pipelines. Only add pads that were
connected to the pad through which the entity was reached to the pipeline.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 71dd14dd6c4d..e2aa14ac8000 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -472,7 +472,7 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			if (iter->pipe && iter->pipe != pipe) {
 				pr_err("Pipe active for %s. Can't start for %s\n",
 				       entity->name, iter->entity->name);
@@ -561,10 +561,9 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad_err->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad_err, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				--iter->stream_count;
@@ -617,10 +616,9 @@ void __media_pipeline_stop(struct media_pad *pad)
 	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				iter->stream_count--;
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 14/27] media: entity: Add debug information in graph walk route check
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (12 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 13/27] media: entity: Add only connected pads to the pipeline Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 15/27] v4l: Add bus type to frame descriptors Tomi Valkeinen
                   ` (14 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add debug printout in graph walk route check.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e2aa14ac8000..f24f3854a8a3 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -365,6 +365,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	 */
 	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
 		link_top(graph) = link_top(graph)->next;
+		dev_dbg(pad->graph_obj.mdev->dev,
+			"walk: skipping \"%s\":%u -> %u (no route)\n",
+			pad->entity->name, pad->index, local->index);
 		return;
 	}
 
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 15/27] v4l: Add bus type to frame descriptors
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (13 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 14/27] media: entity: Add debug information in graph walk route check Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 16/27] v4l: Add CSI-2 bus configuration " Tomi Valkeinen
                   ` (13 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add the media bus type to the frame descriptor. CSI-2 specific
information will be added in next patch to the frame descriptor.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make the bus type a named enum
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 79f9932d2b00..53967379fd08 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -340,12 +340,32 @@ struct v4l2_mbus_frame_desc_entry {
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
 
+/**
+ * enum v4l2_mbus_frame_desc_type - media bus frame description type
+ *
+ * @V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED:
+ *	Undefined frame desc type. Drivers should not use this, it is
+ *	for backwards compatibility.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL:
+ *	Parallel media bus.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_CSI2:
+ *	CSI-2 media bus. Frame desc parameters must be set in
+ *	&struct v4l2_mbus_frame_desc_entry->csi2.
+ */
+enum v4l2_mbus_frame_desc_type {
+	V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED = 0,
+	V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL,
+	V4L2_MBUS_FRAME_DESC_TYPE_CSI2,
+};
+
 /**
  * struct v4l2_mbus_frame_desc - media bus data frame description
+ * @type: type of the bus (enum v4l2_mbus_frame_desc_type)
  * @entry: frame descriptors array
  * @num_entries: number of entries in @entry array
  */
 struct v4l2_mbus_frame_desc {
+	enum v4l2_mbus_frame_desc_type type;
 	struct v4l2_mbus_frame_desc_entry entry[V4L2_FRAME_DESC_ENTRY_MAX];
 	unsigned short num_entries;
 };
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 16/27] v4l: Add CSI-2 bus configuration to frame descriptors
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (14 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 15/27] v4l: Add bus type to frame descriptors Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 17/27] v4l: Add stream to frame descriptor Tomi Valkeinen
                   ` (12 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add CSI-2 bus specific configuration to the frame descriptors. This allows
obtaining the virtual channel and data type information for each stream
the transmitter is sending.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 53967379fd08..59effa111150 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -308,6 +308,17 @@ struct v4l2_subdev_audio_ops {
 	int (*s_stream)(struct v4l2_subdev *sd, int enable);
 };
 
+/**
+ * struct v4l2_mbus_frame_desc_entry_csi2
+ *
+ * @vc: CSI-2 virtual channel
+ * @dt: CSI-2 data type ID
+ */
+struct v4l2_mbus_frame_desc_entry_csi2 {
+	u8 vc;
+	u8 dt;
+};
+
 /**
  * enum v4l2_mbus_frame_desc_flags - media bus frame description flags
  *
@@ -331,11 +342,16 @@ enum v4l2_mbus_frame_desc_flags {
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
  *		%V4L2_MBUS_FRAME_DESC_FL_LEN_MAX is set.
+ * @bus:	Bus-specific frame descriptor parameters
+ * @bus.csi2:	CSI-2-specific bus configuration
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
 	u32 pixelcode;
 	u32 length;
+	union {
+		struct v4l2_mbus_frame_desc_entry_csi2 csi2;
+	} bus;
 };
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 17/27] v4l: Add stream to frame descriptor
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (15 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 16/27] v4l: Add CSI-2 bus configuration " Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:43 ` [PATCH v7 18/27] media: Documentation: Add GS_ROUTING documentation Tomi Valkeinen
                   ` (11 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 59effa111150..b0e7a0e53276 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -338,6 +338,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -347,6 +348,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 18/27] media: Documentation: Add GS_ROUTING documentation
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (16 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 17/27] v4l: Add stream to frame descriptor Tomi Valkeinen
@ 2021-05-24 10:43 ` Tomi Valkeinen
  2021-05-24 10:44 ` [PATCH v7 19/27] v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations Tomi Valkeinen
                   ` (10 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:43 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    | 128 ++++++++++++++++
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 142 ++++++++++++++++++
 3 files changed, 271 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..2868edb83eee 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
@@ -501,3 +503,129 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+
+Multiplexed media pads and internal routing
+-------------------------------------------
+
+Routing Table
+^^^^^^^^^^^^^
+
+Subdevice drivers may expose the internal connections between media pads of an
+entity by exposing a routing table that applications can inspect and manipulate.
+A routing table is described by a struct :c:type:`v4l2_subdev_routing`, which
+contains ``num_routes`` route entries, each one represented by a struct
+:c:type:`v4l2_subdev_route`.
+
+Data routes do not just connect one pad to another in an entity, but they refer
+instead to the ``streams`` a media pad provides. Streams are data connection
+endpoints in a media pad. Multiplexed media pads expose multiple ``streams``
+which represent, when the underlying hardware technology allows that, logical
+data flows transported over a single physical media bus.
+
+A noteworthy example of logical stream multiplexing techniques is represented
+by the data interleaving mechanism implemented by mean of Virtual Channels as
+defined by the MIPI CSI-2 media bus specifications. A subdevice that implements
+support for Virtual Channel data interleaving might expose up to 4 data
+``streams``, one for each available Virtual Channel, on the source media pad
+that represents a CSI-2 connection endpoint.
+
+A route is defined as a connection between a ``(sink_pad, sink_stream)`` pair
+and a ``(source_pad, source_stream)`` pair, where ``sink_pad`` and
+``source_pad`` are the indexes of two media pads part of the same media entity,
+and ``sink_stream`` and ``source_stream`` are the identifiers of the data
+streams to be connected in the media pads. The stream identifiers are arbitrary
+numbers, i.e. they have no relevance to the hardware, but they must be unique on
+a single pad, and the entity on the other side of the link must have a matching
+stream identifier.
+
+The current routes are reported to applications in a routing table which can be
+inspected using the :ref:`routing <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+
+Routes can be added or removed by adding or removing them to/from the routing
+table. Also, a route in the routing table can be activated and deactivated by
+setting or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag in the ``flags``
+field of struct :c:type:`v4l2_subdev_route`.
+
+A subdev driver may create routes that cannot be modified by applications. Such
+routes are identified by the presence of the ``V4L2_SUBDEV_ROUTE_FL_IMMUTABLE``
+flag in the ``flags`` field of struct :c:type:`v4l2_subdev_route`. Immutable
+routes are always active.
+
+A special type of a route is a "source route", marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. Such routes exists in e.g. sensors as the
+routes' origins are internal to the device. A source route has valid
+``source_pad`` and ``source_stream``, but ``sink_pad`` and ``sink_stream`` are
+not used. The purpose of a source route is to describe the streams.
+
+As an example, a subdevice with two sink pads and one output pad has the pads
+defined as follows:
+
+.. flat-table::
+    :header-rows:  1
+
+    * - Pad Index
+      - Function
+    * - 0
+      - SINK
+    * - 1
+      - SINK
+    * - 2
+      - SOURCE
+
+A case where the subdevice would receive a single stream via each sink pad, and
+combine them to the source pad would result in a routing table as follows:
+
+.. flat-table:: routing table
+    :header-rows:  1
+
+    * - Sink Pad/Sink Stream
+      - Source Pad/Source Stream
+    * - 0/0
+      - 2/0
+    * - 1/0
+      - 2/1
+
+Whereas if the same subdev would receive two streams via each sink pad, and
+output the combined 4 streams would result in a routing table as follows:
+
+.. flat-table:: routing table
+    :header-rows:  1
+
+    * - Sink Pad/Sink Stream
+      - Source Pad/Source Stream
+    * - 0/0
+      - 2/0
+    * - 0/1
+      - 2/1
+    * - 1/0
+      - 2/2
+    * - 1/1
+      - 2/3
+
+Some subdevices may have known set of routes, mutable or immutable, dictated by
+the hardware. An example would be a sensor which produces pixel data and
+metadata via CSI-2 bus. In such a case there can ever be only those two streams.
+
+A subdevice that does not produce the data is another matter. Consider a device
+with two CSI-2 sink pads and two CSI-2 source pads, with the ability to route
+streams freely between the sink and source pads based on HW configuration. Each
+sink pad could receive streams for all four CSI-2 virtual channel. If we only
+consider the virtual channels, we would have maximum number of routes of 8.
+
+But CSI-2 also defines a datatype for each CSI-2 packet, allowing one to send,
+say, pixel data and metadata using the same virtual channel but different
+datatype. Now we would have a maximum number of routes of 16.
+
+Generally speaking, the concept of "stream" is very flexible. As a contrived
+example, you might even consider each line of a pixel frame to be a separate
+stream, if your hardware would support such a thing.
+
+Multiplexed Streams
+^^^^^^^^^^^^^^^^^^^
+
+When a subdevice exposes multiple streams in a single pad (multiplexed streams),
+the subdevice driver needs to have ``V4L2_SUBDEV_FL_MULTIPLEXED`` flag set. This
+flag indicates that the subdev supports the uAPI extensions needed to support
+multiple streams, and the driver must handle the ``stream`` field in the various
+subdev ioctls.
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..6d953ee23287
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,142 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the VIDIOC_SUBDEV_S_ROUTING ioctl, by adding or removing routes and setting
+or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of the  ``flags`` field of
+a struct :c:type:`v4l2_subdev_route`.
+
+When inspecting routes through VIDIOC_SUBDEV_G_ROUTING and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call VIDIOC_SUBDEV_G_ROUTING again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 19/27] v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (17 preceding siblings ...)
  2021-05-24 10:43 ` [PATCH v7 18/27] media: Documentation: Add GS_ROUTING documentation Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-05-24 10:44 ` [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE Tomi Valkeinen
                   ` (9 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.get_routing() and
v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 +++++++++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 46 +++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 26 +++++++++++++++
 include/uapi/linux/v4l2-subdev.h      | 48 +++++++++++++++++++++++++++
 4 files changed, 144 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 2673f51aafa4..b7506c0619d5 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/v4l2-common.h>
@@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3326,8 +3342,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index bd996dcace3b..28e84453fe28 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -683,6 +683,52 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting krouting = {
+			.which = routing->which,
+			.num_routes = routing->num_routes,
+			.routes = (struct v4l2_subdev_route *)(uintptr_t)
+					  routing->routes,
+		};
+		int ret;
+
+		ret = v4l2_subdev_call(sd, pad, get_routing, subdev_fh->state, &krouting);
+
+		routing->num_routes = krouting.num_routes;
+
+		return ret;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			if (routes[i].sink_pad >= sd->entity.num_pads ||
+			    routes[i].source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(sd->entity.pads[routes[i].sink_pad].flags &
+			      MEDIA_PAD_FL_SINK) ||
+			    !(sd->entity.pads[routes[i].source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.which = routing->which;
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, subdev_fh->state, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b0e7a0e53276..1cf91372d3cf 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -661,6 +661,22 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @which: format type (from enum v4l2_subdev_format_whence)
+ * @routes: &struct v4l2_subdev_route
+ * @num_routes: number of routes
+ *
+ * This structure is used to translate arguments received from
+ * VIDIOC_SUBDEV_G/S_ROUTING() ioctl to subdev device drivers operations.
+ */
+struct v4l2_subdev_krouting {
+	u32 which;
+	struct v4l2_subdev_route *routes;
+	unsigned int num_routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev information.
  *
@@ -735,6 +751,10 @@ struct v4l2_subdev_state {
  *		     applied to the hardware. The operation shall fail if the
  *		     pad index it has been called on is not valid or in case of
  *		     unrecoverable failures.
+ *
+ * @get_routing: get the subdevice routing table.
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -779,6 +799,12 @@ struct v4l2_subdev_pad_ops {
 			       struct v4l2_mbus_config *config);
 	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*get_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   struct v4l2_subdev_krouting *route);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..45c01799e2cd 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,52 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/**
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)
+
+/**
+ * Is the route immutable, i.e. can it be activated and inactivated?
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @routes: pointer to the routes array
+ * @num_routes: the total number of routes in the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u64 routes;
+	__u32 num_routes;
+	__u32 reserved[5];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -203,6 +249,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (18 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 19/27] v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-06-05 22:44   ` Laurent Pinchart
  2021-05-24 10:44 ` [PATCH v7 21/27] v4l: subdev: routing kernel helper functions Tomi Valkeinen
                   ` (8 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add route flag to indicate that the route is a source route. This means
that the route does not lead anywhere, and the sink_pad and sink_stream
should not be used.

A sensor which provides multiple streams should implement get_routing
and use the flag to mark the routes as sources.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/uapi/linux/v4l2-subdev.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 45c01799e2cd..f20491e1f53f 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -200,6 +200,13 @@ struct v4l2_subdev_capability {
  */
 #define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
 
+/**
+ * Is the route a source endpoint? A source endpoint route doesn't come
+ * from "anywhere", and the sink_pad and sink_stream fields are unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
+
 /**
  * struct v4l2_subdev_route - A route inside a subdev
  *
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 21/27] v4l: subdev: routing kernel helper functions
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (19 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-06-05 23:29   ` Laurent Pinchart
  2021-05-24 10:44 ` [PATCH v7 22/27] v4l: subdev: add stream based configuration Tomi Valkeinen
                   ` (7 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add helper functions for routing. These helpers make it easier for the
drivers to use struct v4l2_subdev_krouting.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 100 ++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           |  69 ++++++++++++++++++
 2 files changed, 169 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 28e84453fe28..ef18682dbc6f 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -912,6 +912,106 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
+int v4l2_subdev_get_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing)
+{
+	int ret;
+
+	routing->which = state ? V4L2_SUBDEV_FORMAT_TRY : V4L2_SUBDEV_FORMAT_ACTIVE;
+	routing->routes = NULL;
+	routing->num_routes = 0;
+
+	ret = v4l2_subdev_call(sd, pad, get_routing, state, routing);
+	if (ret == 0)
+		return 0;
+	if (ret != -ENOSPC)
+		return ret;
+
+	routing->routes = kvmalloc_array(routing->num_routes,
+					 sizeof(*routing->routes), GFP_KERNEL);
+	if (!routing->routes)
+		return -ENOMEM;
+
+	ret = v4l2_subdev_call(sd, pad, get_routing, state, routing);
+	if (ret) {
+		kvfree(routing->routes);
+		return ret;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_routing);
+
+void v4l2_subdev_free_routing(struct v4l2_subdev_krouting *routing)
+{
+	kvfree(routing->routes);
+	routing->routes = NULL;
+	routing->num_routes = 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_free_routing);
+
+int v4l2_subdev_cpy_routing(struct v4l2_subdev_krouting *dst,
+			    const struct v4l2_subdev_krouting *src)
+{
+	if (dst->num_routes < src->num_routes) {
+		dst->num_routes = src->num_routes;
+		return -ENOSPC;
+	}
+
+	memcpy(dst->routes, src->routes,
+	       src->num_routes * sizeof(*src->routes));
+	dst->num_routes = src->num_routes;
+	dst->which = src->which;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_cpy_routing);
+
+int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
+			    const struct v4l2_subdev_krouting *src)
+{
+	v4l2_subdev_free_routing(dst);
+
+	if (src->num_routes == 0) {
+		dst->which = src->which;
+		return 0;
+	}
+
+	dst->routes = kvmalloc_array(src->num_routes, sizeof(*src->routes),
+				     GFP_KERNEL);
+	if (!dst->routes)
+		return -ENOMEM;
+
+	memcpy(dst->routes, src->routes,
+	       src->num_routes * sizeof(*src->routes));
+	dst->num_routes = src->num_routes;
+	dst->which = src->which;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_dup_routing);
+
+bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
+			   unsigned int pad0, unsigned int pad1)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1))
+			return true;
+	}
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
+
 int v4l2_subdev_link_validate(struct media_link *link)
 {
 	struct v4l2_subdev *sink;
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1cf91372d3cf..87f6bb78bbe9 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1248,4 +1248,73 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
 void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 			      const struct v4l2_event *ev);
 
+/**
+ * v4l2_subdev_get_routing() - Get routing from a subdevice
+ *
+ * @sd: The subdev from which to get the routing
+ * @state: Pointer to &struct v4l2_subdev_state
+ * @routing: Pointer to the target &struct v4l2_subdev_krouting
+ *
+ * Get a copy of the subdevice's routing table.
+ *
+ * Must be freed with v4l2_subdev_free_routing after use.
+ */
+int v4l2_subdev_get_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing);
+
+/**
+ * v4l2_subdev_free_routing() - Free the routing
+ *
+ * @routing: The routing to be freed
+ *
+ * Frees the routing data in @routing.
+ */
+void v4l2_subdev_free_routing(struct v4l2_subdev_krouting *routing);
+
+/**
+ * v4l2_subdev_cpy_routing() - Copy the routing
+ *
+ * @dst: The destination routing
+ * @src: The source routing
+ *
+ * Copies routing from @src to @dst without allocating space. If @dst does not
+ * have enough space, set dst->num_routes to the required number of routes, and
+ * return -ENOSPC.
+ *
+ * Can be used in subdevice's v4l2_subdev_pad_ops.get_routing() callback.
+ */
+int v4l2_subdev_cpy_routing(struct v4l2_subdev_krouting *dst,
+			    const struct v4l2_subdev_krouting *src);
+
+/**
+ * v4l2_subdev_dup_routing() - Duplicate the routing
+ *
+ * @dst: The destination routing
+ * @src: The source routing
+ *
+ * Makes a duplicate of the routing from @src to @dst by allocating enough
+ * memory and making a copy of the routing.
+ *
+ * Can be used in subdevice's v4l2_subdev_pad_ops.set_routing() callback
+ * to store the given routing.
+ *
+ * Must be freed with v4l2_subdev_free_routing after use.
+ */
+int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
+			    const struct v4l2_subdev_krouting *src);
+
+/**
+ * v4l2_subdev_has_route() - Check if there is a route between two pads
+ *
+ * @routing: The subdevice's routing
+ * @pad0: First pad
+ * @pad1: Second pad
+ *
+ * Returns whether there is a route between @pad0 and @pad1 of the same
+ * subdevice according to the given routing.
+ */
+bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
+			   unsigned int pad0, unsigned int pad1);
+
 #endif
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 22/27] v4l: subdev: add stream based configuration
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (20 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 21/27] v4l: subdev: routing kernel helper functions Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-06-05 23:42   ` Laurent Pinchart
  2021-05-24 10:44 ` [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls Tomi Valkeinen
                   ` (6 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 62 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 55 ++++++++++++++++++++++++
 2 files changed, 117 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index ef18682dbc6f..da6ea9b14631 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1112,3 +1112,65 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
+
+int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+			     const struct v4l2_subdev_krouting *routing)
+{
+	u32 num_configs = 0;
+	unsigned int i;
+	u32 format_idx = 0;
+
+	v4l2_uninit_stream_configs(stream_configs);
+
+	/* Count number of formats needed */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		/* Each route needs a format on both ends of the route */
+		num_configs += 2;
+	}
+
+	if (num_configs) {
+		stream_configs->configs =
+			kvcalloc(num_configs, sizeof(*stream_configs->configs),
+				 GFP_KERNEL);
+
+		if (!stream_configs->configs)
+			return -ENOMEM;
+
+		stream_configs->num_configs = num_configs;
+	}
+
+	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->sink_pad;
+		stream_configs->configs[idx].stream = route->sink_stream;
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->source_pad;
+		stream_configs->configs[idx].stream = route->source_stream;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_init_stream_configs);
+
+void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs)
+{
+	kvfree(stream_configs->configs);
+	stream_configs->configs = NULL;
+	stream_configs->num_configs = 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_uninit_stream_configs);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 87f6bb78bbe9..39c6b811463a 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -661,6 +661,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @config: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -1317,4 +1348,28 @@ int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
 bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
 			   unsigned int pad0, unsigned int pad1);
 
+
+/**
+ * v4l2_init_stream_configs() - Initialize stream configs according to routing
+ *
+ * @stream_configs: The stream configs to initialize
+ * @routing: The routing used for the stream configs
+ *
+ * Initializes @stream_configs according to @routing, allocating enough
+ * space to hold configuration for each route endpoint.
+ *
+ * Must be freed with v4l2_uninit_stream_configs().
+ */
+int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+			     const struct v4l2_subdev_krouting *routing);
+
+/**
+ * v4l2_uninit_stream_configs() - Uninitialize stream configs
+ *
+ * @stream_configs: The stream configs to uninitialize
+ *
+ * Frees any allocated memory in @stream_configs.
+ */
+void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs);
+
 #endif
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (21 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 22/27] v4l: subdev: add stream based configuration Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-06-05 23:46   ` Laurent Pinchart
  2021-05-24 10:44 ` [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate() Tomi Valkeinen
                   ` (5 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add 'stream' field to all subdev configuration related ioctls.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |  5 ++++-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |  5 ++++-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |  5 ++++-
 .../media/v4l/vidioc-subdev-g-crop.rst        |  5 ++++-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |  5 ++++-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |  5 ++++-
 .../media/v4l/vidioc-subdev-g-selection.rst   |  5 ++++-
 include/uapi/linux/v4l2-subdev.h              | 21 ++++++++++++-------
 8 files changed, 42 insertions(+), 14 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index f20491e1f53f..a6fbdd3456bf 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -50,7 +50,8 @@ struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -64,7 +65,8 @@ struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -88,7 +90,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -112,7 +115,8 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -124,7 +128,8 @@ struct v4l2_subdev_frame_size_enum {
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -146,7 +151,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -170,7 +176,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate()
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (22 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-05-28 11:34   ` Tomi Valkeinen
  2021-05-24 10:44 ` [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state Tomi Valkeinen
                   ` (4 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 184 +++++++++++++++++++++++---
 1 file changed, 166 insertions(+), 18 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index da6ea9b14631..b30b456d8d99 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -894,6 +895,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
 v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+				     u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -902,6 +904,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
+		fmt->stream = stream;
 		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
 	}
 
@@ -1012,31 +1015,176 @@ bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
 
+static int cmp_u32(const void *a, const void *b)
+{
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
+
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
 int v4l2_subdev_link_validate(struct media_link *link)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	int ret;
+	unsigned int i;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	struct v4l2_subdev *source_subdev =
+		media_entity_to_v4l2_subdev(link->source->entity);
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
-		return 0;
+	struct v4l2_subdev_krouting routing;
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	static const u32 default_streams[] = { 0 };
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	u32 num_source_streams = 0;
+	const u32 *source_streams;
+	u32 num_sink_streams = 0;
+	const u32 *sink_streams;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	/* Get source streams */
+
+	memset(&routing, 0, sizeof(routing));
+
+	ret = v4l2_subdev_get_routing(source_subdev, NULL, &routing);
+
+	if (ret && ret != -ENOIOCTLCMD)
+		return ret;
+
+	if (ret == -ENOIOCTLCMD) {
+		num_source_streams = 1;
+		source_streams = default_streams;
+	} else {
+		u32 *streams;
+
+		streams = kmalloc_array(routing.num_routes, sizeof(u32),
+					GFP_KERNEL);
+
+		for (i = 0; i < routing.num_routes; ++i) {
+			struct v4l2_subdev_route *route = &routing.routes[i];
+
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+				continue;
+
+			if (route->source_pad == link->source->index)
+				streams[num_source_streams++] =
+					route->source_stream;
+		}
+
+		sort(streams, num_source_streams, sizeof(u32), &cmp_u32, NULL);
+
+		source_streams = streams;
+
+		v4l2_subdev_free_routing(&routing);
+	}
+
+	/* Get sink streams */
+
+	memset(&routing, 0, sizeof(routing));
+
+	ret = v4l2_subdev_get_routing(sink_subdev, NULL, &routing);
+
+	if (ret && ret != -ENOIOCTLCMD)
+		return ret;
+
+	if (ret == -ENOIOCTLCMD) {
+		num_sink_streams = 1;
+		sink_streams = default_streams;
+	} else {
+		u32 *streams;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+		streams = kmalloc_array(routing.num_routes, sizeof(u32),
+					GFP_KERNEL);
+
+		for (i = 0; i < routing.num_routes; ++i) {
+			struct v4l2_subdev_route *route = &routing.routes[i];
+
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+				continue;
+
+			if (route->sink_pad == link->sink->index)
+				streams[num_sink_streams++] =
+					route->sink_stream;
+		}
+
+		sort(streams, num_sink_streams, sizeof(u32), &cmp_u32, NULL);
+
+		sink_streams = streams;
+
+		v4l2_subdev_free_routing(&routing);
+	}
+
+	if (num_source_streams != num_sink_streams) {
+		dev_err(dev,
+			"Sink and source stream count mismatch: %d vs %d\n",
+			num_source_streams, num_sink_streams);
+		return -EINVAL;
+	}
+
+	/* Validate source and sink stream formats */
+
+	for (i = 0; i < num_source_streams; ++i) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+		int ret;
+
+		if (source_streams[i] != sink_streams[i]) {
+			dev_err(dev, "Sink and source streams do not match\n");
+			return -EINVAL;
+		}
+
+		stream = source_streams[i];
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			return ret;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			return ret;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			return ret;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			return ret;
+	}
+
+	if (source_streams != default_streams)
+		kfree(source_streams);
+
+	if (sink_streams != default_streams)
+		kfree(sink_streams);
+
+	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (23 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate() Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-06-06  0:01   ` Laurent Pinchart
  2021-05-24 10:44 ` [PATCH v7 26/27] v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED Tomi Valkeinen
                   ` (3 subsequent siblings)
  28 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add routing and stream_configs to struct v4l2_subdev_state. This lets
the drivers to implement V4L2_SUBDEV_FORMAT_TRY support for routing and
the stream configurations.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 3 +++
 include/media/v4l2-subdev.h           | 4 ++++
 2 files changed, 7 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index b30b456d8d99..13cffe9d9b89 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1227,6 +1227,9 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
 
 void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
 {
+	v4l2_subdev_free_routing(&state->routing);
+	v4l2_uninit_stream_configs(&state->stream_configs);
+
 	kvfree(state->pads);
 	kvfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 39c6b811463a..973db58c2d9b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -712,6 +712,8 @@ struct v4l2_subdev_krouting {
  * struct v4l2_subdev_state - Used for storing subdev information.
  *
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -719,6 +721,8 @@ struct v4l2_subdev_krouting {
  */
 struct v4l2_subdev_state {
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 26/27] v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (24 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-05-24 10:44 ` [PATCH v7 27/27] v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8 Tomi Valkeinen
                   ` (2 subsequent siblings)
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

Add V4L2_SUBDEV_FL_MULTIPLEXED, which indicates that the subdev supports
routing and per-stream configuration. These drivers do not need the old
pad based configuration, so we can skip the allocation in
v4l2_subdev_alloc_state().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 3 ++-
 include/media/v4l2-subdev.h           | 6 ++++++
 2 files changed, 8 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 13cffe9d9b89..9138434a7a73 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1199,7 +1199,8 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 		goto err;
 	}
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
 					     GFP_KERNEL | __GFP_ZERO);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 973db58c2d9b..f448e7447ff5 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -914,6 +914,12 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers.
+ */
+#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* [PATCH v7 27/27] v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (25 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 26/27] v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED Tomi Valkeinen
@ 2021-05-24 10:44 ` Tomi Valkeinen
  2021-05-26  8:25 ` [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
  2021-06-06  0:06 ` Laurent Pinchart
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-24 10:44 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Tomi Valkeinen,
	Pratyush Yadav, Lokesh Vutla

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f448e7447ff5..3bbc927cedb2 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -356,7 +356,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * FIXME: If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.25.1


^ permalink raw reply related	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (26 preceding siblings ...)
  2021-05-24 10:44 ` [PATCH v7 27/27] v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8 Tomi Valkeinen
@ 2021-05-26  8:25 ` Tomi Valkeinen
  2021-06-06  0:06 ` Laurent Pinchart
  28 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-26  8:25 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Pratyush Yadav, Lokesh Vutla

On 24/05/2021 13:43, Tomi Valkeinen wrote:
> Hi,
> 
> This is v7 of the series, the previous one is:
> 
> https://lore.kernel.org/linux-media/20210427124523.990938-1-tomi.valkeinen@ideasonboard.com/
> 
> In this version I have changed the approach to multiplexed streams, and
> I went with the approach described in the RFC I sent:
> 
> https://lore.kernel.org/linux-media/20210507123558.146948-1-tomi.valkeinen@ideasonboard.com/
> 
> The main change is that in this series each pad+stream combination can
> have its own configuration, versus only pad having its own
> configuration. In other words, a pad with 4 streams will contain 4
> configurations.
> 
> The patches up to and including "v4l: Add stream to frame descriptor"
> are the same as previously, except changes done according to the review
> comments. After that, the new pad+stream approach is implemented.
> 
> This series is based on the subdev-wide state change:
> 
> https://lore.kernel.org/linux-media/20210519091558.562318-1-tomi.valkeinen@ideasonboard.com/

While working on a few prototype bridge and sensor drivers I realized 
that I had been missing one thing here. So far I had always used "stream 
pipelines" which start from the sensor and go to the capture device with 
1-to-1 routes. But I have a bridge which can "split" the input stream 
into two streams, which didn't work with the approach in this series.

The change was a very minor one, essentially just allowing a routing 
table like this:

(0, 0) -> (4, 0)
(0, 0) -> (4, 1)

In other words, stream 0 from pad 0 goes to pad 4 as both stream 0 and 
stream 1. What exactly that means is of course device specific. In my 
case it means that the bridge takes a full frame as input, but outputs 3 
first lines tagged with CSI-2 metadata datatype, and the rest of the 
frame as CSI-2 pixel data.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate()
  2021-05-24 10:44 ` [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate() Tomi Valkeinen
@ 2021-05-28 11:34   ` Tomi Valkeinen
  2021-06-05 23:59     ` Laurent Pinchart
  0 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-05-28 11:34 UTC (permalink / raw)
  To: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas
  Cc: Mauro Carvalho Chehab, Hans Verkuil, Pratyush Yadav, Lokesh Vutla

On 24/05/2021 13:44, Tomi Valkeinen wrote:
> Update v4l2_subdev_link_validate() to use routing and streams for
> validation.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>   drivers/media/v4l2-core/v4l2-subdev.c | 184 +++++++++++++++++++++++---
>   1 file changed, 166 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index da6ea9b14631..b30b456d8d99 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -16,6 +16,7 @@
>   #include <linux/videodev2.h>
>   #include <linux/export.h>
>   #include <linux/version.h>
> +#include <linux/sort.h>
>   
>   #include <media/v4l2-ctrls.h>
>   #include <media/v4l2-device.h>
> @@ -894,6 +895,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>   
>   static int
>   v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> +				     u32 stream,
>   				     struct v4l2_subdev_format *fmt)
>   {
>   	if (is_media_entity_v4l2_subdev(pad->entity)) {
> @@ -902,6 +904,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>   
>   		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>   		fmt->pad = pad->index;
> +		fmt->stream = stream;
>   		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
>   	}
>   
> @@ -1012,31 +1015,176 @@ bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
>   }
>   EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
>   
> +static int cmp_u32(const void *a, const void *b)
> +{
> +	u32 a32 = *(u32 *)a;
> +	u32 b32 = *(u32 *)b;
> +
> +	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
> +}
> +
>   int v4l2_subdev_link_validate(struct media_link *link)
>   {
> -	struct v4l2_subdev *sink;
> -	struct v4l2_subdev_format sink_fmt, source_fmt;
> -	int rval;
> +	int ret;
> +	unsigned int i;
>   
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->source, &source_fmt);
> -	if (rval < 0)
> -		return 0;
> +	struct v4l2_subdev *source_subdev =
> +		media_entity_to_v4l2_subdev(link->source->entity);
> +	struct v4l2_subdev *sink_subdev =
> +		media_entity_to_v4l2_subdev(link->sink->entity);
> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
>   
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->sink, &sink_fmt);
> -	if (rval < 0)
> -		return 0;
> +	struct v4l2_subdev_krouting routing;
>   
> -	sink = media_entity_to_v4l2_subdev(link->sink->entity);
> +	static const u32 default_streams[] = { 0 };
>   
> -	rval = v4l2_subdev_call(sink, pad, link_validate, link,
> -				&source_fmt, &sink_fmt);
> -	if (rval != -ENOIOCTLCMD)
> -		return rval;
> +	u32 num_source_streams = 0;
> +	const u32 *source_streams;
> +	u32 num_sink_streams = 0;
> +	const u32 *sink_streams;
> +
> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> +		link->source->entity->name, link->source->index,
> +		link->sink->entity->name, link->sink->index);
> +
> +	/* Get source streams */
> +
> +	memset(&routing, 0, sizeof(routing));
> +
> +	ret = v4l2_subdev_get_routing(source_subdev, NULL, &routing);
> +
> +	if (ret && ret != -ENOIOCTLCMD)
> +		return ret;
> +
> +	if (ret == -ENOIOCTLCMD) {
> +		num_source_streams = 1;
> +		source_streams = default_streams;
> +	} else {
> +		u32 *streams;
> +
> +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
> +					GFP_KERNEL);
> +
> +		for (i = 0; i < routing.num_routes; ++i) {
> +			struct v4l2_subdev_route *route = &routing.routes[i];
> +
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +				continue;
> +
> +			if (route->source_pad == link->source->index)
> +				streams[num_source_streams++] =
> +					route->source_stream;
> +		}
> +
> +		sort(streams, num_source_streams, sizeof(u32), &cmp_u32, NULL);
> +
> +		source_streams = streams;
> +
> +		v4l2_subdev_free_routing(&routing);
> +	}
> +
> +	/* Get sink streams */
> +
> +	memset(&routing, 0, sizeof(routing));
> +
> +	ret = v4l2_subdev_get_routing(sink_subdev, NULL, &routing);
> +
> +	if (ret && ret != -ENOIOCTLCMD)
> +		return ret;
> +
> +	if (ret == -ENOIOCTLCMD) {
> +		num_sink_streams = 1;
> +		sink_streams = default_streams;
> +	} else {
> +		u32 *streams;
>   
> -	return v4l2_subdev_link_validate_default(
> -		sink, link, &source_fmt, &sink_fmt);
> +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
> +					GFP_KERNEL);
> +
> +		for (i = 0; i < routing.num_routes; ++i) {
> +			struct v4l2_subdev_route *route = &routing.routes[i];
> +
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +				continue;
> +
> +			if (route->sink_pad == link->sink->index)
> +				streams[num_sink_streams++] =
> +					route->sink_stream;
> +		}
> +
> +		sort(streams, num_sink_streams, sizeof(u32), &cmp_u32, NULL);
> +
> +		sink_streams = streams;
> +
> +		v4l2_subdev_free_routing(&routing);
> +	}
> +
> +	if (num_source_streams != num_sink_streams) {
> +		dev_err(dev,
> +			"Sink and source stream count mismatch: %d vs %d\n",
> +			num_source_streams, num_sink_streams);
> +		return -EINVAL;
> +	}
> +
> +	/* Validate source and sink stream formats */
> +
> +	for (i = 0; i < num_source_streams; ++i) {
> +		struct v4l2_subdev_format sink_fmt, source_fmt;
> +		u32 stream;
> +		int ret;
> +
> +		if (source_streams[i] != sink_streams[i]) {
> +			dev_err(dev, "Sink and source streams do not match\n");
> +			return -EINVAL;
> +		}
> +
> +		stream = source_streams[i];
> +
> +		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
> +			link->source->entity->name, link->source->index, stream,
> +			link->sink->entity->name, link->sink->index, stream);
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
> +							   &source_fmt);
> +		if (ret < 0) {
> +			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
> +				link->source->entity->name, link->source->index,
> +				stream);
> +			return ret;
> +		}
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
> +							   &sink_fmt);
> +		if (ret < 0) {
> +			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
> +				link->sink->entity->name, link->sink->index,
> +				stream);
> +			return ret;
> +		}
> +
> +		/* TODO: add stream number to link_validate() */
> +		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
> +				       &source_fmt, &sink_fmt);
> +		if (!ret)
> +			continue;
> +
> +		if (ret != -ENOIOCTLCMD)
> +			return ret;
> +
> +		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
> +							&source_fmt, &sink_fmt);
> +
> +		if (ret)
> +			return ret;
> +	}
> +
> +	if (source_streams != default_streams)
> +		kfree(source_streams);
> +
> +	if (sink_streams != default_streams)
> +		kfree(sink_streams);
> +
> +	return 0;
>   }
>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>   

I noticed two issues with this patch:

- It leaks memory on error cases.

- The previous behavior silently ignored failures to get format from 
subdevs, and returned 0. This one fails.

I'll fix those.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
  2021-05-24 10:44 ` [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE Tomi Valkeinen
@ 2021-06-05 22:44   ` Laurent Pinchart
  2021-06-05 22:46     ` Laurent Pinchart
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 22:44 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Mon, May 24, 2021 at 01:44:01PM +0300, Tomi Valkeinen wrote:
> Add route flag to indicate that the route is a source route. This means
> that the route does not lead anywhere, and the sink_pad and sink_stream
> should not be used.

I don't like this much. It's not a route if it doesn't lead anywhere, so
this flag seems like a hack. If we need a way to discover streams on a
source, I'd rather have an explicit operation to do so. Can't the get
frame descriptor operation be used for this ?

If the need is to find out that we're reaching the end of a pipeline
while going through links and routes, I'd rather have a pad flag to
indicate that the pad is an endpoint.

> A sensor which provides multiple streams should implement get_routing
> and use the flag to mark the routes as sources.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/uapi/linux/v4l2-subdev.h | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 45c01799e2cd..f20491e1f53f 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -200,6 +200,13 @@ struct v4l2_subdev_capability {
>   */
>  #define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
>  
> +/**
> + * Is the route a source endpoint? A source endpoint route doesn't come
> + * from "anywhere", and the sink_pad and sink_stream fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
> +
>  /**
>   * struct v4l2_subdev_route - A route inside a subdev
>   *

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
  2021-06-05 22:44   ` Laurent Pinchart
@ 2021-06-05 22:46     ` Laurent Pinchart
  2021-07-02  7:49       ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 22:46 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

On Sun, Jun 06, 2021 at 01:44:54AM +0300, Laurent Pinchart wrote:
> On Mon, May 24, 2021 at 01:44:01PM +0300, Tomi Valkeinen wrote:
> > Add route flag to indicate that the route is a source route. This means
> > that the route does not lead anywhere, and the sink_pad and sink_stream
> > should not be used.
> 
> I don't like this much. It's not a route if it doesn't lead anywhere, so
> this flag seems like a hack. If we need a way to discover streams on a
> source, I'd rather have an explicit operation to do so. Can't the get
> frame descriptor operation be used for this ?
> 
> If the need is to find out that we're reaching the end of a pipeline
> while going through links and routes, I'd rather have a pad flag to
> indicate that the pad is an endpoint.

Also, I can't see where this is being used, so maybe we can just drop it
:-)

> > A sensor which provides multiple streams should implement get_routing
> > and use the flag to mark the routes as sources.
> > 
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  include/uapi/linux/v4l2-subdev.h | 7 +++++++
> >  1 file changed, 7 insertions(+)
> > 
> > diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> > index 45c01799e2cd..f20491e1f53f 100644
> > --- a/include/uapi/linux/v4l2-subdev.h
> > +++ b/include/uapi/linux/v4l2-subdev.h
> > @@ -200,6 +200,13 @@ struct v4l2_subdev_capability {
> >   */
> >  #define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
> >  
> > +/**
> > + * Is the route a source endpoint? A source endpoint route doesn't come
> > + * from "anywhere", and the sink_pad and sink_stream fields are unused.
> > + * Set by the driver.
> > + */
> > +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
> > +
> >  /**
> >   * struct v4l2_subdev_route - A route inside a subdev
> >   *

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 21/27] v4l: subdev: routing kernel helper functions
  2021-05-24 10:44 ` [PATCH v7 21/27] v4l: subdev: routing kernel helper functions Tomi Valkeinen
@ 2021-06-05 23:29   ` Laurent Pinchart
  2021-07-11 15:48     ` Sakari Ailus
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 23:29 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Mon, May 24, 2021 at 01:44:02PM +0300, Tomi Valkeinen wrote:
> Add helper functions for routing. These helpers make it easier for the
> drivers to use struct v4l2_subdev_krouting.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 100 ++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           |  69 ++++++++++++++++++
>  2 files changed, 169 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 28e84453fe28..ef18682dbc6f 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -912,6 +912,106 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	return -EINVAL;
>  }
>  
> +int v4l2_subdev_get_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing)

This function seems to be ised in v4l2_subdev_link_validate() only,
let's make it static for now.

> +{
> +	int ret;
> +
> +	routing->which = state ? V4L2_SUBDEV_FORMAT_TRY : V4L2_SUBDEV_FORMAT_ACTIVE;

I think we could simplify this if we mandate subdev drivers to embed an
instance of v4l2_subdev_state for the active state. We could then drop
the .get_routing() operation. We could actually drop the .get_fmt()
operation too, but that would be a long term goal. As .get_routing() is
new, I'd rather not introduce it, and push drivers that need routing
support to use v4l2_subdev_state.

> +	routing->routes = NULL;
> +	routing->num_routes = 0;
> +
> +	ret = v4l2_subdev_call(sd, pad, get_routing, state, routing);
> +	if (ret == 0)
> +		return 0;
> +	if (ret != -ENOSPC)
> +		return ret;
> +
> +	routing->routes = kvmalloc_array(routing->num_routes,
> +					 sizeof(*routing->routes), GFP_KERNEL);
> +	if (!routing->routes)
> +		return -ENOMEM;
> +
> +	ret = v4l2_subdev_call(sd, pad, get_routing, state, routing);
> +	if (ret) {
> +		kvfree(routing->routes);
> +		return ret;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_get_routing);
> +
> +void v4l2_subdev_free_routing(struct v4l2_subdev_krouting *routing)
> +{
> +	kvfree(routing->routes);
> +	routing->routes = NULL;
> +	routing->num_routes = 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_free_routing);
> +
> +int v4l2_subdev_cpy_routing(struct v4l2_subdev_krouting *dst,
> +			    const struct v4l2_subdev_krouting *src)
> +{
> +	if (dst->num_routes < src->num_routes) {
> +		dst->num_routes = src->num_routes;
> +		return -ENOSPC;
> +	}
> +
> +	memcpy(dst->routes, src->routes,
> +	       src->num_routes * sizeof(*src->routes));
> +	dst->num_routes = src->num_routes;
> +	dst->which = src->which;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_cpy_routing);

These two functions could also be made internal if we mandate usage of
v4l2_subdev_state.

> +
> +int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
> +			    const struct v4l2_subdev_krouting *src)
> +{
> +	v4l2_subdev_free_routing(dst);
> +
> +	if (src->num_routes == 0) {
> +		dst->which = src->which;
> +		return 0;
> +	}
> +
> +	dst->routes = kvmalloc_array(src->num_routes, sizeof(*src->routes),
> +				     GFP_KERNEL);
> +	if (!dst->routes)
> +		return -ENOMEM;
> +
> +	memcpy(dst->routes, src->routes,
> +	       src->num_routes * sizeof(*src->routes));
> +	dst->num_routes = src->num_routes;
> +	dst->which = src->which;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_dup_routing);
> +
> +bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,

The first argument can be const.

> +			   unsigned int pad0, unsigned int pad1)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
> +		    (route->source_pad == pad0 && route->sink_pad == pad1))
> +			return true;
> +	}
> +
> +	return false;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
> +
>  int v4l2_subdev_link_validate(struct media_link *link)
>  {
>  	struct v4l2_subdev *sink;
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1cf91372d3cf..87f6bb78bbe9 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1248,4 +1248,73 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>  
> +/**
> + * v4l2_subdev_get_routing() - Get routing from a subdevice
> + *
> + * @sd: The subdev from which to get the routing
> + * @state: Pointer to &struct v4l2_subdev_state
> + * @routing: Pointer to the target &struct v4l2_subdev_krouting
> + *
> + * Get a copy of the subdevice's routing table.
> + *
> + * Must be freed with v4l2_subdev_free_routing after use.
> + */
> +int v4l2_subdev_get_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing);
> +
> +/**
> + * v4l2_subdev_free_routing() - Free the routing

Maybe "routing table" (here and where applicable) ? Up to you.

> + *
> + * @routing: The routing to be freed
> + *
> + * Frees the routing data in @routing.
> + */
> +void v4l2_subdev_free_routing(struct v4l2_subdev_krouting *routing);
> +
> +/**
> + * v4l2_subdev_cpy_routing() - Copy the routing
> + *
> + * @dst: The destination routing
> + * @src: The source routing
> + *
> + * Copies routing from @src to @dst without allocating space. If @dst does not
> + * have enough space, set dst->num_routes to the required number of routes, and
> + * return -ENOSPC.
> + *
> + * Can be used in subdevice's v4l2_subdev_pad_ops.get_routing() callback.
> + */
> +int v4l2_subdev_cpy_routing(struct v4l2_subdev_krouting *dst,
> +			    const struct v4l2_subdev_krouting *src);
> +
> +/**
> + * v4l2_subdev_dup_routing() - Duplicate the routing
> + *
> + * @dst: The destination routing
> + * @src: The source routing
> + *
> + * Makes a duplicate of the routing from @src to @dst by allocating enough
> + * memory and making a copy of the routing.
> + *
> + * Can be used in subdevice's v4l2_subdev_pad_ops.set_routing() callback
> + * to store the given routing.
> + *
> + * Must be freed with v4l2_subdev_free_routing after use.
> + */
> +int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
> +			    const struct v4l2_subdev_krouting *src);
> +
> +/**
> + * v4l2_subdev_has_route() - Check if there is a route between two pads

s/a route/an active route/

> + *
> + * @routing: The subdevice's routing
> + * @pad0: First pad
> + * @pad1: Second pad
> + *
> + * Returns whether there is a route between @pad0 and @pad1 of the same

Same here. Maybe we want to also mention "a direct route", as the
function doesn't return true when given pads B and C when there are
direct routes from A to B and A to C.

> + * subdevice according to the given routing.
> + */
> +bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
> +			   unsigned int pad0, unsigned int pad1);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 22/27] v4l: subdev: add stream based configuration
  2021-05-24 10:44 ` [PATCH v7 22/27] v4l: subdev: add stream based configuration Tomi Valkeinen
@ 2021-06-05 23:42   ` Laurent Pinchart
  2021-07-02  8:56     ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 23:42 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Mon, May 24, 2021 at 01:44:03PM +0300, Tomi Valkeinen wrote:
> Add support to manage configurations (format, crop, compose) per stream,
> instead of per pad. This is accomplished with data structures that hold
> an array of all subdev's stream configurations.
> 
> The number of streams can vary at runtime based on routing. Every time
> the routing is changed, the stream configurations need to be
> re-initialized.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 62 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 55 ++++++++++++++++++++++++
>  2 files changed, 117 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index ef18682dbc6f..da6ea9b14631 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1112,3 +1112,65 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
> +			     const struct v4l2_subdev_krouting *routing)
> +{
> +	u32 num_configs = 0;
> +	unsigned int i;
> +	u32 format_idx = 0;
> +
> +	v4l2_uninit_stream_configs(stream_configs);
> +
> +	/* Count number of formats needed */
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		/* Each route needs a format on both ends of the route */
> +		num_configs += 2;
> +	}
> +
> +	if (num_configs) {
> +		stream_configs->configs =
> +			kvcalloc(num_configs, sizeof(*stream_configs->configs),
> +				 GFP_KERNEL);
> +
> +		if (!stream_configs->configs)
> +			return -ENOMEM;
> +
> +		stream_configs->num_configs = num_configs;
> +	}
> +
> +	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +		u32 idx;
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		idx = format_idx++;
> +
> +		stream_configs->configs[idx].pad = route->sink_pad;
> +		stream_configs->configs[idx].stream = route->sink_stream;
> +
> +		idx = format_idx++;
> +
> +		stream_configs->configs[idx].pad = route->source_pad;
> +		stream_configs->configs[idx].stream = route->source_stream;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_init_stream_configs);
> +
> +void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs)
> +{
> +	kvfree(stream_configs->configs);
> +	stream_configs->configs = NULL;
> +	stream_configs->num_configs = 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_uninit_stream_configs);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 87f6bb78bbe9..39c6b811463a 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -661,6 +661,37 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_stream_config - Used for storing stream configuration.
> + *
> + * @pad: pad number
> + * @stream: stream number
> + * @fmt: &struct v4l2_mbus_framefmt
> + * @crop: &struct v4l2_rect to be used for crop
> + * @compose: &struct v4l2_rect to be used for compose
> + *
> + * This structure stores configuration for a stream.
> + */
> +struct v4l2_subdev_stream_config {
> +	u32 pad;
> +	u32 stream;
> +
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_rect crop;
> +	struct v4l2_rect compose;
> +};
> +
> +/**
> + * struct v4l2_subdev_stream_configs - A collection of stream configs.
> + *
> + * @num_configs: number of entries in @config.
> + * @config: an array of &struct v4l2_subdev_stream_configs.
> + */
> +struct v4l2_subdev_stream_configs {
> +	u32 num_configs;
> +	struct v4l2_subdev_stream_config *configs;
> +};

Honestly, this feels over-complicated and under-specified. What happens
to formats that have been previously configured when routing is changed
? What happens when changing routing while streaming is in progress ?
All these use cases need to be clearly specified.

If you want to go in this direction, this needs to be integrated with
v4l2_subdev_state (which I think you do in a later patch), but also
handled completely in the core. There's no way drivers will get this
right, I don't want them to see v4l2_subdev_stream_configs. On the
upside, embedding v4l2_subdev_state in subdevs would help, but I'm not
sure if it will be enough.

Making subdev pad operations handle streams explicitly opens the door to
thousands of questions, and each of them needs to be answered. I fear
that's a daunting task.

> +
>  /**
>   * struct v4l2_subdev_krouting - subdev routing table
>   *
> @@ -1317,4 +1348,28 @@ int v4l2_subdev_dup_routing(struct v4l2_subdev_krouting *dst,
>  bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
>  			   unsigned int pad0, unsigned int pad1);
>  
> +
> +/**
> + * v4l2_init_stream_configs() - Initialize stream configs according to routing
> + *
> + * @stream_configs: The stream configs to initialize
> + * @routing: The routing used for the stream configs
> + *
> + * Initializes @stream_configs according to @routing, allocating enough
> + * space to hold configuration for each route endpoint.
> + *
> + * Must be freed with v4l2_uninit_stream_configs().
> + */
> +int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
> +			     const struct v4l2_subdev_krouting *routing);
> +
> +/**
> + * v4l2_uninit_stream_configs() - Uninitialize stream configs
> + *
> + * @stream_configs: The stream configs to uninitialize
> + *
> + * Frees any allocated memory in @stream_configs.
> + */
> +void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls
  2021-05-24 10:44 ` [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls Tomi Valkeinen
@ 2021-06-05 23:46   ` Laurent Pinchart
  0 siblings, 0 replies; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 23:46 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Mon, May 24, 2021 at 01:44:04PM +0300, Tomi Valkeinen wrote:
> Add 'stream' field to all subdev configuration related ioctls.

This change is simple enough and should be fine, but you're missing the
documentation. The userspace API has to clearly define how streams are
handled. I assume documentation is missing as you're looking for general
feedback on the approach first, but be aware that 500-1000 lines of
documentation will likely be required. The hard part will be to specify
interactions between all the ioctls.

> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../v4l/vidioc-subdev-enum-frame-interval.rst |  5 ++++-
>  .../v4l/vidioc-subdev-enum-frame-size.rst     |  5 ++++-
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      |  5 ++++-
>  .../media/v4l/vidioc-subdev-g-crop.rst        |  5 ++++-
>  .../media/v4l/vidioc-subdev-g-fmt.rst         |  5 ++++-
>  .../v4l/vidioc-subdev-g-frame-interval.rst    |  5 ++++-
>  .../media/v4l/vidioc-subdev-g-selection.rst   |  5 ++++-
>  include/uapi/linux/v4l2-subdev.h              | 21 ++++++++++++-------
>  8 files changed, 42 insertions(+), 14 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> index 3703943b412f..8def4c05d3da 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> @@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
>        - Frame intervals to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> index c25a9896df0e..3ef361c0dca7 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> @@ -97,7 +97,10 @@ information about try formats.
>        - Frame sizes to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> index 417f1a19bcc4..248f6f9ee7c5 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> @@ -73,7 +73,10 @@ information about the try formats.
>        - ``flags``
>        - See :ref:`v4l2-subdev-mbus-code-flags`
>      * - __u32
> -      - ``reserved``\ [7]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [6]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> index bd15c0a5a66b..1d267f7e7991 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> @@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
>        - ``rect``
>        - Crop rectangle boundaries, in pixels.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> index 7acdbb939d89..ed253a1e44b7 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> @@ -102,7 +102,10 @@ should be as close as possible to the original request.
>        - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
>  	details.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> index d7fe7543c506..842f962d2aea 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> @@ -90,7 +90,10 @@ the same sub-device is not defined.
>        - ``interval``
>        - Period, in seconds, between consecutive video frames.
>      * - __u32
> -      - ``reserved``\ [9]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [8]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> index f9172a42f036..6b629c19168c 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> @@ -94,7 +94,10 @@ Selection targets and flags are documented in
>        - ``r``
>        - Selection rectangle, in pixels.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index f20491e1f53f..a6fbdd3456bf 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -50,7 +50,8 @@ struct v4l2_subdev_format {
>  	__u32 which;
>  	__u32 pad;
>  	struct v4l2_mbus_framefmt format;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> @@ -64,7 +65,8 @@ struct v4l2_subdev_crop {
>  	__u32 which;
>  	__u32 pad;
>  	struct v4l2_rect rect;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
> @@ -88,7 +90,8 @@ struct v4l2_subdev_mbus_code_enum {
>  	__u32 code;
>  	__u32 which;
>  	__u32 flags;
> -	__u32 reserved[7];
> +	__u32 stream;
> +	__u32 reserved[6];
>  };
>  
>  /**
> @@ -112,7 +115,8 @@ struct v4l2_subdev_frame_size_enum {
>  	__u32 min_height;
>  	__u32 max_height;
>  	__u32 which;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> @@ -124,7 +128,8 @@ struct v4l2_subdev_frame_size_enum {
>  struct v4l2_subdev_frame_interval {
>  	__u32 pad;
>  	struct v4l2_fract interval;
> -	__u32 reserved[9];
> +	__u32 stream;
> +	__u32 reserved[8];
>  };
>  
>  /**
> @@ -146,7 +151,8 @@ struct v4l2_subdev_frame_interval_enum {
>  	__u32 height;
>  	struct v4l2_fract interval;
>  	__u32 which;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> @@ -170,7 +176,8 @@ struct v4l2_subdev_selection {
>  	__u32 target;
>  	__u32 flags;
>  	struct v4l2_rect r;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate()
  2021-05-28 11:34   ` Tomi Valkeinen
@ 2021-06-05 23:59     ` Laurent Pinchart
  2021-07-09 10:02       ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-05 23:59 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Fri, May 28, 2021 at 02:34:55PM +0300, Tomi Valkeinen wrote:
> On 24/05/2021 13:44, Tomi Valkeinen wrote:
> > Update v4l2_subdev_link_validate() to use routing and streams for
> > validation.
> > 
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >   drivers/media/v4l2-core/v4l2-subdev.c | 184 +++++++++++++++++++++++---
> >   1 file changed, 166 insertions(+), 18 deletions(-)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index da6ea9b14631..b30b456d8d99 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -16,6 +16,7 @@
> >   #include <linux/videodev2.h>
> >   #include <linux/export.h>
> >   #include <linux/version.h>
> > +#include <linux/sort.h>
> >   
> >   #include <media/v4l2-ctrls.h>
> >   #include <media/v4l2-device.h>
> > @@ -894,6 +895,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
> >   
> >   static int
> >   v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> > +				     u32 stream,
> >   				     struct v4l2_subdev_format *fmt)
> >   {
> >   	if (is_media_entity_v4l2_subdev(pad->entity)) {
> > @@ -902,6 +904,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> >   
> >   		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
> >   		fmt->pad = pad->index;
> > +		fmt->stream = stream;
> >   		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
> >   	}
> >   
> > @@ -1012,31 +1015,176 @@ bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
> >   }
> >   EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
> >   
> > +static int cmp_u32(const void *a, const void *b)
> > +{
> > +	u32 a32 = *(u32 *)a;
> > +	u32 b32 = *(u32 *)b;
> > +
> > +	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
> > +}
> > +
> >   int v4l2_subdev_link_validate(struct media_link *link)
> >   {
> > -	struct v4l2_subdev *sink;
> > -	struct v4l2_subdev_format sink_fmt, source_fmt;
> > -	int rval;
> > +	int ret;
> > +	unsigned int i;
> >   

No need for a blank line, and I'd move i and ret (in that order) after
the variables below.

> > -	rval = v4l2_subdev_link_validate_get_format(
> > -		link->source, &source_fmt);
> > -	if (rval < 0)
> > -		return 0;
> > +	struct v4l2_subdev *source_subdev =
> > +		media_entity_to_v4l2_subdev(link->source->entity);
> > +	struct v4l2_subdev *sink_subdev =
> > +		media_entity_to_v4l2_subdev(link->sink->entity);
> > +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> >   

No need for a blank line here either.

> > -	rval = v4l2_subdev_link_validate_get_format(
> > -		link->sink, &sink_fmt);
> > -	if (rval < 0)
> > -		return 0;
> > +	struct v4l2_subdev_krouting routing;
> >   

Same.

> > -	sink = media_entity_to_v4l2_subdev(link->sink->entity);
> > +	static const u32 default_streams[] = { 0 };

I'd move this one at the top.

> >   
> > -	rval = v4l2_subdev_call(sink, pad, link_validate, link,
> > -				&source_fmt, &sink_fmt);
> > -	if (rval != -ENOIOCTLCMD)
> > -		return rval;
> > +	u32 num_source_streams = 0;
> > +	const u32 *source_streams;
> > +	u32 num_sink_streams = 0;
> > +	const u32 *sink_streams;
> > +
> > +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> > +		link->source->entity->name, link->source->index,
> > +		link->sink->entity->name, link->sink->index);
> > +
> > +	/* Get source streams */
> > +
> > +	memset(&routing, 0, sizeof(routing));
> > +
> > +	ret = v4l2_subdev_get_routing(source_subdev, NULL, &routing);
> > +
> > +	if (ret && ret != -ENOIOCTLCMD)
> > +		return ret;
> > +
> > +	if (ret == -ENOIOCTLCMD) {
> > +		num_source_streams = 1;
> > +		source_streams = default_streams;
> > +	} else {
> > +		u32 *streams;
> > +
> > +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
> > +					GFP_KERNEL);
> > +
> > +		for (i = 0; i < routing.num_routes; ++i) {
> > +			struct v4l2_subdev_route *route = &routing.routes[i];
> > +
> > +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> > +				continue;
> > +
> > +			if (route->source_pad == link->source->index)
> > +				streams[num_source_streams++] =
> > +					route->source_stream;
> > +		}
> > +
> > +		sort(streams, num_source_streams, sizeof(u32), &cmp_u32, NULL);
> > +
> > +		source_streams = streams;
> > +
> > +		v4l2_subdev_free_routing(&routing);
> > +	}

Could this be moved to a separate function ? The code below is very
similar.

> > +
> > +	/* Get sink streams */
> > +
> > +	memset(&routing, 0, sizeof(routing));
> > +
> > +	ret = v4l2_subdev_get_routing(sink_subdev, NULL, &routing);
> > +
> > +	if (ret && ret != -ENOIOCTLCMD)
> > +		return ret;
> > +
> > +	if (ret == -ENOIOCTLCMD) {
> > +		num_sink_streams = 1;
> > +		sink_streams = default_streams;
> > +	} else {
> > +		u32 *streams;
> >   
> > -	return v4l2_subdev_link_validate_default(
> > -		sink, link, &source_fmt, &sink_fmt);
> > +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
> > +					GFP_KERNEL);
> > +
> > +		for (i = 0; i < routing.num_routes; ++i) {
> > +			struct v4l2_subdev_route *route = &routing.routes[i];
> > +
> > +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> > +				continue;
> > +
> > +			if (route->sink_pad == link->sink->index)
> > +				streams[num_sink_streams++] =
> > +					route->sink_stream;
> > +		}
> > +
> > +		sort(streams, num_sink_streams, sizeof(u32), &cmp_u32, NULL);

Are you aware that there can be duplicate in the streams array, as a
given stream on a sink pad can be routed to multiple source pads ? I
think that will fail the sink and source streams match test below.

> > +
> > +		sink_streams = streams;
> > +
> > +		v4l2_subdev_free_routing(&routing);
> > +	}
> > +
> > +	if (num_source_streams != num_sink_streams) {
> > +		dev_err(dev,
> > +			"Sink and source stream count mismatch: %d vs %d\n",
> > +			num_source_streams, num_sink_streams);
> > +		return -EINVAL;
> > +	}
> > +
> > +	/* Validate source and sink stream formats */
> > +
> > +	for (i = 0; i < num_source_streams; ++i) {
> > +		struct v4l2_subdev_format sink_fmt, source_fmt;
> > +		u32 stream;
> > +		int ret;
> > +
> > +		if (source_streams[i] != sink_streams[i]) {
> > +			dev_err(dev, "Sink and source streams do not match\n");
> > +			return -EINVAL;
> > +		}

What if the source subdev as a route enabled that produces a stream, and
the sink subdev has not corresponding enabled route ? Isn't this a valid
configuration ? For instance, when a CSI-2 sensor produces image data
and embedded data in two streams with different CSI-2 DT, the embedded
data doesn't have to be captured, it could be dropped in the CSI-2
receiver.

> > +
> > +		stream = source_streams[i];
> > +
> > +		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
> > +			link->source->entity->name, link->source->index, stream,
> > +			link->sink->entity->name, link->sink->index, stream);
> > +
> > +		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
> > +							   &source_fmt);
> > +		if (ret < 0) {
> > +			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
> > +				link->source->entity->name, link->source->index,
> > +				stream);
> > +			return ret;
> > +		}
> > +
> > +		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
> > +							   &sink_fmt);
> > +		if (ret < 0) {
> > +			dev_err(dev, "Failed to get format for \"%s\":%u:%u\n",
> > +				link->sink->entity->name, link->sink->index,
> > +				stream);
> > +			return ret;
> > +		}
> > +
> > +		/* TODO: add stream number to link_validate() */
> > +		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
> > +				       &source_fmt, &sink_fmt);
> > +		if (!ret)
> > +			continue;
> > +
> > +		if (ret != -ENOIOCTLCMD)
> > +			return ret;
> > +
> > +		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
> > +							&source_fmt, &sink_fmt);
> > +
> > +		if (ret)
> > +			return ret;
> > +	}
> > +
> > +	if (source_streams != default_streams)
> > +		kfree(source_streams);
> > +
> > +	if (sink_streams != default_streams)
> > +		kfree(sink_streams);
> > +
> > +	return 0;
> >   }
> >   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
> >   
> 
> I noticed two issues with this patch:
> 
> - It leaks memory on error cases.
> 
> - The previous behavior silently ignored failures to get format from 
> subdevs, and returned 0. This one fails.
> 
> I'll fix those.

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state
  2021-05-24 10:44 ` [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state Tomi Valkeinen
@ 2021-06-06  0:01   ` Laurent Pinchart
  2021-07-02  8:34     ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-06  0:01 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

Thank you for the patch.

On Mon, May 24, 2021 at 01:44:06PM +0300, Tomi Valkeinen wrote:
> Add routing and stream_configs to struct v4l2_subdev_state. This lets
> the drivers to implement V4L2_SUBDEV_FORMAT_TRY support for routing and
> the stream configurations.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 3 +++
>  include/media/v4l2-subdev.h           | 4 ++++
>  2 files changed, 7 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index b30b456d8d99..13cffe9d9b89 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1227,6 +1227,9 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
>  
>  void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
>  {
> +	v4l2_subdev_free_routing(&state->routing);
> +	v4l2_uninit_stream_configs(&state->stream_configs);
> +
>  	kvfree(state->pads);
>  	kvfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 39c6b811463a..973db58c2d9b 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -712,6 +712,8 @@ struct v4l2_subdev_krouting {
>   * struct v4l2_subdev_state - Used for storing subdev information.
>   *
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
> + * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -719,6 +721,8 @@ struct v4l2_subdev_krouting {
>   */
>  struct v4l2_subdev_state {
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
> +	struct v4l2_subdev_stream_configs stream_configs;

stream_configs duplicates the information contained in pads. This is
possibly acceptable for the time being, but needs to be abstracted from
drivers completely.

>  };
>  
>  /**

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
                   ` (27 preceding siblings ...)
  2021-05-26  8:25 ` [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
@ 2021-06-06  0:06 ` Laurent Pinchart
  2021-07-09 15:18   ` Jacopo Mondi
  28 siblings, 1 reply; 60+ messages in thread
From: Laurent Pinchart @ 2021-06-06  0:06 UTC (permalink / raw)
  To: Hans Verkuil, Sakari Ailus
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Pratyush Yadav,
	Lokesh Vutla

Hi Hans, Sakari,

We need your feedback on this series, at least on the general approach.
There are quite a few issues to be addressed, and it makes no sense to
invest time in this if you don't think this is a good direction.

If anyone else wants to give feedback, speak now or forever hold your
peace :-)

On Mon, May 24, 2021 at 01:43:41PM +0300, Tomi Valkeinen wrote:
> Hi,
> 
> This is v7 of the series, the previous one is:
> 
> https://lore.kernel.org/linux-media/20210427124523.990938-1-tomi.valkeinen@ideasonboard.com/
> 
> In this version I have changed the approach to multiplexed streams, and
> I went with the approach described in the RFC I sent:
> 
> https://lore.kernel.org/linux-media/20210507123558.146948-1-tomi.valkeinen@ideasonboard.com/
> 
> The main change is that in this series each pad+stream combination can
> have its own configuration, versus only pad having its own
> configuration. In other words, a pad with 4 streams will contain 4
> configurations.
> 
> The patches up to and including "v4l: Add stream to frame descriptor"
> are the same as previously, except changes done according to the review
> comments. After that, the new pad+stream approach is implemented.
> 
> This series is based on the subdev-wide state change:
> 
> https://lore.kernel.org/linux-media/20210519091558.562318-1-tomi.valkeinen@ideasonboard.com/
> 
>  Tomi
> 
> Jacopo Mondi (2):
>   media: entity: Add iterator helper for entity pads
>   media: Documentation: Add GS_ROUTING documentation
> 
> Laurent Pinchart (4):
>   media: entity: Add has_route entity operation
>   media: entity: Add media_entity_has_route() function
>   media: entity: Use routing information during graph traversal
>   v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations
> 
> Sakari Ailus (13):
>   media: entity: Use pad as a starting point for graph walk
>   media: entity: Use pads instead of entities in the media graph walk
>     stack
>   media: entity: Walk the graph based on pads
>   v4l: mc: Start walk from a specific pad in use count calculation
>   media: entity: Move the pipeline from entity to pads
>   media: entity: Use pad as the starting point for a pipeline
>   media: entity: Skip link validation for pads to which there is no
>     route
>   media: entity: Add an iterator helper for connected pads
>   media: entity: Add only connected pads to the pipeline
>   media: entity: Add debug information in graph walk route check
>   v4l: Add bus type to frame descriptors
>   v4l: Add CSI-2 bus configuration to frame descriptors
>   v4l: Add stream to frame descriptor
> 
> Tomi Valkeinen (8):
>   v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
>   v4l: subdev: routing kernel helper functions
>   v4l: subdev: add stream based configuration
>   v4l: subdev: add 'stream' to subdev ioctls
>   v4l: subdev: use streams in v4l2_subdev_link_validate()
>   v4l: subdev: add routing & stream config to v4l2_subdev_state
>   v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED
>   v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
> 
>  Documentation/driver-api/media/mc-core.rst    |  15 +-
>  .../userspace-api/media/v4l/dev-subdev.rst    | 128 ++++++
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
>  .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
>  .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
>  .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
>  .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 142 +++++++
>  .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
>  drivers/media/mc/mc-device.c                  |  13 +-
>  drivers/media/mc/mc-entity.c                  | 257 +++++++-----
>  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
>  .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
>  .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
>  drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
>  drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
>  drivers/media/platform/exynos4-is/media-dev.c |  20 +-
>  drivers/media/platform/omap3isp/isp.c         |   2 +-
>  drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
>  drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
>  .../media/platform/qcom/camss/camss-video.c   |   6 +-
>  drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
>  drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
>  .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
>  .../media/platform/s3c-camif/camif-capture.c  |   6 +-
>  drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
>  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
>  .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
>  drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
>  drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
>  drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
>  .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
>  drivers/media/usb/au0828/au0828-core.c        |   8 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
>  drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
>  drivers/media/v4l2-core/v4l2-subdev.c         | 396 +++++++++++++++++-
>  drivers/staging/media/imx/imx-media-utils.c   |   8 +-
>  drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
>  drivers/staging/media/omap4iss/iss.c          |   2 +-
>  drivers/staging/media/omap4iss/iss_video.c    |  40 +-
>  drivers/staging/media/omap4iss/iss_video.h    |   2 +-
>  drivers/staging/media/tegra-video/tegra210.c  |   6 +-
>  include/media/media-entity.h                  | 142 +++++--
>  include/media/v4l2-subdev.h                   | 204 ++++++++-
>  include/uapi/linux/v4l2-subdev.h              |  76 +++-
>  48 files changed, 1410 insertions(+), 334 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
Regards,

Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
  2021-06-05 22:46     ` Laurent Pinchart
@ 2021-07-02  7:49       ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-02  7:49 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi,

On 06/06/2021 01:46, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Sun, Jun 06, 2021 at 01:44:54AM +0300, Laurent Pinchart wrote:
>> On Mon, May 24, 2021 at 01:44:01PM +0300, Tomi Valkeinen wrote:
>>> Add route flag to indicate that the route is a source route. This means
>>> that the route does not lead anywhere, and the sink_pad and sink_stream
>>> should not be used.
>>
>> I don't like this much. It's not a route if it doesn't lead anywhere, so
>> this flag seems like a hack. If we need a way to discover streams on a
>> source, I'd rather have an explicit operation to do so. Can't the get
>> frame descriptor operation be used for this ?

If the "route" word is the problem, perhaps we can rename it to... 
"streams"?

If the sensor has one source pad with one or two streams, the userspace 
needs to be able to 1) configure if we have one or two streams 2) 
configure the stream IDs 3) configure the stream format. This is exactly 
the same as when configuring routes for a bridge, the only difference is 
that the sensor's streams come from inside the sensor, not from an 
external source.

Somewhat related, I was wondering if the whole routing table should be 
dropped. Now that the stream endpoints each have a configuration, 
similar to pads in current drivers, perhaps the routing information 
could be just part of the stream endpoints configuration.

So, stream 0 in pad 0 could have a config field that has the destination 
pad+stream tuple. The destination should, of course, have a matching but 
inverse destination configuration. The pad+stream tuples would be 
validated in the normal subdev validation step.

I haven't looked at this closely, so I'm not sure if this would help or not.

>> If the need is to find out that we're reaching the end of a pipeline
>> while going through links and routes, I'd rather have a pad flag to
>> indicate that the pad is an endpoint.

A pad could have both source streams and "routed" streams.

> Also, I can't see where this is being used, so maybe we can just drop it
> :-)

True, I seem to have dropped the use when I dropped my metadata hacks 
from the test sensor driver. But I think the flag, or something similar, 
is needed if this information is in the routing data (or whatever we 
rename it to).

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state
  2021-06-06  0:01   ` Laurent Pinchart
@ 2021-07-02  8:34     ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-02  8:34 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi,

On 06/06/2021 03:01, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, May 24, 2021 at 01:44:06PM +0300, Tomi Valkeinen wrote:
>> Add routing and stream_configs to struct v4l2_subdev_state. This lets
>> the drivers to implement V4L2_SUBDEV_FORMAT_TRY support for routing and
>> the stream configurations.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 3 +++
>>   include/media/v4l2-subdev.h           | 4 ++++
>>   2 files changed, 7 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index b30b456d8d99..13cffe9d9b89 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1227,6 +1227,9 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
>>   
>>   void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
>>   {
>> +	v4l2_subdev_free_routing(&state->routing);
>> +	v4l2_uninit_stream_configs(&state->stream_configs);
>> +
>>   	kvfree(state->pads);
>>   	kvfree(state);
>>   }
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 39c6b811463a..973db58c2d9b 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -712,6 +712,8 @@ struct v4l2_subdev_krouting {
>>    * struct v4l2_subdev_state - Used for storing subdev information.
>>    *
>>    * @pads: &struct v4l2_subdev_pad_config array
>> + * @routing: routing table for the subdev
>> + * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
>>    *
>>    * This structure only needs to be passed to the pad op if the 'which' field
>>    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>> @@ -719,6 +721,8 @@ struct v4l2_subdev_krouting {
>>    */
>>   struct v4l2_subdev_state {
>>   	struct v4l2_subdev_pad_config *pads;
>> +	struct v4l2_subdev_krouting routing;
>> +	struct v4l2_subdev_stream_configs stream_configs;
> 
> stream_configs duplicates the information contained in pads. This is
> possibly acceptable for the time being, but needs to be abstracted from
> drivers completely.

Only one of them will be allocated, and a driver will only use one of 
them, so there's no duplication in practice.

Abstracting this from the drivers is possible but will be challenging, 
as many drivers use the pads array directly.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 22/27] v4l: subdev: add stream based configuration
  2021-06-05 23:42   ` Laurent Pinchart
@ 2021-07-02  8:56     ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-02  8:56 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi,

On 06/06/2021 02:42, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Mon, May 24, 2021 at 01:44:03PM +0300, Tomi Valkeinen wrote:
>> Add support to manage configurations (format, crop, compose) per stream,
>> instead of per pad. This is accomplished with data structures that hold
>> an array of all subdev's stream configurations.
>>
>> The number of streams can vary at runtime based on routing. Every time
>> the routing is changed, the stream configurations need to be
>> re-initialized.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 62 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 55 ++++++++++++++++++++++++
>>   2 files changed, 117 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index ef18682dbc6f..da6ea9b14631 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1112,3 +1112,65 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
>> +			     const struct v4l2_subdev_krouting *routing)
>> +{
>> +	u32 num_configs = 0;
>> +	unsigned int i;
>> +	u32 format_idx = 0;
>> +
>> +	v4l2_uninit_stream_configs(stream_configs);
>> +
>> +	/* Count number of formats needed */
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routing->routes[i];
>> +
>> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +			continue;
>> +
>> +		/* Each route needs a format on both ends of the route */
>> +		num_configs += 2;
>> +	}
>> +
>> +	if (num_configs) {
>> +		stream_configs->configs =
>> +			kvcalloc(num_configs, sizeof(*stream_configs->configs),
>> +				 GFP_KERNEL);
>> +
>> +		if (!stream_configs->configs)
>> +			return -ENOMEM;
>> +
>> +		stream_configs->num_configs = num_configs;
>> +	}
>> +
>> +	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routing->routes[i];
>> +		u32 idx;
>> +
>> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>> +			continue;
>> +
>> +		idx = format_idx++;
>> +
>> +		stream_configs->configs[idx].pad = route->sink_pad;
>> +		stream_configs->configs[idx].stream = route->sink_stream;
>> +
>> +		idx = format_idx++;
>> +
>> +		stream_configs->configs[idx].pad = route->source_pad;
>> +		stream_configs->configs[idx].stream = route->source_stream;
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_init_stream_configs);
>> +
>> +void v4l2_uninit_stream_configs(struct v4l2_subdev_stream_configs *stream_configs)
>> +{
>> +	kvfree(stream_configs->configs);
>> +	stream_configs->configs = NULL;
>> +	stream_configs->num_configs = 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_uninit_stream_configs);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 87f6bb78bbe9..39c6b811463a 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -661,6 +661,37 @@ struct v4l2_subdev_pad_config {
>>   	struct v4l2_rect try_compose;
>>   };
>>   
>> +/**
>> + * struct v4l2_subdev_stream_config - Used for storing stream configuration.
>> + *
>> + * @pad: pad number
>> + * @stream: stream number
>> + * @fmt: &struct v4l2_mbus_framefmt
>> + * @crop: &struct v4l2_rect to be used for crop
>> + * @compose: &struct v4l2_rect to be used for compose
>> + *
>> + * This structure stores configuration for a stream.
>> + */
>> +struct v4l2_subdev_stream_config {
>> +	u32 pad;
>> +	u32 stream;
>> +
>> +	struct v4l2_mbus_framefmt fmt;
>> +	struct v4l2_rect crop;
>> +	struct v4l2_rect compose;
>> +};
>> +
>> +/**
>> + * struct v4l2_subdev_stream_configs - A collection of stream configs.
>> + *
>> + * @num_configs: number of entries in @config.
>> + * @config: an array of &struct v4l2_subdev_stream_configs.
>> + */
>> +struct v4l2_subdev_stream_configs {
>> +	u32 num_configs;
>> +	struct v4l2_subdev_stream_config *configs;
>> +};
> 
> Honestly, this feels over-complicated and under-specified. What happens

Under-specified I agree, but why is it over-complicated? It's just a 
dynamic map of (pad+stream) -> stream-config. Or do you mean the whole 
approach to multiplexed streams?

> to formats that have been previously configured when routing is changed

Changing routing is a drastic operation, so all formats are reset to 
default.

> ? What happens when changing routing while streaming is in progress ?

It's not allowed (probably my code doesn't check this at the moment).

> All these use cases need to be clearly specified.
> 
> If you want to go in this direction, this needs to be integrated with
> v4l2_subdev_state (which I think you do in a later patch), but also
> handled completely in the core. There's no way drivers will get this
> right, I don't want them to see v4l2_subdev_stream_configs. On the
> upside, embedding v4l2_subdev_state in subdevs would help, but I'm not
> sure if it will be enough.
> 
> Making subdev pad operations handle streams explicitly opens the door to
> thousands of questions, and each of them needs to be answered. I fear
> that's a daunting task.

Yes, that's multiplexed streams for you =).

The previous approach I tried, where pads were either single-stream and 
configurable or multistream and non-configurable, also raised lots of 
questions and the implementation complexity was getting quite horrible. 
E.g. a simple sensor driver that also has metadata had to be split into 
three subdevs, and the need to walk the media graph back and forth lots 
of times was very ugly.

My opinion is that this version is much better. It's quite 
understandable and easy to use from the userspace (taking into account 
that the issue of multiplexed streams is complex, and that the APIs are 
perhaps not yet optimal), and the code on the kernel side resembles the 
non-multiplexed operations, and is more manageable in the drivers than 
with the previous version.

Now that the subdev state has been merged, I can work towards handling 
many of these operations in the v4l2 core. But that's essentially "just" 
kernel cleanup, I don't think that affects much whether this multiplexed 
streams approach in general is ok or not.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk
  2021-05-24 10:43 ` [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk Tomi Valkeinen
@ 2021-07-08 10:45   ` Jacopo Mondi
  0 siblings, 0 replies; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-08 10:45 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,
  a small drive-by comment

On Mon, May 24, 2021 at 01:43:42PM +0300, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> With the upcoming use of the recently added has_route() media entity op, all

Probably due to re-sorting, has_route() will be introduced later

> the pads in an entity will no longer be considered interconnected. This has
> an effect where the media graph is traversed: the starting pad does make a
> difference.
>
> Prepare for this change by using pad instead of the entity as an argument
> for the graph walk operations. The actual graph traversal algorithm change
> is in further patches.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst    |  2 +-
>  drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
>  drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
>  drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
>  drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
>  drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
>  drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
>  include/media/media-entity.h                  | 10 ++++------
>  9 files changed, 23 insertions(+), 26 deletions(-)
>
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 57b5bbba944e..ba0aee982124 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
>  :c:func:`media_graph_walk_start()`
>
>  The graph structure, provided by the caller, is initialized to start graph
> -traversal at the given entity.
> +traversal at the given pad in an entity.
>
>  Drivers can then retrieve the next entity by calling
>  :c:func:`media_graph_walk_next()`
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 678b99771cfa..b18502b61396 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
>  }
>  EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
>
> -void media_graph_walk_start(struct media_graph *graph,
> -			    struct media_entity *entity)
> +void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
>  {
>  	media_entity_enum_zero(&graph->ent_enum);
> -	media_entity_enum_set(&graph->ent_enum, entity);
> +	media_entity_enum_set(&graph->ent_enum, pad->entity);
>
>  	graph->top = 0;
>  	graph->stack[graph->top].entity = NULL;
> -	stack_push(graph, entity);
> -	dev_dbg(entity->graph_obj.mdev->dev,
> -		"begin graph walk at '%s'\n", entity->name);
> +	stack_push(graph, pad->entity);
> +	dev_dbg(pad->graph_obj.mdev->dev,
> +		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
>  }
>  EXPORT_SYMBOL_GPL(media_graph_walk_start);
>
> @@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  			goto error_graph_walk_start;
>  	}
>
> -	media_graph_walk_start(&pipe->graph, entity);
> +	media_graph_walk_start(&pipe->graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(graph))) {
>  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
> @@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	 * Link validation on graph failed. We revert what we did and
>  	 * return the error.
>  	 */
> -	media_graph_walk_start(graph, entity_err);
> +	media_graph_walk_start(graph, entity_err->pads);
>
>  	while ((entity_err = media_graph_walk_next(graph))) {
>  		/* Sanity check for negative stream_count */
> @@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
>  	if (WARN_ON(!pipe))
>  		return;
>
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(graph))) {
>  		/* Sanity check for negative stream_count */
> diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
> index e025178db06c..5aaecdf92c53 100644
> --- a/drivers/media/platform/exynos4-is/media-dev.c
> +++ b/drivers/media/platform/exynos4-is/media-dev.c
> @@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	 * through active links. This is needed as we cannot power on/off the
>  	 * subdevs in random order.
>  	 */
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(graph))) {
>  		if (!is_media_entity_v4l2_video_device(entity))
> @@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	return 0;
>
>  err:
> -	media_graph_walk_start(graph, entity_err);
> +	media_graph_walk_start(graph, entity_err->pads);
>
>  	while ((entity_err = media_graph_walk_next(graph))) {
>  		if (!is_media_entity_v4l2_video_device(entity_err))
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index 8811d6dd4ee7..3c1485d59404 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
>  		return ret;
>  	}
>
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct isp_video *__video;
> diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
> index 044eb5778820..61e4fbaba7b7 100644
> --- a/drivers/media/platform/vsp1/vsp1_video.c
> +++ b/drivers/media/platform/vsp1/vsp1_video.c
> @@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
>  	if (ret)
>  		return ret;
>
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct v4l2_subdev *subdev;
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 2a56201cb853..d64c3bee8b95 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -190,7 +190,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
>  		return ret;
>  	}
>
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct xvip_dma *dma;
> diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
> index b01474717dca..d215fe31b9a2 100644
> --- a/drivers/media/v4l2-core/v4l2-mc.c
> +++ b/drivers/media/v4l2-core/v4l2-mc.c
> @@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
>  {
>  	int use = 0;
>
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(graph))) {
>  		if (is_media_entity_v4l2_video_device(entity))
> @@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
>  	if (!change)
>  		return 0;
>
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>
>  	while (!ret && (entity = media_graph_walk_next(graph)))
>  		if (is_media_entity_v4l2_subdev(entity))
> @@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
>  	if (!ret)
>  		return ret;
>
> -	media_graph_walk_start(graph, first);
> +	media_graph_walk_start(graph, first->pads);
>
>  	while ((first = media_graph_walk_next(graph))
>  	       && first != entity)
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index d0da083deed5..760cd0ab1feb 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
>  		return NULL;
>  	}
>
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		if (entity == &video->video.entity)
> @@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  		goto err_media_pipeline_start;
>
>  	mutex_lock(&mdev->graph_mutex);
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  	while ((entity = media_graph_walk_next(&graph)))
>  		media_entity_enum_set(&pipe->ent_enum, entity);
>  	mutex_unlock(&mdev->graph_mutex);
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 09737b47881f..b9bfcf34eb0a 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
>  void media_graph_walk_cleanup(struct media_graph *graph);
>
>  /**
> - * media_graph_walk_start - Start walking the media graph at a
> - *	given entity
> + * media_graph_walk_start - Start walking the media graph at a given pad
>   *
>   * @graph: Media graph structure that will be used to walk the graph
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
>   * Before using this function, media_graph_walk_init() must be
>   * used to allocate resources used for walking the graph. This
>   * function initializes the graph traversal structure to walk the
> - * entities graph starting at the given entity. The traversal
> + * entities graph starting at the given pad. The traversal
>   * structure must not be modified by the caller during graph
>   * traversal. After the graph walk, the resources must be released
>   * using media_graph_walk_cleanup().
>   */
> -void media_graph_walk_start(struct media_graph *graph,
> -			    struct media_entity *entity);
> +void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
>
>  /**
>   * media_graph_walk_next - Get the next entity in the graph
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 03/27] media: entity: Walk the graph based on pads
  2021-05-24 10:43 ` [PATCH v7 03/27] media: entity: Walk the graph based on pads Tomi Valkeinen
@ 2021-07-08 10:48   ` Jacopo Mondi
  0 siblings, 0 replies; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-08 10:48 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,

On Mon, May 24, 2021 at 01:43:44PM +0300, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> Instead of iterating over graph entities during the walk, iterate the pads
> through which the entity was first reached. This is required in order to
> make the entity pipeline pad-based rather than entity based.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst    |  7 ++-
>  drivers/media/mc/mc-entity.c                  | 51 +++++++++++--------
>  drivers/media/platform/exynos4-is/media-dev.c | 20 ++++----
>  drivers/media/platform/omap3isp/ispvideo.c    | 17 ++++---
>  drivers/media/platform/vsp1/vsp1_video.c      | 12 ++---
>  drivers/media/platform/xilinx/xilinx-dma.c    | 12 ++---
>  drivers/media/v4l2-core/v4l2-mc.c             | 24 ++++-----
>  drivers/staging/media/omap4iss/iss_video.c    | 36 ++++++-------
>  include/media/media-entity.h                  |  7 +--
>  9 files changed, 101 insertions(+), 85 deletions(-)
>
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index ba0aee982124..8a13640bed56 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -169,8 +169,11 @@ Drivers initiate a graph traversal by calling
>  The graph structure, provided by the caller, is initialized to start graph
>  traversal at the given pad in an entity.
>
> -Drivers can then retrieve the next entity by calling
> -:c:func:`media_graph_walk_next()`
> +Drivers can then retrieve the next pad by calling
> +:c:func:`media_graph_walk_next()`. Only the pad through which the entity
> +is first reached is returned. If the caller is interested in knowing which
> +further pads would be connected, the :c:func:`media_entity_has_route()`

Not a big deal either, but this will be introduced later too

> +function can be used for that.
>
>  When the graph traversal is complete the function will return ``NULL``.
>
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 0d58634b6fa8..32045671eaff 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -341,9 +341,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
>  	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
>  }
>
> -struct media_entity *media_graph_walk_next(struct media_graph *graph)
> +struct media_pad *media_graph_walk_next(struct media_graph *graph)
>  {
> -	struct media_entity *entity;
> +	struct media_pad *pad;
>
>  	if (stack_top(graph) == NULL)
>  		return NULL;
> @@ -356,11 +356,11 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
>  	while (link_top(graph) != &stack_top(graph)->entity->links)
>  		media_graph_walk_iter(graph);
>
> -	entity = stack_pop(graph)->entity;
> -	dev_dbg(entity->graph_obj.mdev->dev,
> -		"walk: returning entity '%s'\n", entity->name);
> +	pad = stack_pop(graph);
> +	dev_dbg(pad->graph_obj.mdev->dev,
> +		"walk: returning pad '%s':%u\n", pad->entity->name, pad->index);
>
> -	return entity;
> +	return pad;
>  }
>  EXPORT_SYMBOL_GPL(media_graph_walk_next);
>
> @@ -408,7 +408,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  {
>  	struct media_device *mdev = entity->graph_obj.mdev;
>  	struct media_graph *graph = &pipe->graph;
> -	struct media_entity *entity_err = entity;
> +	struct media_pad *pad = entity->pads;
> +	struct media_pad *pad_err = pad;
>  	struct media_link *link;
>  	int ret;
>
> @@ -418,9 +419,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  			goto error_graph_walk_start;
>  	}
>
> -	media_graph_walk_start(&pipe->graph, entity->pads);
> +	media_graph_walk_start(&pipe->graph, pad);
> +
> +	while ((pad = media_graph_walk_next(graph))) {
> +		struct media_entity *entity = pad->entity;
>
> -	while ((entity = media_graph_walk_next(graph))) {
>  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
>  		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
>
> @@ -429,7 +432,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  		if (entity->pipe && entity->pipe != pipe) {
>  			pr_err("Pipe active for %s. Can't start for %s\n",
>  				entity->name,
> -				entity_err->name);
> +				pad_err->entity->name);
>  			ret = -EBUSY;
>  			goto error;
>  		}
> @@ -447,26 +450,27 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  		bitmap_fill(has_no_links, entity->num_pads);
>
>  		list_for_each_entry(link, &entity->links, list) {
> -			struct media_pad *pad = link->sink->entity == entity
> -						? link->sink : link->source;
> +			struct media_pad *other_pad =
> +				link->sink->entity == entity ?
> +				link->sink : link->source;
>
>  			/* Mark that a pad is connected by a link. */
> -			bitmap_clear(has_no_links, pad->index, 1);
> +			bitmap_clear(has_no_links, other_pad->index, 1);
>
>  			/*
>  			 * Pads that either do not need to connect or
>  			 * are connected through an enabled link are
>  			 * fine.
>  			 */
> -			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
> +			if (!(other_pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
>  			    link->flags & MEDIA_LNK_FL_ENABLED)
> -				bitmap_set(active, pad->index, 1);
> +				bitmap_set(active, other_pad->index, 1);
>
>  			/*
>  			 * Link validation will only take place for
>  			 * sink ends of the link that are enabled.
>  			 */
> -			if (link->sink != pad ||
> +			if (link->sink != other_pad ||
>  			    !(link->flags & MEDIA_LNK_FL_ENABLED))
>  				continue;
>
> @@ -502,9 +506,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	 * Link validation on graph failed. We revert what we did and
>  	 * return the error.
>  	 */
> -	media_graph_walk_start(graph, entity_err->pads);
> +	media_graph_walk_start(graph, pad_err);
> +
> +	while ((pad_err = media_graph_walk_next(graph))) {
> +		struct media_entity *entity_err = pad_err->entity;
>
> -	while ((entity_err = media_graph_walk_next(graph))) {
>  		/* Sanity check for negative stream_count */
>  		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
>  			entity_err->stream_count--;
> @@ -516,7 +522,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  		 * We haven't increased stream_count further than this
>  		 * so we quit here.
>  		 */
> -		if (entity_err == entity)
> +		if (pad_err == pad)
>  			break;
>  	}
>
> @@ -543,8 +549,9 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
>
>  void __media_pipeline_stop(struct media_entity *entity)
>  {
> -	struct media_graph *graph = &entity->pipe->graph;
>  	struct media_pipeline *pipe = entity->pipe;
> +	struct media_graph *graph = &pipe->graph;
> +	struct media_pad *pad;
>
>  	/*
>  	 * If the following check fails, the driver has performed an
> @@ -555,7 +562,9 @@ void __media_pipeline_stop(struct media_entity *entity)
>
>  	media_graph_walk_start(graph, entity->pads);
>
> -	while ((entity = media_graph_walk_next(graph))) {
> +	while ((pad = media_graph_walk_next(graph))) {
> +		struct media_entity *entity = pad->entity;
> +
>  		/* Sanity check for negative stream_count */
>  		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
>  			entity->stream_count--;
> diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
> index 5aaecdf92c53..8938a6367703 100644
> --- a/drivers/media/platform/exynos4-is/media-dev.c
> +++ b/drivers/media/platform/exynos4-is/media-dev.c
> @@ -1164,7 +1164,7 @@ static int __fimc_md_modify_pipeline(struct media_entity *entity, bool enable)
>  static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  				      struct media_graph *graph)
>  {
> -	struct media_entity *entity_err = entity;
> +	struct media_pad *pad, *pad_err = entity->pads;
>  	int ret;
>
>  	/*
> @@ -1173,13 +1173,13 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	 * through active links. This is needed as we cannot power on/off the
>  	 * subdevs in random order.
>  	 */
> -	media_graph_walk_start(graph, entity->pads);
> +	media_graph_walk_start(graph, pad_err);
>
> -	while ((entity = media_graph_walk_next(graph))) {
> -		if (!is_media_entity_v4l2_video_device(entity))
> +	while ((pad = media_graph_walk_next(graph))) {
> +		if (!is_media_entity_v4l2_video_device(pad->entity))
>  			continue;
>
> -		ret  = __fimc_md_modify_pipeline(entity, enable);
> +		ret  = __fimc_md_modify_pipeline(pad->entity, enable);
>
>  		if (ret < 0)
>  			goto err;
> @@ -1188,15 +1188,15 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	return 0;
>
>  err:
> -	media_graph_walk_start(graph, entity_err->pads);
> +	media_graph_walk_start(graph, pad_err);
>
> -	while ((entity_err = media_graph_walk_next(graph))) {
> -		if (!is_media_entity_v4l2_video_device(entity_err))
> +	while ((pad_err = media_graph_walk_next(graph))) {
> +		if (!is_media_entity_v4l2_video_device(pad_err->entity))
>  			continue;
>
> -		__fimc_md_modify_pipeline(entity_err, !enable);
> +		__fimc_md_modify_pipeline(pad_err->entity, !enable);
>
> -		if (entity_err == entity)
> +		if (pad_err == pad)
>  			break;
>  	}
>
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index 3c1485d59404..5c1cbb1a9003 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -222,8 +222,8 @@ static int isp_video_get_graph_data(struct isp_video *video,
>  				    struct isp_pipeline *pipe)
>  {
>  	struct media_graph graph;
> -	struct media_entity *entity = &video->video.entity;
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pad *pad = video->video.entity.pads;
> +	struct media_device *mdev = video->video.entity.graph_obj.mdev;
>  	struct isp_video *far_end = NULL;
>  	int ret;
>
> @@ -234,23 +234,24 @@ static int isp_video_get_graph_data(struct isp_video *video,
>  		return ret;
>  	}
>
> -	media_graph_walk_start(&graph, entity->pads);
> +	media_graph_walk_start(&graph, pad);
>
> -	while ((entity = media_graph_walk_next(&graph))) {
> +	while ((pad = media_graph_walk_next(&graph))) {
>  		struct isp_video *__video;
>
> -		media_entity_enum_set(&pipe->ent_enum, entity);
> +		media_entity_enum_set(&pipe->ent_enum, pad->entity);
>
>  		if (far_end != NULL)
>  			continue;
>
> -		if (entity == &video->video.entity)
> +		if (pad == video->video.entity.pads)
>  			continue;
>
> -		if (!is_media_entity_v4l2_video_device(entity))
> +		if (!is_media_entity_v4l2_video_device(pad->entity))
>  			continue;
>
> -		__video = to_isp_video(media_entity_to_video_device(entity));
> +		__video = to_isp_video(media_entity_to_video_device(
> +					       pad->entity));
>  		if (__video->type != video->type)
>  			far_end = __video;
>  	}
> diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
> index 61e4fbaba7b7..f2c36f2fdf53 100644
> --- a/drivers/media/platform/vsp1/vsp1_video.c
> +++ b/drivers/media/platform/vsp1/vsp1_video.c
> @@ -559,8 +559,8 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
>  				     struct vsp1_video *video)
>  {
>  	struct media_graph graph;
> -	struct media_entity *entity = &video->video.entity;
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pad *pad = video->video.entity.pads;
> +	struct media_device *mdev = video->video.entity.graph_obj.mdev;
>  	unsigned int i;
>  	int ret;
>
> @@ -569,17 +569,17 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
>  	if (ret)
>  		return ret;
>
> -	media_graph_walk_start(&graph, entity->pads);
> +	media_graph_walk_start(&graph, pad);
>
> -	while ((entity = media_graph_walk_next(&graph))) {
> +	while ((pad = media_graph_walk_next(&graph))) {
>  		struct v4l2_subdev *subdev;
>  		struct vsp1_rwpf *rwpf;
>  		struct vsp1_entity *e;
>
> -		if (!is_media_entity_v4l2_subdev(entity))
> +		if (!is_media_entity_v4l2_subdev(pad->entity))
>  			continue;
>
> -		subdev = media_entity_to_v4l2_subdev(entity);
> +		subdev = media_entity_to_v4l2_subdev(pad->entity);
>  		e = to_vsp1_entity(subdev);
>  		list_add_tail(&e->list_pipe, &pipe->entities);
>  		e->pipe = pipe;
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index d64c3bee8b95..8df3c43aecbe 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -175,8 +175,8 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
>  				  struct xvip_dma *start)
>  {
>  	struct media_graph graph;
> -	struct media_entity *entity = &start->video.entity;
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pad *pad = start->video.entity.pads;
> +	struct media_device *mdev = pad->entity->graph_obj.mdev;
>  	unsigned int num_inputs = 0;
>  	unsigned int num_outputs = 0;
>  	int ret;
> @@ -190,15 +190,15 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
>  		return ret;
>  	}
>
> -	media_graph_walk_start(&graph, entity->pads);
> +	media_graph_walk_start(&graph, pad);
>
> -	while ((entity = media_graph_walk_next(&graph))) {
> +	while ((pad = media_graph_walk_next(&graph))) {
>  		struct xvip_dma *dma;
>
> -		if (entity->function != MEDIA_ENT_F_IO_V4L)
> +		if (pad->entity->function != MEDIA_ENT_F_IO_V4L)
>  			continue;
>
> -		dma = to_xvip_dma(media_entity_to_video_device(entity));
> +		dma = to_xvip_dma(media_entity_to_video_device(pad->entity));
>
>  		if (dma->pad.flags & MEDIA_PAD_FL_SINK) {
>  			pipe->output = dma;
> diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
> index d215fe31b9a2..cbeb580c6754 100644
> --- a/drivers/media/v4l2-core/v4l2-mc.c
> +++ b/drivers/media/v4l2-core/v4l2-mc.c
> @@ -434,13 +434,14 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
>  static int pipeline_pm_use_count(struct media_entity *entity,
>  	struct media_graph *graph)
>  {
> +	struct media_pad *pad;
>  	int use = 0;
>
>  	media_graph_walk_start(graph, entity->pads);
>
> -	while ((entity = media_graph_walk_next(graph))) {
> -		if (is_media_entity_v4l2_video_device(entity))
> -			use += entity->use_count;
> +	while ((pad = media_graph_walk_next(graph))) {
> +		if (is_media_entity_v4l2_video_device(pad->entity))
> +			use += pad->entity->use_count;
>  	}
>
>  	return use;
> @@ -493,7 +494,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
>  static int pipeline_pm_power(struct media_entity *entity, int change,
>  	struct media_graph *graph)
>  {
> -	struct media_entity *first = entity;
> +	struct media_pad *tmp_pad, *pad;
>  	int ret = 0;
>
>  	if (!change)
> @@ -501,19 +502,18 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
>
>  	media_graph_walk_start(graph, entity->pads);
>
> -	while (!ret && (entity = media_graph_walk_next(graph)))
> -		if (is_media_entity_v4l2_subdev(entity))
> -			ret = pipeline_pm_power_one(entity, change);
> +	while (!ret && (pad = media_graph_walk_next(graph)))
> +		if (is_media_entity_v4l2_subdev(pad->entity))
> +			ret = pipeline_pm_power_one(pad->entity, change);
>
>  	if (!ret)
>  		return ret;
>
> -	media_graph_walk_start(graph, first->pads);
> +	media_graph_walk_start(graph, entity->pads);
>
> -	while ((first = media_graph_walk_next(graph))
> -	       && first != entity)
> -		if (is_media_entity_v4l2_subdev(first))
> -			pipeline_pm_power_one(first, -change);
> +	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
> +		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
> +			pipeline_pm_power_one(tmp_pad->entity, -change);
>
>  	return ret;
>  }
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index 760cd0ab1feb..8c25ad73a81e 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -206,8 +206,8 @@ static struct iss_video *
>  iss_video_far_end(struct iss_video *video)
>  {
>  	struct media_graph graph;
> -	struct media_entity *entity = &video->video.entity;
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pad *pad = video->video.entity.pads;
> +	struct media_device *mdev = video->video.entity.graph_obj.mdev;
>  	struct iss_video *far_end = NULL;
>
>  	mutex_lock(&mdev->graph_mutex);
> @@ -217,16 +217,17 @@ iss_video_far_end(struct iss_video *video)
>  		return NULL;
>  	}
>
> -	media_graph_walk_start(&graph, entity->pads);
> +	media_graph_walk_start(&graph, pad);
>
> -	while ((entity = media_graph_walk_next(&graph))) {
> -		if (entity == &video->video.entity)
> +	while ((pad = media_graph_walk_next(&graph))) {
> +		if (pad->entity == &video->video.entity)
>  			continue;
>
> -		if (!is_media_entity_v4l2_video_device(entity))
> +		if (!is_media_entity_v4l2_video_device(pad->entity))
>  			continue;
>
> -		far_end = to_iss_video(media_entity_to_video_device(entity));
> +		far_end = to_iss_video(media_entity_to_video_device(
> +						pad->entity));
>  		if (far_end->type != video->type)
>  			break;
>
> @@ -853,8 +854,8 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	struct iss_video_fh *vfh = to_iss_video_fh(fh);
>  	struct iss_video *video = video_drvdata(file);
>  	struct media_graph graph;
> -	struct media_entity *entity = &video->video.entity;
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_pad *pad = video->video.entity.pads;
> +	struct media_device *mdev = video->video.entity.graph_obj.mdev;
>  	enum iss_pipeline_state state;
>  	struct iss_pipeline *pipe;
>  	struct iss_video *far_end;
> @@ -870,31 +871,32 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	 * Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = entity->pipe
> -	     ? to_iss_pipeline(entity) : &video->pipe;
> +	pipe = pad->entity->pipe
> +	     ? to_iss_pipeline(pad->entity) : &video->pipe;
>  	pipe->external = NULL;
>  	pipe->external_rate = 0;
>  	pipe->external_bpp = 0;
>
> -	ret = media_entity_enum_init(&pipe->ent_enum, entity->graph_obj.mdev);
> +	ret = media_entity_enum_init(&pipe->ent_enum,
> +				     pad->entity->graph_obj.mdev);
>  	if (ret)
>  		goto err_graph_walk_init;
>
> -	ret = media_graph_walk_init(&graph, entity->graph_obj.mdev);
> +	ret = media_graph_walk_init(&graph, pad->entity->graph_obj.mdev);
>  	if (ret)
>  		goto err_graph_walk_init;
>
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, true);
>
> -	ret = media_pipeline_start(entity, &pipe->pipe);
> +	ret = media_pipeline_start(pad->entity, &pipe->pipe);
>  	if (ret < 0)
>  		goto err_media_pipeline_start;
>
>  	mutex_lock(&mdev->graph_mutex);
> -	media_graph_walk_start(&graph, entity->pads);
> -	while ((entity = media_graph_walk_next(&graph)))
> -		media_entity_enum_set(&pipe->ent_enum, entity);
> +	media_graph_walk_start(&graph, pad);
> +	while ((pad = media_graph_walk_next(&graph)))
> +		media_entity_enum_set(&pipe->ent_enum, pad->entity);
>  	mutex_unlock(&mdev->graph_mutex);
>
>  	/*
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 5b55d6179e13..926fd201eae3 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -926,10 +926,11 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
>   * The graph structure must have been previously initialized with a call to
>   * media_graph_walk_start().
>   *
> - * Return: returns the next entity in the graph or %NULL if the whole graph
> - * have been traversed.
> + * Return: returns the next entity in the graph, identified by the pad through
> + * which it has been reached. If the whole graph has been traversed, return
> + * %NULL.
>   */
> -struct media_entity *media_graph_walk_next(struct media_graph *graph);
> +struct media_pad *media_graph_walk_next(struct media_graph *graph);
>
>  /**
>   * media_pipeline_start - Mark a pipeline as streaming
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline
  2021-05-24 10:43 ` [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline Tomi Valkeinen
@ 2021-07-08 12:36   ` Jacopo Mondi
  2021-07-11 15:25     ` Sakari Ailus
  2021-07-16  6:35     ` Tomi Valkeinen
  0 siblings, 2 replies; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-08 12:36 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hello Tomi,
    A few minors and a question below

On Mon, May 24, 2021 at 01:43:48PM +0300, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> The pipeline has been moved from the entity to the pads; reflect this in
> the media pipeline function API.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst    |  6 ++--
>  drivers/media/mc/mc-entity.c                  | 24 ++++++-------
>  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
>  .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
>  .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
>  drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
>  drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
>  .../media/platform/qcom/camss/camss-video.c   |  6 ++--
>  drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
>  .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
>  .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
>  drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
>  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
>  .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
>  drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
>  drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
>  drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
>  .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
>  drivers/media/usb/au0828/au0828-core.c        |  8 ++---
>  drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
>  drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
>  drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
>  drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
>  include/media/media-entity.h                  | 34 +++++++++----------
>  24 files changed, 98 insertions(+), 100 deletions(-)
>
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 8a13640bed56..69a64279a61f 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -213,11 +213,11 @@ When starting streaming, drivers must notify all entities in the pipeline to
>  prevent link states from being modified during streaming by calling
>  :c:func:`media_pipeline_start()`.
>
> -The function will mark all entities connected to the given entity through
> -enabled links, either directly or indirectly, as streaming.
> +The function will mark all entities connected to the given pad through

As the stream_count counter is now moved to the pads, should this be

+The function will mark all the pads connected to the given pad through

> +enabled routes and links, either directly or indirectly, as streaming.
>
>  The struct media_pipeline instance pointed to by
> -the pipe argument will be stored in every entity in the pipeline.
> +the pipe argument will be stored in every pad in the pipeline.
>  Drivers should embed the struct media_pipeline

Does this still apply ?

>  in higher-level pipeline structures and can then access the
>  pipeline through the struct media_entity

This sentence should probably be changed to

pipeline through the struct media_pad pipe field.

> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index ea1cf7f63ae8..e6451903359c 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
>   * Pipeline management
>   */
>
> -__must_check int __media_pipeline_start(struct media_entity *entity,
> +__must_check int __media_pipeline_start(struct media_pad *pad,
>  					struct media_pipeline *pipe)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->graph_obj.mdev;
>  	struct media_graph *graph = &pipe->graph;
> -	struct media_pad *pad = entity->pads;
>  	struct media_pad *pad_err = pad;
>  	struct media_link *link;
>  	int ret;
> @@ -542,24 +541,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_start);
>
> -__must_check int media_pipeline_start(struct media_entity *entity,
> +__must_check int media_pipeline_start(struct media_pad *pad,
>  				      struct media_pipeline *pipe)

As it seems that even with the full series applied
media_pipeline_start() is always called with entity->pads as its first
argument, I wonder if it wouldn't be more linear for a driver to keep
using entity and have this function here pass the entity's pads to
__media_pipeline_start().

Do we expect drivers to actually start the pipeline using a specific
pad ?

Thanks
   j

>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->graph_obj.mdev;
>  	int ret;
>
>  	mutex_lock(&mdev->graph_mutex);
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(pad, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	return ret;
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_start);
>
> -void __media_pipeline_stop(struct media_entity *entity)
> +void __media_pipeline_stop(struct media_pad *pad)
>  {
> -	struct media_pipeline *pipe = entity->pads->pipe;
> +	struct media_pipeline *pipe = pad->pipe;
>  	struct media_graph *graph = &pipe->graph;
> -	struct media_pad *pad;
>
>  	/*
>  	 * If the following check fails, the driver has performed an
> @@ -568,7 +566,7 @@ void __media_pipeline_stop(struct media_entity *entity)
>  	if (WARN_ON(!pipe))
>  		return;
>
> -	media_graph_walk_start(graph, entity->pads);
> +	media_graph_walk_start(graph, pad);
>
>  	while ((pad = media_graph_walk_next(graph))) {
>  		struct media_entity *entity = pad->entity;
> @@ -590,12 +588,12 @@ void __media_pipeline_stop(struct media_entity *entity)
>  }
>  EXPORT_SYMBOL_GPL(__media_pipeline_stop);
>
> -void media_pipeline_stop(struct media_entity *entity)
> +void media_pipeline_stop(struct media_pad *pad)
>  {
> -	struct media_device *mdev = entity->graph_obj.mdev;
> +	struct media_device *mdev = pad->graph_obj.mdev;
>
>  	mutex_lock(&mdev->graph_mutex);
> -	__media_pipeline_stop(entity);
> +	__media_pipeline_stop(pad);
>  	mutex_unlock(&mdev->graph_mutex);
>  }
>  EXPORT_SYMBOL_GPL(media_pipeline_stop);
> diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> index c2644efe4dbd..e0e1d8da70af 100644
> --- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> +++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
> @@ -982,7 +982,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		return r;
>  	}
>
> -	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
> +	r = media_pipeline_start(q->vdev.entity.pads, &q->pipe);
>  	if (r)
>  		goto fail_pipeline;
>
> @@ -1002,7 +1002,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  fail_csi2_subdev:
>  	cio2_hw_exit(cio2, q);
>  fail_hw:
> -	media_pipeline_stop(&q->vdev.entity);
> +	media_pipeline_stop(q->vdev.entity.pads);
>  fail_pipeline:
>  	dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r);
>  	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
> @@ -1023,7 +1023,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
>  	cio2_hw_exit(cio2, q);
>  	synchronize_irq(cio2->pci_dev->irq);
>  	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
> -	media_pipeline_stop(&q->vdev.entity);
> +	media_pipeline_stop(q->vdev.entity.pads);
>  	pm_runtime_put(&cio2->pci_dev->dev);
>  	cio2->streaming = false;
>  }
> diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
> index 7ff4024003f4..eaac0ac4e406 100644
> --- a/drivers/media/platform/exynos4-is/fimc-capture.c
> +++ b/drivers/media/platform/exynos4-is/fimc-capture.c
> @@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
>  	mutex_lock(&fimc->lock);
>
>  	if (close && vc->streaming) {
> -		media_pipeline_stop(&vc->ve.vdev.entity);
> +		media_pipeline_stop(vc->ve.vdev.entity.pads);
>  		vc->streaming = false;
>  	}
>
> @@ -1184,7 +1184,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
>  	if (fimc_capture_active(fimc))
>  		return -EBUSY;
>
> -	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
> +	ret = media_pipeline_start(entity->pads, &vc->ve.pipe->mp);
>  	if (ret < 0)
>  		return ret;
>
> @@ -1218,7 +1218,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
>  	}
>
>  err_p_stop:
> -	media_pipeline_stop(entity);
> +	media_pipeline_stop(entity->pads);
>  	return ret;
>  }
>
> @@ -1234,7 +1234,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
>  		return ret;
>
>  	if (vc->streaming) {
> -		media_pipeline_stop(&vc->ve.vdev.entity);
> +		media_pipeline_stop(vc->ve.vdev.entity.pads);
>  		vc->streaming = false;
>  	}
>
> diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
> index 8d9dc597deaa..ad8195e0b8f2 100644
> --- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
> +++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
> @@ -309,7 +309,7 @@ static int isp_video_release(struct file *file)
>  	mutex_lock(&isp->video_lock);
>
>  	if (v4l2_fh_is_singular_file(file) && ivc->streaming) {
> -		media_pipeline_stop(entity);
> +		media_pipeline_stop(entity->pads);
>  		ivc->streaming = 0;
>  	}
>
> @@ -490,7 +490,7 @@ static int isp_video_streamon(struct file *file, void *priv,
>  	struct media_entity *me = &ve->vdev.entity;
>  	int ret;
>
> -	ret = media_pipeline_start(me, &ve->pipe->mp);
> +	ret = media_pipeline_start(me->pads, &ve->pipe->mp);
>  	if (ret < 0)
>  		return ret;
>
> @@ -505,7 +505,7 @@ static int isp_video_streamon(struct file *file, void *priv,
>  	isp->video_capture.streaming = 1;
>  	return 0;
>  p_stop:
> -	media_pipeline_stop(me);
> +	media_pipeline_stop(me->pads);
>  	return ret;
>  }
>
> @@ -520,7 +520,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
>  	if (ret < 0)
>  		return ret;
>
> -	media_pipeline_stop(&video->ve.vdev.entity);
> +	media_pipeline_stop(video->ve.vdev.entity.pads);
>  	video->streaming = 0;
>  	return 0;
>  }
> diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
> index 67bfb1ad2ba2..a979600ff6e6 100644
> --- a/drivers/media/platform/exynos4-is/fimc-lite.c
> +++ b/drivers/media/platform/exynos4-is/fimc-lite.c
> @@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
>  	if (v4l2_fh_is_singular_file(file) &&
>  	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
>  		if (fimc->streaming) {
> -			media_pipeline_stop(entity);
> +			media_pipeline_stop(entity->pads);
>  			fimc->streaming = false;
>  		}
>  		fimc_lite_stop_capture(fimc, false);
> @@ -822,7 +822,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
>  	if (fimc_lite_active(fimc))
>  		return -EBUSY;
>
> -	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
> +	ret = media_pipeline_start(entity->pads, &fimc->ve.pipe->mp);
>  	if (ret < 0)
>  		return ret;
>
> @@ -839,7 +839,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
>  	}
>
>  err_p_stop:
> -	media_pipeline_stop(entity);
> +	media_pipeline_stop(entity->pads);
>  	return 0;
>  }
>
> @@ -853,7 +853,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
>  	if (ret < 0)
>  		return ret;
>
> -	media_pipeline_stop(&fimc->ve.vdev.entity);
> +	media_pipeline_stop(fimc->ve.vdev.entity.pads);
>  	fimc->streaming = false;
>  	return 0;
>  }
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index a8438040c4aa..87334477f223 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -1105,7 +1105,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
>  	pipe->max_rate = pipe->l3_ick;
>
> -	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
> +	ret = media_pipeline_start(video->video.entity.pads, &pipe->pipe);
>  	if (ret < 0)
>  		goto err_pipeline_start;
>
> @@ -1162,7 +1162,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	return 0;
>
>  err_check_format:
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>  err_pipeline_start:
>  	/* TODO: Implement PM QoS */
>  	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
> @@ -1229,7 +1229,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
>  	video->error = false;
>
>  	/* TODO: Implement PM QoS */
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>
>  	media_entity_enum_cleanup(&pipe->ent_enum);
>
> diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
> index f282275af626..5cd494f17589 100644
> --- a/drivers/media/platform/qcom/camss/camss-video.c
> +++ b/drivers/media/platform/qcom/camss/camss-video.c
> @@ -491,7 +491,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
>  	struct v4l2_subdev *subdev;
>  	int ret;
>
> -	ret = media_pipeline_start(&vdev->entity, &video->pipe);
> +	ret = media_pipeline_start(vdev->entity.pads, &video->pipe);
>  	if (ret < 0)
>  		return ret;
>
> @@ -520,7 +520,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
>  	return 0;
>
>  error:
> -	media_pipeline_stop(&vdev->entity);
> +	media_pipeline_stop(vdev->entity.pads);
>
>  	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
>
> @@ -551,7 +551,7 @@ static void video_stop_streaming(struct vb2_queue *q)
>  		v4l2_subdev_call(subdev, video, s_stream, 0);
>  	}
>
> -	media_pipeline_stop(&vdev->entity);
> +	media_pipeline_stop(vdev->entity.pads);
>
>  	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
>  }
> diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
> index 7994262c9b63..2d930e53c1e2 100644
> --- a/drivers/media/platform/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/rcar-vin/rcar-dma.c
> @@ -1215,7 +1215,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	sd = media_entity_to_v4l2_subdev(pad->entity);
>
>  	if (!on) {
> -		media_pipeline_stop(&vin->vdev.entity);
> +		media_pipeline_stop(vin->vdev.entity.pads);
>  		return v4l2_subdev_call(sd, video, s_stream, 0);
>  	}
>
> @@ -1232,7 +1232,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	mdev = vin->vdev.entity.graph_obj.mdev;
>  	mutex_lock(&mdev->graph_mutex);
>  	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
> -	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
> +	ret = __media_pipeline_start(vin->vdev.entity.pads, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	if (ret)
>  		return ret;
> @@ -1241,7 +1241,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	if (ret == -ENOIOCTLCMD)
>  		ret = 0;
>  	if (ret)
> -		media_pipeline_stop(&vin->vdev.entity);
> +		media_pipeline_stop(vin->vdev.entity.pads);
>
>  	return ret;
>  }
> diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> index 5f6c9d1623e4..c2a11fcc9709 100644
> --- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> +++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
> @@ -979,7 +979,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
>
>  	rkisp1_dummy_buf_destroy(cap);
>
> -	media_pipeline_stop(&node->vdev.entity);
> +	media_pipeline_stop(node->vdev.entity.pads);
>
>  	mutex_unlock(&cap->rkisp1->stream_lock);
>  }
> @@ -993,7 +993,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
>
>  	mutex_lock(&cap->rkisp1->stream_lock);
>
> -	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
> +	ret = media_pipeline_start(entity->pads, &cap->rkisp1->pipe);
>  	if (ret) {
>  		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
>  		goto err_ret_buffers;
> @@ -1030,7 +1030,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
>  err_destroy_dummy:
>  	rkisp1_dummy_buf_destroy(cap);
>  err_pipeline_stop:
> -	media_pipeline_stop(entity);
> +	media_pipeline_stop(entity->pads);
>  err_ret_buffers:
>  	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
>  	mutex_unlock(&cap->rkisp1->stream_lock);
> diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
> index 3a8d0b05c117..77ad4d3d0d48 100644
> --- a/drivers/media/platform/s3c-camif/camif-capture.c
> +++ b/drivers/media/platform/s3c-camif/camif-capture.c
> @@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
>  	if (s3c_vp_active(vp))
>  		return 0;
>
> -	ret = media_pipeline_start(sensor, camif->m_pipeline);
> +	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
>  	if (ret < 0)
>  		return ret;
>
>  	ret = camif_pipeline_validate(camif);
>  	if (ret < 0) {
> -		media_pipeline_stop(sensor);
> +		media_pipeline_stop(sensor->pads);
>  		return ret;
>  	}
>
> @@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
>
>  	ret = vb2_streamoff(&vp->vb_queue, type);
>  	if (ret == 0)
> -		media_pipeline_stop(&camif->sensor.sd->entity);
> +		media_pipeline_stop(camif->sensor.sd->entity.pads);
>  	return ret;
>  }
>
> diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c
> index 6f163e301eb1..14a86f3826b8 100644
> --- a/drivers/media/platform/stm32/stm32-dcmi.c
> +++ b/drivers/media/platform/stm32/stm32-dcmi.c
> @@ -730,7 +730,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		goto err_pm_put;
>  	}
>
> -	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
> +	ret = media_pipeline_start(dcmi->vdev->entity.pads, &dcmi->pipeline);
>  	if (ret < 0) {
>  		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
>  			__func__, ret);
> @@ -844,7 +844,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	dcmi_pipeline_stop(dcmi);
>
>  err_media_pipeline_stop:
> -	media_pipeline_stop(&dcmi->vdev->entity);
> +	media_pipeline_stop(dcmi->vdev->entity.pads);
>
>  err_pm_put:
>  	pm_runtime_put(dcmi->dev);
> @@ -870,7 +870,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
>
>  	dcmi_pipeline_stop(dcmi);
>
> -	media_pipeline_stop(&dcmi->vdev->entity);
> +	media_pipeline_stop(dcmi->vdev->entity.pads);
>
>  	spin_lock_irq(&dcmi->irqlock);
>
> diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> index 2c39cd7f2862..be0defdf74f1 100644
> --- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> +++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
> @@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  		goto err_clear_dma_queue;
>  	}
>
> -	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
> +	ret = media_pipeline_start(csi->vdev.entity.pads, &csi->vdev.pipe);
>  	if (ret < 0)
>  		goto err_free_scratch_buffer;
>
> @@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	sun4i_csi_capture_stop(csi);
>
>  err_disable_pipeline:
> -	media_pipeline_stop(&csi->vdev.entity);
> +	media_pipeline_stop(csi->vdev.entity.pads);
>
>  err_free_scratch_buffer:
>  	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
> @@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
>  	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
>  	spin_unlock_irqrestore(&csi->qlock, flags);
>
> -	media_pipeline_stop(&csi->vdev.entity);
> +	media_pipeline_stop(csi->vdev.entity.pads);
>
>  	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
>  			  csi->scratch.paddr);
> diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> index 3181d0781b61..537057a75eaa 100644
> --- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> +++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
> @@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
>
>  	video->sequence = 0;
>
> -	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
> +	ret = media_pipeline_start(video->vdev.entity.pads, &video->vdev.pipe);
>  	if (ret < 0)
>  		goto clear_dma_queue;
>
> @@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
>  stop_csi_stream:
>  	sun6i_csi_set_stream(video->csi, false);
>  stop_media_pipeline:
> -	media_pipeline_stop(&video->vdev.entity);
> +	media_pipeline_stop(video->vdev.entity.pads);
>  clear_dma_queue:
>  	spin_lock_irqsave(&video->dma_queue_lock, flags);
>  	list_for_each_entry(buf, &video->dma_queue, list)
> @@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
>
>  	sun6i_csi_set_stream(video->csi, false);
>
> -	media_pipeline_stop(&video->vdev.entity);
> +	media_pipeline_stop(video->vdev.entity.pads);
>
>  	/* Release all active buffers */
>  	spin_lock_irqsave(&video->dma_queue_lock, flags);
> diff --git a/drivers/media/platform/ti-vpe/cal-video.c b/drivers/media/platform/ti-vpe/cal-video.c
> index 7b7436a355ee..ca75b54311a8 100644
> --- a/drivers/media/platform/ti-vpe/cal-video.c
> +++ b/drivers/media/platform/ti-vpe/cal-video.c
> @@ -675,7 +675,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	dma_addr_t addr;
>  	int ret;
>
> -	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
> +	ret = media_pipeline_start(ctx->vdev.entity.pads, &ctx->phy->pipe);
>  	if (ret < 0) {
>  		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
>  		goto error_release_buffers;
> @@ -719,7 +719,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	pm_runtime_put_sync(ctx->cal->dev);
>
>  error_pipeline:
> -	media_pipeline_stop(&ctx->vdev.entity);
> +	media_pipeline_stop(ctx->vdev.entity.pads);
>  error_release_buffers:
>  	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
>
> @@ -738,7 +738,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
>
>  	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
>
> -	media_pipeline_stop(&ctx->vdev.entity);
> +	media_pipeline_stop(ctx->vdev.entity.pads);
>  }
>
>  static const struct vb2_ops cal_video_qops = {
> diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
> index f2c36f2fdf53..978f820b0f34 100644
> --- a/drivers/media/platform/vsp1/vsp1_video.c
> +++ b/drivers/media/platform/vsp1/vsp1_video.c
> @@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
>  	}
>  	mutex_unlock(&pipe->lock);
>
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>  	vsp1_video_release_buffers(video);
>  	vsp1_video_pipeline_put(pipe);
>  }
> @@ -1048,7 +1048,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  		return PTR_ERR(pipe);
>  	}
>
> -	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
> +	ret = __media_pipeline_start(video->video.entity.pads, &pipe->pipe);
>  	if (ret < 0) {
>  		mutex_unlock(&mdev->graph_mutex);
>  		goto err_pipe;
> @@ -1072,7 +1072,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	return 0;
>
>  err_stop:
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>  err_pipe:
>  	vsp1_video_pipeline_put(pipe);
>  	return ret;
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 7fa0467dddde..07074eda5f70 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -406,7 +406,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	pipe = dma->video.entity.pads->pipe
>  	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
>
> -	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
> +	ret = media_pipeline_start(dma->video.entity.pads, &pipe->pipe);
>  	if (ret < 0)
>  		goto error;
>
> @@ -432,7 +432,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	return 0;
>
>  error_stop:
> -	media_pipeline_stop(&dma->video.entity);
> +	media_pipeline_stop(dma->video.entity.pads);
>
>  error:
>  	/* Give back all queued buffers to videobuf2. */
> @@ -460,7 +460,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
>
>  	/* Cleanup the pipeline and mark it as being stopped. */
>  	xvip_pipeline_cleanup(pipe);
> -	media_pipeline_stop(&dma->video.entity);
> +	media_pipeline_stop(dma->video.entity.pads);
>
>  	/* Give back all queued buffers to videobuf2. */
>  	spin_lock_irq(&dma->queued_lock);
> diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
> index 5e9fd902cd37..10724b0a868c 100644
> --- a/drivers/media/test-drivers/vimc/vimc-capture.c
> +++ b/drivers/media/test-drivers/vimc/vimc-capture.c
> @@ -246,7 +246,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	vcap->sequence = 0;
>
>  	/* Start the media pipeline */
> -	ret = media_pipeline_start(entity, &vcap->stream.pipe);
> +	ret = media_pipeline_start(entity->pads, &vcap->stream.pipe);
>  	if (ret) {
>  		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
>  		return ret;
> @@ -254,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
>
>  	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
>  	if (ret) {
> -		media_pipeline_stop(entity);
> +		media_pipeline_stop(entity->pads);
>  		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
>  		return ret;
>  	}
> @@ -273,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
>  	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
>
>  	/* Stop the media pipeline */
> -	media_pipeline_stop(&vcap->vdev.entity);
> +	media_pipeline_stop(vcap->vdev.entity.pads);
>
>  	/* Release all active buffers */
>  	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
> diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
> index a8a72d5fbd12..93dd7bb0ece0 100644
> --- a/drivers/media/usb/au0828/au0828-core.c
> +++ b/drivers/media/usb/au0828/au0828-core.c
> @@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
>  		goto end;
>  	}
>
> -	ret = __media_pipeline_start(entity, pipe);
> +	ret = __media_pipeline_start(entity->pads, pipe);
>  	if (ret) {
>  		pr_err("Start Pipeline: %s->%s Error %d\n",
>  			source->name, entity->name, ret);
> @@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
>  				return;
>
>  			/* stop pipeline */
> -			__media_pipeline_stop(dev->active_link_owner);
> +			__media_pipeline_stop(dev->active_link_owner->pads);
>  			pr_debug("Pipeline stop for %s\n",
>  				dev->active_link_owner->name);
>
>  			ret = __media_pipeline_start(
> -					dev->active_link_user,
> +					dev->active_link_user->pads,
>  					dev->active_link_user_pipe);
>  			if (ret) {
>  				pr_err("Start Pipeline: %s->%s %d\n",
> @@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
>  			return;
>
>  		/* stop pipeline */
> -		__media_pipeline_stop(dev->active_link_owner);
> +		__media_pipeline_stop(dev->active_link_owner->pads);
>  		pr_debug("Pipeline stop for %s\n",
>  			dev->active_link_owner->name);
>
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 535da4dda3c6..74218af45551 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  	mutex_lock(&imxmd->md.graph_mutex);
>
>  	if (on) {
> -		ret = __media_pipeline_start(entity, &imxmd->pipe);
> +		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
>  		if (ret)
>  			goto out;
>  		ret = v4l2_subdev_call(sd, video, s_stream, 1);
>  		if (ret)
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
>  		if (entity->pads->pipe)
> -			__media_pipeline_stop(entity);
> +			__media_pipeline_stop(entity->pads);
>  	}
>
>  out:
> diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
> index 38a240764509..db5867da3f11 100644
> --- a/drivers/staging/media/ipu3/ipu3-v4l2.c
> +++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
> @@ -485,7 +485,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>
>  	pipe = node->pipe;
>  	imgu_pipe = &imgu->imgu_pipe[pipe];
> -	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
> +	r = media_pipeline_start(node->vdev.entity.pads, &imgu_pipe->pipeline);
>  	if (r < 0)
>  		goto fail_return_bufs;
>
> @@ -510,7 +510,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	return 0;
>
>  fail_stop_pipeline:
> -	media_pipeline_stop(&node->vdev.entity);
> +	media_pipeline_stop(node->vdev.entity.pads);
>  fail_return_bufs:
>  	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
>
> @@ -550,7 +550,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
>  	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
>  	mutex_unlock(&imgu->streaming_lock);
>
> -	media_pipeline_stop(&node->vdev.entity);
> +	media_pipeline_stop(node->vdev.entity.pads);
>  }
>
>  /******************** v4l2_ioctl_ops ********************/
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index b74f7891711d..20fac40581c6 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -889,7 +889,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, true);
>
> -	ret = media_pipeline_start(pad->entity, &pipe->pipe);
> +	ret = media_pipeline_start(pad, &pipe->pipe);
>  	if (ret < 0)
>  		goto err_media_pipeline_start;
>
> @@ -980,7 +980,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  err_omap4iss_set_stream:
>  	vb2_streamoff(&vfh->queue, type);
>  err_iss_video_check_format:
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>  err_media_pipeline_start:
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, false);
> @@ -1034,7 +1034,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
>
>  	if (video->iss->pdata->set_constraints)
>  		video->iss->pdata->set_constraints(video->iss, false);
> -	media_pipeline_stop(&video->video.entity);
> +	media_pipeline_stop(video->video.entity.pads);
>
>  done:
>  	mutex_unlock(&video->stream_lock);
> diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
> index f10a041e3e6c..d2d7dd0e8624 100644
> --- a/drivers/staging/media/tegra-video/tegra210.c
> +++ b/drivers/staging/media/tegra-video/tegra210.c
> @@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
>  		       VI_INCR_SYNCPT_NO_STALL);
>
>  	/* start the pipeline */
> -	ret = media_pipeline_start(&chan->video.entity, pipe);
> +	ret = media_pipeline_start(chan->video.entity.pads, pipe);
>  	if (ret < 0)
>  		goto error_pipeline_start;
>
> @@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
>  error_kthread_start:
>  	tegra_channel_set_stream(chan, false);
>  error_set_stream:
> -	media_pipeline_stop(&chan->video.entity);
> +	media_pipeline_stop(chan->video.entity.pads);
>  error_pipeline_start:
>  	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
>  	return ret;
> @@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
>
>  	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
>  	tegra_channel_set_stream(chan, false);
> -	media_pipeline_stop(&chan->video.entity);
> +	media_pipeline_stop(chan->video.entity.pads);
>  }
>
>  /*
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index c9d97c902d05..516d73a2941e 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -937,53 +937,53 @@ struct media_pad *media_graph_walk_next(struct media_graph *graph);
>
>  /**
>   * media_pipeline_start - Mark a pipeline as streaming
> - * @entity: Starting entity
> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + * @pad: Starting pad
> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>   *
> - * Mark all entities connected to a given entity through enabled links, either
> - * directly or indirectly, as streaming. The given pipeline object is assigned
> - * to every entity in the pipeline and stored in the media_entity pipe field.
> + * Mark all pads connected to a given pad through enabled routes or links,
> + * either directly or indirectly, as streaming. The given pipeline object is
> + * assigned to every pad in the pipeline and stored in the media_pad pipe
> + * field.
>   *
>   * Calls to this function can be nested, in which case the same number of
>   * media_pipeline_stop() calls will be required to stop streaming. The
>   * pipeline pointer must be identical for all nested calls to
>   * media_pipeline_start().
>   */
> -__must_check int media_pipeline_start(struct media_entity *entity,
> +__must_check int media_pipeline_start(struct media_pad *pad,
>  				      struct media_pipeline *pipe);
>  /**
>   * __media_pipeline_start - Mark a pipeline as streaming
>   *
> - * @entity: Starting entity
> - * @pipe: Media pipeline to be assigned to all entities in the pipeline.
> + * @pad: Starting pad
> + * @pipe: Media pipeline to be assigned to all pads in the pipeline.
>   *
>   * ..note:: This is the non-locking version of media_pipeline_start()
>   */
> -__must_check int __media_pipeline_start(struct media_entity *entity,
> +__must_check int __media_pipeline_start(struct media_pad *pad,
>  					struct media_pipeline *pipe);
>
>  /**
>   * media_pipeline_stop - Mark a pipeline as not streaming
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
> - * Mark all entities connected to a given entity through enabled links, either
> - * directly or indirectly, as not streaming. The media_entity pipe field is
> - * reset to %NULL.
> + * Mark all pads connected to a given pad through enabled routes or links,
> + * either directly or indirectly, as not streaming.
>   *
>   * If multiple calls to media_pipeline_start() have been made, the same
>   * number of calls to this function are required to mark the pipeline as not
> - * streaming.
> + * streaming and reset the media_pad pipe field to %NULL.
>   */
> -void media_pipeline_stop(struct media_entity *entity);
> +void media_pipeline_stop(struct media_pad *pad);
>
>  /**
>   * __media_pipeline_stop - Mark a pipeline as not streaming
>   *
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
>   * .. note:: This is the non-locking version of media_pipeline_stop()
>   */
> -void __media_pipeline_stop(struct media_entity *entity);
> +void __media_pipeline_stop(struct media_pad *pad);
>
>  /**
>   * media_devnode_create() - creates and initializes a device node interface
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 08/27] media: entity: Add has_route entity operation
  2021-05-24 10:43 ` [PATCH v7 08/27] media: entity: Add has_route entity operation Tomi Valkeinen
@ 2021-07-08 12:43   ` Jacopo Mondi
  2021-07-11 15:26     ` Sakari Ailus
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-08 12:43 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

Hi Tomi,
   a small note

On Mon, May 24, 2021 at 01:43:49PM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> The optional operation can be used by entities to report whether two
> pads are internally connected.
>
> While at there, fix a Sphinx compiler warning in a comment block a few
> lines above.
>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/media-entity.h | 7 +++++++
>  1 file changed, 7 insertions(+)
>
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 516d73a2941e..ad4020b2df65 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -187,6 +187,7 @@ enum media_pad_signal_type {
>   * @flags:	Pad flags, as defined in
>   *		:ref:`include/uapi/linux/media.h <media_header>`
>   *		(seek for ``MEDIA_PAD_FL_*``)
> + *
>   * .. note::
>   *
>   *    @stream_count reference count must never be negative, but is a signed
> @@ -214,6 +215,10 @@ struct media_pad {
>   * @link_validate:	Return whether a link is valid from the entity point of
>   *			view. The media_pipeline_start() function
>   *			validates all links by calling this operation. Optional.
> + * @has_route:		Return whether a route exists inside the entity between
> + *			two given pads. Pads are passed to the operation ordered
> + *			by index. Optional: If the operation isn't implemented

According to the next patch, this doesn't seem to be 'Optional:' :)

Thanks
   j

> + *			all pads will be considered as connected.
>   *
>   * .. note::
>   *
> @@ -227,6 +232,8 @@ struct media_entity_operations {
>  			  const struct media_pad *local,
>  			  const struct media_pad *remote, u32 flags);
>  	int (*link_validate)(struct media_link *link);
> +	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
> +			  unsigned int pad1);
>  };
>
>  /**
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads
  2021-05-24 10:43 ` [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads Tomi Valkeinen
@ 2021-07-08 13:11   ` Jacopo Mondi
  2021-07-16  6:19     ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-08 13:11 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hello again,

On Mon, May 24, 2021 at 01:43:47PM +0300, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> This moves the pipe and stream_count fields from struct media_entity to
> struct media_pad. Effectively streams become pad-specific rather than
> being entity specific, allowing several independent streams to traverse a
> single entity and an entity to be part of several streams.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
>
> - Update documentation to use 'pads'
> - Use the media pad iterator in media_entity.c
> - Update rcar-dma.c to use the new per-pad stream count
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>
> - Fix cleanup in the error path of __media_pipeline_start()
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> ---
>  drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
>  drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
>  drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
>  drivers/media/platform/omap3isp/isp.c         |  2 +-
>  drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
>  drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
>  drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
>  drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
>  drivers/staging/media/imx/imx-media-utils.c   |  2 +-
>  drivers/staging/media/omap4iss/iss.c          |  2 +-
>  drivers/staging/media/omap4iss/iss_video.c    |  2 +-
>  drivers/staging/media/omap4iss/iss_video.h    |  2 +-
>  include/media/media-entity.h                  | 21 +++---
>  15 files changed, 73 insertions(+), 56 deletions(-)
>
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index 40ae9b6bac47..ea1cf7f63ae8 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -424,24 +424,28 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>
>  	while ((pad = media_graph_walk_next(graph))) {
>  		struct media_entity *entity = pad->entity;
> +		bool skip_validation = pad->pipe != NULL;
> +		struct media_pad *iter;
>
>  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
>  		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
>
> -		entity->stream_count++;
> -
> -		if (entity->pipe && entity->pipe != pipe) {
> -			pr_err("Pipe active for %s. Can't start for %s\n",
> -				entity->name,
> -				pad_err->entity->name);
> -			ret = -EBUSY;
> -			goto error;
> +		media_entity_for_each_pad(entity, iter) {
> +			if (iter->pipe && iter->pipe != pipe) {
> +				pr_err("Pipe active for %s. Can't start for %s\n",
> +				       entity->name, iter->entity->name);
> +				ret = -EBUSY;
> +			} else {
> +				iter->pipe = pipe;
> +			}
> +			iter->stream_count++;
>  		}
>
> -		entity->pipe = pipe;
> +		if (ret)

ret is not initialized when declared and there is a code path that
could lead here without assigning it. I would initialize it to 0 when
declaring it, or even re-assign it to 0 at the beginning of this while
loop.

Thanks
   j

> +			goto error;
>
> -		/* Already streaming --- no need to check. */
> -		if (entity->stream_count > 1)
> +		/* Already part of the pipeline, skip validation. */
> +		if (skip_validation)
>  			continue;
>
>  		if (!entity->ops || !entity->ops->link_validate)
> @@ -510,20 +514,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	media_graph_walk_start(graph, pad_err);
>
>  	while ((pad_err = media_graph_walk_next(graph))) {
> -		struct media_entity *entity_err = pad_err->entity;
> -
> -		/* Sanity check for negative stream_count */
> -		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
> -			entity_err->stream_count--;
> -			if (entity_err->stream_count == 0)
> -				entity_err->pipe = NULL;
> +		struct media_entity *entity = pad_err->entity;
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			/* Sanity check for negative stream_count */
> +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> +				--iter->stream_count;
> +				if (iter->stream_count == 0)
> +					iter->pipe = NULL;
> +			}
>  		}
>
>  		/*
>  		 * We haven't increased stream_count further than this
>  		 * so we quit here.
>  		 */
> -		if (pad_err == pad)
> +		if (pad_err->entity == pad->entity)
>  			break;
>  	}
>
> @@ -550,7 +557,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
>
>  void __media_pipeline_stop(struct media_entity *entity)
>  {
> -	struct media_pipeline *pipe = entity->pipe;
> +	struct media_pipeline *pipe = entity->pads->pipe;
>  	struct media_graph *graph = &pipe->graph;
>  	struct media_pad *pad;
>
> @@ -565,12 +572,15 @@ void __media_pipeline_stop(struct media_entity *entity)
>
>  	while ((pad = media_graph_walk_next(graph))) {
>  		struct media_entity *entity = pad->entity;
> -
> -		/* Sanity check for negative stream_count */
> -		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
> -			entity->stream_count--;
> -			if (entity->stream_count == 0)
> -				entity->pipe = NULL;
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			/* Sanity check for negative stream_count */
> +			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
> +				iter->stream_count--;
> +				if (iter->stream_count == 0)
> +					iter->pipe = NULL;
> +			}
>  		}
>  	}
>
> @@ -840,7 +850,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  {
>  	const u32 mask = MEDIA_LNK_FL_ENABLED;
>  	struct media_device *mdev;
> -	struct media_entity *source, *sink;
> +	struct media_pad *source, *sink;
>  	int ret = -EBUSY;
>
>  	if (link == NULL)
> @@ -856,8 +866,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
>  	if (link->flags == flags)
>  		return 0;
>
> -	source = link->source->entity;
> -	sink = link->sink->entity;
> +	source = link->source;
> +	sink = link->sink;
>
>  	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
>  	    (source->stream_count || sink->stream_count))
> diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
> index 855235bea46d..80274e29ccc5 100644
> --- a/drivers/media/platform/exynos4-is/fimc-isp.c
> +++ b/drivers/media/platform/exynos4-is/fimc-isp.c
> @@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
>  			}
>  		}
>  	} else {
> -		if (sd->entity.stream_count == 0) {
> +		if (sd->entity.pads->stream_count == 0) {
>  			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
>  				struct v4l2_subdev_format format = *fmt;
>
> diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
> index aaa3af0493ce..67bfb1ad2ba2 100644
> --- a/drivers/media/platform/exynos4-is/fimc-lite.c
> +++ b/drivers/media/platform/exynos4-is/fimc-lite.c
> @@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
>  	mutex_lock(&fimc->lock);
>
>  	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
> -	    sd->entity.stream_count > 0) ||
> +	    sd->entity.pads->stream_count > 0) ||
>  	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
>  	    vb2_is_busy(&fimc->vb_queue))) {
>  		mutex_unlock(&fimc->lock);
> diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
> index 53025c8c7531..5d6021ac708b 100644
> --- a/drivers/media/platform/omap3isp/isp.c
> +++ b/drivers/media/platform/omap3isp/isp.c
> @@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
>  	struct isp_pipeline *pipe;
>  	struct media_pad *pad;
>
> -	if (!me->pipe)
> +	if (!me->pads->pipe)
>  		return 0;
>  	pipe = to_isp_pipeline(me);
>  	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index 5c1cbb1a9003..a8438040c4aa 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	/* Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = video->video.entity.pipe
> +	pipe = video->video.entity.pads->pipe
>  	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
>
>  	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
> diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
> index a0908670c0cf..4c9c5b719ec5 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.h
> +++ b/drivers/media/platform/omap3isp/ispvideo.h
> @@ -100,7 +100,7 @@ struct isp_pipeline {
>  };
>
>  #define to_isp_pipeline(__e) \
> -	container_of((__e)->pipe, struct isp_pipeline, pipe)
> +	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
>
>  static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
>  {
> diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
> index cb3025992817..07ec008aacc4 100644
> --- a/drivers/media/platform/rcar-vin/rcar-core.c
> +++ b/drivers/media/platform/rcar-vin/rcar-core.c
> @@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
>  		return 0;
>
>  	/*
> -	 * Don't allow link changes if any entity in the graph is
> -	 * streaming, modifying the CHSEL register fields can disrupt
> -	 * running streams.
> +	 * Don't allow link changes if any stream in the graph is active as
> +	 * modifying the CHSEL register fields can disrupt running streams.
>  	 */
> -	media_device_for_each_entity(entity, &group->mdev)
> -		if (entity->stream_count)
> -			return -EBUSY;
> +	media_device_for_each_entity(entity, &group->mdev) {
> +		struct media_pad *iter;
> +
> +		media_entity_for_each_pad(entity, iter) {
> +			if (iter->stream_count)
> +				return -EBUSY;
> +		}
> +	}
>
>  	mutex_lock(&group->lock);
>
> diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
> index f30dafbdf61c..7994262c9b63 100644
> --- a/drivers/media/platform/rcar-vin/rcar-dma.c
> +++ b/drivers/media/platform/rcar-vin/rcar-dma.c
> @@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
>  	 */
>  	mdev = vin->vdev.entity.graph_obj.mdev;
>  	mutex_lock(&mdev->graph_mutex);
> -	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
> +	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
>  	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
>  	mutex_unlock(&mdev->graph_mutex);
>  	if (ret)
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 8df3c43aecbe..7fa0467dddde 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -403,7 +403,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
>  	 * Use the pipeline object embedded in the first DMA object that starts
>  	 * streaming.
>  	 */
> -	pipe = dma->video.entity.pipe
> +	pipe = dma->video.entity.pads->pipe
>  	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
>
>  	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
> index 2378bdae57ae..69ced71a5696 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.h
> +++ b/drivers/media/platform/xilinx/xilinx-dma.h
> @@ -47,7 +47,7 @@ struct xvip_pipeline {
>
>  static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
>  {
> -	return container_of(e->pipe, struct xvip_pipeline, pipe);
> +	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
>  }
>
>  /**
> diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
> index 6f90acf9c725..535da4dda3c6 100644
> --- a/drivers/staging/media/imx/imx-media-utils.c
> +++ b/drivers/staging/media/imx/imx-media-utils.c
> @@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
>  			__media_pipeline_stop(entity);
>  	} else {
>  		v4l2_subdev_call(sd, video, s_stream, 0);
> -		if (entity->pipe)
> +		if (entity->pads->pipe)
>  			__media_pipeline_stop(entity);
>  	}
>
> diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
> index 68588e9dab0b..4c6f25aa8b57 100644
> --- a/drivers/staging/media/omap4iss/iss.c
> +++ b/drivers/staging/media/omap4iss/iss.c
> @@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
>  	struct iss_pipeline *pipe;
>  	struct media_pad *pad;
>
> -	if (!me->pipe)
> +	if (!me->pads->pipe)
>  		return 0;
>  	pipe = to_iss_pipeline(me);
>  	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index 8c25ad73a81e..b74f7891711d 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  	 * Start streaming on the pipeline. No link touching an entity in the
>  	 * pipeline can be activated or deactivated once streaming is started.
>  	 */
> -	pipe = pad->entity->pipe
> +	pipe = pad->pipe
>  	     ? to_iss_pipeline(pad->entity) : &video->pipe;
>  	pipe->external = NULL;
>  	pipe->external_rate = 0;
> diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
> index 526281bf0051..9b8ec27bf87d 100644
> --- a/drivers/staging/media/omap4iss/iss_video.h
> +++ b/drivers/staging/media/omap4iss/iss_video.h
> @@ -91,7 +91,7 @@ struct iss_pipeline {
>  };
>
>  #define to_iss_pipeline(__e) \
> -	container_of((__e)->pipe, struct iss_pipeline, pipe)
> +	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
>
>  static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
>  {
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 5f6eed24e63f..c9d97c902d05 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -180,15 +180,24 @@ enum media_pad_signal_type {
>   *
>   * @graph_obj:	Embedded structure containing the media object common data
>   * @entity:	Entity this pad belongs to
> + * @pipe:	Pipeline this pad belongs to
> + * @stream_count: Stream count for the pad
>   * @index:	Pad index in the entity pads array, numbered from 0 to n
>   * @sig_type:	Type of the signal inside a media pad
>   * @flags:	Pad flags, as defined in
>   *		:ref:`include/uapi/linux/media.h <media_header>`
>   *		(seek for ``MEDIA_PAD_FL_*``)
> + * .. note::
> + *
> + *    @stream_count reference count must never be negative, but is a signed
> + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> + *    detect reference count bugs that would make it negative.
>   */
>  struct media_pad {
>  	struct media_gobj graph_obj;	/* must be first field in struct */
>  	struct media_entity *entity;
> +	struct media_pipeline *pipe;
> +	int stream_count;
>  	u16 index;
>  	enum media_pad_signal_type sig_type;
>  	unsigned long flags;
> @@ -267,9 +276,7 @@ enum media_entity_type {
>   * @pads:	Pads array with the size defined by @num_pads.
>   * @links:	List of data links.
>   * @ops:	Entity operations.
> - * @stream_count: Stream count for the entity.
>   * @use_count:	Use count for the entity.
> - * @pipe:	Pipeline this entity belongs to.
>   * @info:	Union with devnode information.  Kept just for backward
>   *		compatibility.
>   * @info.dev:	Contains device major and minor info.
> @@ -282,10 +289,9 @@ enum media_entity_type {
>   *
>   * .. note::
>   *
> - *    @stream_count and @use_count reference counts must never be
> - *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
> - *    check can be used to detect reference count bugs that would make them
> - *    negative.
> + *    @use_count reference count must never be negative, but is a signed
> + *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
> + *    detect reference count bugs that would make it negative.
>   */
>  struct media_entity {
>  	struct media_gobj graph_obj;	/* must be first field in struct */
> @@ -304,11 +310,8 @@ struct media_entity {
>
>  	const struct media_entity_operations *ops;
>
> -	int stream_count;
>  	int use_count;
>
> -	struct media_pipeline *pipe;
> -
>  	union {
>  		struct {
>  			u32 major;
> --
> 2.25.1
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate()
  2021-06-05 23:59     ` Laurent Pinchart
@ 2021-07-09 10:02       ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-09 10:02 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: linux-media, sakari.ailus, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi,

On 06/06/2021 02:59, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Fri, May 28, 2021 at 02:34:55PM +0300, Tomi Valkeinen wrote:
>> On 24/05/2021 13:44, Tomi Valkeinen wrote:
>>> Update v4l2_subdev_link_validate() to use routing and streams for
>>> validation.
>>>
>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>> ---
>>>    drivers/media/v4l2-core/v4l2-subdev.c | 184 +++++++++++++++++++++++---
>>>    1 file changed, 166 insertions(+), 18 deletions(-)
>>>
>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>> index da6ea9b14631..b30b456d8d99 100644
>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>> @@ -16,6 +16,7 @@
>>>    #include <linux/videodev2.h>
>>>    #include <linux/export.h>
>>>    #include <linux/version.h>
>>> +#include <linux/sort.h>
>>>    
>>>    #include <media/v4l2-ctrls.h>
>>>    #include <media/v4l2-device.h>
>>> @@ -894,6 +895,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>>>    
>>>    static int
>>>    v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>> +				     u32 stream,
>>>    				     struct v4l2_subdev_format *fmt)
>>>    {
>>>    	if (is_media_entity_v4l2_subdev(pad->entity)) {
>>> @@ -902,6 +904,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>>>    
>>>    		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>>>    		fmt->pad = pad->index;
>>> +		fmt->stream = stream;
>>>    		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
>>>    	}
>>>    
>>> @@ -1012,31 +1015,176 @@ bool v4l2_subdev_has_route(struct v4l2_subdev_krouting *routing,
>>>    }
>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
>>>    
>>> +static int cmp_u32(const void *a, const void *b)
>>> +{
>>> +	u32 a32 = *(u32 *)a;
>>> +	u32 b32 = *(u32 *)b;
>>> +
>>> +	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
>>> +}
>>> +
>>>    int v4l2_subdev_link_validate(struct media_link *link)
>>>    {
>>> -	struct v4l2_subdev *sink;
>>> -	struct v4l2_subdev_format sink_fmt, source_fmt;
>>> -	int rval;
>>> +	int ret;
>>> +	unsigned int i;
>>>    
> 
> No need for a blank line, and I'd move i and ret (in that order) after
> the variables below.
> 
>>> -	rval = v4l2_subdev_link_validate_get_format(
>>> -		link->source, &source_fmt);
>>> -	if (rval < 0)
>>> -		return 0;
>>> +	struct v4l2_subdev *source_subdev =
>>> +		media_entity_to_v4l2_subdev(link->source->entity);
>>> +	struct v4l2_subdev *sink_subdev =
>>> +		media_entity_to_v4l2_subdev(link->sink->entity);
>>> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
>>>    
> 
> No need for a blank line here either.
> 
>>> -	rval = v4l2_subdev_link_validate_get_format(
>>> -		link->sink, &sink_fmt);
>>> -	if (rval < 0)
>>> -		return 0;
>>> +	struct v4l2_subdev_krouting routing;
>>>    
> 
> Same.
> 
>>> -	sink = media_entity_to_v4l2_subdev(link->sink->entity);
>>> +	static const u32 default_streams[] = { 0 };
> 
> I'd move this one at the top.

Ok (to all above).

>>>    
>>> -	rval = v4l2_subdev_call(sink, pad, link_validate, link,
>>> -				&source_fmt, &sink_fmt);
>>> -	if (rval != -ENOIOCTLCMD)
>>> -		return rval;
>>> +	u32 num_source_streams = 0;
>>> +	const u32 *source_streams;
>>> +	u32 num_sink_streams = 0;
>>> +	const u32 *sink_streams;
>>> +
>>> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
>>> +		link->source->entity->name, link->source->index,
>>> +		link->sink->entity->name, link->sink->index);
>>> +
>>> +	/* Get source streams */
>>> +
>>> +	memset(&routing, 0, sizeof(routing));
>>> +
>>> +	ret = v4l2_subdev_get_routing(source_subdev, NULL, &routing);
>>> +
>>> +	if (ret && ret != -ENOIOCTLCMD)
>>> +		return ret;
>>> +
>>> +	if (ret == -ENOIOCTLCMD) {
>>> +		num_source_streams = 1;
>>> +		source_streams = default_streams;
>>> +	} else {
>>> +		u32 *streams;
>>> +
>>> +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
>>> +					GFP_KERNEL);
>>> +
>>> +		for (i = 0; i < routing.num_routes; ++i) {
>>> +			struct v4l2_subdev_route *route = &routing.routes[i];
>>> +
>>> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>>> +				continue;
>>> +
>>> +			if (route->source_pad == link->source->index)
>>> +				streams[num_source_streams++] =
>>> +					route->source_stream;
>>> +		}
>>> +
>>> +		sort(streams, num_source_streams, sizeof(u32), &cmp_u32, NULL);
>>> +
>>> +		source_streams = streams;
>>> +
>>> +		v4l2_subdev_free_routing(&routing);
>>> +	}
> 
> Could this be moved to a separate function ? The code below is very
> similar.

Yes. It makes handling of the default_streams a bit more complex, but 
it's still cleaner to have a separate func.

>>> +
>>> +	/* Get sink streams */
>>> +
>>> +	memset(&routing, 0, sizeof(routing));
>>> +
>>> +	ret = v4l2_subdev_get_routing(sink_subdev, NULL, &routing);
>>> +
>>> +	if (ret && ret != -ENOIOCTLCMD)
>>> +		return ret;
>>> +
>>> +	if (ret == -ENOIOCTLCMD) {
>>> +		num_sink_streams = 1;
>>> +		sink_streams = default_streams;
>>> +	} else {
>>> +		u32 *streams;
>>>    
>>> -	return v4l2_subdev_link_validate_default(
>>> -		sink, link, &source_fmt, &sink_fmt);
>>> +		streams = kmalloc_array(routing.num_routes, sizeof(u32),
>>> +					GFP_KERNEL);
>>> +
>>> +		for (i = 0; i < routing.num_routes; ++i) {
>>> +			struct v4l2_subdev_route *route = &routing.routes[i];
>>> +
>>> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
>>> +				continue;
>>> +
>>> +			if (route->sink_pad == link->sink->index)
>>> +				streams[num_sink_streams++] =
>>> +					route->sink_stream;
>>> +		}
>>> +
>>> +		sort(streams, num_sink_streams, sizeof(u32), &cmp_u32, NULL);
> 
> Are you aware that there can be duplicate in the streams array, as a
> given stream on a sink pad can be routed to multiple source pads ? I
> think that will fail the sink and source streams match test below.

Yes, I have fixed that issue.

>>> +
>>> +		sink_streams = streams;
>>> +
>>> +		v4l2_subdev_free_routing(&routing);
>>> +	}
>>> +
>>> +	if (num_source_streams != num_sink_streams) {
>>> +		dev_err(dev,
>>> +			"Sink and source stream count mismatch: %d vs %d\n",
>>> +			num_source_streams, num_sink_streams);
>>> +		return -EINVAL;
>>> +	}
>>> +
>>> +	/* Validate source and sink stream formats */
>>> +
>>> +	for (i = 0; i < num_source_streams; ++i) {
>>> +		struct v4l2_subdev_format sink_fmt, source_fmt;
>>> +		u32 stream;
>>> +		int ret;
>>> +
>>> +		if (source_streams[i] != sink_streams[i]) {
>>> +			dev_err(dev, "Sink and source streams do not match\n");
>>> +			return -EINVAL;
>>> +		}
> 
> What if the source subdev as a route enabled that produces a stream, and
> the sink subdev has not corresponding enabled route ? Isn't this a valid
> configuration ? For instance, when a CSI-2 sensor produces image data
> and embedded data in two streams with different CSI-2 DT, the embedded
> data doesn't have to be captured, it could be dropped in the CSI-2
> receiver.

Yes. Although in some cases the sink may not be able to drop the stream, 
but I don't think we can know that here. So I've fixed this to allow 
more source streams than sink streams.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-06-06  0:06 ` Laurent Pinchart
@ 2021-07-09 15:18   ` Jacopo Mondi
  2021-07-09 18:26     ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-09 15:18 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Hans Verkuil, Sakari Ailus, Tomi Valkeinen, linux-media,
	Jacopo Mondi, niklas.soderlund+renesas, Mauro Carvalho Chehab,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi, Laurent,

On Sun, Jun 06, 2021 at 03:06:18AM +0300, Laurent Pinchart wrote:
> Hi Hans, Sakari,
>
> We need your feedback on this series, at least on the general approach.
> There are quite a few issues to be addressed, and it makes no sense to
> invest time in this if you don't think this is a good direction.
>
> If anyone else wants to give feedback, speak now or forever hold your
> peace :-)

Since you ask...

Having been involved a bit as the n-th person that tried to bring this
to completion I spent a bit of time trying to recollect how the
previous approach worked and how it compares to this one. Sorry if
this goes in length.

I share Tomi's concern on one part of the previous version:

- The resulting device topology gets complicated in a non-trivial way.

  The typical example of having to model one image sensor that sends
  embedded data and images with three sub-devices speaks for itself, I
  presume.

  However in one way, I feel like this is somehow correct and provides
  a more accurate representation of the actual sensor architecture.
  Splitting a sensor into components would allow to better handle
  devices which supports multiple buses (typically CSI-2 and
  parallel) through the internal routing tables, and allows
  better control of the components of the image sensor. [1]

- Multiplexed source pads do not accept a format or any other configuration
  like crop/composing. Again this might seem odd, and it might be
  worth considering if those pads shouldn't be made 'special' somehow,
  but I again think it models a multiplexed bus quite accurately,
  doesn't it ? It's weird that the format of, in example, a CSI-2
  receiver source pad has to be propagated from the image sensor
  entity sink pad, crossing two entities, two routes and one
  media link. This makes rather complex to automate format propagation along
  pipelines, not only when done by abusing media-ctl like most people do,
  but also when done programmatically the task is not easy (I know I'm
  contradicting my [1] point here :)

  Also link validation is of course a bit more complex as shown by
  731facccc987 ("v4l: subdev: Take routing information into account in link validation")
  which was part of the previous series, but it's totally up to the
  core..

Moving everything to the pads by adding a 'stream' field basically
makes all pads potentially multiplexed, reducing the problem of format
configuration/validation to a 1-to-1 {pad, stream} pair validation
which allows to collapse the topology and maintain the current one.

Apart from the concerns expressed by Laurent (which I share but only
partially understand, as the implications of bulk moving the
v4l2-subdev configuration API to be stream-aware are not totally clear
to me yet) what I'm not convinced of is that now cross-entities
"routes" (or "streams") on a multiplexed bus do require a format
assigned, effectively exposing them to userspace, with the consequence
that the format configuration influences the routes setup up to the
point the two have to be kept consistent. The concept
could even be extended to inter-entities routes, as you suggested the
routing tables could even be dropped completely in this case, but I
feel mixing routing and format setup is a bit a layer violation and
forbids, in example, routing two streams to the same endpoint, which I
feel will be required to perform DT multiplexing on the same virtual
channel. The previous version had the multiplexed link configuration
completely hidden from userspace and controlled solely by the routing API,
which seems a tad more linear and offers more flexibility for drivers.

I'm not strongly pushing for one solution over the other, the only use
case I can reason on at the moment is a simple single-stream VC
multiplexing and both solutions works equally fine for that. This one
is certainly simpler regarding the required device topology.

Btw Tomi, do you have examples of drivers ported to your new proposal ?

Just my 2 cents, and sorry for the wall of text :)
    j

[1]
  The counter-argument about the additional complexity doesn't apply
  to drivers if not marginally but impacts userspace in a non
  negligeable way. To be honest, I do think this is only marginally an
  issue. As long as the single V4L2 apis (and v4l2-ctrls) were
  enough to be able to capture anything from the camera system the
  argument of the additional complexity held: a generic
  camera application could have worked with any (most) kind of devices and the
  platform specificities were abstracted away enough for such generic
  applications to exists. Honestly, with the introduction of the
  media-controller API and the v4l2-subdev APIs, I think we're well
  past that point. Userspace that controls complex devices
  has to be specialized to a point that an additional IOCTL and a more
  detailed knowledge of the topology is a rather small burden compared
  to the quantum leap the subsystem went through with the introduction
  of complex devices support.

>
> On Mon, May 24, 2021 at 01:43:41PM +0300, Tomi Valkeinen wrote:
> > Hi,
> >
> > This is v7 of the series, the previous one is:
> >
> > https://lore.kernel.org/linux-media/20210427124523.990938-1-tomi.valkeinen@ideasonboard.com/
> >
> > In this version I have changed the approach to multiplexed streams, and
> > I went with the approach described in the RFC I sent:
> >
> > https://lore.kernel.org/linux-media/20210507123558.146948-1-tomi.valkeinen@ideasonboard.com/
> >
> > The main change is that in this series each pad+stream combination can
> > have its own configuration, versus only pad having its own
> > configuration. In other words, a pad with 4 streams will contain 4
> > configurations.
> >
> > The patches up to and including "v4l: Add stream to frame descriptor"
> > are the same as previously, except changes done according to the review
> > comments. After that, the new pad+stream approach is implemented.
> >
> > This series is based on the subdev-wide state change:
> >
> > https://lore.kernel.org/linux-media/20210519091558.562318-1-tomi.valkeinen@ideasonboard.com/
> >
> >  Tomi
> >
> > Jacopo Mondi (2):
> >   media: entity: Add iterator helper for entity pads
> >   media: Documentation: Add GS_ROUTING documentation
> >
> > Laurent Pinchart (4):
> >   media: entity: Add has_route entity operation
> >   media: entity: Add media_entity_has_route() function
> >   media: entity: Use routing information during graph traversal
> >   v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations
> >
> > Sakari Ailus (13):
> >   media: entity: Use pad as a starting point for graph walk
> >   media: entity: Use pads instead of entities in the media graph walk
> >     stack
> >   media: entity: Walk the graph based on pads
> >   v4l: mc: Start walk from a specific pad in use count calculation
> >   media: entity: Move the pipeline from entity to pads
> >   media: entity: Use pad as the starting point for a pipeline
> >   media: entity: Skip link validation for pads to which there is no
> >     route
> >   media: entity: Add an iterator helper for connected pads
> >   media: entity: Add only connected pads to the pipeline
> >   media: entity: Add debug information in graph walk route check
> >   v4l: Add bus type to frame descriptors
> >   v4l: Add CSI-2 bus configuration to frame descriptors
> >   v4l: Add stream to frame descriptor
> >
> > Tomi Valkeinen (8):
> >   v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE
> >   v4l: subdev: routing kernel helper functions
> >   v4l: subdev: add stream based configuration
> >   v4l: subdev: add 'stream' to subdev ioctls
> >   v4l: subdev: use streams in v4l2_subdev_link_validate()
> >   v4l: subdev: add routing & stream config to v4l2_subdev_state
> >   v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED
> >   v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
> >
> >  Documentation/driver-api/media/mc-core.rst    |  15 +-
> >  .../userspace-api/media/v4l/dev-subdev.rst    | 128 ++++++
> >  .../userspace-api/media/v4l/user-func.rst     |   1 +
> >  .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
> >  .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
> >  .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
> >  .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
> >  .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
> >  .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
> >  .../media/v4l/vidioc-subdev-g-routing.rst     | 142 +++++++
> >  .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
> >  drivers/media/mc/mc-device.c                  |  13 +-
> >  drivers/media/mc/mc-entity.c                  | 257 +++++++-----
> >  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
> >  .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
> >  .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
> >  drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
> >  drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
> >  drivers/media/platform/exynos4-is/media-dev.c |  20 +-
> >  drivers/media/platform/omap3isp/isp.c         |   2 +-
> >  drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
> >  drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
> >  .../media/platform/qcom/camss/camss-video.c   |   6 +-
> >  drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
> >  drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
> >  .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
> >  .../media/platform/s3c-camif/camif-capture.c  |   6 +-
> >  drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
> >  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
> >  .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
> >  drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
> >  drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
> >  drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
> >  drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
> >  .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
> >  drivers/media/usb/au0828/au0828-core.c        |   8 +-
> >  drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
> >  drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
> >  drivers/media/v4l2-core/v4l2-subdev.c         | 396 +++++++++++++++++-
> >  drivers/staging/media/imx/imx-media-utils.c   |   8 +-
> >  drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
> >  drivers/staging/media/omap4iss/iss.c          |   2 +-
> >  drivers/staging/media/omap4iss/iss_video.c    |  40 +-
> >  drivers/staging/media/omap4iss/iss_video.h    |   2 +-
> >  drivers/staging/media/tegra-video/tegra210.c  |   6 +-
> >  include/media/media-entity.h                  | 142 +++++--
> >  include/media/v4l2-subdev.h                   | 204 ++++++++-
> >  include/uapi/linux/v4l2-subdev.h              |  76 +++-
> >  48 files changed, 1410 insertions(+), 334 deletions(-)
> >  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
>
> --
> Regards,
>
> Laurent Pinchart

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-07-09 15:18   ` Jacopo Mondi
@ 2021-07-09 18:26     ` Tomi Valkeinen
  2021-07-10  8:42       ` Jacopo Mondi
  0 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-09 18:26 UTC (permalink / raw)
  To: Jacopo Mondi, Laurent Pinchart
  Cc: Hans Verkuil, Sakari Ailus, linux-media, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Pratyush Yadav,
	Lokesh Vutla

Hi Jacopo,

On 09/07/2021 18:18, Jacopo Mondi wrote:
> Hi Tomi, Laurent,
> 
> On Sun, Jun 06, 2021 at 03:06:18AM +0300, Laurent Pinchart wrote:
>> Hi Hans, Sakari,
>>
>> We need your feedback on this series, at least on the general approach.
>> There are quite a few issues to be addressed, and it makes no sense to
>> invest time in this if you don't think this is a good direction.
>>
>> If anyone else wants to give feedback, speak now or forever hold your
>> peace :-)
> 
> Since you ask...
> 
> Having been involved a bit as the n-th person that tried to bring this
> to completion I spent a bit of time trying to recollect how the
> previous approach worked and how it compares to this one. Sorry if
> this goes in length.
> 
> I share Tomi's concern on one part of the previous version:
> 
> - The resulting device topology gets complicated in a non-trivial way.
> 
>    The typical example of having to model one image sensor that sends
>    embedded data and images with three sub-devices speaks for itself, I
>    presume.
> 
>    However in one way, I feel like this is somehow correct and provides
>    a more accurate representation of the actual sensor architecture.
>    Splitting a sensor into components would allow to better handle
>    devices which supports multiple buses (typically CSI-2 and
>    parallel) through the internal routing tables, and allows
>    better control of the components of the image sensor. [1]

I'm not sure what kind of setup you mean, but nothing prevents you from 
splitting devices into multiple subdevs with the new approach if it 
makes sense on your HW.

I have a parallel sensor that provides metadata on a line before the 
actual frame. I have hard time understanding why that should be split 
into 3 subdevs.

> - Multiplexed source pads do not accept a format or any other configuration
>    like crop/composing. Again this might seem odd, and it might be
>    worth considering if those pads shouldn't be made 'special' somehow,
>    but I again think it models a multiplexed bus quite accurately,
>    doesn't it ? It's weird that the format of, in example, a CSI-2
>    receiver source pad has to be propagated from the image sensor
>    entity sink pad, crossing two entities, two routes and one
>    media link. This makes rather complex to automate format propagation along
>    pipelines, not only when done by abusing media-ctl like most people do,
>    but also when done programmatically the task is not easy (I know I'm
>    contradicting my [1] point here :)

Hmm, but is it easy in the kernel side, then? I didn't feel so with the 
previous version. The kernel needed to travel the graph back and forth 
"all the time", just to figure out what's going on and where.

If the userspace understands the HW topology (as it more or less must), 
and it configures the routes (as it has to), and sets the formats on 
certain subdevs, then I don't see that it would have any issues in 
propagating the formats.

>    Also link validation is of course a bit more complex as shown by
>    731facccc987 ("v4l: subdev: Take routing information into account in link validation")
>    which was part of the previous series, but it's totally up to the
>    core..
> 
> Moving everything to the pads by adding a 'stream' field basically
> makes all pads potentially multiplexed, reducing the problem of format
> configuration/validation to a 1-to-1 {pad, stream} pair validation
> which allows to collapse the topology and maintain the current one.

Yes. I think I have problem understanding the counter arguments as I 
don't really see a difference with a) two subdevs, each with two 
non-multiplexed pads, linked 1-to-1 and b) two subdevs, each with one 
multiplexed pad, with two routes.

There is one particular issue I had with the previous version, which I 
think is a big reason I like the new approach:

I'm using TI CAL driver, which already exists in upstreams and supports 
both non-MC and MC-without-streams. Adding support for streams, i.e 
supporting non-MC, MC-without-streams and MC-with-streams made the 
driver an unholy mess (including a new module parameter to enable 
streams). With the new approach, the changes were relatively minor, as 
MC with and without streams are really the same thing.

With the previous approach you couldn't e.g. have a CSI2-RX bridge 
driver that would support both old, non-multiplexed CSI2 sensor drivers 
and multiplexed CSI2 sensor drivers. Unless you had something like the 
module parameter mentioned above. Or perhaps a DT property to define 
which mode the pad is in.

Also, one problem is that I really only have a single multiplexed HW 
setup, which limits my testing and the way I see multiplexed streams. 
That setup is "luckily" not the simplest one:

SoC CSI-2 RX <-> FPDLink Deserializer <-> FPDLink Serializer <-> Sensor

4 serializer+sensor cameras can be connected to the deserializer. Each 
sensor provides 2 streams (pixel and metadata). So I have 8 streams 
coming in to the SoC.

> Apart from the concerns expressed by Laurent (which I share but only
> partially understand, as the implications of bulk moving the
> v4l2-subdev configuration API to be stream-aware are not totally clear
> to me yet) what I'm not convinced of is that now cross-entities
> "routes" (or "streams") on a multiplexed bus do require a format
> assigned, effectively exposing them to userspace, with the consequence
> that the format configuration influences the routes setup up to the
> point the two have to be kept consistent. The concept
> could even be extended to inter-entities routes, as you suggested the
> routing tables could even be dropped completely in this case, but I
> feel mixing routing and format setup is a bit a layer violation and
> forbids, in example, routing two streams to the same endpoint, which I
> feel will be required to perform DT multiplexing on the same virtual
> channel. The previous version had the multiplexed link configuration
> completely hidden from userspace and controlled solely by the routing API,
> which seems a tad more linear and offers more flexibility for drivers.
> 
> I'm not strongly pushing for one solution over the other, the only use
> case I can reason on at the moment is a simple single-stream VC
> multiplexing and both solutions works equally fine for that. This one
> is certainly simpler regarding the required device topology.
> 
> Btw Tomi, do you have examples of drivers ported to your new proposal ?

Yes. They're a bit messy, but I can share them with the next version. 
I'm currently fixing a lot of things, and making full use of the new 
v4l2_subdev_state.

   Tomi



^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-07-09 18:26     ` Tomi Valkeinen
@ 2021-07-10  8:42       ` Jacopo Mondi
  2021-07-12  8:19         ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-10  8:42 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: Laurent Pinchart, Hans Verkuil, Sakari Ailus, linux-media,
	Jacopo Mondi, niklas.soderlund+renesas, Mauro Carvalho Chehab,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,
   thanks for you reply

On Fri, Jul 09, 2021 at 09:26:03PM +0300, Tomi Valkeinen wrote:
> Hi Jacopo,
>
> On 09/07/2021 18:18, Jacopo Mondi wrote:
> > Hi Tomi, Laurent,
> >
> > On Sun, Jun 06, 2021 at 03:06:18AM +0300, Laurent Pinchart wrote:
> > > Hi Hans, Sakari,
> > >
> > > We need your feedback on this series, at least on the general approach.
> > > There are quite a few issues to be addressed, and it makes no sense to
> > > invest time in this if you don't think this is a good direction.
> > >
> > > If anyone else wants to give feedback, speak now or forever hold your
> > > peace :-)
> >
> > Since you ask...
> >
> > Having been involved a bit as the n-th person that tried to bring this
> > to completion I spent a bit of time trying to recollect how the
> > previous approach worked and how it compares to this one. Sorry if
> > this goes in length.
> >
> > I share Tomi's concern on one part of the previous version:
> >
> > - The resulting device topology gets complicated in a non-trivial way.
> >
> >    The typical example of having to model one image sensor that sends
> >    embedded data and images with three sub-devices speaks for itself, I
> >    presume.
> >
> >    However in one way, I feel like this is somehow correct and provides
> >    a more accurate representation of the actual sensor architecture.
> >    Splitting a sensor into components would allow to better handle
> >    devices which supports multiple buses (typically CSI-2 and
> >    parallel) through the internal routing tables, and allows
> >    better control of the components of the image sensor. [1]
>
> I'm not sure what kind of setup you mean, but nothing prevents you from
> splitting devices into multiple subdevs with the new approach if it makes
> sense on your HW.

Nothing prevents it it from being done today, my point was that having
to do so to support mulitplexed streams is an incentive to get to a
more precise representation of the sensor architecture, not only a
cons :)

>
> I have a parallel sensor that provides metadata on a line before the actual
> frame. I have hard time understanding why that should be split into 3
> subdevs.
>

As I guess there's no way to extract that line of embedded data if not
from the frame when already in memory, I won't consider this the best
example of a multiplexed bus :)

> > - Multiplexed source pads do not accept a format or any other configuration
> >    like crop/composing. Again this might seem odd, and it might be
> >    worth considering if those pads shouldn't be made 'special' somehow,
> >    but I again think it models a multiplexed bus quite accurately,
> >    doesn't it ? It's weird that the format of, in example, a CSI-2
> >    receiver source pad has to be propagated from the image sensor
> >    entity sink pad, crossing two entities, two routes and one
> >    media link. This makes rather complex to automate format propagation along
> >    pipelines, not only when done by abusing media-ctl like most people do,
> >    but also when done programmatically the task is not easy (I know I'm
> >    contradicting my [1] point here :)
>
> Hmm, but is it easy in the kernel side, then? I didn't feel so with the
> previous version. The kernel needed to travel the graph back and forth "all
> the time", just to figure out what's going on and where.

Not for the core. You see the patch I referenced, I praise Sakari for
getting there, the validation is indeed complex.

I mean that for drivers it would be easier as the routing management
is separate from format management, and drivers do not have to match
endpoints by the format they have applied to infer routes.

>
> If the userspace understands the HW topology (as it more or less must), and
> it configures the routes (as it has to), and sets the formats on certain
> subdevs, then I don't see that it would have any issues in propagating the
> formats.
>

As I've said the fact that setting up a route is accomplished by
setting the same format on two endpoints feels like a layer violation.
For userspace traversing a route means matching the formats on a
possibly high number of {pad, stream} pairs. It won't be easy without
a dedicated API and feels rather error prone for drivers too if they
have to configure they internal routing based on format information

> >    Also link validation is of course a bit more complex as shown by
> >    731facccc987 ("v4l: subdev: Take routing information into account in link validation")
> >    which was part of the previous series, but it's totally up to the
> >    core..
> >
> > Moving everything to the pads by adding a 'stream' field basically
> > makes all pads potentially multiplexed, reducing the problem of format
> > configuration/validation to a 1-to-1 {pad, stream} pair validation
> > which allows to collapse the topology and maintain the current one.
>
> Yes. I think I have problem understanding the counter arguments as I don't
> really see a difference with a) two subdevs, each with two non-multiplexed
> pads, linked 1-to-1 and b) two subdevs, each with one multiplexed pad, with
> two routes.

My main concerns are:

- Usage of format configuration to establish routing as per above.
  Format assignment gets a routing semantic associated, which is an
  implicit behavior difficult to control and inspect for applications.

- Userspace is in control of connecting endpoints on the multiplexed
  bus by assigning formats, this has two consequences:
  - A 1-to-1 mapping between streams on the two sides of the
    multiplexed bus which prevents routing multiple streams to the
    same endpoint (is this correct ?)
  - As the only information about a 'stream' on the multiplexed bus is
    the format it transports, it is required to assign to the stream
    identifier a semantic (ie stream 0 = virtual channel 0). The
    previous version had the information of what's transported on the
    multiplexed bus hidden from userspace and delegated to the
    frame_desc kAPI. This way it was possible to describe precisely
    what's sent on the bus, with bus-specific structures (ie struct
    v4l2_mbus_frame_desc_entry.bus.csi2)
  - This might seem a bit pedantic, but, setting image formats and
    sizes on the endpoints of a multiplexed bus such as CSI-2 is not
    technically correct. CSI-2 transports packets tagged with
    identifiers for the virtual channel and data type they transport
    (and identifiers for the packet type, but that's part of the bus
    protocol). The format and size is relevant for configuring the
    size of the memory area where the receiver dumps the received
    packets, but it's not part of the link configuration itself.
    This was better represented by using the information from the
    remote side frame_desc.

>
> There is one particular issue I had with the previous version, which I think
> is a big reason I like the new approach:
>
> I'm using TI CAL driver, which already exists in upstreams and supports both
> non-MC and MC-without-streams. Adding support for streams, i.e supporting
> non-MC, MC-without-streams and MC-with-streams made the driver an unholy
> mess (including a new module parameter to enable streams). With the new
> approach, the changes were relatively minor, as MC with and without streams
> are really the same thing.

I can only agree about the fact your solution is indeed simpler
regarding the topology handling.

>
> With the previous approach you couldn't e.g. have a CSI2-RX bridge driver
> that would support both old, non-multiplexed CSI2 sensor drivers and
> multiplexed CSI2 sensor drivers. Unless you had something like the module
> parameter mentioned above. Or perhaps a DT property to define which mode the
> pad is in.

Agreed again, with the previous version a new subdev would have been
required, right ?

>
> Also, one problem is that I really only have a single multiplexed HW setup,
> which limits my testing and the way I see multiplexed streams. That setup is
> "luckily" not the simplest one:

Luckily, yes :)

>
> SoC CSI-2 RX <-> FPDLink Deserializer <-> FPDLink Serializer <-> Sensor
>
> 4 serializer+sensor cameras can be connected to the deserializer. Each
> sensor provides 2 streams (pixel and metadata). So I have 8 streams coming
> in to the SoC.

That's great, we have a very similar GMSL setup we could use to
compare. I had not considered metadata in my mental picture of how to
handle this kind of setups so far. For the simpler case I imagine it could
have been handled by making the deserializer source pad a multiplexed
pad with 4 endpoints (one for each virtual channel) where to route the
streams received on the 4 sink pads (one for each stream serialized on
the GMSL/FDP bus).

Userspace configures routing on the deser, directing each input stream
to one stream on the multiplexed source pad, effectively configuring
on which VC each stream is put on the multiplexed bus. Please note
that in your example, unless your deser can do demuxing on DT, each
stream at this stage will contain 2 datatypes, images and metadata.

The CSI-2 receiver would fetch the frame descriptor to learn what is
about to be sent on the bus and creates its routing table accordingly.
In the simplest example it can simply route stream n to its n-th
source pad. If your CSI-2 receiver can route VC differently the
routing table can be manipulated by userspace. If your CSI-2 receiver
can do DT demultiplexing (not even sure if a CSI-2 receiver could do
so or it happens at a later stage in the pipeline) each {VC, DT} pair will be
represented as an endpoint in your multiplexed sink pad to be routed to
a different source pad (or whatever is next in your pipeline).

I wish someone could disprove my understanding of how the previous version
worked as it is based on my mental picture only, which might of course
be faulty.

How would you model that with stream formats, for a comparison ?

>
> > Apart from the concerns expressed by Laurent (which I share but only
> > partially understand, as the implications of bulk moving the
> > v4l2-subdev configuration API to be stream-aware are not totally clear
> > to me yet) what I'm not convinced of is that now cross-entities
> > "routes" (or "streams") on a multiplexed bus do require a format
> > assigned, effectively exposing them to userspace, with the consequence
> > that the format configuration influences the routes setup up to the
> > point the two have to be kept consistent. The concept
> > could even be extended to inter-entities routes, as you suggested the
> > routing tables could even be dropped completely in this case, but I
> > feel mixing routing and format setup is a bit a layer violation and
> > forbids, in example, routing two streams to the same endpoint, which I
> > feel will be required to perform DT multiplexing on the same virtual
> > channel. The previous version had the multiplexed link configuration
> > completely hidden from userspace and controlled solely by the routing API,
> > which seems a tad more linear and offers more flexibility for drivers.
> >
> > I'm not strongly pushing for one solution over the other, the only use
> > case I can reason on at the moment is a simple single-stream VC
> > multiplexing and both solutions works equally fine for that. This one
> > is certainly simpler regarding the required device topology.
> >
> > Btw Tomi, do you have examples of drivers ported to your new proposal ?
>
> Yes. They're a bit messy, but I can share them with the next version. I'm
> currently fixing a lot of things, and making full use of the new
> v4l2_subdev_state.

Thanks, it would help!

>
>   Tomi
>
>

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline
  2021-07-08 12:36   ` Jacopo Mondi
@ 2021-07-11 15:25     ` Sakari Ailus
  2021-07-16  6:35     ` Tomi Valkeinen
  1 sibling, 0 replies; 60+ messages in thread
From: Sakari Ailus @ 2021-07-11 15:25 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Jacopo,

On Thu, Jul 08, 2021 at 02:36:20PM +0200, Jacopo Mondi wrote:
> Hello Tomi,
>     A few minors and a question below
> 
> On Mon, May 24, 2021 at 01:43:48PM +0300, Tomi Valkeinen wrote:
> > From: Sakari Ailus <sakari.ailus@linux.intel.com>
> >
> > The pipeline has been moved from the entity to the pads; reflect this in
> > the media pipeline function API.
> >
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> > Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  Documentation/driver-api/media/mc-core.rst    |  6 ++--
> >  drivers/media/mc/mc-entity.c                  | 24 ++++++-------
> >  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
> >  .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
> >  .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
> >  drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
> >  drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
> >  .../media/platform/qcom/camss/camss-video.c   |  6 ++--
> >  drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
> >  .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
> >  .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
> >  drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
> >  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
> >  .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
> >  drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
> >  drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
> >  drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
> >  .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
> >  drivers/media/usb/au0828/au0828-core.c        |  8 ++---
> >  drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
> >  drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
> >  drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
> >  drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
> >  include/media/media-entity.h                  | 34 +++++++++----------
> >  24 files changed, 98 insertions(+), 100 deletions(-)
> >
> > diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> > index 8a13640bed56..69a64279a61f 100644
> > --- a/Documentation/driver-api/media/mc-core.rst
> > +++ b/Documentation/driver-api/media/mc-core.rst
> > @@ -213,11 +213,11 @@ When starting streaming, drivers must notify all entities in the pipeline to
> >  prevent link states from being modified during streaming by calling
> >  :c:func:`media_pipeline_start()`.
> >
> > -The function will mark all entities connected to the given entity through
> > -enabled links, either directly or indirectly, as streaming.
> > +The function will mark all entities connected to the given pad through
> 
> As the stream_count counter is now moved to the pads, should this be
> 
> +The function will mark all the pads connected to the given pad through

Yes.

> 
> > +enabled routes and links, either directly or indirectly, as streaming.
> >
> >  The struct media_pipeline instance pointed to by
> > -the pipe argument will be stored in every entity in the pipeline.
> > +the pipe argument will be stored in every pad in the pipeline.
> >  Drivers should embed the struct media_pipeline
> 
> Does this still apply ?

Yes.

> 
> >  in higher-level pipeline structures and can then access the
> >  pipeline through the struct media_entity
> 
> This sentence should probably be changed to
> 
> pipeline through the struct media_pad pipe field.

Yes.

> 
> > diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> > index ea1cf7f63ae8..e6451903359c 100644
> > --- a/drivers/media/mc/mc-entity.c
> > +++ b/drivers/media/mc/mc-entity.c
> > @@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
> >   * Pipeline management
> >   */
> >
> > -__must_check int __media_pipeline_start(struct media_entity *entity,
> > +__must_check int __media_pipeline_start(struct media_pad *pad,
> >  					struct media_pipeline *pipe)
> >  {
> > -	struct media_device *mdev = entity->graph_obj.mdev;
> > +	struct media_device *mdev = pad->graph_obj.mdev;
> >  	struct media_graph *graph = &pipe->graph;
> > -	struct media_pad *pad = entity->pads;
> >  	struct media_pad *pad_err = pad;
> >  	struct media_link *link;
> >  	int ret;
> > @@ -542,24 +541,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
> >  }
> >  EXPORT_SYMBOL_GPL(__media_pipeline_start);
> >
> > -__must_check int media_pipeline_start(struct media_entity *entity,
> > +__must_check int media_pipeline_start(struct media_pad *pad,
> >  				      struct media_pipeline *pipe)
> 
> As it seems that even with the full series applied
> media_pipeline_start() is always called with entity->pads as its first
> argument, I wonder if it wouldn't be more linear for a driver to keep
> using entity and have this function here pass the entity's pads to
> __media_pipeline_start().
> 
> Do we expect drivers to actually start the pipeline using a specific
> pad ?

The pipeline is moved to pads with this patch, and therefore it's logical
to do start from pads, too. Should there be more than one, figuring out
which one to use would not be possible.

-- 
Sakari Ailus

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 08/27] media: entity: Add has_route entity operation
  2021-07-08 12:43   ` Jacopo Mondi
@ 2021-07-11 15:26     ` Sakari Ailus
  2021-07-12  7:42       ` Jacopo Mondi
  0 siblings, 1 reply; 60+ messages in thread
From: Sakari Ailus @ 2021-07-11 15:26 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

On Thu, Jul 08, 2021 at 02:43:10PM +0200, Jacopo Mondi wrote:
> Hi Tomi,
>    a small note
> 
> On Mon, May 24, 2021 at 01:43:49PM +0300, Tomi Valkeinen wrote:
> > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> >
> > The optional operation can be used by entities to report whether two
> > pads are internally connected.
> >
> > While at there, fix a Sphinx compiler warning in a comment block a few
> > lines above.
> >
> > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  include/media/media-entity.h | 7 +++++++
> >  1 file changed, 7 insertions(+)
> >
> > diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> > index 516d73a2941e..ad4020b2df65 100644
> > --- a/include/media/media-entity.h
> > +++ b/include/media/media-entity.h
> > @@ -187,6 +187,7 @@ enum media_pad_signal_type {
> >   * @flags:	Pad flags, as defined in
> >   *		:ref:`include/uapi/linux/media.h <media_header>`
> >   *		(seek for ``MEDIA_PAD_FL_*``)
> > + *
> >   * .. note::
> >   *
> >   *    @stream_count reference count must never be negative, but is a signed
> > @@ -214,6 +215,10 @@ struct media_pad {
> >   * @link_validate:	Return whether a link is valid from the entity point of
> >   *			view. The media_pipeline_start() function
> >   *			validates all links by calling this operation. Optional.
> > + * @has_route:		Return whether a route exists inside the entity between
> > + *			two given pads. Pads are passed to the operation ordered
> > + *			by index. Optional: If the operation isn't implemented
> 
> According to the next patch, this doesn't seem to be 'Optional:' :)

How? Few drivers will implement this in the end, and that's been taken into
account in code AFAIU.

-- 
Sakari Ailus

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 21/27] v4l: subdev: routing kernel helper functions
  2021-06-05 23:29   ` Laurent Pinchart
@ 2021-07-11 15:48     ` Sakari Ailus
  0 siblings, 0 replies; 60+ messages in thread
From: Sakari Ailus @ 2021-07-11 15:48 UTC (permalink / raw)
  To: Laurent Pinchart
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

Hi Laurent,

On Sun, Jun 06, 2021 at 02:29:42AM +0300, Laurent Pinchart wrote:
> > +{
> > +	int ret;
> > +
> > +	routing->which = state ? V4L2_SUBDEV_FORMAT_TRY : V4L2_SUBDEV_FORMAT_ACTIVE;
> 
> I think we could simplify this if we mandate subdev drivers to embed an
> instance of v4l2_subdev_state for the active state. We could then drop
> the .get_routing() operation. We could actually drop the .get_fmt()
> operation too, but that would be a long term goal. As .get_routing() is
> new, I'd rather not introduce it, and push drivers that need routing
> support to use v4l2_subdev_state.

Sounds good to me.

-- 
Sakari Ailus

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 08/27] media: entity: Add has_route entity operation
  2021-07-11 15:26     ` Sakari Ailus
@ 2021-07-12  7:42       ` Jacopo Mondi
  2021-07-26 18:13         ` Sakari Ailus
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-12  7:42 UTC (permalink / raw)
  To: Sakari Ailus
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

Hi Sakari,

On Sun, Jul 11, 2021 at 06:26:26PM +0300, Sakari Ailus wrote:
> On Thu, Jul 08, 2021 at 02:43:10PM +0200, Jacopo Mondi wrote:
> > Hi Tomi,
> >    a small note
> >
> > On Mon, May 24, 2021 at 01:43:49PM +0300, Tomi Valkeinen wrote:
> > > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > >
> > > The optional operation can be used by entities to report whether two
> > > pads are internally connected.
> > >
> > > While at there, fix a Sphinx compiler warning in a comment block a few
> > > lines above.
> > >
> > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > ---
> > >  include/media/media-entity.h | 7 +++++++
> > >  1 file changed, 7 insertions(+)
> > >
> > > diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> > > index 516d73a2941e..ad4020b2df65 100644
> > > --- a/include/media/media-entity.h
> > > +++ b/include/media/media-entity.h
> > > @@ -187,6 +187,7 @@ enum media_pad_signal_type {
> > >   * @flags:	Pad flags, as defined in
> > >   *		:ref:`include/uapi/linux/media.h <media_header>`
> > >   *		(seek for ``MEDIA_PAD_FL_*``)
> > > + *
> > >   * .. note::
> > >   *
> > >   *    @stream_count reference count must never be negative, but is a signed
> > > @@ -214,6 +215,10 @@ struct media_pad {
> > >   * @link_validate:	Return whether a link is valid from the entity point of
> > >   *			view. The media_pipeline_start() function
> > >   *			validates all links by calling this operation. Optional.
> > > + * @has_route:		Return whether a route exists inside the entity between
> > > + *			two given pads. Pads are passed to the operation ordered
> > > + *			by index. Optional: If the operation isn't implemented
> >
> > According to the next patch, this doesn't seem to be 'Optional:' :)
>
> How? Few drivers will implement this in the end, and that's been taken into
> account in code AFAIU.

What I meant is that it's not optional that all pads are considered
connected if the op is not implemented. Or maybe this should have read
as "Implementing the operation is optional" ?

>
> --
> Sakari Ailus

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-07-10  8:42       ` Jacopo Mondi
@ 2021-07-12  8:19         ` Tomi Valkeinen
  2021-07-23 10:21           ` Jacopo Mondi
  0 siblings, 1 reply; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-12  8:19 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: Laurent Pinchart, Hans Verkuil, Sakari Ailus, linux-media,
	Jacopo Mondi, niklas.soderlund+renesas, Mauro Carvalho Chehab,
	Pratyush Yadav, Lokesh Vutla

[-- Attachment #1: Type: text/plain, Size: 14589 bytes --]

Hi,

On 10/07/2021 11:42, Jacopo Mondi wrote:
> Hi Tomi,
>     thanks for you reply
> 
> On Fri, Jul 09, 2021 at 09:26:03PM +0300, Tomi Valkeinen wrote:
>> Hi Jacopo,
>>
>> On 09/07/2021 18:18, Jacopo Mondi wrote:
>>> Hi Tomi, Laurent,
>>>
>>> On Sun, Jun 06, 2021 at 03:06:18AM +0300, Laurent Pinchart wrote:
>>>> Hi Hans, Sakari,
>>>>
>>>> We need your feedback on this series, at least on the general approach.
>>>> There are quite a few issues to be addressed, and it makes no sense to
>>>> invest time in this if you don't think this is a good direction.
>>>>
>>>> If anyone else wants to give feedback, speak now or forever hold your
>>>> peace :-)
>>>
>>> Since you ask...
>>>
>>> Having been involved a bit as the n-th person that tried to bring this
>>> to completion I spent a bit of time trying to recollect how the
>>> previous approach worked and how it compares to this one. Sorry if
>>> this goes in length.
>>>
>>> I share Tomi's concern on one part of the previous version:
>>>
>>> - The resulting device topology gets complicated in a non-trivial way.
>>>
>>>     The typical example of having to model one image sensor that sends
>>>     embedded data and images with three sub-devices speaks for itself, I
>>>     presume.
>>>
>>>     However in one way, I feel like this is somehow correct and provides
>>>     a more accurate representation of the actual sensor architecture.
>>>     Splitting a sensor into components would allow to better handle
>>>     devices which supports multiple buses (typically CSI-2 and
>>>     parallel) through the internal routing tables, and allows
>>>     better control of the components of the image sensor. [1]
>>
>> I'm not sure what kind of setup you mean, but nothing prevents you from
>> splitting devices into multiple subdevs with the new approach if it makes
>> sense on your HW.
> 
> Nothing prevents it it from being done today, my point was that having
> to do so to support mulitplexed streams is an incentive to get to a
> more precise representation of the sensor architecture, not only a
> cons :)
> 
>>
>> I have a parallel sensor that provides metadata on a line before the actual
>> frame. I have hard time understanding why that should be split into 3
>> subdevs.
>>
> 
> As I guess there's no way to extract that line of embedded data if not
> from the frame when already in memory, I won't consider this the best
> example of a multiplexed bus :)

The FPDLink Deserializer does it, it can mark first N lines with a DT 
for embedded data.

Yes, it's not as fancy as with CSI-2, but it is essentially a 
multiplexed bus, with two streams.

>>> - Multiplexed source pads do not accept a format or any other configuration
>>>     like crop/composing. Again this might seem odd, and it might be
>>>     worth considering if those pads shouldn't be made 'special' somehow,
>>>     but I again think it models a multiplexed bus quite accurately,
>>>     doesn't it ? It's weird that the format of, in example, a CSI-2
>>>     receiver source pad has to be propagated from the image sensor
>>>     entity sink pad, crossing two entities, two routes and one
>>>     media link. This makes rather complex to automate format propagation along
>>>     pipelines, not only when done by abusing media-ctl like most people do,
>>>     but also when done programmatically the task is not easy (I know I'm
>>>     contradicting my [1] point here :)
>>
>> Hmm, but is it easy in the kernel side, then? I didn't feel so with the
>> previous version. The kernel needed to travel the graph back and forth "all
>> the time", just to figure out what's going on and where.
> 
> Not for the core. You see the patch I referenced, I praise Sakari for
> getting there, the validation is indeed complex.
> 
> I mean that for drivers it would be easier as the routing management
> is separate from format management, and drivers do not have to match
> endpoints by the format they have applied to infer routes.

I'm not sure what you mean here with "do not have to match endpoints by 
the format they have applied to infer routes".

The routing is set with the ioctl, it's not inferred in any way.

>> If the userspace understands the HW topology (as it more or less must), and
>> it configures the routes (as it has to), and sets the formats on certain
>> subdevs, then I don't see that it would have any issues in propagating the
>> formats.
>>
> 
> As I've said the fact that setting up a route is accomplished by
> setting the same format on two endpoints feels like a layer violation.
> For userspace traversing a route means matching the formats on a
> possibly high number of {pad, stream} pairs. It won't be easy without
> a dedicated API and feels rather error prone for drivers too if they
> have to configure they internal routing based on format information

Hmm, are you talking about the method I suggested in my earlier mail, 
where I was thinking out loud if the routing endpoint information could 
be set to a (pad, stream) pair? That is not implemented.

This current series version has a routing table, set with the 
set-routing ioctl. When the routing is set, you could think that a set 
of "virtual" pads is created (identified by (pad, stream) pair), where 
each route endpoint has a pad. Those pads can then be configured 
similarly to the "normal" pads.

>>>     Also link validation is of course a bit more complex as shown by
>>>     731facccc987 ("v4l: subdev: Take routing information into account in link validation")
>>>     which was part of the previous series, but it's totally up to the
>>>     core..
>>>
>>> Moving everything to the pads by adding a 'stream' field basically
>>> makes all pads potentially multiplexed, reducing the problem of format
>>> configuration/validation to a 1-to-1 {pad, stream} pair validation
>>> which allows to collapse the topology and maintain the current one.
>>
>> Yes. I think I have problem understanding the counter arguments as I don't
>> really see a difference with a) two subdevs, each with two non-multiplexed
>> pads, linked 1-to-1 and b) two subdevs, each with one multiplexed pad, with
>> two routes.
> 
> My main concerns are:
> 
> - Usage of format configuration to establish routing as per above.
>    Format assignment gets a routing semantic associated, which is an
>    implicit behavior difficult to control and inspect for applications.

Again, either I'm totally misunderstanding what you're saying, or you 
are talking about the method that has not been implemented.

> - Userspace is in control of connecting endpoints on the multiplexed
>    bus by assigning formats, this has two consequences:
>    - A 1-to-1 mapping between streams on the two sides of the
>      multiplexed bus which prevents routing multiple streams to the
>      same endpoint (is this correct ?)

No, you can have multiple streams with the same endpoint (i.e. the same 
(pad, stream) for source/sink side).

>    - As the only information about a 'stream' on the multiplexed bus is
>      the format it transports, it is required to assign to the stream
>      identifier a semantic (ie stream 0 = virtual channel 0). The
>      previous version had the information of what's transported on the
>      multiplexed bus hidden from userspace and delegated to the
>      frame_desc kAPI. This way it was possible to describe precisely
>      what's sent on the bus, with bus-specific structures (ie struct
>      v4l2_mbus_frame_desc_entry.bus.csi2)

That is how it's in this series too. The difference is that in the 
previous version, when a driver needed to know something about the 
stream which was not in the frame_desc, it had to start traversing the 
graph to find out a non-multiplexed pad. With this version the driver 
has the information in its (pad, stream) pair.

>    - This might seem a bit pedantic, but, setting image formats and
>      sizes on the endpoints of a multiplexed bus such as CSI-2 is not
>      technically correct. CSI-2 transports packets tagged with
>      identifiers for the virtual channel and data type they transport
>      (and identifiers for the packet type, but that's part of the bus
>      protocol). The format and size is relevant for configuring the
>      size of the memory area where the receiver dumps the received
>      packets, but it's not part of the link configuration itself.
>      This was better represented by using the information from the
>      remote side frame_desc.

Why is a multiplexed CSI-2 bus different than a non-multiplexed parallel 
bus? Or more specifically, why is a single stream in a multiplexed CSI-2 
bus different than the stream in non-multiplexed parallel bus? It's the 
same data, transported in a slightly different manner.

One could, of course, argue that they are not different, and pad 
configuration for non-multiplexed pads should also be removed.

This reminds me of one more problem I had in the previous version: 
supporting TRY. I couldn't implement TRY support as the subdevs didn't 
have the information needed. With this version, they do have the 
information, and can independently say if the subdev's routing + format 
configuration is valid or not.

>> There is one particular issue I had with the previous version, which I think
>> is a big reason I like the new approach:
>>
>> I'm using TI CAL driver, which already exists in upstreams and supports both
>> non-MC and MC-without-streams. Adding support for streams, i.e supporting
>> non-MC, MC-without-streams and MC-with-streams made the driver an unholy
>> mess (including a new module parameter to enable streams). With the new
>> approach, the changes were relatively minor, as MC with and without streams
>> are really the same thing.
> 
> I can only agree about the fact your solution is indeed simpler
> regarding the topology handling.
> 
>>
>> With the previous approach you couldn't e.g. have a CSI2-RX bridge driver
>> that would support both old, non-multiplexed CSI2 sensor drivers and
>> multiplexed CSI2 sensor drivers. Unless you had something like the module
>> parameter mentioned above. Or perhaps a DT property to define which mode the
>> pad is in.
> 
> Agreed again, with the previous version a new subdev would have been
> required, right ?

The previous version needed some way to create or set up the pads 
differently based on the future usage. The subdev's pad had to be in 
either non-multiplexed or multiplexed mode, and this choice had to be 
made "early", before using the subdev.

Or, yes, I guess one option would have been to split the device into 
multiple subdevs, one subdev with multiplexed pads, the other with 
non-multiplexed pads. That would have been horribly confusing.

>> Also, one problem is that I really only have a single multiplexed HW setup,
>> which limits my testing and the way I see multiplexed streams. That setup is
>> "luckily" not the simplest one:
> 
> Luckily, yes :)
> 
>>
>> SoC CSI-2 RX <-> FPDLink Deserializer <-> FPDLink Serializer <-> Sensor
>>
>> 4 serializer+sensor cameras can be connected to the deserializer. Each
>> sensor provides 2 streams (pixel and metadata). So I have 8 streams coming
>> in to the SoC.
> 
> That's great, we have a very similar GMSL setup we could use to
> compare. I had not considered metadata in my mental picture of how to
> handle this kind of setups so far. For the simpler case I imagine it could
> have been handled by making the deserializer source pad a multiplexed
> pad with 4 endpoints (one for each virtual channel) where to route the
> streams received on the 4 sink pads (one for each stream serialized on
> the GMSL/FDP bus).
> 
> Userspace configures routing on the deser, directing each input stream
> to one stream on the multiplexed source pad, effectively configuring
> on which VC each stream is put on the multiplexed bus. Please note
> that in your example, unless your deser can do demuxing on DT, each
> stream at this stage will contain 2 datatypes, images and metadata.

No, a "stream" is an independent set of data. Pixel data is its own 
stream, and metadata is its own stream, even if they're on the same 
CSI-2 link (or parallel link).

> The CSI-2 receiver would fetch the frame descriptor to learn what is
> about to be sent on the bus and creates its routing table accordingly.
> In the simplest example it can simply route stream n to its n-th
> source pad. If your CSI-2 receiver can route VC differently the
> routing table can be manipulated by userspace. If your CSI-2 receiver
> can do DT demultiplexing (not even sure if a CSI-2 receiver could do
> so or it happens at a later stage in the pipeline) each {VC, DT} pair will be
> represented as an endpoint in your multiplexed sink pad to be routed to
> a different source pad (or whatever is next in your pipeline).
> 
> I wish someone could disprove my understanding of how the previous version
> worked as it is based on my mental picture only, which might of course
> be faulty.
> 
> How would you model that with stream formats, for a comparison ?

I've attached a picture that perhaps helps.

On the left side it has the previous version, on the right side this new 
version. Note that the picture is just partially drawn to avoid needless 
repetition. The second CAL RX doesn't have anything connected, I haven't 
drawn all the links between CAL RX0 and CAL Video nodes, and I have 
drawn only a few of the optional routes/links (drawn in dotted lines).

The picture is also missing the serializers. I should add them, but they 
are just pass-through components and do not bring much into the picture.

On the left side, the blue-ish pads are multiplexed pads (i.e. they 
cannot be configured). The sensor is also split only into two subdevs, 
as it was easier to implement than a three-subdev-model.

Also note that e.g. the link between UB960 pad4 and CAL RX0 pad0 is 
drawn instead using streams. In other words, there is only one media 
link between those pads, but in that link there are 8 streams, which are 
drawn here.

The CAL videoX nodes are the /dev/videoX nodes. CAL has 8 dma engines, 
so there are 8 video nodes. Any of the video nodes can be connected to 
any one of the source pads on either CAL RX subdev.

The UB960 routes all inputs into the output port, and tags first N lines 
with embedded DT, the rest with pixel data DT. And VC is set matching 
the input port.

CAL will route each stream, based on the DT and VC, to a separate DMA 
engine, which then goes to memory buffers.

The differences between the old and new model look minor in the picture, 
but in the code they are quite huge.

  Tomi

[-- Attachment #2: fpdlink-Comparison.png --]
[-- Type: image/png, Size: 99881 bytes --]

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads
  2021-07-08 13:11   ` Jacopo Mondi
@ 2021-07-16  6:19     ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-16  6:19 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

On 08/07/2021 16:11, Jacopo Mondi wrote:
> Hello again,
> 
> On Mon, May 24, 2021 at 01:43:47PM +0300, Tomi Valkeinen wrote:
>> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>>
>> This moves the pipe and stream_count fields from struct media_entity to
>> struct media_pad. Effectively streams become pad-specific rather than
>> being entity specific, allowing several independent streams to traverse a
>> single entity and an entity to be part of several streams.
>>
>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
>>
>> - Update documentation to use 'pads'
>> - Use the media pad iterator in media_entity.c
>> - Update rcar-dma.c to use the new per-pad stream count
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>
>> - Fix cleanup in the error path of __media_pipeline_start()
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> ---
>>   drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
>>   drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
>>   drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
>>   drivers/media/platform/omap3isp/isp.c         |  2 +-
>>   drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
>>   drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
>>   drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
>>   drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
>>   drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
>>   drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
>>   drivers/staging/media/imx/imx-media-utils.c   |  2 +-
>>   drivers/staging/media/omap4iss/iss.c          |  2 +-
>>   drivers/staging/media/omap4iss/iss_video.c    |  2 +-
>>   drivers/staging/media/omap4iss/iss_video.h    |  2 +-
>>   include/media/media-entity.h                  | 21 +++---
>>   15 files changed, 73 insertions(+), 56 deletions(-)
>>
>> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
>> index 40ae9b6bac47..ea1cf7f63ae8 100644
>> --- a/drivers/media/mc/mc-entity.c
>> +++ b/drivers/media/mc/mc-entity.c
>> @@ -424,24 +424,28 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>>
>>   	while ((pad = media_graph_walk_next(graph))) {
>>   		struct media_entity *entity = pad->entity;
>> +		bool skip_validation = pad->pipe != NULL;
>> +		struct media_pad *iter;
>>
>>   		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
>>   		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
>>
>> -		entity->stream_count++;
>> -
>> -		if (entity->pipe && entity->pipe != pipe) {
>> -			pr_err("Pipe active for %s. Can't start for %s\n",
>> -				entity->name,
>> -				pad_err->entity->name);
>> -			ret = -EBUSY;
>> -			goto error;
>> +		media_entity_for_each_pad(entity, iter) {
>> +			if (iter->pipe && iter->pipe != pipe) {
>> +				pr_err("Pipe active for %s. Can't start for %s\n",
>> +				       entity->name, iter->entity->name);
>> +				ret = -EBUSY;
>> +			} else {
>> +				iter->pipe = pipe;
>> +			}
>> +			iter->stream_count++;
>>   		}
>>
>> -		entity->pipe = pipe;
>> +		if (ret)
> 
> ret is not initialized when declared and there is a code path that
> could lead here without assigning it. I would initialize it to 0 when
> declaring it, or even re-assign it to 0 at the beginning of this while
> loop.

Good catch! I'll fix that.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline
  2021-07-08 12:36   ` Jacopo Mondi
  2021-07-11 15:25     ` Sakari Ailus
@ 2021-07-16  6:35     ` Tomi Valkeinen
  1 sibling, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-16  6:35 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: linux-media, sakari.ailus, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla

On 08/07/2021 15:36, Jacopo Mondi wrote:
> Hello Tomi,
>      A few minors and a question below
> 
> On Mon, May 24, 2021 at 01:43:48PM +0300, Tomi Valkeinen wrote:
>> From: Sakari Ailus <sakari.ailus@linux.intel.com>
>>
>> The pipeline has been moved from the entity to the pads; reflect this in
>> the media pipeline function API.
>>
>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>> Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
>> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   Documentation/driver-api/media/mc-core.rst    |  6 ++--
>>   drivers/media/mc/mc-entity.c                  | 24 ++++++-------
>>   drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
>>   .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
>>   .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
>>   drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
>>   drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
>>   .../media/platform/qcom/camss/camss-video.c   |  6 ++--
>>   drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
>>   .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
>>   .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
>>   drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
>>   .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
>>   .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
>>   drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
>>   drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
>>   drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
>>   .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
>>   drivers/media/usb/au0828/au0828-core.c        |  8 ++---
>>   drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
>>   drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
>>   drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
>>   drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
>>   include/media/media-entity.h                  | 34 +++++++++----------
>>   24 files changed, 98 insertions(+), 100 deletions(-)
>>
>> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
>> index 8a13640bed56..69a64279a61f 100644
>> --- a/Documentation/driver-api/media/mc-core.rst
>> +++ b/Documentation/driver-api/media/mc-core.rst
>> @@ -213,11 +213,11 @@ When starting streaming, drivers must notify all entities in the pipeline to
>>   prevent link states from being modified during streaming by calling
>>   :c:func:`media_pipeline_start()`.
>>
>> -The function will mark all entities connected to the given entity through
>> -enabled links, either directly or indirectly, as streaming.
>> +The function will mark all entities connected to the given pad through
> 
> As the stream_count counter is now moved to the pads, should this be
> 
> +The function will mark all the pads connected to the given pad through
> 
>> +enabled routes and links, either directly or indirectly, as streaming.
>>
>>   The struct media_pipeline instance pointed to by
>> -the pipe argument will be stored in every entity in the pipeline.
>> +the pipe argument will be stored in every pad in the pipeline.
>>   Drivers should embed the struct media_pipeline
> 
> Does this still apply ?
> 
>>   in higher-level pipeline structures and can then access the
>>   pipeline through the struct media_entity
> 
> This sentence should probably be changed to
> 
> pipeline through the struct media_pad pipe field.

Thanks! I've made these two changes.

  Tomi


^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-07-12  8:19         ` Tomi Valkeinen
@ 2021-07-23 10:21           ` Jacopo Mondi
  2021-07-26 10:49             ` Tomi Valkeinen
  0 siblings, 1 reply; 60+ messages in thread
From: Jacopo Mondi @ 2021-07-23 10:21 UTC (permalink / raw)
  To: Tomi Valkeinen
  Cc: Laurent Pinchart, Hans Verkuil, Sakari Ailus, linux-media,
	Jacopo Mondi, niklas.soderlund+renesas, Mauro Carvalho Chehab,
	Pratyush Yadav, Lokesh Vutla

Hi Tomi,
   sorry for the late reply

On Mon, Jul 12, 2021 at 11:19:08AM +0300, Tomi Valkeinen wrote:
> Hi,
>
> On 10/07/2021 11:42, Jacopo Mondi wrote:
> > Hi Tomi,
> >     thanks for you reply
> >
> > On Fri, Jul 09, 2021 at 09:26:03PM +0300, Tomi Valkeinen wrote:
> > > Hi Jacopo,
> > >
> > > On 09/07/2021 18:18, Jacopo Mondi wrote:
> > > > Hi Tomi, Laurent,
> > > >
> > > > On Sun, Jun 06, 2021 at 03:06:18AM +0300, Laurent Pinchart wrote:
> > > > > Hi Hans, Sakari,
> > > > >
> > > > > We need your feedback on this series, at least on the general approach.
> > > > > There are quite a few issues to be addressed, and it makes no sense to
> > > > > invest time in this if you don't think this is a good direction.
> > > > >
> > > > > If anyone else wants to give feedback, speak now or forever hold your
> > > > > peace :-)
> > > >
> > > > Since you ask...
> > > >
> > > > Having been involved a bit as the n-th person that tried to bring this
> > > > to completion I spent a bit of time trying to recollect how the
> > > > previous approach worked and how it compares to this one. Sorry if
> > > > this goes in length.
> > > >
> > > > I share Tomi's concern on one part of the previous version:
> > > >
> > > > - The resulting device topology gets complicated in a non-trivial way.
> > > >
> > > >     The typical example of having to model one image sensor that sends
> > > >     embedded data and images with three sub-devices speaks for itself, I
> > > >     presume.
> > > >
> > > >     However in one way, I feel like this is somehow correct and provides
> > > >     a more accurate representation of the actual sensor architecture.
> > > >     Splitting a sensor into components would allow to better handle
> > > >     devices which supports multiple buses (typically CSI-2 and
> > > >     parallel) through the internal routing tables, and allows
> > > >     better control of the components of the image sensor. [1]
> > >
> > > I'm not sure what kind of setup you mean, but nothing prevents you from
> > > splitting devices into multiple subdevs with the new approach if it makes
> > > sense on your HW.
> >
> > Nothing prevents it it from being done today, my point was that having
> > to do so to support mulitplexed streams is an incentive to get to a
> > more precise representation of the sensor architecture, not only a
> > cons :)
> >
> > >
> > > I have a parallel sensor that provides metadata on a line before the actual
> > > frame. I have hard time understanding why that should be split into 3
> > > subdevs.
> > >
> >
> > As I guess there's no way to extract that line of embedded data if not
> > from the frame when already in memory, I won't consider this the best
> > example of a multiplexed bus :)
>
> The FPDLink Deserializer does it, it can mark first N lines with a DT for
> embedded data.
>
> Yes, it's not as fancy as with CSI-2, but it is essentially a multiplexed
> bus, with two streams.
>

I concur that the deser source pad is actually multiplexed then

> > > > - Multiplexed source pads do not accept a format or any other configuration
> > > >     like crop/composing. Again this might seem odd, and it might be
> > > >     worth considering if those pads shouldn't be made 'special' somehow,
> > > >     but I again think it models a multiplexed bus quite accurately,
> > > >     doesn't it ? It's weird that the format of, in example, a CSI-2
> > > >     receiver source pad has to be propagated from the image sensor
> > > >     entity sink pad, crossing two entities, two routes and one
> > > >     media link. This makes rather complex to automate format propagation along
> > > >     pipelines, not only when done by abusing media-ctl like most people do,
> > > >     but also when done programmatically the task is not easy (I know I'm
> > > >     contradicting my [1] point here :)
> > >
> > > Hmm, but is it easy in the kernel side, then? I didn't feel so with the
> > > previous version. The kernel needed to travel the graph back and forth "all
> > > the time", just to figure out what's going on and where.
> >
> > Not for the core. You see the patch I referenced, I praise Sakari for
> > getting there, the validation is indeed complex.
> >
> > I mean that for drivers it would be easier as the routing management
> > is separate from format management, and drivers do not have to match
> > endpoints by the format they have applied to infer routes.
>
> I'm not sure what you mean here with "do not have to match endpoints by the
> format they have applied to infer routes".
>
> The routing is set with the ioctl, it's not inferred in any way.
>

You are right, the GS_ROUTING ioctls are still in place, so an entity
internal routing is managed as it used to be

> > > If the userspace understands the HW topology (as it more or less must), and
> > > it configures the routes (as it has to), and sets the formats on certain
> > > subdevs, then I don't see that it would have any issues in propagating the
> > > formats.
> > >
> >
> > As I've said the fact that setting up a route is accomplished by
> > setting the same format on two endpoints feels like a layer violation.
> > For userspace traversing a route means matching the formats on a
> > possibly high number of {pad, stream} pairs. It won't be easy without
> > a dedicated API and feels rather error prone for drivers too if they
> > have to configure they internal routing based on format information
>
> Hmm, are you talking about the method I suggested in my earlier mail, where
> I was thinking out loud if the routing endpoint information could be set to
> a (pad, stream) pair? That is not implemented.

Yes, I was referring to the idea of collapsing routing configuration into
format configuration.

>
> This current series version has a routing table, set with the set-routing
> ioctl. When the routing is set, you could think that a set of "virtual" pads
> is created (identified by (pad, stream) pair), where each route endpoint has
> a pad. Those pads can then be configured similarly to the "normal" pads.
>

And that's for routes inside an entity. Am I wrong that in this
version multiplexed (or virtual) pads identified by the (pad, stream)
pair have a format assigned at both ends of a link ?

> > > >     Also link validation is of course a bit more complex as shown by
> > > >     731facccc987 ("v4l: subdev: Take routing information into account in link validation")
> > > >     which was part of the previous series, but it's totally up to the
> > > >     core..
> > > >
> > > > Moving everything to the pads by adding a 'stream' field basically
> > > > makes all pads potentially multiplexed, reducing the problem of format
> > > > configuration/validation to a 1-to-1 {pad, stream} pair validation
> > > > which allows to collapse the topology and maintain the current one.
> > >
> > > Yes. I think I have problem understanding the counter arguments as I don't
> > > really see a difference with a) two subdevs, each with two non-multiplexed
> > > pads, linked 1-to-1 and b) two subdevs, each with one multiplexed pad, with
> > > two routes.
> >
> > My main concerns are:
> >
> > - Usage of format configuration to establish routing as per above.
> >    Format assignment gets a routing semantic associated, which is an
> >    implicit behavior difficult to control and inspect for applications.
>
> Again, either I'm totally misunderstanding what you're saying, or you are
> talking about the method that has not been implemented.
>

For routing internal to entities as you said GS_ROUTING is still in
place, so my argument is moot. However I think it still applies to
multiplexed ends of a cross-entity link (see below).


> > - Userspace is in control of connecting endpoints on the multiplexed
> >    bus by assigning formats, this has two consequences:
> >    - A 1-to-1 mapping between streams on the two sides of the
> >      multiplexed bus which prevents routing multiple streams to the
> >      same endpoint (is this correct ?)
>
> No, you can have multiple streams with the same endpoint (i.e. the same
> (pad, stream) for source/sink side).
>
> >    - As the only information about a 'stream' on the multiplexed bus is
> >      the format it transports, it is required to assign to the stream
> >      identifier a semantic (ie stream 0 = virtual channel 0). The
> >      previous version had the information of what's transported on the
> >      multiplexed bus hidden from userspace and delegated to the
> >      frame_desc kAPI. This way it was possible to describe precisely
> >      what's sent on the bus, with bus-specific structures (ie struct
> >      v4l2_mbus_frame_desc_entry.bus.csi2)
>
> That is how it's in this series too. The difference is that in the previous
> version, when a driver needed to know something about the stream which was
> not in the frame_desc, it had to start traversing the graph to find out a
> non-multiplexed pad. With this version the driver has the information in its
> (pad, stream) pair.

That's a (desirable) consequence of the fact multiplexed ends of a
link have a format assigned, right ?

>
> >    - This might seem a bit pedantic, but, setting image formats and
> >      sizes on the endpoints of a multiplexed bus such as CSI-2 is not
> >      technically correct. CSI-2 transports packets tagged with
> >      identifiers for the virtual channel and data type they transport
> >      (and identifiers for the packet type, but that's part of the bus
> >      protocol). The format and size is relevant for configuring the
> >      size of the memory area where the receiver dumps the received
> >      packets, but it's not part of the link configuration itself.
> >      This was better represented by using the information from the
> >      remote side frame_desc.
>
> Why is a multiplexed CSI-2 bus different than a non-multiplexed parallel
> bus? Or more specifically, why is a single stream in a multiplexed CSI-2 bus
> different than the stream in non-multiplexed parallel bus? It's the same
> data, transported in a slightly different manner.
>
> One could, of course, argue that they are not different, and pad
> configuration for non-multiplexed pads should also be removed.

While I get where you're going, I don't completely agree. The format
set on the ends of a non-multiplexed link does not represent what is
transported there but instructs the receiver (less so the transmitter)
about what data it should expects to receive and allows drivers to
prepare for it. The same doesn't apply to multiplexed pads, where
'what is expected to receive' is given by the union of the formats of
the several (pad, stream) endpoints. Assuming my understanding is
correct, that's what I don't like about having formats on multiplexed
pads.

>
> This reminds me of one more problem I had in the previous version:
> supporting TRY. I couldn't implement TRY support as the subdevs didn't have
> the information needed. With this version, they do have the information, and
> can independently say if the subdev's routing + format configuration is
> valid or not.
>
> > > There is one particular issue I had with the previous version, which I think
> > > is a big reason I like the new approach:
> > >
> > > I'm using TI CAL driver, which already exists in upstreams and supports both
> > > non-MC and MC-without-streams. Adding support for streams, i.e supporting
> > > non-MC, MC-without-streams and MC-with-streams made the driver an unholy
> > > mess (including a new module parameter to enable streams). With the new
> > > approach, the changes were relatively minor, as MC with and without streams
> > > are really the same thing.
> >
> > I can only agree about the fact your solution is indeed simpler
> > regarding the topology handling.
> >
> > >
> > > With the previous approach you couldn't e.g. have a CSI2-RX bridge driver
> > > that would support both old, non-multiplexed CSI2 sensor drivers and
> > > multiplexed CSI2 sensor drivers. Unless you had something like the module
> > > parameter mentioned above. Or perhaps a DT property to define which mode the
> > > pad is in.
> >
> > Agreed again, with the previous version a new subdev would have been
> > required, right ?
>
> The previous version needed some way to create or set up the pads
> differently based on the future usage. The subdev's pad had to be in either
> non-multiplexed or multiplexed mode, and this choice had to be made "early",
> before using the subdev.
>
> Or, yes, I guess one option would have been to split the device into
> multiple subdevs, one subdev with multiplexed pads, the other with
> non-multiplexed pads. That would have been horribly confusing.
>
> > > Also, one problem is that I really only have a single multiplexed HW setup,
> > > which limits my testing and the way I see multiplexed streams. That setup is
> > > "luckily" not the simplest one:
> >
> > Luckily, yes :)
> >
> > >
> > > SoC CSI-2 RX <-> FPDLink Deserializer <-> FPDLink Serializer <-> Sensor
> > >
> > > 4 serializer+sensor cameras can be connected to the deserializer. Each
> > > sensor provides 2 streams (pixel and metadata). So I have 8 streams coming
> > > in to the SoC.
> >
> > That's great, we have a very similar GMSL setup we could use to
> > compare. I had not considered metadata in my mental picture of how to
> > handle this kind of setups so far. For the simpler case I imagine it could
> > have been handled by making the deserializer source pad a multiplexed
> > pad with 4 endpoints (one for each virtual channel) where to route the
> > streams received on the 4 sink pads (one for each stream serialized on
> > the GMSL/FDP bus).
> >
> > Userspace configures routing on the deser, directing each input stream
> > to one stream on the multiplexed source pad, effectively configuring
> > on which VC each stream is put on the multiplexed bus. Please note
> > that in your example, unless your deser can do demuxing on DT, each
> > stream at this stage will contain 2 datatypes, images and metadata.
>
> No, a "stream" is an independent set of data. Pixel data is its own stream,
> and metadata is its own stream, even if they're on the same CSI-2 link (or
> parallel link).
>
> > The CSI-2 receiver would fetch the frame descriptor to learn what is
> > about to be sent on the bus and creates its routing table accordingly.
> > In the simplest example it can simply route stream n to its n-th
> > source pad. If your CSI-2 receiver can route VC differently the
> > routing table can be manipulated by userspace. If your CSI-2 receiver
> > can do DT demultiplexing (not even sure if a CSI-2 receiver could do
> > so or it happens at a later stage in the pipeline) each {VC, DT} pair will be
> > represented as an endpoint in your multiplexed sink pad to be routed to
> > a different source pad (or whatever is next in your pipeline).
> >
> > I wish someone could disprove my understanding of how the previous version
> > worked as it is based on my mental picture only, which might of course
> > be faulty.
> >
> > How would you model that with stream formats, for a comparison ?
>
> I've attached a picture that perhaps helps.
>
> On the left side it has the previous version, on the right side this new
> version. Note that the picture is just partially drawn to avoid needless
> repetition. The second CAL RX doesn't have anything connected, I haven't
> drawn all the links between CAL RX0 and CAL Video nodes, and I have drawn
> only a few of the optional routes/links (drawn in dotted lines).
>
> The picture is also missing the serializers. I should add them, but they are
> just pass-through components and do not bring much into the picture.
>
> On the left side, the blue-ish pads are multiplexed pads (i.e. they cannot
> be configured). The sensor is also split only into two subdevs, as it was
> easier to implement than a three-subdev-model.
>
> Also note that e.g. the link between UB960 pad4 and CAL RX0 pad0 is drawn
> instead using streams. In other words, there is only one media link between
> those pads, but in that link there are 8 streams, which are drawn here.
>
> The CAL videoX nodes are the /dev/videoX nodes. CAL has 8 dma engines, so
> there are 8 video nodes. Any of the video nodes can be connected to any one
> of the source pads on either CAL RX subdev.
>
> The UB960 routes all inputs into the output port, and tags first N lines
> with embedded DT, the rest with pixel data DT. And VC is set matching the
> input port.
>
> CAL will route each stream, based on the DT and VC, to a separate DMA
> engine, which then goes to memory buffers.
>
> The differences between the old and new model look minor in the picture, but
> in the code they are quite huge.
>

Thanks, this helps. Do we concur that the main difference between the
two version (let's call them v0 and v1) at least from an uAPI perspective,
is basically that the ends of a multiplexed link:

- in v0 had no formats assigned, configuration of the receiver end was
  performed in-kernel through get_frame_desc()

- in v1 each (pad, stream) has a format assigned by userspace and the
  receiver configuration is deduced from the formats assigned to the
  several endpoints ?

I would like to thank you for having bear with me so far and clarify my
doubts, I hope you'll find energy to clarify my last questions
here and I hope my understanding is not completely wrong, this is a
non-easy topic to deal with.

However, I don't want to continue to argue because what I care about
is getting to have some solution merged, and I feel this discussion is
not getting any productive for that. What I can offer, if anyway
helpful, is in the next weeks (or months?) rebase the work we've done
on R-Car in the past to support VC multiplexing on your series to have
more material for comparison. Then the ideal solution would be to pick
a conference/event most of the interested parties could attend, sit
down for a few days and find a way to move forward. The linux-media
mini conferences held around ELC/plumbers had this purpose and helped
finalize direction for other topics in the past. Considering the
current situation that's very unlikely to happen, but a virtual
version of the same could be considered.

Alternatively if the majority of maintainers reach consensus earlier
on this version I'll be happy to shut up, but that requires to rope
them in to review this series :)

Thanks
   j


>  Tomi



^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 00/27] v4l: subdev internal routing and streams
  2021-07-23 10:21           ` Jacopo Mondi
@ 2021-07-26 10:49             ` Tomi Valkeinen
  0 siblings, 0 replies; 60+ messages in thread
From: Tomi Valkeinen @ 2021-07-26 10:49 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: Laurent Pinchart, Hans Verkuil, Sakari Ailus, linux-media,
	Jacopo Mondi, niklas.soderlund+renesas, Mauro Carvalho Chehab,
	Pratyush Yadav, Lokesh Vutla

Hi Jacopo,

On 23/07/2021 13:21, Jacopo Mondi wrote:

>> This current series version has a routing table, set with the set-routing
>> ioctl. When the routing is set, you could think that a set of "virtual" pads
>> is created (identified by (pad, stream) pair), where each route endpoint has
>> a pad. Those pads can then be configured similarly to the "normal" pads.
>>
> 
> And that's for routes inside an entity. Am I wrong that in this
> version multiplexed (or virtual) pads identified by the (pad, stream)
> pair have a format assigned at both ends of a link ?

Yes, that is correct.

>>>>>      Also link validation is of course a bit more complex as shown by
>>>>>      731facccc987 ("v4l: subdev: Take routing information into account in link validation")
>>>>>      which was part of the previous series, but it's totally up to the
>>>>>      core..
>>>>>
>>>>> Moving everything to the pads by adding a 'stream' field basically
>>>>> makes all pads potentially multiplexed, reducing the problem of format
>>>>> configuration/validation to a 1-to-1 {pad, stream} pair validation
>>>>> which allows to collapse the topology and maintain the current one.
>>>>
>>>> Yes. I think I have problem understanding the counter arguments as I don't
>>>> really see a difference with a) two subdevs, each with two non-multiplexed
>>>> pads, linked 1-to-1 and b) two subdevs, each with one multiplexed pad, with
>>>> two routes.
>>>
>>> My main concerns are:
>>>
>>> - Usage of format configuration to establish routing as per above.
>>>     Format assignment gets a routing semantic associated, which is an
>>>     implicit behavior difficult to control and inspect for applications.
>>
>> Again, either I'm totally misunderstanding what you're saying, or you are
>> talking about the method that has not been implemented.
>>
> 
> For routing internal to entities as you said GS_ROUTING is still in
> place, so my argument is moot. However I think it still applies to
> multiplexed ends of a cross-entity link (see below).
> 
> 
>>> - Userspace is in control of connecting endpoints on the multiplexed
>>>     bus by assigning formats, this has two consequences:
>>>     - A 1-to-1 mapping between streams on the two sides of the
>>>       multiplexed bus which prevents routing multiple streams to the
>>>       same endpoint (is this correct ?)
>>
>> No, you can have multiple streams with the same endpoint (i.e. the same
>> (pad, stream) for source/sink side).
>>
>>>     - As the only information about a 'stream' on the multiplexed bus is
>>>       the format it transports, it is required to assign to the stream
>>>       identifier a semantic (ie stream 0 = virtual channel 0). The
>>>       previous version had the information of what's transported on the
>>>       multiplexed bus hidden from userspace and delegated to the
>>>       frame_desc kAPI. This way it was possible to describe precisely
>>>       what's sent on the bus, with bus-specific structures (ie struct
>>>       v4l2_mbus_frame_desc_entry.bus.csi2)
>>
>> That is how it's in this series too. The difference is that in the previous
>> version, when a driver needed to know something about the stream which was
>> not in the frame_desc, it had to start traversing the graph to find out a
>> non-multiplexed pad. With this version the driver has the information in its
>> (pad, stream) pair.
> 
> That's a (desirable) consequence of the fact multiplexed ends of a
> link have a format assigned, right ?

Yes.

>>
>>>     - This might seem a bit pedantic, but, setting image formats and
>>>       sizes on the endpoints of a multiplexed bus such as CSI-2 is not
>>>       technically correct. CSI-2 transports packets tagged with
>>>       identifiers for the virtual channel and data type they transport
>>>       (and identifiers for the packet type, but that's part of the bus
>>>       protocol). The format and size is relevant for configuring the
>>>       size of the memory area where the receiver dumps the received
>>>       packets, but it's not part of the link configuration itself.
>>>       This was better represented by using the information from the
>>>       remote side frame_desc.
>>
>> Why is a multiplexed CSI-2 bus different than a non-multiplexed parallel
>> bus? Or more specifically, why is a single stream in a multiplexed CSI-2 bus
>> different than the stream in non-multiplexed parallel bus? It's the same
>> data, transported in a slightly different manner.
>>
>> One could, of course, argue that they are not different, and pad
>> configuration for non-multiplexed pads should also be removed.
> 
> While I get where you're going, I don't completely agree. The format
> set on the ends of a non-multiplexed link does not represent what is
> transported there but instructs the receiver (less so the transmitter)
> about what data it should expects to receive and allows drivers to
> prepare for it. The same doesn't apply to multiplexed pads, where
> 'what is expected to receive' is given by the union of the formats of
> the several (pad, stream) endpoints. Assuming my understanding is
> correct, that's what I don't like about having formats on multiplexed
> pads.

Hmm, I don't quite understand your point. Do you mean that in 
non-multiplexed case the format is used to configure the receiver 
hardware, whereas with multiplexed case that is not true?

The format describes the content of a stream. In non-multiplexed case 
there's only one stream. In both cases the frame-desc can be used to 
find out details about the transmission.

Maybe this becomes more clear if you can give some practical examples to 
highlight your concerns?

> Thanks, this helps. Do we concur that the main difference between the
> two version (let's call them v0 and v1) at least from an uAPI perspective,

Let's rather call them v1 and v2, as I've uesd those terms elsewhere 
already =)

> is basically that the ends of a multiplexed link:
> 
> - in v0 had no formats assigned, configuration of the receiver end was
>    performed in-kernel through get_frame_desc()

Yes, although if the driver needs to know details found in the format, 
it has to traverse through the graph to find the data.

> - in v1 each (pad, stream) has a format assigned by userspace and the
>    receiver configuration is deduced from the formats assigned to the
>    several endpoints ?

The receiver configuration needs to be decided based on frame desc, 
formats, link freq control. This is the same for v0.

> I would like to thank you for having bear with me so far and clarify my
> doubts, I hope you'll find energy to clarify my last questions
> here and I hope my understanding is not completely wrong, this is a
> non-easy topic to deal with.

This will perhaps be easier to understand when I provide the drivers 
I've been using for testing. I'll do that for the next version.

> However, I don't want to continue to argue because what I care about
> is getting to have some solution merged, and I feel this discussion is
> not getting any productive for that. What I can offer, if anyway

I disagree, I think it is productive =). I am relatively new to cameras 
and V4L2, so it's important to get opinions. It is also good to get 
questions so that I have to explain what I'm doing, as it often forces 
me to re-think topics that I have already "finished".

> helpful, is in the next weeks (or months?) rebase the work we've done
> on R-Car in the past to support VC multiplexing on your series to have
> more material for comparison. Then the ideal solution would be to pick

This is a good idea, but please wait until the next version of the 
series. While there are currently no changes in the concepts, there are 
quite a lot of kernel side changes due to the new subdev state and using 
that.

> a conference/event most of the interested parties could attend, sit
> down for a few days and find a way to move forward. The linux-media
> mini conferences held around ELC/plumbers had this purpose and helped
> finalize direction for other topics in the past. Considering the
> current situation that's very unlikely to happen, but a virtual
> version of the same could be considered.

I'm happy to have meetings, one-to-one or bigger ones, to discuss this.

  Tomi

^ permalink raw reply	[flat|nested] 60+ messages in thread

* Re: [PATCH v7 08/27] media: entity: Add has_route entity operation
  2021-07-12  7:42       ` Jacopo Mondi
@ 2021-07-26 18:13         ` Sakari Ailus
  0 siblings, 0 replies; 60+ messages in thread
From: Sakari Ailus @ 2021-07-26 18:13 UTC (permalink / raw)
  To: Jacopo Mondi
  Cc: Tomi Valkeinen, linux-media, Jacopo Mondi, Laurent Pinchart,
	niklas.soderlund+renesas, Mauro Carvalho Chehab, Hans Verkuil,
	Pratyush Yadav, Lokesh Vutla, Michal Simek

On Mon, Jul 12, 2021 at 09:42:20AM +0200, Jacopo Mondi wrote:
> Hi Sakari,
> 
> On Sun, Jul 11, 2021 at 06:26:26PM +0300, Sakari Ailus wrote:
> > On Thu, Jul 08, 2021 at 02:43:10PM +0200, Jacopo Mondi wrote:
> > > Hi Tomi,
> > >    a small note
> > >
> > > On Mon, May 24, 2021 at 01:43:49PM +0300, Tomi Valkeinen wrote:
> > > > From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > >
> > > > The optional operation can be used by entities to report whether two
> > > > pads are internally connected.
> > > >
> > > > While at there, fix a Sphinx compiler warning in a comment block a few
> > > > lines above.
> > > >
> > > > Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> > > > Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> > > > Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> > > > Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> > > > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > > > ---
> > > >  include/media/media-entity.h | 7 +++++++
> > > >  1 file changed, 7 insertions(+)
> > > >
> > > > diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> > > > index 516d73a2941e..ad4020b2df65 100644
> > > > --- a/include/media/media-entity.h
> > > > +++ b/include/media/media-entity.h
> > > > @@ -187,6 +187,7 @@ enum media_pad_signal_type {
> > > >   * @flags:	Pad flags, as defined in
> > > >   *		:ref:`include/uapi/linux/media.h <media_header>`
> > > >   *		(seek for ``MEDIA_PAD_FL_*``)
> > > > + *
> > > >   * .. note::
> > > >   *
> > > >   *    @stream_count reference count must never be negative, but is a signed
> > > > @@ -214,6 +215,10 @@ struct media_pad {
> > > >   * @link_validate:	Return whether a link is valid from the entity point of
> > > >   *			view. The media_pipeline_start() function
> > > >   *			validates all links by calling this operation. Optional.
> > > > + * @has_route:		Return whether a route exists inside the entity between
> > > > + *			two given pads. Pads are passed to the operation ordered
> > > > + *			by index. Optional: If the operation isn't implemented
> > >
> > > According to the next patch, this doesn't seem to be 'Optional:' :)
> >
> > How? Few drivers will implement this in the end, and that's been taken into
> > account in code AFAIU.
> 
> What I meant is that it's not optional that all pads are considered
> connected if the op is not implemented. Or maybe this should have read
> as "Implementing the operation is optional" ?

Fine for me. I don't think the old text was bad either though.

-- 
Sakari Ailus

^ permalink raw reply	[flat|nested] 60+ messages in thread

end of thread, other threads:[~2021-07-26 18:13 UTC | newest]

Thread overview: 60+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2021-05-24 10:43 [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 01/27] media: entity: Use pad as a starting point for graph walk Tomi Valkeinen
2021-07-08 10:45   ` Jacopo Mondi
2021-05-24 10:43 ` [PATCH v7 02/27] media: entity: Use pads instead of entities in the media graph walk stack Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 03/27] media: entity: Walk the graph based on pads Tomi Valkeinen
2021-07-08 10:48   ` Jacopo Mondi
2021-05-24 10:43 ` [PATCH v7 04/27] v4l: mc: Start walk from a specific pad in use count calculation Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 05/27] media: entity: Add iterator helper for entity pads Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 06/27] media: entity: Move the pipeline from entity to pads Tomi Valkeinen
2021-07-08 13:11   ` Jacopo Mondi
2021-07-16  6:19     ` Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 07/27] media: entity: Use pad as the starting point for a pipeline Tomi Valkeinen
2021-07-08 12:36   ` Jacopo Mondi
2021-07-11 15:25     ` Sakari Ailus
2021-07-16  6:35     ` Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 08/27] media: entity: Add has_route entity operation Tomi Valkeinen
2021-07-08 12:43   ` Jacopo Mondi
2021-07-11 15:26     ` Sakari Ailus
2021-07-12  7:42       ` Jacopo Mondi
2021-07-26 18:13         ` Sakari Ailus
2021-05-24 10:43 ` [PATCH v7 09/27] media: entity: Add media_entity_has_route() function Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 10/27] media: entity: Use routing information during graph traversal Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 11/27] media: entity: Skip link validation for pads to which there is no route Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 12/27] media: entity: Add an iterator helper for connected pads Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 13/27] media: entity: Add only connected pads to the pipeline Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 14/27] media: entity: Add debug information in graph walk route check Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 15/27] v4l: Add bus type to frame descriptors Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 16/27] v4l: Add CSI-2 bus configuration " Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 17/27] v4l: Add stream to frame descriptor Tomi Valkeinen
2021-05-24 10:43 ` [PATCH v7 18/27] media: Documentation: Add GS_ROUTING documentation Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 19/27] v4l: subdev: Add [GS]_ROUTING subdev ioctls and operations Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 20/27] v4l: subdev: add V4L2_SUBDEV_ROUTE_FL_SOURCE Tomi Valkeinen
2021-06-05 22:44   ` Laurent Pinchart
2021-06-05 22:46     ` Laurent Pinchart
2021-07-02  7:49       ` Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 21/27] v4l: subdev: routing kernel helper functions Tomi Valkeinen
2021-06-05 23:29   ` Laurent Pinchart
2021-07-11 15:48     ` Sakari Ailus
2021-05-24 10:44 ` [PATCH v7 22/27] v4l: subdev: add stream based configuration Tomi Valkeinen
2021-06-05 23:42   ` Laurent Pinchart
2021-07-02  8:56     ` Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 23/27] v4l: subdev: add 'stream' to subdev ioctls Tomi Valkeinen
2021-06-05 23:46   ` Laurent Pinchart
2021-05-24 10:44 ` [PATCH v7 24/27] v4l: subdev: use streams in v4l2_subdev_link_validate() Tomi Valkeinen
2021-05-28 11:34   ` Tomi Valkeinen
2021-06-05 23:59     ` Laurent Pinchart
2021-07-09 10:02       ` Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 25/27] v4l: subdev: add routing & stream config to v4l2_subdev_state Tomi Valkeinen
2021-06-06  0:01   ` Laurent Pinchart
2021-07-02  8:34     ` Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 26/27] v4l: subdev: add V4L2_SUBDEV_FL_MULTIPLEXED Tomi Valkeinen
2021-05-24 10:44 ` [PATCH v7 27/27] v4l: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8 Tomi Valkeinen
2021-05-26  8:25 ` [PATCH v7 00/27] v4l: subdev internal routing and streams Tomi Valkeinen
2021-06-06  0:06 ` Laurent Pinchart
2021-07-09 15:18   ` Jacopo Mondi
2021-07-09 18:26     ` Tomi Valkeinen
2021-07-10  8:42       ` Jacopo Mondi
2021-07-12  8:19         ` Tomi Valkeinen
2021-07-23 10:21           ` Jacopo Mondi
2021-07-26 10:49             ` Tomi Valkeinen

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).