[PATCH v9 00/36] v4l: subdev internal routing and streams

[PATCH v9 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

Hi,

This is v9 of the multiplexed streams series. v8 can be found from:

https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/

I have pushed my work branch to:

git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v9

which contains the patches in this series, along with subdev drivers
using multiplexed streams.

As can be guessed from the work branch, I have been testing this series
with TI's FPDLink setup. I have also done a "backwards compatibility"
test by dropping all multiplexed streams patches from the CAL driver
(the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
single-stream configuration.

The changes since v8:
- Fixed the use of BIT() in uAPI header.
- Documentation updates.
- Added documentation about multiplexed streams to dev-subdev.rst (I had
  this for v8, but missed it from the series...).
- 'which' has been dropped from the subdev state. This affects the
  subdev state allocation (no need to give the try/active parameter) and
  set_routing() subdev op (need to pass 'which').
- Dropped v4l2_subdev_validate_state(), as it's not needed.
- Renamed v4l2_subdev_alloc_state() and v4l2_subdev_free_state() to
  v4l2_subdev_init_finalize() and v4l2_subdev_cleanup(). I'm not super
  happy with the names, but I think it's better to have general init &
  cleanup functions, which can be used for other purposes too.
- Renamed v4l2_alloc_subdev_state() and v4l2_free_subdev_state() to
  __v4l2_subdev_state_alloc() and __v4l2_subdev_state_free() to give an
  indication that they're not supposed to be used from the drivers.

 Tomi

Jacopo Mondi (2):
  media: entity: Add iterator helper for entity pads
  media: Documentation: Add GS_ROUTING documentation

Laurent Pinchart (4):
  media: entity: Add has_route entity operation
  media: entity: Add media_entity_has_route() function
  media: entity: Use routing information during graph traversal
  media: subdev: Add [GS]_ROUTING subdev ioctls and operations

Sakari Ailus (13):
  media: entity: Use pad as a starting point for graph walk
  media: entity: Use pads instead of entities in the media graph walk
    stack
  media: entity: Walk the graph based on pads
  media: mc: Start walk from a specific pad in use count calculation
  media: entity: Move the pipeline from entity to pads
  media: entity: Use pad as the starting point for a pipeline
  media: entity: Skip link validation for pads to which there is no
    route
  media: entity: Add an iterator helper for connected pads
  media: entity: Add only connected pads to the pipeline
  media: entity: Add debug information in graph walk route check
  media: Add bus type to frame descriptors
  media: Add CSI-2 bus configuration to frame descriptors
  media: Add stream to frame descriptor

Tomi Valkeinen (17):
  media: subdev: rename subdev-state alloc & free
  media: subdev: add active state to struct v4l2_subdev
  media: subdev: pass also the active state to subdevs from ioctls
  media: subdev: add subdev state locking
  media: subdev: Add v4l2_subdev_validate_and_lock_state()
  media: Documentation: add documentation about subdev state
  media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
  media: add V4L2_SUBDEV_FL_MULTIPLEXED
  media: subdev: add v4l2_subdev_has_route()
  media: subdev: add v4l2_subdev_set_routing helper()
  media: Documentation: add multiplexed streams documentation
  media: subdev: add stream based configuration
  media: subdev: use streams in v4l2_subdev_link_validate()
  media: subdev: add "opposite" stream helper funcs
  media: subdev: add v4l2_subdev_get_fmt() helper function
  media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
  media: subdev: add v4l2_routing_simple_verify() helper

 Documentation/driver-api/media/mc-core.rst    |  18 +-
 .../driver-api/media/v4l2-subdev.rst          |  36 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 166 ++++
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/mc/mc-device.c                  |  13 +-
 drivers/media/mc/mc-entity.c                  | 257 +++---
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
 .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
 .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
 drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
 drivers/media/platform/exynos4-is/media-dev.c |  20 +-
 drivers/media/platform/omap3isp/isp.c         |   2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
 drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
 .../media/platform/qcom/camss/camss-video.c   |   6 +-
 drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
 drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
 drivers/media/platform/rcar-vin/rcar-v4l2.c   |   5 +-
 .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
 .../media/platform/s3c-camif/camif-capture.c  |   6 +-
 drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
 drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_entity.c     |   6 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
 .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
 drivers/media/usb/au0828/au0828-core.c        |   8 +-
 drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 732 +++++++++++++++++-
 drivers/staging/media/imx/imx-media-utils.c   |   8 +-
 drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
 drivers/staging/media/omap4iss/iss.c          |   2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  40 +-
 drivers/staging/media/omap4iss/iss_video.h    |   2 +-
 drivers/staging/media/tegra-video/tegra210.c  |   6 +-
 drivers/staging/media/tegra-video/vi.c        |   6 +-
 include/media/media-entity.h                  | 143 +++-
 include/media/v4l2-subdev.h                   | 374 ++++++++-
 include/uapi/linux/v4l2-subdev.h              |  85 +-
 52 files changed, 1989 insertions(+), 369 deletions(-)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

-- 
2.25.1

[PATCH v9 01/36] media: subdev: rename subdev-state alloc & free

[Tomi Valkeinen]

v4l2_subdev_alloc_state() and v4l2_subdev_free_state() are not supposed
to be used by the drivers. However, we do have a few drivers that use
those at the moment, so we need to expose these functions for the time
being.

Prefix the functions with __ to mark the functions as internal.

At the same time, rename them to v4l2_subdev_state_alloc and
v4l2_subdev_state_free to match the style used for other functions like
video_device_alloc() and media_request_alloc().

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
 drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
 drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
 drivers/staging/media/tegra-video/vi.c      |  4 ++--
 include/media/v4l2-subdev.h                 | 10 +++++-----
 5 files changed, 17 insertions(+), 17 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index 0d141155f0e3..ba1d16ab1651 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = v4l2_subdev_alloc_state(sd);
+	sd_state = __v4l2_subdev_state_alloc(sd);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
@@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 
 	rvin_format_align(vin, pix);
 done:
-	v4l2_subdev_free_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
 
 	return ret;
 }
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 823c15facd1b..869cadc1468d 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
+	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
@@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
 		entity->ops->destroy(entity);
 	if (entity->subdev.ctrl_handler)
 		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
-	v4l2_subdev_free_state(entity->config);
+	__v4l2_subdev_state_free(entity->config);
 	media_entity_cleanup(&entity->subdev.entity);
 }
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 5d27a27cc2f2..fe49c86a9b02 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 
-	state = v4l2_subdev_alloc_state(sd);
+	state = __v4l2_subdev_state_alloc(sd);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 
 static void subdev_fh_free(struct v4l2_subdev_fh *fh)
 {
-	v4l2_subdev_free_state(fh->state);
+	__v4l2_subdev_state_free(fh->state);
 	fh->state = NULL;
 }
 
@@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
 
 	return ERR_PTR(ret);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_alloc);
 
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 {
 	if (!state)
 		return;
@@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
 	kvfree(state->pads);
 	kfree(state);
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_free);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index d321790b07d9..66b9ce160472 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = v4l2_subdev_alloc_state(subdev);
+	sd_state = __v4l2_subdev_state_alloc(subdev);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
@@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	v4l2_fill_pix_format(pix, &fmt.format);
 	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
 
-	v4l2_subdev_free_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
 
 	return 0;
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 95ec18c2f49c..e52bf508c75b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
 int v4l2_subdev_link_validate(struct media_link *link);
 
 /**
- * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
+ * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
  *
- * Must call v4l2_subdev_free_state() when state is no longer needed.
+ * Must call __v4l2_subdev_state_free() when state is no longer needed.
  */
-struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
 
 /**
- * v4l2_subdev_free_state - free a v4l2_subdev_state
+ * __v4l2_subdev_state_free - free a v4l2_subdev_state
  *
  * @state: v4l2_subdev_state to be freed.
  */
-void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state);
 
 #endif /* CONFIG_MEDIA_CONTROLLER */
 
-- 
2.25.1

[PATCH v9 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

Add a new 'state' field to struct v4l2_subdev to which we can store the
active state of a subdev. This will place the subdev configuration into
a known place, allowing us to use the state directly from the v4l2
framework, thus simplifying the drivers.

Also add functions v4l2_subdev_init_finalize() and
v4l2_subdev_cleanup(), which will allocate and free the active state.
The functions are named in a generic way so that they can be also used
for other subdev initialization work.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
 include/media/v4l2-subdev.h           | 51 +++++++++++++++++++++++++++
 2 files changed, 72 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index fe49c86a9b02..bcaf66a1e3d9 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
+
+int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
+{
+	struct v4l2_subdev_state *state;
+
+	state = __v4l2_subdev_state_alloc(sd);
+	if (IS_ERR(state))
+		return PTR_ERR(state);
+
+	sd->state = state;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
+
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
+{
+	__v4l2_subdev_state_free(sd->state);
+	sd->state = NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index e52bf508c75b..3aaa7146e5ff 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
  * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
  *		     device using v4l2_async_register_subdev_sensor().
  * @pdata: common part of subdevice platform data
+ * @state: active state for the subdev (NULL for subdevs tracking the state
+ *	   internally)
  *
  * Each instance of a subdev driver should create this struct, either
  * stand-alone or embedded in a larger struct.
@@ -929,6 +931,19 @@ struct v4l2_subdev {
 	struct v4l2_async_notifier *notifier;
 	struct v4l2_async_notifier *subdev_notifier;
 	struct v4l2_subdev_platform_data *pdata;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+
+	/*
+	 * TODO: state should most likely be changed from a pointer to an
+	 * embedded field. For the time being it's kept as a pointer to more
+	 * easily catch uses of state in the cases where the driver doesn't
+	 * support it.
+	 */
+	struct v4l2_subdev_state *state;
 };
 
 
@@ -1217,4 +1232,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
 void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 			      const struct v4l2_event *ev);
 
+/**
+ * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
+ * @sd: The subdev
+ *
+ * This finalizes the initialization of the subdev, including allocation of
+ * the active state for the subdev.
+ *
+ * This must be called by the subdev drivers that use the centralized active
+ * state, after the subdev struct has been initialized and
+ * media_entity_pads_init() has been called.
+ *
+ * Must call v4l2_subdev_cleanup() when the subdev is being removed.
+ */
+int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
+ * @sd: The subdevice
+ *
+ * This will release the resources allocated in v4l2_subdev_init_finalize.
+ */
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
+ * @sd: The subdevice
+ *
+ * Return the active state for the subdevice, or NULL if the subdev does not
+ * support active state.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
+{
+	return sd->state;
+}
+
 #endif
-- 
2.25.1

[PATCH v9 03/36] media: subdev: pass also the active state to subdevs from ioctls

[Tomi Valkeinen]

At the moment when a subdev op is called, the TRY subdev state
(subdev_fh->state) is passed as a parameter even for ACTIVE case, or
alternatively a NULL can be passed for ACTIVE case. This used to make
sense, as the ACTIVE state was handled internally by the subdev drivers.

We now have a state for ACTIVE case in a standard place, and can pass
that alto to the drivers. This patch changes the subdev ioctls to either
pass the TRY or ACTIVE state to the subdev.

Unfortunately many drivers call ops from other subdevs, and implicitly
pass NULL as the state, so this is just a partial solution. A coccinelle
spatch could perhaps be created which fixes the drivers' subdev calls.

For all current upstream drivers this doesn't matter, as they do not
expect to get a valid state for ACTIVE case. But future drivers which
support multiplexed streaming and routing will depend on getting a state
for both active and try cases.

For new drivers we can mandate that the pipelines where the drivers are
used need to pass the state properly, or preferably, not call such
subdev ops at all.

However, if an existing subdev driver is changed to support multiplexed
streams, the driver has to consider cases where its ops will be called
with NULL state. The problem can easily be solved by using the
v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up
patch.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
 1 file changed, 63 insertions(+), 10 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index bcaf66a1e3d9..aa757cc28879 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
 EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
 
 #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
+
+static struct v4l2_subdev_state *
+subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
+		       unsigned int cmd, void *arg)
+{
+	u32 which;
+
+	switch (cmd) {
+	default:
+		return NULL;
+
+	case VIDIOC_SUBDEV_G_FMT:
+	case VIDIOC_SUBDEV_S_FMT: {
+		which = ((struct v4l2_subdev_format *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_G_CROP:
+	case VIDIOC_SUBDEV_S_CROP: {
+		which = ((struct v4l2_subdev_crop *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
+		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
+		break;
+	}
+	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
+		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
+		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
+		break;
+	}
+
+	case VIDIOC_SUBDEV_G_SELECTION:
+	case VIDIOC_SUBDEV_S_SELECTION: {
+		which = ((struct v4l2_subdev_selection *)arg)->which;
+		break;
+	}
+	}
+
+	return which == V4L2_SUBDEV_FORMAT_TRY ?
+			     subdev_fh->state :
+			     v4l2_subdev_get_active_state(sd);
+}
+
 static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 {
 	struct video_device *vdev = video_devdata(file);
@@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	struct v4l2_fh *vfh = file->private_data;
 	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
 	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
+	struct v4l2_subdev_state *state;
 	int rval;
 
+	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
+
 	switch (cmd) {
 	case VIDIOC_SUBDEV_QUERYCAP: {
 		struct v4l2_subdev_capability *cap = arg;
@@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_S_FMT: {
@@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(format->reserved, 0, sizeof(format->reserved));
 		memset(format->format.reserved, 0, sizeof(format->format.reserved));
-		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
+		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
 	}
 
 	case VIDIOC_SUBDEV_G_CROP: {
@@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.target = V4L2_SEL_TGT_CROP;
 
 		rval = v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, &sel);
+			sd, pad, get_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		sel.r = crop->rect;
 
 		rval = v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, &sel);
+			sd, pad, set_selection, state, &sel);
 
 		crop->rect = sel.r;
 
@@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_mbus_code_enum *code = arg;
 
 		memset(code->reserved, 0, sizeof(code->reserved));
-		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
 					code);
 	}
 
@@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_size_enum *fse = arg;
 
 		memset(fse->reserved, 0, sizeof(fse->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
 					fse);
 	}
 
@@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 		struct v4l2_subdev_frame_interval_enum *fie = arg;
 
 		memset(fie->reserved, 0, sizeof(fie->reserved));
-		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
+		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
 					fie);
 	}
 
@@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, get_selection, subdev_fh->state, sel);
+			sd, pad, get_selection, state, sel);
 	}
 
 	case VIDIOC_SUBDEV_S_SELECTION: {
@@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 
 		memset(sel->reserved, 0, sizeof(sel->reserved));
 		return v4l2_subdev_call(
-			sd, pad, set_selection, subdev_fh->state, sel);
+			sd, pad, set_selection, state, sel);
 	}
 
 	case VIDIOC_G_EDID: {
@@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
 		struct v4l2_subdev *sd =
 			media_entity_to_v4l2_subdev(pad->entity);
+		struct v4l2_subdev_state *state;
+
+		state = v4l2_subdev_get_active_state(sd);
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
-		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
+		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
 	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
-- 
2.25.1

[PATCH v9 04/36] media: subdev: add subdev state locking

[Tomi Valkeinen]

The V4L2 subdevs have managed without centralized locking for the state
(previously pad_config), as the TRY state is supposedly safe (although I
believe two TRY ioctls for the same fd would race), and the ACTIVE
state, and its locking, is managed by the drivers internally.

We now have ACTIVE state in a centralized position, and need locking.
Strictly speaking the locking is only needed for new drivers that use
the new state, as the current drivers continue behaving as they used to.

Add a mutex to the struct v4l2_subdev_state, along with a few helper
functions for locking/unlocking.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
 drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
 drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
 drivers/staging/media/tegra-video/vi.c      |  4 +-
 include/media/v4l2-subdev.h                 | 49 ++++++++++++++++++++-
 5 files changed, 88 insertions(+), 10 deletions(-)

diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index ba1d16ab1651..e6bd94d63e4f 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 {
 	struct v4l2_subdev *sd = vin_to_source(vin);
 	struct v4l2_subdev_state *sd_state;
+	static struct lock_class_key key;
 	struct v4l2_subdev_format format = {
 		.which = which,
 		.pad = vin->parallel.source_pad,
@@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
 	u32 width, height;
 	int ret;
 
-	sd_state = __v4l2_subdev_state_alloc(sd);
+	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 869cadc1468d..e607c3ae2520 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 		     const char *name, unsigned int num_pads,
 		     const struct v4l2_subdev_ops *ops, u32 function)
 {
+	static struct lock_class_key key;
 	struct v4l2_subdev *subdev;
 	unsigned int i;
 	int ret;
@@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
 	 * Allocate the pad configuration to store formats and selection
 	 * rectangles.
 	 */
-	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
+	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
+						   "vsp1:config->lock", &key);
 	if (IS_ERR(entity->config)) {
 		media_entity_cleanup(&entity->subdev.entity);
 		return PTR_ERR(entity->config);
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index aa757cc28879..52309a299b03 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -27,8 +27,9 @@
 static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
 {
 	struct v4l2_subdev_state *state;
+	static struct lock_class_key key;
 
-	state = __v4l2_subdev_state_alloc(sd);
+	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
+struct v4l2_subdev_state *
+__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
+			  struct lock_class_key *lock_key)
 {
 	struct v4l2_subdev_state *state;
 	int ret;
@@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
 	if (!state)
 		return ERR_PTR(-ENOMEM);
 
+	__mutex_init(&state->lock, lock_name, lock_key);
+
 	if (sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
@@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	if (!state)
 		return;
 
+	mutex_destroy(&state->lock);
+
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
 
-int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key)
 {
 	struct v4l2_subdev_state *state;
 
-	state = __v4l2_subdev_state_alloc(sd);
+	state = __v4l2_subdev_state_alloc(sd, name, key);
 	if (IS_ERR(state))
 		return PTR_ERR(state);
 
@@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
 
 	return 0;
 }
-EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
 
 void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
 {
@@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
 	sd->state = NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
+
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
+{
+	mutex_lock(&sd->state->lock);
+
+	return sd->state;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
+
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
+{
+	mutex_lock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
+
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
+{
+	mutex_unlock(&state->lock);
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index 66b9ce160472..3516735f0cad 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 				      struct v4l2_pix_format *pix)
 {
 	const struct tegra_video_format *fmtinfo;
+	static struct lock_class_key key;
 	struct v4l2_subdev *subdev;
 	struct v4l2_subdev_format fmt;
 	struct v4l2_subdev_state *sd_state;
@@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
 	if (!subdev)
 		return -ENODEV;
 
-	sd_state = __v4l2_subdev_state_alloc(subdev);
+	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
+					     &key);
 	if (IS_ERR(sd_state))
 		return PTR_ERR(sd_state);
 	/*
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 3aaa7146e5ff..a6e8f531a396 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
+ * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
  *
  * This structure only needs to be passed to the pad op if the 'which' field
@@ -662,6 +663,7 @@ struct v4l2_subdev_pad_config {
  * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
  */
 struct v4l2_subdev_state {
+	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
 };
 
@@ -1153,10 +1155,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
  * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
+ * @lock_name: name of the state lock
+ * @key: lock_class_key for the lock
  *
  * Must call __v4l2_subdev_state_free() when state is no longer needed.
  */
-struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
+						    const char *lock_name,
+						    struct lock_class_key *key);
 
 /**
  * __v4l2_subdev_state_free - free a v4l2_subdev_state
@@ -1245,7 +1251,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
  *
  * Must call v4l2_subdev_cleanup() when the subdev is being removed.
  */
-int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
+#define v4l2_subdev_init_finalize(sd)                                          \
+	({                                                                     \
+		static struct lock_class_key __key;                            \
+		const char *name = KBUILD_BASENAME                             \
+			":" __stringify(__LINE__) ":subdev->state->lock";      \
+		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
+	})
+
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key);
 
 /**
  * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
@@ -1268,4 +1283,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
 	return sd->state;
 }
 
+/**
+ * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
+ *				     subdevice
+ * @sd: The subdevice
+ *
+ * Return the locked active state for the subdevice, or NULL if the subdev
+ * does not support active state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
+
+/**
+ * v4l2_subdev_lock_state() - Lock the subdev state
+ * @state: The subdevice state
+ *
+ * Lock the given subdev state.
+ *
+ * Must be unlocked with v4l2_subdev_unlock_state() after use.
+ */
+void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
+
+/**
+ * v4l2_subdev_unlock_state() - Unlock the subdev state
+ * @state: The subdevice state
+ *
+ * Unlock the given subdev state.
+ */
+void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
+
 #endif
-- 
2.25.1

[PATCH v9 05/36] media: subdev: Add v4l2_subdev_validate_and_lock_state()

[Tomi Valkeinen]

All suitable subdev ops are now passed either the TRY or the ACTIVE
state by the v4l2 core. However, other subdev drivers can still call the
ops passing NULL as the state, implying the active case.

For all current upstream drivers this doesn't matter, as they do not
expect to get a valid state for ACTIVE case. But future drivers which
support multiplexed streaming and routing will depend on getting a state
for both active and try cases.

For new drivers we can mandate that the pipelines where the drivers are
used need to pass the state properly, or preferably, not call such
subdev ops at all.

However, if an existing subdev driver is changed to support multiplexed
streams, the driver has to consider cases where its ops will be called
with NULL state. The problem can easily be solved by using the
v4l2_subdev_validate_and_lock_state() helper, introduced here.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 33 +++++++++++++++++++++++++++++++++
 1 file changed, 33 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index a6e8f531a396..b75eeaee492b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1313,4 +1313,37 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
  */
 void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
 
+/**
+ * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev
+ *					   state
+ * @sd: subdevice
+ * @state: subdevice state as passed to the subdev op
+ *
+ * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
+ * NULL as the state parameter, as subdevs always used to have their active
+ * state stored privately.
+ *
+ * However, newer state-aware subdev drivers, which store their active state in
+ * a common place, subdev->state, expect to always get a proper state as a
+ * parameter.
+ *
+ * These state-aware drivers can use v4l2_subdev_validate_and_lock_state()
+ * instead of v4l2_subdev_lock_state(). v4l2_subdev_validate_and_lock_state()
+ * solves the issue by using subdev->state in case the passed state is
+ * NULL.
+ *
+ * This is a temporary helper function, and should be removed when we can ensure
+ * that all drivers pass proper state when calling other subdevs.
+ */
+static inline struct v4l2_subdev_state *
+v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
+				    struct v4l2_subdev_state *state)
+{
+	state = state ? state : sd->state;
+
+	v4l2_subdev_lock_state(state);
+
+	return state;
+}
+
 #endif
-- 
2.25.1

[PATCH v9 06/36] media: Documentation: add documentation about subdev state

[Tomi Valkeinen]

Add documentation about centrally managed subdev state.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
 1 file changed, 28 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index 7736da077fb8..c3fd57fff668 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
 :c:type:`i2c_board_info` structure using the ``client_type`` and the
 ``addr`` to fill it.
 
+Centrally managed subdev active state
+-------------------------------------
+
+Traditionally V4L2 subdev drivers maintained internal state for the active
+configuration for the subdev. This is often implemented as an array of struct
+v4l2_mbus_framefmt, one entry for each pad.
+
+In addition to the active configuration, each subdev filehandle has contained
+an array of struct v4l2_subdev_pad_config, managed by V4L2 core, which
+contains the TRY configuration.
+
+To simplify the subdev drivers the V4L2 subdev API now optionally supports a
+centrally managed active configuration. A subdev driver must use
+v4l2_subdev_init_finalize() to initialize the active state between calls to
+media_entity_pads_init() and v4l2_*_register_subdev(), and must call
+v4l2_subdev_cleanup() to free the state.
+
+The active state must be locked before access, and can be done with
+v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
+
+The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
+Unfortunately all the subdev drivers do not comply with this yet, and may pass
+NULL for the ACTIVE case. This is only a problem for subdev drivers which use
+the cetrally managed active state and are used in media pipelines with older
+subdev drivers. In these cases the called subdev ops must also handle the NULL
+case. This can be easily managed by the use of
+v4l2_subdev_validate_and_lock_state() helper.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
-- 
2.25.1

[PATCH v9 07/36] media: entity: Use pad as a starting point for graph walk

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the upcoming use of the recently added has_route() media entity op,
all the pads in an entity will no longer be considered interconnected.
This has an effect where the media graph is traversed: the starting pad
does make a difference.

Prepare for this change by using pad instead of the entity as an argument
for the graph walk operations. The actual graph traversal algorithm change
is in further patches.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  2 +-
 drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
 drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
 drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
 include/media/media-entity.h                  | 10 ++++------
 9 files changed, 23 insertions(+), 26 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 57b5bbba944e..ba0aee982124 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
 :c:func:`media_graph_walk_start()`
 
 The graph structure, provided by the caller, is initialized to start graph
-traversal at the given entity.
+traversal at the given pad in an entity.
 
 Drivers can then retrieve the next entity by calling
 :c:func:`media_graph_walk_next()`
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index f40f41977142..47cdaa301838 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
 
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity)
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 {
 	media_entity_enum_zero(&graph->ent_enum);
-	media_entity_enum_set(&graph->ent_enum, entity);
+	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
 	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, entity);
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"begin graph walk at '%s'\n", entity->name);
+	stack_push(graph, pad->entity);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
@@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity);
+	media_graph_walk_start(&pipe->graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
@@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
@@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index fa648721eaab..4ec7f7a6b0f6 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity))
@@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err);
+	media_graph_walk_start(graph, entity_err->pads);
 
 	while ((entity_err = media_graph_walk_next(graph))) {
 		if (!is_media_entity_v4l2_video_device(entity_err))
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 8811d6dd4ee7..3c1485d59404 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 044eb5778820..61e4fbaba7b7 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 338c3661d809..af0d55ab6c15 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -189,7 +189,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index b01474717dca..d215fe31b9a2 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
 {
 	int use = 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(entity))
@@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity);
+	media_graph_walk_start(graph, entity->pads);
 
 	while (!ret && (entity = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(entity))
@@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first);
+	media_graph_walk_start(graph, first->pads);
 
 	while ((first = media_graph_walk_next(graph))
 	       && first != entity)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index d0da083deed5..760cd0ab1feb 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 
 	while ((entity = media_graph_walk_next(&graph))) {
 		if (entity == &video->video.entity)
@@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity);
+	media_graph_walk_start(&graph, entity->pads);
 	while ((entity = media_graph_walk_next(&graph)))
 		media_entity_enum_set(&pipe->ent_enum, entity);
 	mutex_unlock(&mdev->graph_mutex);
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 09737b47881f..b9bfcf34eb0a 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
 void media_graph_walk_cleanup(struct media_graph *graph);
 
 /**
- * media_graph_walk_start - Start walking the media graph at a
- *	given entity
+ * media_graph_walk_start - Start walking the media graph at a given pad
  *
  * @graph: Media graph structure that will be used to walk the graph
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * Before using this function, media_graph_walk_init() must be
  * used to allocate resources used for walking the graph. This
  * function initializes the graph traversal structure to walk the
- * entities graph starting at the given entity. The traversal
+ * entities graph starting at the given pad. The traversal
  * structure must not be modified by the caller during graph
  * traversal. After the graph walk, the resources must be released
  * using media_graph_walk_cleanup().
  */
-void media_graph_walk_start(struct media_graph *graph,
-			    struct media_entity *entity);
+void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
 
 /**
  * media_graph_walk_next - Get the next entity in the graph
-- 
2.25.1

[PATCH v9 08/36] media: entity: Use pads instead of entities in the media graph walk stack

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Change the media graph walk stack structure to use media pads instead of
using media entities. In addition to the entity, the pad contains the
information which pad in the entity are being dealt with.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 55 ++++++++++++++++++------------------
 include/media/media-entity.h |  8 +++---
 2 files changed, 31 insertions(+), 32 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 47cdaa301838..e3a932ab9abd 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -228,40 +228,39 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
-static struct media_entity *
-media_entity_other(struct media_entity *entity, struct media_link *link)
+static struct media_pad *
+media_pad_other(struct media_pad *pad, struct media_link *link)
 {
-	if (link->source->entity == entity)
-		return link->sink->entity;
+	if (link->source == pad)
+		return link->sink;
 	else
-		return link->source->entity;
+		return link->source;
 }
 
 /* push an entity to traversal stack */
-static void stack_push(struct media_graph *graph,
-		       struct media_entity *entity)
+static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
 	if (graph->top == MEDIA_ENTITY_ENUM_MAX_DEPTH - 1) {
 		WARN_ON(1);
 		return;
 	}
 	graph->top++;
-	graph->stack[graph->top].link = entity->links.next;
-	graph->stack[graph->top].entity = entity;
+	graph->stack[graph->top].link = pad->entity->links.next;
+	graph->stack[graph->top].pad = pad;
 }
 
-static struct media_entity *stack_pop(struct media_graph *graph)
+static struct media_pad *stack_pop(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
-	entity = graph->stack[graph->top].entity;
+	pad = graph->stack[graph->top].pad;
 	graph->top--;
 
-	return entity;
+	return pad;
 }
 
 #define link_top(en)	((en)->stack[(en)->top].link)
-#define stack_top(en)	((en)->stack[(en)->top].entity)
+#define stack_top(en)	((en)->stack[(en)->top].pad)
 
 /**
  * media_graph_walk_init - Allocate resources for graph walk
@@ -297,8 +296,8 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
 	media_entity_enum_set(&graph->ent_enum, pad->entity);
 
 	graph->top = 0;
-	graph->stack[graph->top].entity = NULL;
-	stack_push(graph, pad->entity);
+	graph->stack[graph->top].pad = NULL;
+	stack_push(graph, pad);
 	dev_dbg(pad->graph_obj.mdev->dev,
 		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
 }
@@ -306,16 +305,16 @@ EXPORT_SYMBOL_GPL(media_graph_walk_start);
 
 static void media_graph_walk_iter(struct media_graph *graph)
 {
-	struct media_entity *entity = stack_top(graph);
+	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_entity *next;
+	struct media_pad *next;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
 	/* The link is not enabled so we do not follow. */
 	if (!(link->flags & MEDIA_LNK_FL_ENABLED)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping disabled link '%s':%u -> '%s':%u\n",
 			link->source->entity->name, link->source->index,
 			link->sink->entity->name, link->sink->index);
@@ -323,23 +322,23 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	}
 
 	/* Get the entity at the other end of the link. */
-	next = media_entity_other(entity, link);
+	next = media_pad_other(pad, link);
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
 		link_top(graph) = link_top(graph)->next;
-		dev_dbg(entity->graph_obj.mdev->dev,
+		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->name);
+			next->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
 	stack_push(graph, next);
-	dev_dbg(entity->graph_obj.mdev->dev, "walk: pushing '%s' on stack\n",
-		next->name);
-	lockdep_assert_held(&entity->graph_obj.mdev->graph_mutex);
+	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		next->entity->name, next->index);
+	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
 struct media_entity *media_graph_walk_next(struct media_graph *graph)
@@ -354,10 +353,10 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	 * top of the stack until no more entities on the level can be
 	 * found.
 	 */
-	while (link_top(graph) != &stack_top(graph)->links)
+	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph);
+	entity = stack_pop(graph)->entity;
 	dev_dbg(entity->graph_obj.mdev->dev,
 		"walk: returning entity '%s'\n", entity->name);
 
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b9bfcf34eb0a..5b55d6179e13 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -78,16 +78,16 @@ struct media_entity_enum {
  * struct media_graph - Media graph traversal state
  *
  * @stack:		Graph traversal stack; the stack contains information
- *			on the path the media entities to be walked and the
- *			links through which they were reached.
- * @stack.entity:	pointer to &struct media_entity at the graph.
+ *			on the media pads to be walked and the links through
+ *			which they were reached.
+ * @stack.pad:		pointer to &struct media_pad at the graph.
  * @stack.link:		pointer to &struct list_head.
  * @ent_enum:		Visited entities
  * @top:		The top of the stack
  */
 struct media_graph {
 	struct {
-		struct media_entity *entity;
+		struct media_pad *pad;
 		struct list_head *link;
 	} stack[MEDIA_ENTITY_ENUM_MAX_DEPTH];
 
-- 
2.25.1

[PATCH v9 09/36] media: entity: Walk the graph based on pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Instead of iterating over graph entities during the walk, iterate the pads
through which the entity was first reached. This is required in order to
make the entity pipeline pad-based rather than entity based.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  7 ++-
 drivers/media/mc/mc-entity.c                  | 51 +++++++++++--------
 drivers/media/platform/exynos4-is/media-dev.c | 20 ++++----
 drivers/media/platform/omap3isp/ispvideo.c    | 17 ++++---
 drivers/media/platform/vsp1/vsp1_video.c      | 12 ++---
 drivers/media/platform/xilinx/xilinx-dma.c    | 12 ++---
 drivers/media/v4l2-core/v4l2-mc.c             | 24 ++++-----
 drivers/staging/media/omap4iss/iss_video.c    | 36 ++++++-------
 include/media/media-entity.h                  |  7 +--
 9 files changed, 101 insertions(+), 85 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index ba0aee982124..8a13640bed56 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -169,8 +169,11 @@ Drivers initiate a graph traversal by calling
 The graph structure, provided by the caller, is initialized to start graph
 traversal at the given pad in an entity.
 
-Drivers can then retrieve the next entity by calling
-:c:func:`media_graph_walk_next()`
+Drivers can then retrieve the next pad by calling
+:c:func:`media_graph_walk_next()`. Only the pad through which the entity
+is first reached is returned. If the caller is interested in knowing which
+further pads would be connected, the :c:func:`media_entity_has_route()`
+function can be used for that.
 
 When the graph traversal is complete the function will return ``NULL``.
 
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e3a932ab9abd..06147f38ce2e 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -341,9 +341,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
 }
 
-struct media_entity *media_graph_walk_next(struct media_graph *graph)
+struct media_pad *media_graph_walk_next(struct media_graph *graph)
 {
-	struct media_entity *entity;
+	struct media_pad *pad;
 
 	if (stack_top(graph) == NULL)
 		return NULL;
@@ -356,11 +356,11 @@ struct media_entity *media_graph_walk_next(struct media_graph *graph)
 	while (link_top(graph) != &stack_top(graph)->entity->links)
 		media_graph_walk_iter(graph);
 
-	entity = stack_pop(graph)->entity;
-	dev_dbg(entity->graph_obj.mdev->dev,
-		"walk: returning entity '%s'\n", entity->name);
+	pad = stack_pop(graph);
+	dev_dbg(pad->graph_obj.mdev->dev,
+		"walk: returning pad '%s':%u\n", pad->entity->name, pad->index);
 
-	return entity;
+	return pad;
 }
 EXPORT_SYMBOL_GPL(media_graph_walk_next);
 
@@ -408,7 +408,8 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad = entity->pads;
+	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
 
@@ -418,9 +419,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 			goto error_graph_walk_start;
 	}
 
-	media_graph_walk_start(&pipe->graph, entity->pads);
+	media_graph_walk_start(&pipe->graph, pad);
+
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
 
-	while ((entity = media_graph_walk_next(graph))) {
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
@@ -429,7 +432,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		if (entity->pipe && entity->pipe != pipe) {
 			pr_err("Pipe active for %s. Can't start for %s\n",
 				entity->name,
-				entity_err->name);
+				pad_err->entity->name);
 			ret = -EBUSY;
 			goto error;
 		}
@@ -447,26 +450,27 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		bitmap_fill(has_no_links, entity->num_pads);
 
 		list_for_each_entry(link, &entity->links, list) {
-			struct media_pad *pad = link->sink->entity == entity
-						? link->sink : link->source;
+			struct media_pad *other_pad =
+				link->sink->entity == entity ?
+				link->sink : link->source;
 
 			/* Mark that a pad is connected by a link. */
-			bitmap_clear(has_no_links, pad->index, 1);
+			bitmap_clear(has_no_links, other_pad->index, 1);
 
 			/*
 			 * Pads that either do not need to connect or
 			 * are connected through an enabled link are
 			 * fine.
 			 */
-			if (!(pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
+			if (!(other_pad->flags & MEDIA_PAD_FL_MUST_CONNECT) ||
 			    link->flags & MEDIA_LNK_FL_ENABLED)
-				bitmap_set(active, pad->index, 1);
+				bitmap_set(active, other_pad->index, 1);
 
 			/*
 			 * Link validation will only take place for
 			 * sink ends of the link that are enabled.
 			 */
-			if (link->sink != pad ||
+			if (link->sink != other_pad ||
 			    !(link->flags & MEDIA_LNK_FL_ENABLED))
 				continue;
 
@@ -502,9 +506,11 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	 * Link validation on graph failed. We revert what we did and
 	 * return the error.
 	 */
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
+
+	while ((pad_err = media_graph_walk_next(graph))) {
+		struct media_entity *entity_err = pad_err->entity;
 
-	while ((entity_err = media_graph_walk_next(graph))) {
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
 			entity_err->stream_count--;
@@ -516,7 +522,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
@@ -543,8 +549,9 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_graph *graph = &entity->pipe->graph;
 	struct media_pipeline *pipe = entity->pipe;
+	struct media_graph *graph = &pipe->graph;
+	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -555,7 +562,9 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
+	while ((pad = media_graph_walk_next(graph))) {
+		struct media_entity *entity = pad->entity;
+
 		/* Sanity check for negative stream_count */
 		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
 			entity->stream_count--;
diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
index 4ec7f7a6b0f6..10429466a394 100644
--- a/drivers/media/platform/exynos4-is/media-dev.c
+++ b/drivers/media/platform/exynos4-is/media-dev.c
@@ -1164,7 +1164,7 @@ static int __fimc_md_modify_pipeline(struct media_entity *entity, bool enable)
 static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 				      struct media_graph *graph)
 {
-	struct media_entity *entity_err = entity;
+	struct media_pad *pad, *pad_err = entity->pads;
 	int ret;
 
 	/*
@@ -1173,13 +1173,13 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	 * through active links. This is needed as we cannot power on/off the
 	 * subdevs in random order.
 	 */
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity))
+	while ((pad = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		ret  = __fimc_md_modify_pipeline(entity, enable);
+		ret  = __fimc_md_modify_pipeline(pad->entity, enable);
 
 		if (ret < 0)
 			goto err;
@@ -1188,15 +1188,15 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
 	return 0;
 
 err:
-	media_graph_walk_start(graph, entity_err->pads);
+	media_graph_walk_start(graph, pad_err);
 
-	while ((entity_err = media_graph_walk_next(graph))) {
-		if (!is_media_entity_v4l2_video_device(entity_err))
+	while ((pad_err = media_graph_walk_next(graph))) {
+		if (!is_media_entity_v4l2_video_device(pad_err->entity))
 			continue;
 
-		__fimc_md_modify_pipeline(entity_err, !enable);
+		__fimc_md_modify_pipeline(pad_err->entity, !enable);
 
-		if (entity_err == entity)
+		if (pad_err == pad)
 			break;
 	}
 
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 3c1485d59404..5c1cbb1a9003 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -222,8 +222,8 @@ static int isp_video_get_graph_data(struct isp_video *video,
 				    struct isp_pipeline *pipe)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct isp_video *far_end = NULL;
 	int ret;
 
@@ -234,23 +234,24 @@ static int isp_video_get_graph_data(struct isp_video *video,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct isp_video *__video;
 
-		media_entity_enum_set(&pipe->ent_enum, entity);
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 
 		if (far_end != NULL)
 			continue;
 
-		if (entity == &video->video.entity)
+		if (pad == video->video.entity.pads)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		__video = to_isp_video(media_entity_to_video_device(entity));
+		__video = to_isp_video(media_entity_to_video_device(
+					       pad->entity));
 		if (__video->type != video->type)
 			far_end = __video;
 	}
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index 61e4fbaba7b7..f2c36f2fdf53 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -559,8 +559,8 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 				     struct vsp1_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	unsigned int i;
 	int ret;
 
@@ -569,17 +569,17 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
 	if (ret)
 		return ret;
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct v4l2_subdev *subdev;
 		struct vsp1_rwpf *rwpf;
 		struct vsp1_entity *e;
 
-		if (!is_media_entity_v4l2_subdev(entity))
+		if (!is_media_entity_v4l2_subdev(pad->entity))
 			continue;
 
-		subdev = media_entity_to_v4l2_subdev(entity);
+		subdev = media_entity_to_v4l2_subdev(pad->entity);
 		e = to_vsp1_entity(subdev);
 		list_add_tail(&e->list_pipe, &pipe->entities);
 		e->pipe = pipe;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index af0d55ab6c15..d33f99c6ffa4 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -174,8 +174,8 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 				  struct xvip_dma *start)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &start->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = start->video.entity.pads;
+	struct media_device *mdev = pad->entity->graph_obj.mdev;
 	unsigned int num_inputs = 0;
 	unsigned int num_outputs = 0;
 	int ret;
@@ -189,15 +189,15 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
 		return ret;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
+	while ((pad = media_graph_walk_next(&graph))) {
 		struct xvip_dma *dma;
 
-		if (entity->function != MEDIA_ENT_F_IO_V4L)
+		if (pad->entity->function != MEDIA_ENT_F_IO_V4L)
 			continue;
 
-		dma = to_xvip_dma(media_entity_to_video_device(entity));
+		dma = to_xvip_dma(media_entity_to_video_device(pad->entity));
 
 		if (dma->pad.flags & MEDIA_PAD_FL_SINK) {
 			pipe->output = dma;
diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index d215fe31b9a2..cbeb580c6754 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -434,13 +434,14 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 static int pipeline_pm_use_count(struct media_entity *entity,
 	struct media_graph *graph)
 {
+	struct media_pad *pad;
 	int use = 0;
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while ((entity = media_graph_walk_next(graph))) {
-		if (is_media_entity_v4l2_video_device(entity))
-			use += entity->use_count;
+	while ((pad = media_graph_walk_next(graph))) {
+		if (is_media_entity_v4l2_video_device(pad->entity))
+			use += pad->entity->use_count;
 	}
 
 	return use;
@@ -493,7 +494,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 static int pipeline_pm_power(struct media_entity *entity, int change,
 	struct media_graph *graph)
 {
-	struct media_entity *first = entity;
+	struct media_pad *tmp_pad, *pad;
 	int ret = 0;
 
 	if (!change)
@@ -501,19 +502,18 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 
 	media_graph_walk_start(graph, entity->pads);
 
-	while (!ret && (entity = media_graph_walk_next(graph)))
-		if (is_media_entity_v4l2_subdev(entity))
-			ret = pipeline_pm_power_one(entity, change);
+	while (!ret && (pad = media_graph_walk_next(graph)))
+		if (is_media_entity_v4l2_subdev(pad->entity))
+			ret = pipeline_pm_power_one(pad->entity, change);
 
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, first->pads);
+	media_graph_walk_start(graph, entity->pads);
 
-	while ((first = media_graph_walk_next(graph))
-	       && first != entity)
-		if (is_media_entity_v4l2_subdev(first))
-			pipeline_pm_power_one(first, -change);
+	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
+		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
+			pipeline_pm_power_one(tmp_pad->entity, -change);
 
 	return ret;
 }
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 760cd0ab1feb..8c25ad73a81e 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -206,8 +206,8 @@ static struct iss_video *
 iss_video_far_end(struct iss_video *video)
 {
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	struct iss_video *far_end = NULL;
 
 	mutex_lock(&mdev->graph_mutex);
@@ -217,16 +217,17 @@ iss_video_far_end(struct iss_video *video)
 		return NULL;
 	}
 
-	media_graph_walk_start(&graph, entity->pads);
+	media_graph_walk_start(&graph, pad);
 
-	while ((entity = media_graph_walk_next(&graph))) {
-		if (entity == &video->video.entity)
+	while ((pad = media_graph_walk_next(&graph))) {
+		if (pad->entity == &video->video.entity)
 			continue;
 
-		if (!is_media_entity_v4l2_video_device(entity))
+		if (!is_media_entity_v4l2_video_device(pad->entity))
 			continue;
 
-		far_end = to_iss_video(media_entity_to_video_device(entity));
+		far_end = to_iss_video(media_entity_to_video_device(
+						pad->entity));
 		if (far_end->type != video->type)
 			break;
 
@@ -853,8 +854,8 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	struct iss_video_fh *vfh = to_iss_video_fh(fh);
 	struct iss_video *video = video_drvdata(file);
 	struct media_graph graph;
-	struct media_entity *entity = &video->video.entity;
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_pad *pad = video->video.entity.pads;
+	struct media_device *mdev = video->video.entity.graph_obj.mdev;
 	enum iss_pipeline_state state;
 	struct iss_pipeline *pipe;
 	struct iss_video *far_end;
@@ -870,31 +871,32 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = entity->pipe
-	     ? to_iss_pipeline(entity) : &video->pipe;
+	pipe = pad->entity->pipe
+	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
 	pipe->external_bpp = 0;
 
-	ret = media_entity_enum_init(&pipe->ent_enum, entity->graph_obj.mdev);
+	ret = media_entity_enum_init(&pipe->ent_enum,
+				     pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
-	ret = media_graph_walk_init(&graph, entity->graph_obj.mdev);
+	ret = media_graph_walk_init(&graph, pad->entity->graph_obj.mdev);
 	if (ret)
 		goto err_graph_walk_init;
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(entity, &pipe->pipe);
+	ret = media_pipeline_start(pad->entity, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
 	mutex_lock(&mdev->graph_mutex);
-	media_graph_walk_start(&graph, entity->pads);
-	while ((entity = media_graph_walk_next(&graph)))
-		media_entity_enum_set(&pipe->ent_enum, entity);
+	media_graph_walk_start(&graph, pad);
+	while ((pad = media_graph_walk_next(&graph)))
+		media_entity_enum_set(&pipe->ent_enum, pad->entity);
 	mutex_unlock(&mdev->graph_mutex);
 
 	/*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5b55d6179e13..926fd201eae3 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -926,10 +926,11 @@ void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
  * The graph structure must have been previously initialized with a call to
  * media_graph_walk_start().
  *
- * Return: returns the next entity in the graph or %NULL if the whole graph
- * have been traversed.
+ * Return: returns the next entity in the graph, identified by the pad through
+ * which it has been reached. If the whole graph has been traversed, return
+ * %NULL.
  */
-struct media_entity *media_graph_walk_next(struct media_graph *graph);
+struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
-- 
2.25.1

[PATCH v9 10/36] media: mc: Start walk from a specific pad in use count calculation

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

With the addition of the recent has_route() media entity op, the pads of a
media entity are no longer all interconnected. This has to be taken into
account in power management.

Prepare for the addition of a helper function supporting S_ROUTING.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-mc.c | 27 +++++++++++++--------------
 1 file changed, 13 insertions(+), 14 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
index cbeb580c6754..35d18ed89fa5 100644
--- a/drivers/media/v4l2-core/v4l2-mc.c
+++ b/drivers/media/v4l2-core/v4l2-mc.c
@@ -427,17 +427,16 @@ EXPORT_SYMBOL_GPL(v4l2_create_fwnode_links);
 
 /*
  * pipeline_pm_use_count - Count the number of users of a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  *
  * Return the total number of users of all video device nodes in the pipeline.
  */
-static int pipeline_pm_use_count(struct media_entity *entity,
-	struct media_graph *graph)
+static int pipeline_pm_use_count(struct media_pad *pad,
+				 struct media_graph *graph)
 {
-	struct media_pad *pad;
 	int use = 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		if (is_media_entity_v4l2_video_device(pad->entity))
@@ -483,7 +482,7 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
 
 /*
  * pipeline_pm_power - Apply power change to all entities in a pipeline
- * @entity: The entity
+ * @pad: Any pad along the pipeline
  * @change: Use count change
  *
  * Walk the pipeline to update the use count and the power state of all non-node
@@ -491,16 +490,16 @@ static int pipeline_pm_power_one(struct media_entity *entity, int change)
  *
  * Return 0 on success or a negative error code on failure.
  */
-static int pipeline_pm_power(struct media_entity *entity, int change,
-	struct media_graph *graph)
+static int pipeline_pm_power(struct media_pad *pad, int change,
+			     struct media_graph *graph)
 {
-	struct media_pad *tmp_pad, *pad;
+	struct media_pad *tmp_pad, *first = pad;
 	int ret = 0;
 
 	if (!change)
 		return 0;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while (!ret && (pad = media_graph_walk_next(graph)))
 		if (is_media_entity_v4l2_subdev(pad->entity))
@@ -509,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
 	if (!ret)
 		return ret;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, first);
 
 	while ((tmp_pad = media_graph_walk_next(graph)) && tmp_pad != pad)
 		if (is_media_entity_v4l2_subdev(tmp_pad->entity))
@@ -531,7 +530,7 @@ static int v4l2_pipeline_pm_use(struct media_entity *entity, unsigned int use)
 	WARN_ON(entity->use_count < 0);
 
 	/* Apply power change to connected non-nodes. */
-	ret = pipeline_pm_power(entity, change, &mdev->pm_count_walk);
+	ret = pipeline_pm_power(entity->pads, change, &mdev->pm_count_walk);
 	if (ret < 0)
 		entity->use_count -= change;
 
@@ -557,8 +556,8 @@ int v4l2_pipeline_link_notify(struct media_link *link, u32 flags,
 			      unsigned int notification)
 {
 	struct media_graph *graph = &link->graph_obj.mdev->pm_count_walk;
-	struct media_entity *source = link->source->entity;
-	struct media_entity *sink = link->sink->entity;
+	struct media_pad *source = link->source;
+	struct media_pad *sink = link->sink;
 	int source_use;
 	int sink_use;
 	int ret = 0;
-- 
2.25.1

[PATCH v9 11/36] media: entity: Add iterator helper for entity pads

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add an iterator helper to easily cycle through all pads in an entity and
use it in media-entity and media-device code where appropriate.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-device.c | 13 ++++++-------
 drivers/media/mc/mc-entity.c | 11 ++++++-----
 include/media/media-entity.h | 12 ++++++++++++
 3 files changed, 24 insertions(+), 12 deletions(-)

diff --git a/drivers/media/mc/mc-device.c b/drivers/media/mc/mc-device.c
index cf5e459b1d96..cb569beab99e 100644
--- a/drivers/media/mc/mc-device.c
+++ b/drivers/media/mc/mc-device.c
@@ -581,7 +581,7 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	struct media_device *mdev = entity->graph_obj.mdev;
 	struct media_link *link, *tmp;
 	struct media_interface *intf;
-	unsigned int i;
+	struct media_pad *iter;
 
 	ida_free(&mdev->entity_internal_idx, entity->internal_idx);
 
@@ -597,8 +597,8 @@ static void __media_device_unregister_entity(struct media_entity *entity)
 	__media_entity_remove_links(entity);
 
 	/* Remove all pads that belong to this entity */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_destroy(&entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_destroy(&iter->graph_obj);
 
 	/* Remove the entity */
 	media_gobj_destroy(&entity->graph_obj);
@@ -617,7 +617,7 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 					      struct media_entity *entity)
 {
 	struct media_entity_notify *notify, *next;
-	unsigned int i;
+	struct media_pad *iter;
 	int ret;
 
 	if (entity->function == MEDIA_ENT_F_V4L2_SUBDEV_UNKNOWN ||
@@ -646,9 +646,8 @@ int __must_check media_device_register_entity(struct media_device *mdev,
 	media_gobj_create(mdev, MEDIA_GRAPH_ENTITY, &entity->graph_obj);
 
 	/* Initialize objects at the pads */
-	for (i = 0; i < entity->num_pads; i++)
-		media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-			       &entity->pads[i].graph_obj);
+	media_entity_for_each_pad(entity, iter)
+		media_gobj_create(mdev, MEDIA_GRAPH_PAD, &iter->graph_obj);
 
 	/* invoke entity_notify callbacks */
 	list_for_each_entry_safe(notify, next, &mdev->entity_notify, list)
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 06147f38ce2e..2b438c481812 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -198,7 +198,8 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 			   struct media_pad *pads)
 {
 	struct media_device *mdev = entity->graph_obj.mdev;
-	unsigned int i;
+	struct media_pad *iter;
+	unsigned int i = 0;
 
 	if (num_pads >= MEDIA_ENTITY_MAX_PADS)
 		return -E2BIG;
@@ -209,12 +210,12 @@ int media_entity_pads_init(struct media_entity *entity, u16 num_pads,
 	if (mdev)
 		mutex_lock(&mdev->graph_mutex);
 
-	for (i = 0; i < num_pads; i++) {
-		pads[i].entity = entity;
-		pads[i].index = i;
+	media_entity_for_each_pad(entity, iter) {
+		iter->entity = entity;
+		iter->index = i++;
 		if (mdev)
 			media_gobj_create(mdev, MEDIA_GRAPH_PAD,
-					&entity->pads[i].graph_obj);
+					&iter->graph_obj);
 	}
 
 	if (mdev)
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 926fd201eae3..5f6eed24e63f 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -1107,3 +1107,15 @@ void media_remove_intf_links(struct media_interface *intf);
 	 (entity)->ops->operation((entity) , ##args) : -ENOIOCTLCMD)
 
 #endif
+
+/**
+ * media_entity_for_each_pad - Iterate on all pads in an entity
+ * @entity: The entity the pads belong to
+ * @iter: The iterator pad
+ *
+ * Iterate on all pads in a media entity.
+ */
+#define media_entity_for_each_pad(entity, iter)			\
+	for (iter = (entity)->pads;				\
+	     iter < &(entity)->pads[(entity)->num_pads];	\
+	     ++iter)
-- 
2.25.1

[PATCH v9 12/36] media: entity: Move the pipeline from entity to pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

This moves the pipe and stream_count fields from struct media_entity to
struct media_pad. Effectively streams become pad-specific rather than
being entity specific, allowing several independent streams to traverse a
single entity and an entity to be part of several streams.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Update documentation to use 'pads'
- Use the media pad iterator in media_entity.c
- Update rcar-dma.c to use the new per-pad stream count
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix cleanup in the error path of __media_pipeline_start()
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c                  | 68 +++++++++++--------
 drivers/media/platform/exynos4-is/fimc-isp.c  |  2 +-
 drivers/media/platform/exynos4-is/fimc-lite.c |  2 +-
 drivers/media/platform/omap3isp/isp.c         |  2 +-
 drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
 drivers/media/platform/omap3isp/ispvideo.h    |  2 +-
 drivers/media/platform/rcar-vin/rcar-core.c   | 16 +++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
 drivers/media/platform/xilinx/xilinx-dma.h    |  2 +-
 drivers/staging/media/imx/imx-media-utils.c   |  2 +-
 drivers/staging/media/omap4iss/iss.c          |  2 +-
 drivers/staging/media/omap4iss/iss_video.c    |  2 +-
 drivers/staging/media/omap4iss/iss_video.h    |  2 +-
 include/media/media-entity.h                  | 21 +++---
 15 files changed, 74 insertions(+), 55 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 2b438c481812..8ad4cb845f4a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -424,24 +424,30 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
+		bool skip_validation = pad->pipe != NULL;
+		struct media_pad *iter;
 
 		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
 		DECLARE_BITMAP(has_no_links, MEDIA_ENTITY_MAX_PADS);
 
-		entity->stream_count++;
+		ret = 0;
 
-		if (entity->pipe && entity->pipe != pipe) {
-			pr_err("Pipe active for %s. Can't start for %s\n",
-				entity->name,
-				pad_err->entity->name);
-			ret = -EBUSY;
-			goto error;
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->pipe && iter->pipe != pipe) {
+				pr_err("Pipe active for %s. Can't start for %s\n",
+				       entity->name, iter->entity->name);
+				ret = -EBUSY;
+			} else {
+				iter->pipe = pipe;
+			}
+			iter->stream_count++;
 		}
 
-		entity->pipe = pipe;
+		if (ret)
+			goto error;
 
-		/* Already streaming --- no need to check. */
-		if (entity->stream_count > 1)
+		/* Already part of the pipeline, skip validation. */
+		if (skip_validation)
 			continue;
 
 		if (!entity->ops || !entity->ops->link_validate)
@@ -510,20 +516,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity_err = pad_err->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity_err->stream_count <= 0)) {
-			entity_err->stream_count--;
-			if (entity_err->stream_count == 0)
-				entity_err->pipe = NULL;
+		struct media_entity *entity = pad_err->entity;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				--iter->stream_count;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 
 		/*
 		 * We haven't increased stream_count further than this
 		 * so we quit here.
 		 */
-		if (pad_err == pad)
+		if (pad_err->entity == pad->entity)
 			break;
 	}
 
@@ -550,7 +559,7 @@ EXPORT_SYMBOL_GPL(media_pipeline_start);
 
 void __media_pipeline_stop(struct media_entity *entity)
 {
-	struct media_pipeline *pipe = entity->pipe;
+	struct media_pipeline *pipe = entity->pads->pipe;
 	struct media_graph *graph = &pipe->graph;
 	struct media_pad *pad;
 
@@ -565,12 +574,15 @@ void __media_pipeline_stop(struct media_entity *entity)
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
-
-		/* Sanity check for negative stream_count */
-		if (!WARN_ON_ONCE(entity->stream_count <= 0)) {
-			entity->stream_count--;
-			if (entity->stream_count == 0)
-				entity->pipe = NULL;
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			/* Sanity check for negative stream_count */
+			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
+				iter->stream_count--;
+				if (iter->stream_count == 0)
+					iter->pipe = NULL;
+			}
 		}
 	}
 
@@ -840,7 +852,7 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 {
 	const u32 mask = MEDIA_LNK_FL_ENABLED;
 	struct media_device *mdev;
-	struct media_entity *source, *sink;
+	struct media_pad *source, *sink;
 	int ret = -EBUSY;
 
 	if (link == NULL)
@@ -856,8 +868,8 @@ int __media_entity_setup_link(struct media_link *link, u32 flags)
 	if (link->flags == flags)
 		return 0;
 
-	source = link->source->entity;
-	sink = link->sink->entity;
+	source = link->source;
+	sink = link->sink;
 
 	if (!(link->flags & MEDIA_LNK_FL_DYNAMIC) &&
 	    (source->stream_count || sink->stream_count))
diff --git a/drivers/media/platform/exynos4-is/fimc-isp.c b/drivers/media/platform/exynos4-is/fimc-isp.c
index 855235bea46d..80274e29ccc5 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp.c
@@ -226,7 +226,7 @@ static int fimc_isp_subdev_set_fmt(struct v4l2_subdev *sd,
 			}
 		}
 	} else {
-		if (sd->entity.stream_count == 0) {
+		if (sd->entity.pads->stream_count == 0) {
 			if (fmt->pad == FIMC_ISP_SD_PAD_SINK) {
 				struct v4l2_subdev_format format = *fmt;
 
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index aaa3af0493ce..67bfb1ad2ba2 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -1073,7 +1073,7 @@ static int fimc_lite_subdev_set_fmt(struct v4l2_subdev *sd,
 	mutex_lock(&fimc->lock);
 
 	if ((atomic_read(&fimc->out_path) == FIMC_IO_ISP &&
-	    sd->entity.stream_count > 0) ||
+	    sd->entity.pads->stream_count > 0) ||
 	    (atomic_read(&fimc->out_path) == FIMC_IO_DMA &&
 	    vb2_is_busy(&fimc->vb_queue))) {
 		mutex_unlock(&fimc->lock);
diff --git a/drivers/media/platform/omap3isp/isp.c b/drivers/media/platform/omap3isp/isp.c
index 20f59c59ff8a..28aab16d7662 100644
--- a/drivers/media/platform/omap3isp/isp.c
+++ b/drivers/media/platform/omap3isp/isp.c
@@ -936,7 +936,7 @@ static int isp_pipeline_is_last(struct media_entity *me)
 	struct isp_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_isp_pipeline(me);
 	if (pipe->stream_state == ISP_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index 5c1cbb1a9003..a8438040c4aa 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1094,7 +1094,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	/* Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = video->video.entity.pipe
+	pipe = video->video.entity.pads->pipe
 	     ? to_isp_pipeline(&video->video.entity) : &video->pipe;
 
 	ret = media_entity_enum_init(&pipe->ent_enum, &video->isp->media_dev);
diff --git a/drivers/media/platform/omap3isp/ispvideo.h b/drivers/media/platform/omap3isp/ispvideo.h
index a0908670c0cf..4c9c5b719ec5 100644
--- a/drivers/media/platform/omap3isp/ispvideo.h
+++ b/drivers/media/platform/omap3isp/ispvideo.h
@@ -100,7 +100,7 @@ struct isp_pipeline {
 };
 
 #define to_isp_pipeline(__e) \
-	container_of((__e)->pipe, struct isp_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct isp_pipeline, pipe)
 
 static inline int isp_pipeline_ready(struct isp_pipeline *pipe)
 {
diff --git a/drivers/media/platform/rcar-vin/rcar-core.c b/drivers/media/platform/rcar-vin/rcar-core.c
index 33957cc9118c..e59453d6b7c3 100644
--- a/drivers/media/platform/rcar-vin/rcar-core.c
+++ b/drivers/media/platform/rcar-vin/rcar-core.c
@@ -132,13 +132,17 @@ static int rvin_group_link_notify(struct media_link *link, u32 flags,
 		return 0;
 
 	/*
-	 * Don't allow link changes if any entity in the graph is
-	 * streaming, modifying the CHSEL register fields can disrupt
-	 * running streams.
+	 * Don't allow link changes if any stream in the graph is active as
+	 * modifying the CHSEL register fields can disrupt running streams.
 	 */
-	media_device_for_each_entity(entity, &group->mdev)
-		if (entity->stream_count)
-			return -EBUSY;
+	media_device_for_each_entity(entity, &group->mdev) {
+		struct media_pad *iter;
+
+		media_entity_for_each_pad(entity, iter) {
+			if (iter->stream_count)
+				return -EBUSY;
+		}
+	}
 
 	mutex_lock(&group->lock);
 
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index f5f722ab1d4e..80b7ae47d165 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1231,7 +1231,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	 */
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
-	pipe = sd->entity.pipe ? sd->entity.pipe : &vin->vdev.pipe;
+	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
 	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index d33f99c6ffa4..03ee19d00041 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -402,7 +402,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	 * Use the pipeline object embedded in the first DMA object that starts
 	 * streaming.
 	 */
-	pipe = dma->video.entity.pipe
+	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
 	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
diff --git a/drivers/media/platform/xilinx/xilinx-dma.h b/drivers/media/platform/xilinx/xilinx-dma.h
index 2378bdae57ae..69ced71a5696 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.h
+++ b/drivers/media/platform/xilinx/xilinx-dma.h
@@ -47,7 +47,7 @@ struct xvip_pipeline {
 
 static inline struct xvip_pipeline *to_xvip_pipeline(struct media_entity *e)
 {
-	return container_of(e->pipe, struct xvip_pipeline, pipe);
+	return container_of(e->pads->pipe, struct xvip_pipeline, pipe);
 }
 
 /**
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 6f90acf9c725..535da4dda3c6 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -913,7 +913,7 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 			__media_pipeline_stop(entity);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
-		if (entity->pipe)
+		if (entity->pads->pipe)
 			__media_pipeline_stop(entity);
 	}
 
diff --git a/drivers/staging/media/omap4iss/iss.c b/drivers/staging/media/omap4iss/iss.c
index 68588e9dab0b..4c6f25aa8b57 100644
--- a/drivers/staging/media/omap4iss/iss.c
+++ b/drivers/staging/media/omap4iss/iss.c
@@ -548,7 +548,7 @@ static int iss_pipeline_is_last(struct media_entity *me)
 	struct iss_pipeline *pipe;
 	struct media_pad *pad;
 
-	if (!me->pipe)
+	if (!me->pads->pipe)
 		return 0;
 	pipe = to_iss_pipeline(me);
 	if (pipe->stream_state == ISS_PIPELINE_STREAM_STOPPED)
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index 8c25ad73a81e..b74f7891711d 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -871,7 +871,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	 * Start streaming on the pipeline. No link touching an entity in the
 	 * pipeline can be activated or deactivated once streaming is started.
 	 */
-	pipe = pad->entity->pipe
+	pipe = pad->pipe
 	     ? to_iss_pipeline(pad->entity) : &video->pipe;
 	pipe->external = NULL;
 	pipe->external_rate = 0;
diff --git a/drivers/staging/media/omap4iss/iss_video.h b/drivers/staging/media/omap4iss/iss_video.h
index 526281bf0051..9b8ec27bf87d 100644
--- a/drivers/staging/media/omap4iss/iss_video.h
+++ b/drivers/staging/media/omap4iss/iss_video.h
@@ -91,7 +91,7 @@ struct iss_pipeline {
 };
 
 #define to_iss_pipeline(__e) \
-	container_of((__e)->pipe, struct iss_pipeline, pipe)
+	container_of((__e)->pads->pipe, struct iss_pipeline, pipe)
 
 static inline int iss_pipeline_ready(struct iss_pipeline *pipe)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 5f6eed24e63f..c9d97c902d05 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -180,15 +180,24 @@ enum media_pad_signal_type {
  *
  * @graph_obj:	Embedded structure containing the media object common data
  * @entity:	Entity this pad belongs to
+ * @pipe:	Pipeline this pad belongs to
+ * @stream_count: Stream count for the pad
  * @index:	Pad index in the entity pads array, numbered from 0 to n
  * @sig_type:	Type of the signal inside a media pad
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ * .. note::
+ *
+ *    @stream_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_pad {
 	struct media_gobj graph_obj;	/* must be first field in struct */
 	struct media_entity *entity;
+	struct media_pipeline *pipe;
+	int stream_count;
 	u16 index;
 	enum media_pad_signal_type sig_type;
 	unsigned long flags;
@@ -267,9 +276,7 @@ enum media_entity_type {
  * @pads:	Pads array with the size defined by @num_pads.
  * @links:	List of data links.
  * @ops:	Entity operations.
- * @stream_count: Stream count for the entity.
  * @use_count:	Use count for the entity.
- * @pipe:	Pipeline this entity belongs to.
  * @info:	Union with devnode information.  Kept just for backward
  *		compatibility.
  * @info.dev:	Contains device major and minor info.
@@ -282,10 +289,9 @@ enum media_entity_type {
  *
  * .. note::
  *
- *    @stream_count and @use_count reference counts must never be
- *    negative, but are signed integers on purpose: a simple ``WARN_ON(<0)``
- *    check can be used to detect reference count bugs that would make them
- *    negative.
+ *    @use_count reference count must never be negative, but is a signed
+ *    integer on purpose: a simple ``WARN_ON(<0)`` check can be used to
+ *    detect reference count bugs that would make it negative.
  */
 struct media_entity {
 	struct media_gobj graph_obj;	/* must be first field in struct */
@@ -304,11 +310,8 @@ struct media_entity {
 
 	const struct media_entity_operations *ops;
 
-	int stream_count;
 	int use_count;
 
-	struct media_pipeline *pipe;
-
 	union {
 		struct {
 			u32 major;
-- 
2.25.1

[PATCH v9 13/36] media: entity: Use pad as the starting point for a pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The pipeline has been moved from the entity to the pads; reflect this in
the media pipeline function API.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 Documentation/driver-api/media/mc-core.rst    |  9 +++--
 drivers/media/mc/mc-entity.c                  | 24 ++++++-------
 drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |  6 ++--
 .../media/platform/exynos4-is/fimc-capture.c  |  8 ++---
 .../platform/exynos4-is/fimc-isp-video.c      |  8 ++---
 drivers/media/platform/exynos4-is/fimc-lite.c |  8 ++---
 drivers/media/platform/omap3isp/ispvideo.c    |  6 ++--
 .../media/platform/qcom/camss/camss-video.c   |  6 ++--
 drivers/media/platform/rcar-vin/rcar-dma.c    |  6 ++--
 .../platform/rockchip/rkisp1/rkisp1-capture.c |  6 ++--
 .../media/platform/s3c-camif/camif-capture.c  |  6 ++--
 drivers/media/platform/stm32/stm32-dcmi.c     |  6 ++--
 .../platform/sunxi/sun4i-csi/sun4i_dma.c      |  6 ++--
 .../platform/sunxi/sun6i-csi/sun6i_video.c    |  6 ++--
 drivers/media/platform/ti-vpe/cal-video.c     |  6 ++--
 drivers/media/platform/vsp1/vsp1_video.c      |  6 ++--
 drivers/media/platform/xilinx/xilinx-dma.c    |  6 ++--
 .../media/test-drivers/vimc/vimc-capture.c    |  6 ++--
 drivers/media/usb/au0828/au0828-core.c        |  8 ++---
 drivers/staging/media/imx/imx-media-utils.c   |  6 ++--
 drivers/staging/media/ipu3/ipu3-v4l2.c        |  6 ++--
 drivers/staging/media/omap4iss/iss_video.c    |  6 ++--
 drivers/staging/media/tegra-video/tegra210.c  |  6 ++--
 include/media/media-entity.h                  | 34 +++++++++----------
 24 files changed, 99 insertions(+), 102 deletions(-)

diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
index 8a13640bed56..136047a22744 100644
--- a/Documentation/driver-api/media/mc-core.rst
+++ b/Documentation/driver-api/media/mc-core.rst
@@ -213,15 +213,14 @@ When starting streaming, drivers must notify all entities in the pipeline to
 prevent link states from being modified during streaming by calling
 :c:func:`media_pipeline_start()`.
 
-The function will mark all entities connected to the given entity through
-enabled links, either directly or indirectly, as streaming.
+The function will mark all the pads connected to the given pad through
+enabled routes and links, either directly or indirectly, as streaming.
 
 The struct media_pipeline instance pointed to by
-the pipe argument will be stored in every entity in the pipeline.
+the pipe argument will be stored in every pad in the pipeline.
 Drivers should embed the struct media_pipeline
 in higher-level pipeline structures and can then access the
-pipeline through the struct media_entity
-pipe field.
+pipeline through the struct media_pad pipe field.
 
 Calls to :c:func:`media_pipeline_start()` can be nested.
 The pipeline pointer must be identical for all nested calls to the function.
diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 8ad4cb845f4a..b44ab423b49b 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -404,12 +404,11 @@ EXPORT_SYMBOL_GPL(media_entity_get_fwnode_pad);
  * Pipeline management
  */
 
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad = entity->pads;
 	struct media_pad *pad_err = pad;
 	struct media_link *link;
 	int ret;
@@ -544,24 +543,23 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_start);
 
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 	int ret;
 
 	mutex_lock(&mdev->graph_mutex);
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(pad, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	return ret;
 }
 EXPORT_SYMBOL_GPL(media_pipeline_start);
 
-void __media_pipeline_stop(struct media_entity *entity)
+void __media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_pipeline *pipe = entity->pads->pipe;
+	struct media_pipeline *pipe = pad->pipe;
 	struct media_graph *graph = &pipe->graph;
-	struct media_pad *pad;
 
 	/*
 	 * If the following check fails, the driver has performed an
@@ -570,7 +568,7 @@ void __media_pipeline_stop(struct media_entity *entity)
 	if (WARN_ON(!pipe))
 		return;
 
-	media_graph_walk_start(graph, entity->pads);
+	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
 		struct media_entity *entity = pad->entity;
@@ -592,12 +590,12 @@ void __media_pipeline_stop(struct media_entity *entity)
 }
 EXPORT_SYMBOL_GPL(__media_pipeline_stop);
 
-void media_pipeline_stop(struct media_entity *entity)
+void media_pipeline_stop(struct media_pad *pad)
 {
-	struct media_device *mdev = entity->graph_obj.mdev;
+	struct media_device *mdev = pad->graph_obj.mdev;
 
 	mutex_lock(&mdev->graph_mutex);
-	__media_pipeline_stop(entity);
+	__media_pipeline_stop(pad);
 	mutex_unlock(&mdev->graph_mutex);
 }
 EXPORT_SYMBOL_GPL(media_pipeline_stop);
diff --git a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
index 47db0ee0fcbf..74db79b2990f 100644
--- a/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
+++ b/drivers/media/pci/intel/ipu3/ipu3-cio2-main.c
@@ -981,7 +981,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 		return r;
 	}
 
-	r = media_pipeline_start(&q->vdev.entity, &q->pipe);
+	r = media_pipeline_start(q->vdev.entity.pads, &q->pipe);
 	if (r)
 		goto fail_pipeline;
 
@@ -1001,7 +1001,7 @@ static int cio2_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 fail_csi2_subdev:
 	cio2_hw_exit(cio2, q);
 fail_hw:
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 fail_pipeline:
 	dev_dbg(&cio2->pci_dev->dev, "failed to start streaming (%d)\n", r);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_QUEUED);
@@ -1022,7 +1022,7 @@ static void cio2_vb2_stop_streaming(struct vb2_queue *vq)
 	cio2_hw_exit(cio2, q);
 	synchronize_irq(cio2->pci_dev->irq);
 	cio2_vb2_return_all_buffers(q, VB2_BUF_STATE_ERROR);
-	media_pipeline_stop(&q->vdev.entity);
+	media_pipeline_stop(q->vdev.entity.pads);
 	pm_runtime_put(&cio2->pci_dev->dev);
 	cio2->streaming = false;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-capture.c b/drivers/media/platform/exynos4-is/fimc-capture.c
index 7ff4024003f4..eaac0ac4e406 100644
--- a/drivers/media/platform/exynos4-is/fimc-capture.c
+++ b/drivers/media/platform/exynos4-is/fimc-capture.c
@@ -524,7 +524,7 @@ static int fimc_capture_release(struct file *file)
 	mutex_lock(&fimc->lock);
 
 	if (close && vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
@@ -1184,7 +1184,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	if (fimc_capture_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &vc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &vc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -1218,7 +1218,7 @@ static int fimc_cap_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return ret;
 }
 
@@ -1234,7 +1234,7 @@ static int fimc_cap_streamoff(struct file *file, void *priv,
 		return ret;
 
 	if (vc->streaming) {
-		media_pipeline_stop(&vc->ve.vdev.entity);
+		media_pipeline_stop(vc->ve.vdev.entity.pads);
 		vc->streaming = false;
 	}
 
diff --git a/drivers/media/platform/exynos4-is/fimc-isp-video.c b/drivers/media/platform/exynos4-is/fimc-isp-video.c
index 83688a7982f7..e2862b3dcdfc 100644
--- a/drivers/media/platform/exynos4-is/fimc-isp-video.c
+++ b/drivers/media/platform/exynos4-is/fimc-isp-video.c
@@ -312,7 +312,7 @@ static int isp_video_release(struct file *file)
 	is_singular_file = v4l2_fh_is_singular_file(file);
 
 	if (is_singular_file && ivc->streaming) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		ivc->streaming = 0;
 	}
 
@@ -493,7 +493,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	struct media_entity *me = &ve->vdev.entity;
 	int ret;
 
-	ret = media_pipeline_start(me, &ve->pipe->mp);
+	ret = media_pipeline_start(me->pads, &ve->pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -508,7 +508,7 @@ static int isp_video_streamon(struct file *file, void *priv,
 	isp->video_capture.streaming = 1;
 	return 0;
 p_stop:
-	media_pipeline_stop(me);
+	media_pipeline_stop(me->pads);
 	return ret;
 }
 
@@ -523,7 +523,7 @@ static int isp_video_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&video->ve.vdev.entity);
+	media_pipeline_stop(video->ve.vdev.entity.pads);
 	video->streaming = 0;
 	return 0;
 }
diff --git a/drivers/media/platform/exynos4-is/fimc-lite.c b/drivers/media/platform/exynos4-is/fimc-lite.c
index 67bfb1ad2ba2..a979600ff6e6 100644
--- a/drivers/media/platform/exynos4-is/fimc-lite.c
+++ b/drivers/media/platform/exynos4-is/fimc-lite.c
@@ -516,7 +516,7 @@ static int fimc_lite_release(struct file *file)
 	if (v4l2_fh_is_singular_file(file) &&
 	    atomic_read(&fimc->out_path) == FIMC_IO_DMA) {
 		if (fimc->streaming) {
-			media_pipeline_stop(entity);
+			media_pipeline_stop(entity->pads);
 			fimc->streaming = false;
 		}
 		fimc_lite_stop_capture(fimc, false);
@@ -822,7 +822,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	if (fimc_lite_active(fimc))
 		return -EBUSY;
 
-	ret = media_pipeline_start(entity, &fimc->ve.pipe->mp);
+	ret = media_pipeline_start(entity->pads, &fimc->ve.pipe->mp);
 	if (ret < 0)
 		return ret;
 
@@ -839,7 +839,7 @@ static int fimc_lite_streamon(struct file *file, void *priv,
 	}
 
 err_p_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 	return 0;
 }
 
@@ -853,7 +853,7 @@ static int fimc_lite_streamoff(struct file *file, void *priv,
 	if (ret < 0)
 		return ret;
 
-	media_pipeline_stop(&fimc->ve.vdev.entity);
+	media_pipeline_stop(fimc->ve.vdev.entity.pads);
 	fimc->streaming = false;
 	return 0;
 }
diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
index a8438040c4aa..87334477f223 100644
--- a/drivers/media/platform/omap3isp/ispvideo.c
+++ b/drivers/media/platform/omap3isp/ispvideo.c
@@ -1105,7 +1105,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	pipe->l3_ick = clk_get_rate(video->isp->clock[ISP_CLK_L3_ICK]);
 	pipe->max_rate = pipe->l3_ick;
 
-	ret = media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto err_pipeline_start;
 
@@ -1162,7 +1162,7 @@ isp_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipeline_start:
 	/* TODO: Implement PM QoS */
 	/* The DMA queue must be emptied here, otherwise CCDC interrupts that
@@ -1229,7 +1229,7 @@ isp_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 	video->error = false;
 
 	/* TODO: Implement PM QoS */
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 	media_entity_enum_cleanup(&pipe->ent_enum);
 
diff --git a/drivers/media/platform/qcom/camss/camss-video.c b/drivers/media/platform/qcom/camss/camss-video.c
index f282275af626..5cd494f17589 100644
--- a/drivers/media/platform/qcom/camss/camss-video.c
+++ b/drivers/media/platform/qcom/camss/camss-video.c
@@ -491,7 +491,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	struct v4l2_subdev *subdev;
 	int ret;
 
-	ret = media_pipeline_start(&vdev->entity, &video->pipe);
+	ret = media_pipeline_start(vdev->entity.pads, &video->pipe);
 	if (ret < 0)
 		return ret;
 
@@ -520,7 +520,7 @@ static int video_start_streaming(struct vb2_queue *q, unsigned int count)
 	return 0;
 
 error:
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_QUEUED);
 
@@ -551,7 +551,7 @@ static void video_stop_streaming(struct vb2_queue *q)
 		v4l2_subdev_call(subdev, video, s_stream, 0);
 	}
 
-	media_pipeline_stop(&vdev->entity);
+	media_pipeline_stop(vdev->entity.pads);
 
 	video->ops->flush_buffers(video, VB2_BUF_STATE_ERROR);
 }
diff --git a/drivers/media/platform/rcar-vin/rcar-dma.c b/drivers/media/platform/rcar-vin/rcar-dma.c
index 80b7ae47d165..83b2f923cf98 100644
--- a/drivers/media/platform/rcar-vin/rcar-dma.c
+++ b/drivers/media/platform/rcar-vin/rcar-dma.c
@@ -1215,7 +1215,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	sd = media_entity_to_v4l2_subdev(pad->entity);
 
 	if (!on) {
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 		return v4l2_subdev_call(sd, video, s_stream, 0);
 	}
 
@@ -1232,7 +1232,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	mdev = vin->vdev.entity.graph_obj.mdev;
 	mutex_lock(&mdev->graph_mutex);
 	pipe = sd->entity.pads->pipe ? sd->entity.pads->pipe : &vin->vdev.pipe;
-	ret = __media_pipeline_start(&vin->vdev.entity, pipe);
+	ret = __media_pipeline_start(vin->vdev.entity.pads, pipe);
 	mutex_unlock(&mdev->graph_mutex);
 	if (ret)
 		return ret;
@@ -1241,7 +1241,7 @@ static int rvin_set_stream(struct rvin_dev *vin, int on)
 	if (ret == -ENOIOCTLCMD)
 		ret = 0;
 	if (ret)
-		media_pipeline_stop(&vin->vdev.entity);
+		media_pipeline_stop(vin->vdev.entity.pads);
 
 	return ret;
 }
diff --git a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
index 41988eb0ec0a..2efbe181b6f1 100644
--- a/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
+++ b/drivers/media/platform/rockchip/rkisp1/rkisp1-capture.c
@@ -988,7 +988,7 @@ static void rkisp1_vb2_stop_streaming(struct vb2_queue *queue)
 
 	rkisp1_dummy_buf_destroy(cap);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 
 	mutex_unlock(&cap->rkisp1->stream_lock);
 }
@@ -1002,7 +1002,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 
 	mutex_lock(&cap->rkisp1->stream_lock);
 
-	ret = media_pipeline_start(entity, &cap->rkisp1->pipe);
+	ret = media_pipeline_start(entity->pads, &cap->rkisp1->pipe);
 	if (ret) {
 		dev_err(cap->rkisp1->dev, "start pipeline failed %d\n", ret);
 		goto err_ret_buffers;
@@ -1038,7 +1038,7 @@ rkisp1_vb2_start_streaming(struct vb2_queue *queue, unsigned int count)
 err_destroy_dummy:
 	rkisp1_dummy_buf_destroy(cap);
 err_pipeline_stop:
-	media_pipeline_stop(entity);
+	media_pipeline_stop(entity->pads);
 err_ret_buffers:
 	rkisp1_return_all_buffers(cap, VB2_BUF_STATE_QUEUED);
 	mutex_unlock(&cap->rkisp1->stream_lock);
diff --git a/drivers/media/platform/s3c-camif/camif-capture.c b/drivers/media/platform/s3c-camif/camif-capture.c
index 140854ab4dd8..0189b8a33032 100644
--- a/drivers/media/platform/s3c-camif/camif-capture.c
+++ b/drivers/media/platform/s3c-camif/camif-capture.c
@@ -848,13 +848,13 @@ static int s3c_camif_streamon(struct file *file, void *priv,
 	if (s3c_vp_active(vp))
 		return 0;
 
-	ret = media_pipeline_start(sensor, camif->m_pipeline);
+	ret = media_pipeline_start(sensor->pads, camif->m_pipeline);
 	if (ret < 0)
 		return ret;
 
 	ret = camif_pipeline_validate(camif);
 	if (ret < 0) {
-		media_pipeline_stop(sensor);
+		media_pipeline_stop(sensor->pads);
 		return ret;
 	}
 
@@ -878,7 +878,7 @@ static int s3c_camif_streamoff(struct file *file, void *priv,
 
 	ret = vb2_streamoff(&vp->vb_queue, type);
 	if (ret == 0)
-		media_pipeline_stop(&camif->sensor.sd->entity);
+		media_pipeline_stop(camif->sensor.sd->entity.pads);
 	return ret;
 }
 
diff --git a/drivers/media/platform/stm32/stm32-dcmi.c b/drivers/media/platform/stm32/stm32-dcmi.c
index d914ccef9831..457bc838bb73 100644
--- a/drivers/media/platform/stm32/stm32-dcmi.c
+++ b/drivers/media/platform/stm32/stm32-dcmi.c
@@ -730,7 +730,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_unlocked;
 	}
 
-	ret = media_pipeline_start(&dcmi->vdev->entity, &dcmi->pipeline);
+	ret = media_pipeline_start(dcmi->vdev->entity.pads, &dcmi->pipeline);
 	if (ret < 0) {
 		dev_err(dcmi->dev, "%s: Failed to start streaming, media pipeline start error (%d)\n",
 			__func__, ret);
@@ -844,7 +844,7 @@ static int dcmi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dcmi_pipeline_stop(dcmi);
 
 err_media_pipeline_stop:
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 err_pm_put:
 	pm_runtime_put(dcmi->dev);
@@ -871,7 +871,7 @@ static void dcmi_stop_streaming(struct vb2_queue *vq)
 
 	dcmi_pipeline_stop(dcmi);
 
-	media_pipeline_stop(&dcmi->vdev->entity);
+	media_pipeline_stop(dcmi->vdev->entity.pads);
 
 	spin_lock_irq(&dcmi->irqlock);
 
diff --git a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
index 2c39cd7f2862..be0defdf74f1 100644
--- a/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
+++ b/drivers/media/platform/sunxi/sun4i-csi/sun4i_dma.c
@@ -266,7 +266,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 		goto err_clear_dma_queue;
 	}
 
-	ret = media_pipeline_start(&csi->vdev.entity, &csi->vdev.pipe);
+	ret = media_pipeline_start(csi->vdev.entity.pads, &csi->vdev.pipe);
 	if (ret < 0)
 		goto err_free_scratch_buffer;
 
@@ -330,7 +330,7 @@ static int sun4i_csi_start_streaming(struct vb2_queue *vq, unsigned int count)
 	sun4i_csi_capture_stop(csi);
 
 err_disable_pipeline:
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 err_free_scratch_buffer:
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
@@ -359,7 +359,7 @@ static void sun4i_csi_stop_streaming(struct vb2_queue *vq)
 	return_all_buffers(csi, VB2_BUF_STATE_ERROR);
 	spin_unlock_irqrestore(&csi->qlock, flags);
 
-	media_pipeline_stop(&csi->vdev.entity);
+	media_pipeline_stop(csi->vdev.entity.pads);
 
 	dma_free_coherent(csi->dev, csi->scratch.size, csi->scratch.vaddr,
 			  csi->scratch.paddr);
diff --git a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
index 07b2161392d2..6baa597a4dad 100644
--- a/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
+++ b/drivers/media/platform/sunxi/sun6i-csi/sun6i_video.c
@@ -141,7 +141,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	video->sequence = 0;
 
-	ret = media_pipeline_start(&video->vdev.entity, &video->vdev.pipe);
+	ret = media_pipeline_start(video->vdev.entity.pads, &video->vdev.pipe);
 	if (ret < 0)
 		goto clear_dma_queue;
 
@@ -207,7 +207,7 @@ static int sun6i_video_start_streaming(struct vb2_queue *vq, unsigned int count)
 stop_csi_stream:
 	sun6i_csi_set_stream(video->csi, false);
 stop_media_pipeline:
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 clear_dma_queue:
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
 	list_for_each_entry(buf, &video->dma_queue, list)
@@ -231,7 +231,7 @@ static void sun6i_video_stop_streaming(struct vb2_queue *vq)
 
 	sun6i_csi_set_stream(video->csi, false);
 
-	media_pipeline_stop(&video->vdev.entity);
+	media_pipeline_stop(video->vdev.entity.pads);
 
 	/* Release all active buffers */
 	spin_lock_irqsave(&video->dma_queue_lock, flags);
diff --git a/drivers/media/platform/ti-vpe/cal-video.c b/drivers/media/platform/ti-vpe/cal-video.c
index 7799da1cc261..861a20ee6885 100644
--- a/drivers/media/platform/ti-vpe/cal-video.c
+++ b/drivers/media/platform/ti-vpe/cal-video.c
@@ -711,7 +711,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	dma_addr_t addr;
 	int ret;
 
-	ret = media_pipeline_start(&ctx->vdev.entity, &ctx->phy->pipe);
+	ret = media_pipeline_start(ctx->vdev.entity.pads, &ctx->phy->pipe);
 	if (ret < 0) {
 		ctx_err(ctx, "Failed to start media pipeline: %d\n", ret);
 		goto error_release_buffers;
@@ -764,7 +764,7 @@ static int cal_start_streaming(struct vb2_queue *vq, unsigned int count)
 	cal_ctx_unprepare(ctx);
 
 error_pipeline:
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 error_release_buffers:
 	cal_release_buffers(ctx, VB2_BUF_STATE_QUEUED);
 
@@ -785,7 +785,7 @@ static void cal_stop_streaming(struct vb2_queue *vq)
 
 	cal_release_buffers(ctx, VB2_BUF_STATE_ERROR);
 
-	media_pipeline_stop(&ctx->vdev.entity);
+	media_pipeline_stop(ctx->vdev.entity.pads);
 }
 
 static const struct vb2_ops cal_video_qops = {
diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
index f2c36f2fdf53..978f820b0f34 100644
--- a/drivers/media/platform/vsp1/vsp1_video.c
+++ b/drivers/media/platform/vsp1/vsp1_video.c
@@ -927,7 +927,7 @@ static void vsp1_video_stop_streaming(struct vb2_queue *vq)
 	}
 	mutex_unlock(&pipe->lock);
 
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 	vsp1_video_release_buffers(video);
 	vsp1_video_pipeline_put(pipe);
 }
@@ -1048,7 +1048,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 		return PTR_ERR(pipe);
 	}
 
-	ret = __media_pipeline_start(&video->video.entity, &pipe->pipe);
+	ret = __media_pipeline_start(video->video.entity.pads, &pipe->pipe);
 	if (ret < 0) {
 		mutex_unlock(&mdev->graph_mutex);
 		goto err_pipe;
@@ -1072,7 +1072,7 @@ vsp1_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	return 0;
 
 err_stop:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_pipe:
 	vsp1_video_pipeline_put(pipe);
 	return ret;
diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
index 03ee19d00041..f04b3d190562 100644
--- a/drivers/media/platform/xilinx/xilinx-dma.c
+++ b/drivers/media/platform/xilinx/xilinx-dma.c
@@ -405,7 +405,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	pipe = dma->video.entity.pads->pipe
 	     ? to_xvip_pipeline(&dma->video.entity) : &dma->pipe;
 
-	ret = media_pipeline_start(&dma->video.entity, &pipe->pipe);
+	ret = media_pipeline_start(dma->video.entity.pads, &pipe->pipe);
 	if (ret < 0)
 		goto error;
 
@@ -431,7 +431,7 @@ static int xvip_dma_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 error_stop:
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 error:
 	/* Give back all queued buffers to videobuf2. */
@@ -459,7 +459,7 @@ static void xvip_dma_stop_streaming(struct vb2_queue *vq)
 
 	/* Cleanup the pipeline and mark it as being stopped. */
 	xvip_pipeline_cleanup(pipe);
-	media_pipeline_stop(&dma->video.entity);
+	media_pipeline_stop(dma->video.entity.pads);
 
 	/* Give back all queued buffers to videobuf2. */
 	spin_lock_irq(&dma->queued_lock);
diff --git a/drivers/media/test-drivers/vimc/vimc-capture.c b/drivers/media/test-drivers/vimc/vimc-capture.c
index 5e9fd902cd37..10724b0a868c 100644
--- a/drivers/media/test-drivers/vimc/vimc-capture.c
+++ b/drivers/media/test-drivers/vimc/vimc-capture.c
@@ -246,7 +246,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 	vcap->sequence = 0;
 
 	/* Start the media pipeline */
-	ret = media_pipeline_start(entity, &vcap->stream.pipe);
+	ret = media_pipeline_start(entity->pads, &vcap->stream.pipe);
 	if (ret) {
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
@@ -254,7 +254,7 @@ static int vimc_cap_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	ret = vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 1);
 	if (ret) {
-		media_pipeline_stop(entity);
+		media_pipeline_stop(entity->pads);
 		vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_QUEUED);
 		return ret;
 	}
@@ -273,7 +273,7 @@ static void vimc_cap_stop_streaming(struct vb2_queue *vq)
 	vimc_streamer_s_stream(&vcap->stream, &vcap->ved, 0);
 
 	/* Stop the media pipeline */
-	media_pipeline_stop(&vcap->vdev.entity);
+	media_pipeline_stop(vcap->vdev.entity.pads);
 
 	/* Release all active buffers */
 	vimc_cap_return_all_buffers(vcap, VB2_BUF_STATE_ERROR);
diff --git a/drivers/media/usb/au0828/au0828-core.c b/drivers/media/usb/au0828/au0828-core.c
index caefac07af92..877e85a451cb 100644
--- a/drivers/media/usb/au0828/au0828-core.c
+++ b/drivers/media/usb/au0828/au0828-core.c
@@ -410,7 +410,7 @@ static int au0828_enable_source(struct media_entity *entity,
 		goto end;
 	}
 
-	ret = __media_pipeline_start(entity, pipe);
+	ret = __media_pipeline_start(entity->pads, pipe);
 	if (ret) {
 		pr_err("Start Pipeline: %s->%s Error %d\n",
 			source->name, entity->name, ret);
@@ -501,12 +501,12 @@ static void au0828_disable_source(struct media_entity *entity)
 				return;
 
 			/* stop pipeline */
-			__media_pipeline_stop(dev->active_link_owner);
+			__media_pipeline_stop(dev->active_link_owner->pads);
 			pr_debug("Pipeline stop for %s\n",
 				dev->active_link_owner->name);
 
 			ret = __media_pipeline_start(
-					dev->active_link_user,
+					dev->active_link_user->pads,
 					dev->active_link_user_pipe);
 			if (ret) {
 				pr_err("Start Pipeline: %s->%s %d\n",
@@ -532,7 +532,7 @@ static void au0828_disable_source(struct media_entity *entity)
 			return;
 
 		/* stop pipeline */
-		__media_pipeline_stop(dev->active_link_owner);
+		__media_pipeline_stop(dev->active_link_owner->pads);
 		pr_debug("Pipeline stop for %s\n",
 			dev->active_link_owner->name);
 
diff --git a/drivers/staging/media/imx/imx-media-utils.c b/drivers/staging/media/imx/imx-media-utils.c
index 535da4dda3c6..74218af45551 100644
--- a/drivers/staging/media/imx/imx-media-utils.c
+++ b/drivers/staging/media/imx/imx-media-utils.c
@@ -905,16 +905,16 @@ int imx_media_pipeline_set_stream(struct imx_media_dev *imxmd,
 	mutex_lock(&imxmd->md.graph_mutex);
 
 	if (on) {
-		ret = __media_pipeline_start(entity, &imxmd->pipe);
+		ret = __media_pipeline_start(entity->pads, &imxmd->pipe);
 		if (ret)
 			goto out;
 		ret = v4l2_subdev_call(sd, video, s_stream, 1);
 		if (ret)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	} else {
 		v4l2_subdev_call(sd, video, s_stream, 0);
 		if (entity->pads->pipe)
-			__media_pipeline_stop(entity);
+			__media_pipeline_stop(entity->pads);
 	}
 
 out:
diff --git a/drivers/staging/media/ipu3/ipu3-v4l2.c b/drivers/staging/media/ipu3/ipu3-v4l2.c
index 38a240764509..db5867da3f11 100644
--- a/drivers/staging/media/ipu3/ipu3-v4l2.c
+++ b/drivers/staging/media/ipu3/ipu3-v4l2.c
@@ -485,7 +485,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 
 	pipe = node->pipe;
 	imgu_pipe = &imgu->imgu_pipe[pipe];
-	r = media_pipeline_start(&node->vdev.entity, &imgu_pipe->pipeline);
+	r = media_pipeline_start(node->vdev.entity.pads, &imgu_pipe->pipeline);
 	if (r < 0)
 		goto fail_return_bufs;
 
@@ -510,7 +510,7 @@ static int imgu_vb2_start_streaming(struct vb2_queue *vq, unsigned int count)
 	return 0;
 
 fail_stop_pipeline:
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 fail_return_bufs:
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_QUEUED);
 
@@ -550,7 +550,7 @@ static void imgu_vb2_stop_streaming(struct vb2_queue *vq)
 	imgu_return_all_buffers(imgu, node, VB2_BUF_STATE_ERROR);
 	mutex_unlock(&imgu->streaming_lock);
 
-	media_pipeline_stop(&node->vdev.entity);
+	media_pipeline_stop(node->vdev.entity.pads);
 }
 
 /******************** v4l2_ioctl_ops ********************/
diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
index b74f7891711d..20fac40581c6 100644
--- a/drivers/staging/media/omap4iss/iss_video.c
+++ b/drivers/staging/media/omap4iss/iss_video.c
@@ -889,7 +889,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, true);
 
-	ret = media_pipeline_start(pad->entity, &pipe->pipe);
+	ret = media_pipeline_start(pad, &pipe->pipe);
 	if (ret < 0)
 		goto err_media_pipeline_start;
 
@@ -980,7 +980,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
 err_omap4iss_set_stream:
 	vb2_streamoff(&vfh->queue, type);
 err_iss_video_check_format:
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 err_media_pipeline_start:
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
@@ -1034,7 +1034,7 @@ iss_video_streamoff(struct file *file, void *fh, enum v4l2_buf_type type)
 
 	if (video->iss->pdata->set_constraints)
 		video->iss->pdata->set_constraints(video->iss, false);
-	media_pipeline_stop(&video->video.entity);
+	media_pipeline_stop(video->video.entity.pads);
 
 done:
 	mutex_unlock(&video->stream_lock);
diff --git a/drivers/staging/media/tegra-video/tegra210.c b/drivers/staging/media/tegra-video/tegra210.c
index f10a041e3e6c..d2d7dd0e8624 100644
--- a/drivers/staging/media/tegra-video/tegra210.c
+++ b/drivers/staging/media/tegra-video/tegra210.c
@@ -547,7 +547,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 		       VI_INCR_SYNCPT_NO_STALL);
 
 	/* start the pipeline */
-	ret = media_pipeline_start(&chan->video.entity, pipe);
+	ret = media_pipeline_start(chan->video.entity.pads, pipe);
 	if (ret < 0)
 		goto error_pipeline_start;
 
@@ -595,7 +595,7 @@ static int tegra210_vi_start_streaming(struct vb2_queue *vq, u32 count)
 error_kthread_start:
 	tegra_channel_set_stream(chan, false);
 error_set_stream:
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 error_pipeline_start:
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_QUEUED);
 	return ret;
@@ -617,7 +617,7 @@ static void tegra210_vi_stop_streaming(struct vb2_queue *vq)
 
 	tegra_channel_release_buffers(chan, VB2_BUF_STATE_ERROR);
 	tegra_channel_set_stream(chan, false);
-	media_pipeline_stop(&chan->video.entity);
+	media_pipeline_stop(chan->video.entity.pads);
 }
 
 /*
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index c9d97c902d05..516d73a2941e 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -937,53 +937,53 @@ struct media_pad *media_graph_walk_next(struct media_graph *graph);
 
 /**
  * media_pipeline_start - Mark a pipeline as streaming
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as streaming. The given pipeline object is assigned
- * to every entity in the pipeline and stored in the media_entity pipe field.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as streaming. The given pipeline object is
+ * assigned to every pad in the pipeline and stored in the media_pad pipe
+ * field.
  *
  * Calls to this function can be nested, in which case the same number of
  * media_pipeline_stop() calls will be required to stop streaming. The
  * pipeline pointer must be identical for all nested calls to
  * media_pipeline_start().
  */
-__must_check int media_pipeline_start(struct media_entity *entity,
+__must_check int media_pipeline_start(struct media_pad *pad,
 				      struct media_pipeline *pipe);
 /**
  * __media_pipeline_start - Mark a pipeline as streaming
  *
- * @entity: Starting entity
- * @pipe: Media pipeline to be assigned to all entities in the pipeline.
+ * @pad: Starting pad
+ * @pipe: Media pipeline to be assigned to all pads in the pipeline.
  *
  * ..note:: This is the non-locking version of media_pipeline_start()
  */
-__must_check int __media_pipeline_start(struct media_entity *entity,
+__must_check int __media_pipeline_start(struct media_pad *pad,
 					struct media_pipeline *pipe);
 
 /**
  * media_pipeline_stop - Mark a pipeline as not streaming
- * @entity: Starting entity
+ * @pad: Starting pad
  *
- * Mark all entities connected to a given entity through enabled links, either
- * directly or indirectly, as not streaming. The media_entity pipe field is
- * reset to %NULL.
+ * Mark all pads connected to a given pad through enabled routes or links,
+ * either directly or indirectly, as not streaming.
  *
  * If multiple calls to media_pipeline_start() have been made, the same
  * number of calls to this function are required to mark the pipeline as not
- * streaming.
+ * streaming and reset the media_pad pipe field to %NULL.
  */
-void media_pipeline_stop(struct media_entity *entity);
+void media_pipeline_stop(struct media_pad *pad);
 
 /**
  * __media_pipeline_stop - Mark a pipeline as not streaming
  *
- * @entity: Starting entity
+ * @pad: Starting pad
  *
  * .. note:: This is the non-locking version of media_pipeline_stop()
  */
-void __media_pipeline_stop(struct media_entity *entity);
+void __media_pipeline_stop(struct media_pad *pad);
 
 /**
  * media_devnode_create() - creates and initializes a device node interface
-- 
2.25.1

[PATCH v9 14/36] media: entity: Add has_route entity operation

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

The optional operation can be used by entities to report whether two
pads are internally connected.

While at there, fix a Sphinx compiler warning in a comment block a few
lines above.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/media-entity.h | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index 516d73a2941e..ad4020b2df65 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -187,6 +187,7 @@ enum media_pad_signal_type {
  * @flags:	Pad flags, as defined in
  *		:ref:`include/uapi/linux/media.h <media_header>`
  *		(seek for ``MEDIA_PAD_FL_*``)
+ *
  * .. note::
  *
  *    @stream_count reference count must never be negative, but is a signed
@@ -214,6 +215,10 @@ struct media_pad {
  * @link_validate:	Return whether a link is valid from the entity point of
  *			view. The media_pipeline_start() function
  *			validates all links by calling this operation. Optional.
+ * @has_route:		Return whether a route exists inside the entity between
+ *			two given pads. Pads are passed to the operation ordered
+ *			by index. Optional: If the operation isn't implemented
+ *			all pads will be considered as connected.
  *
  * .. note::
  *
@@ -227,6 +232,8 @@ struct media_entity_operations {
 			  const struct media_pad *local,
 			  const struct media_pad *remote, u32 flags);
 	int (*link_validate)(struct media_link *link);
+	bool (*has_route)(struct media_entity *entity, unsigned int pad0,
+			  unsigned int pad1);
 };
 
 /**
-- 
2.25.1

[PATCH v9 15/36] media: entity: Add media_entity_has_route() function

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

This is a wrapper around the media entity has_route operation.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 19 +++++++++++++++++++
 include/media/media-entity.h | 17 +++++++++++++++++
 2 files changed, 36 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index b44ab423b49b..a83f004efd37 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -229,6 +229,25 @@ EXPORT_SYMBOL_GPL(media_entity_pads_init);
  * Graph traversal
  */
 
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1)
+{
+	if (pad0 >= entity->num_pads || pad1 >= entity->num_pads)
+		return false;
+
+	if (pad0 == pad1)
+		return true;
+
+	if (!entity->ops || !entity->ops->has_route)
+		return true;
+
+	if (entity->pads[pad1].index < entity->pads[pad0].index)
+		swap(pad0, pad1);
+
+	return entity->ops->has_route(entity, pad0, pad1);
+}
+EXPORT_SYMBOL_GPL(media_entity_has_route);
+
 static struct media_pad *
 media_pad_other(struct media_pad *pad, struct media_link *link)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index ad4020b2df65..b3069eef7fdb 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -904,6 +904,23 @@ int media_entity_get_fwnode_pad(struct media_entity *entity,
 __must_check int media_graph_walk_init(
 	struct media_graph *graph, struct media_device *mdev);
 
+/**
+ * media_entity_has_route - Check if two entity pads are connected internally
+ *
+ * @entity: The entity
+ * @pad0: The first pad index
+ * @pad1: The second pad index
+ *
+ * This function can be used to check whether two pads of an entity are
+ * connected internally in the entity.
+ *
+ * The caller must hold entity->graph_obj.mdev->mutex.
+ *
+ * Return: true if the pads are connected internally and false otherwise.
+ */
+bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
+			    unsigned int pad1);
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v9 16/36] media: entity: Use routing information during graph traversal

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Take internal routing information as reported by the entity has_route
operation into account during graph traversal to avoid following
unrelated links.

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>
Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 46 ++++++++++++++++++++++--------------
 1 file changed, 28 insertions(+), 18 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index a83f004efd37..58cdc9c6b342 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,15 +248,6 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
-static struct media_pad *
-media_pad_other(struct media_pad *pad, struct media_link *link)
-{
-	if (link->source == pad)
-		return link->sink;
-	else
-		return link->source;
-}
-
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
@@ -327,7 +318,8 @@ static void media_graph_walk_iter(struct media_graph *graph)
 {
 	struct media_pad *pad = stack_top(graph);
 	struct media_link *link;
-	struct media_pad *next;
+	struct media_pad *remote;
+	struct media_pad *local;
 
 	link = list_entry(link_top(graph), typeof(*link), list);
 
@@ -341,24 +333,42 @@ static void media_graph_walk_iter(struct media_graph *graph)
 		return;
 	}
 
-	/* Get the entity at the other end of the link. */
-	next = media_pad_other(pad, link);
+	/*
+	 * Get the local pad, the remote pad and the entity at the other
+	 * end of the link.
+	 */
+	if (link->source->entity == pad->entity) {
+		remote = link->sink;
+		local = link->source;
+	} else {
+		remote = link->source;
+		local = link->sink;
+	}
+
+	/*
+	 * Are the local pad and the pad we came from connected
+	 * internally in the entity ?
+	 */
+	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
+		link_top(graph) = link_top(graph)->next;
+		return;
+	}
 
 	/* Has the entity already been visited? */
-	if (media_entity_enum_test_and_set(&graph->ent_enum, next->entity)) {
+	if (media_entity_enum_test_and_set(&graph->ent_enum, remote->entity)) {
 		link_top(graph) = link_top(graph)->next;
 		dev_dbg(pad->graph_obj.mdev->dev,
 			"walk: skipping entity '%s' (already seen)\n",
-			next->entity->name);
+			remote->entity->name);
 		return;
 	}
 
 	/* Push the new entity to stack and start over. */
 	link_top(graph) = link_top(graph)->next;
-	stack_push(graph, next);
-	dev_dbg(next->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
-		next->entity->name, next->index);
-	lockdep_assert_held(&next->graph_obj.mdev->graph_mutex);
+	stack_push(graph, remote);
+	dev_dbg(remote->graph_obj.mdev->dev, "walk: pushing '%s':%u on stack\n",
+		remote->entity->name, remote->index);
+	lockdep_assert_held(&remote->graph_obj.mdev->graph_mutex);
 }
 
 struct media_pad *media_graph_walk_next(struct media_graph *graph)
-- 
2.25.1

[PATCH v9 17/36] media: entity: Skip link validation for pads to which there is no route

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Links are validated along the pipeline which is about to start streaming.
Not all the pads in entities that are traversed along that pipeline are
part of the pipeline, however. Skip the link validation for such pads.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 5 +++++
 1 file changed, 5 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 58cdc9c6b342..e963850b32df 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -489,6 +489,11 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 				link->sink->entity == entity ?
 				link->sink : link->source;
 
+			/* Ignore pads to which there is no route. */
+			if (!media_entity_has_route(entity, pad->index,
+						    other_pad->index))
+				continue;
+
 			/* Mark that a pad is connected by a link. */
 			bitmap_clear(has_no_links, other_pad->index, 1);
 
-- 
2.25.1

[PATCH v9 18/36] media: entity: Add an iterator helper for connected pads

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add a helper macro for iterating over pads that are connected through
enabled routes. This can be used to find all the connected pads within an
entity, for instance starting from the pad which has been obtained during
the graph walk.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make __media_entity_next_routed_pad() return NULL and adjust the
  iterator to handle that
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 14 ++++++++++++++
 include/media/media-entity.h | 27 +++++++++++++++++++++++++++
 2 files changed, 41 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index e963850b32df..072f017b399a 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -248,6 +248,20 @@ bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 }
 EXPORT_SYMBOL_GPL(media_entity_has_route);
 
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter)
+{
+	struct media_entity *entity = root->entity;
+
+	for (; iter < &entity->pads[entity->num_pads]; iter++) {
+		if (media_entity_has_route(entity, root->index, iter->index))
+			return iter;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(__media_entity_next_routed_pad);
+
 /* push an entity to traversal stack */
 static void stack_push(struct media_graph *graph, struct media_pad *pad)
 {
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index b3069eef7fdb..f90cc6417d30 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -921,6 +921,33 @@ __must_check int media_graph_walk_init(
 bool media_entity_has_route(struct media_entity *entity, unsigned int pad0,
 			    unsigned int pad1);
 
+/**
+ * __media_entity_next_routed_pad - Get next pad connected to @root
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Get next pad which has a route to @root.
+ */
+struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
+						 struct media_pad *iter);
+
+/**
+ * media_entity_for_each_routed_pad - Iterate over entity pads connected by
+ *				      routes
+ *
+ * @root: The root pad to which the iterated pads have a route
+ * @iter: The iterator pad
+ *
+ * Iterate over all pads of an entity which have an internal route to @root pad.
+ * The iteration will include the @root pad itself.
+ */
+#define media_entity_for_each_routed_pad(root, iter)                           \
+	for (iter = __media_entity_next_routed_pad(root,                       \
+						   (root)->entity->pads);      \
+	     iter != NULL;                                                     \
+	     iter = __media_entity_next_routed_pad(root, iter + 1))
+
 /**
  * media_graph_walk_cleanup - Release resources used by graph walk.
  *
-- 
2.25.1

[PATCH v9 19/36] media: entity: Add only connected pads to the pipeline

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

A single entity may contain multiple pipelines. Only add pads that were
connected to the pad through which the entity was reached to the pipeline.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 8 +++-----
 1 file changed, 3 insertions(+), 5 deletions(-)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 072f017b399a..4eb4b94c09e2 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -474,7 +474,7 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 
 		ret = 0;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			if (iter->pipe && iter->pipe != pipe) {
 				pr_err("Pipe active for %s. Can't start for %s\n",
 				       entity->name, iter->entity->name);
@@ -563,10 +563,9 @@ __must_check int __media_pipeline_start(struct media_pad *pad,
 	media_graph_walk_start(graph, pad_err);
 
 	while ((pad_err = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad_err->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad_err, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				--iter->stream_count;
@@ -619,10 +618,9 @@ void __media_pipeline_stop(struct media_pad *pad)
 	media_graph_walk_start(graph, pad);
 
 	while ((pad = media_graph_walk_next(graph))) {
-		struct media_entity *entity = pad->entity;
 		struct media_pad *iter;
 
-		media_entity_for_each_pad(entity, iter) {
+		media_entity_for_each_routed_pad(pad, iter) {
 			/* Sanity check for negative stream_count */
 			if (!WARN_ON_ONCE(iter->stream_count <= 0)) {
 				iter->stream_count--;
-- 
2.25.1

[PATCH v9 20/36] media: entity: Add debug information in graph walk route check

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add debug printout in graph walk route check.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/mc/mc-entity.c | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
index 4eb4b94c09e2..663773f6bb2f 100644
--- a/drivers/media/mc/mc-entity.c
+++ b/drivers/media/mc/mc-entity.c
@@ -365,6 +365,9 @@ static void media_graph_walk_iter(struct media_graph *graph)
 	 */
 	if (!media_entity_has_route(pad->entity, pad->index, local->index)) {
 		link_top(graph) = link_top(graph)->next;
+		dev_dbg(pad->graph_obj.mdev->dev,
+			"walk: skipping \"%s\":%u -> %u (no route)\n",
+			pad->entity->name, pad->index, local->index);
 		return;
 	}
 
-- 
2.25.1

[PATCH v9 21/36] media: Add bus type to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add the media bus type to the frame descriptor. CSI-2 specific
information will be added in next patch to the frame descriptor.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>

- Make the bus type a named enum
Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 20 ++++++++++++++++++++
 1 file changed, 20 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b75eeaee492b..4a04ef9c3000 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -344,12 +344,32 @@ struct v4l2_mbus_frame_desc_entry {
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
 
+/**
+ * enum v4l2_mbus_frame_desc_type - media bus frame description type
+ *
+ * @V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED:
+ *	Undefined frame desc type. Drivers should not use this, it is
+ *	for backwards compatibility.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL:
+ *	Parallel media bus.
+ * @V4L2_MBUS_FRAME_DESC_TYPE_CSI2:
+ *	CSI-2 media bus. Frame desc parameters must be set in
+ *	&struct v4l2_mbus_frame_desc_entry->csi2.
+ */
+enum v4l2_mbus_frame_desc_type {
+	V4L2_MBUS_FRAME_DESC_TYPE_UNDEFINED = 0,
+	V4L2_MBUS_FRAME_DESC_TYPE_PARALLEL,
+	V4L2_MBUS_FRAME_DESC_TYPE_CSI2,
+};
+
 /**
  * struct v4l2_mbus_frame_desc - media bus data frame description
+ * @type: type of the bus (enum v4l2_mbus_frame_desc_type)
  * @entry: frame descriptors array
  * @num_entries: number of entries in @entry array
  */
 struct v4l2_mbus_frame_desc {
+	enum v4l2_mbus_frame_desc_type type;
 	struct v4l2_mbus_frame_desc_entry entry[V4L2_FRAME_DESC_ENTRY_MAX];
 	unsigned short num_entries;
 };
-- 
2.25.1

[PATCH v9 22/36] media: Add CSI-2 bus configuration to frame descriptors

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

Add CSI-2 bus specific configuration to the frame descriptors. This allows
obtaining the virtual channel and data type information for each stream
the transmitter is sending.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 16 ++++++++++++++++
 1 file changed, 16 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 4a04ef9c3000..4836a9fbe725 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -312,6 +312,17 @@ struct v4l2_subdev_audio_ops {
 	int (*s_stream)(struct v4l2_subdev *sd, int enable);
 };
 
+/**
+ * struct v4l2_mbus_frame_desc_entry_csi2
+ *
+ * @vc: CSI-2 virtual channel
+ * @dt: CSI-2 data type ID
+ */
+struct v4l2_mbus_frame_desc_entry_csi2 {
+	u8 vc;
+	u8 dt;
+};
+
 /**
  * enum v4l2_mbus_frame_desc_flags - media bus frame description flags
  *
@@ -335,11 +346,16 @@ enum v4l2_mbus_frame_desc_flags {
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
  *		%V4L2_MBUS_FRAME_DESC_FL_LEN_MAX is set.
+ * @bus:	Bus-specific frame descriptor parameters
+ * @bus.csi2:	CSI-2-specific bus configuration
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
 	u32 pixelcode;
 	u32 length;
+	union {
+		struct v4l2_mbus_frame_desc_entry_csi2 csi2;
+	} bus;
 };
 
 #define V4L2_FRAME_DESC_ENTRY_MAX	4
-- 
2.25.1

[PATCH v9 23/36] media: Add stream to frame descriptor

[Tomi Valkeinen]

From: Sakari Ailus <sakari.ailus@linux.intel.com>

The stream field identifies the stream this frame descriptor applies to in
routing configuration across a multiplexed link.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 4836a9fbe725..f8443475651d 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -342,6 +342,7 @@ enum v4l2_mbus_frame_desc_flags {
  * struct v4l2_mbus_frame_desc_entry - media bus frame description structure
  *
  * @flags:	bitmask flags, as defined by &enum v4l2_mbus_frame_desc_flags.
+ * @stream:	stream in routing configuration
  * @pixelcode:	media bus pixel code, valid if @flags
  *		%FRAME_DESC_FL_BLOB is not set.
  * @length:	number of octets per frame, valid if @flags
@@ -351,6 +352,7 @@ enum v4l2_mbus_frame_desc_flags {
  */
 struct v4l2_mbus_frame_desc_entry {
 	enum v4l2_mbus_frame_desc_flags flags;
+	u32 stream;
 	u32 pixelcode;
 	u32 length;
 	union {
-- 
2.25.1

[PATCH v9 24/36] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Tomi Valkeinen]

V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
to have an arbitrary amount of streams in a single pad, so preferably
there should be no hardcoded maximum number.

However, I believe a reasonable max is 8, which would cover a CSI-2 pad
with 4 streams of pixel data and 4 streams of metadata.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 6 +++++-
 1 file changed, 5 insertions(+), 1 deletion(-)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f8443475651d..b3a72c98a2d3 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
 	} bus;
 };
 
-#define V4L2_FRAME_DESC_ENTRY_MAX	4
+ /*
+  * FIXME: If this number is too small, it should be dropped altogether and the
+  * API switched to a dynamic number of frame descriptor entries.
+  */
+#define V4L2_FRAME_DESC_ENTRY_MAX	8
 
 /**
  * enum v4l2_mbus_frame_desc_type - media bus frame description type
-- 
2.25.1

[PATCH v9 25/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
the subdev supports the new API with multiplexed streams (routing,
stream configs).

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 include/media/v4l2-subdev.h | 11 +++++++++++
 1 file changed, 11 insertions(+)

diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b3a72c98a2d3..7f9c83fac020 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -888,6 +888,17 @@ struct v4l2_subdev_internal_ops {
  * should set this flag.
  */
 #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
+/*
+ * Set this flag if this subdev supports multiplexed streams. This means
+ * that the driver supports routing and handles the stream parameter in its
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
+ */
+#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
 
 struct regulator_bulk_data;
 
-- 
2.25.1

[PATCH v9 26/36] media: Documentation: Add GS_ROUTING documentation

[Tomi Valkeinen]

From: Jacopo Mondi <jacopo+renesas@jmondi.org>

Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
description of multiplexed media pads and internal routing to the
V4L2-subdev documentation section.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
 .../userspace-api/media/v4l/user-func.rst     |   1 +
 .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++++++++++++++++
 3 files changed, 149 insertions(+)
 create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst

diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index fd1de0a73a9f..a67c2749089a 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
 
 -  negotiate image formats on individual pads
 
+-  inspect and modify internal data routing between pads of the same entity
+
 Sub-device character device nodes, conventionally named
 ``/dev/v4l-subdev*``, use major number 81.
 
diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
index 53e604bd7d60..228c1521f190 100644
--- a/Documentation/userspace-api/media/v4l/user-func.rst
+++ b/Documentation/userspace-api/media/v4l/user-func.rst
@@ -70,6 +70,7 @@ Function Reference
     vidioc-subdev-g-crop
     vidioc-subdev-g-fmt
     vidioc-subdev-g-frame-interval
+    vidioc-subdev-g-routing
     vidioc-subdev-g-selection
     vidioc-subdev-querycap
     vidioc-subscribe-event
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
new file mode 100644
index 000000000000..41f4873c49f7
--- /dev/null
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
@@ -0,0 +1,146 @@
+.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
+.. c:namespace:: V4L
+
+.. _VIDIOC_SUBDEV_G_ROUTING:
+
+******************************************************
+ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
+******************************************************
+
+Name
+====
+
+VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
+
+
+Synopsis
+========
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_G_ROUTING
+
+.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
+    :name: VIDIOC_SUBDEV_S_ROUTING
+
+
+Arguments
+=========
+
+``fd``
+    File descriptor returned by :ref:`open() <func-open>`.
+
+``argp``
+    Pointer to struct :c:type:`v4l2_subdev_routing`.
+
+
+Description
+===========
+
+These ioctls are used to get and set the routing in a media entity.
+The routing configuration determines the flows of data inside an entity.
+
+Drivers report their current routing tables using the
+``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
+with the VIDIOC_SUBDEV_S_ROUTING ioctl, by adding or removing routes and setting
+or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of the  ``flags`` field of
+a struct :c:type:`v4l2_subdev_route`.
+
+A special case for routing are routes marked with
+``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
+source endpoints on sensors and the sink fields are unused.
+
+When inspecting routes through VIDIOC_SUBDEV_G_ROUTING and the application
+provided ``num_routes`` is not big enough to contain all the available routes
+the subdevice exposes, drivers return the ENOSPC error code and adjust the
+value of the ``num_routes`` field. Application should then reserve enough memory
+for all the route entries and call VIDIOC_SUBDEV_G_ROUTING again.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_routing
+
+.. flat-table:: struct v4l2_subdev_routing
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``which``
+      - Format to modified, from enum
+        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
+    * - struct :c:type:`v4l2_subdev_route`
+      - ``routes[]``
+      - Array of struct :c:type:`v4l2_subdev_route` entries
+    * - __u32
+      - ``num_routes``
+      - Number of entries of the routes array
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
+
+.. c:type:: v4l2_subdev_route
+
+.. flat-table:: struct v4l2_subdev_route
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       1 1 2
+
+    * - __u32
+      - ``sink_pad``
+      - Sink pad number.
+    * - __u32
+      - ``sink_stream``
+      - Sink pad stream number.
+    * - __u32
+      - ``source_pad``
+      - Source pad number.
+    * - __u32
+      - ``source_stream``
+      - Source pad stream number.
+    * - __u32
+      - ``flags``
+      - Route enable/disable flags
+	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
+    * - __u32
+      - ``reserved``\ [5]
+      - Reserved for future extensions. Applications and drivers must set
+	the array to zero.
+
+.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
+
+.. _v4l2-subdev-routing-flags:
+
+.. flat-table:: enum v4l2_subdev_routing_flags
+    :header-rows:  0
+    :stub-columns: 0
+    :widths:       3 1 4
+
+    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - 0
+      - The route is enabled. Set by applications.
+    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
+      - 1
+      - The route is immutable. Set by the driver.
+    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - 2
+      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
+        fields are unused. Set by the driver.
+
+Return Value
+============
+
+On success 0 is returned, on error -1 and the ``errno`` variable is set
+appropriately. The generic error codes are described at the
+:ref:`Generic Error Codes <gen-errors>` chapter.
+
+ENOSPC
+   The number of provided route entries is less than the available ones.
+
+EINVAL
+   The sink or source pad identifiers reference a non-existing pad, or reference
+   pads of different types (ie. the sink_pad identifiers refers to a source pad)
+   or the sink or source stream identifiers reference a non-existing stream on
+   the sink or source pad.
-- 
2.25.1

[PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Add support for subdev internal routing. A route is defined as a single
stream from a sink pad to a source pad.

The userspace can configure the routing via two new ioctls,
VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
implement the functionality with v4l2_subdev_pad_ops.set_routing().

Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
Signed-off-by: Michal Simek <michal.simek@xilinx.com>

- Add sink and source streams for multiplexed links
- Copy the argument back in case of an error. This is needed to let the
  caller know the number of routes.

Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>

- Expand and refine documentation.
- Make the 'routes' pointer a __u64 __user pointer so that a compat32
  version of the ioctl is not required.
- Add struct v4l2_subdev_krouting to be used for subdevice operations.

Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>

- Fix typecasing warnings
- Check sink & source pad types
- Add 'which' field
- Add V4L2_SUBDEV_ROUTE_FL_SOURCE
- Routing to subdev state
- Dropped get_routing subdev op

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
 drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 22 ++++++++
 include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
 4 files changed, 178 insertions(+), 1 deletion(-)

diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
index 05d5db3d85e5..8e9315ffcb99 100644
--- a/drivers/media/v4l2-core/v4l2-ioctl.c
+++ b/drivers/media/v4l2-core/v4l2-ioctl.c
@@ -16,6 +16,7 @@
 #include <linux/kernel.h>
 #include <linux/version.h>
 
+#include <linux/v4l2-subdev.h>
 #include <linux/videodev2.h>
 
 #include <media/v4l2-common.h>
@@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
 		ret = 1;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = parg;
+
+		if (routing->num_routes > 256)
+			return -EINVAL;
+
+		*user_ptr = u64_to_user_ptr(routing->routes);
+		*kernel_ptr = (void **)&routing->routes;
+		*array_size = sizeof(struct v4l2_subdev_route)
+			    * routing->num_routes;
+		ret = 1;
+		break;
+	}
 	}
 
 	return ret;
@@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
 	/*
 	 * Some ioctls can return an error, but still have valid
 	 * results that must be returned.
+	 *
+	 * FIXME: subdev IOCTLS are partially handled here and partially in
+	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
+	 * defined here as part of the 'v4l2_ioctls' array. As
+	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
+	 * in case of failure, but it is not defined here as part of the
+	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
 	 */
-	if (err < 0 && !always_copy)
+	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
 		goto out;
 
 out_array_args:
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 52309a299b03..b4e1f8772d96 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
 		which = ((struct v4l2_subdev_selection *)arg)->which;
 		break;
 	}
+
+	case VIDIOC_SUBDEV_G_ROUTING:
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		which = ((struct v4l2_subdev_routing *)arg)->which;
+		break;
+	}
 	}
 
 	return which == V4L2_SUBDEV_FORMAT_TRY ?
@@ -710,6 +716,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 	case VIDIOC_SUBDEV_QUERYSTD:
 		return v4l2_subdev_call(sd, video, querystd, arg);
 
+	case VIDIOC_SUBDEV_G_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_krouting *krouting;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		krouting = &state->routing;
+
+		if (routing->num_routes < krouting->num_routes) {
+			routing->num_routes = krouting->num_routes;
+			return -ENOSPC;
+		}
+
+		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
+		       krouting->routes,
+		       krouting->num_routes * sizeof(*krouting->routes));
+		routing->num_routes = krouting->num_routes;
+
+		return 0;
+	}
+
+	case VIDIOC_SUBDEV_S_ROUTING: {
+		struct v4l2_subdev_routing *routing = arg;
+		struct v4l2_subdev_route *routes =
+			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
+		struct v4l2_subdev_krouting krouting = {};
+		unsigned int i;
+
+		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+			return -ENOIOCTLCMD;
+
+		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
+			return -EPERM;
+
+		memset(routing->reserved, 0, sizeof(routing->reserved));
+
+		for (i = 0; i < routing->num_routes; ++i) {
+			const struct v4l2_subdev_route *route = &routes[i];
+			const struct media_pad *pads = sd->entity.pads;
+
+			/* Do not check sink pad for source routes */
+			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+				if (route->sink_pad >= sd->entity.num_pads)
+					return -EINVAL;
+
+				if (!(pads[route->sink_pad].flags &
+				      MEDIA_PAD_FL_SINK))
+					return -EINVAL;
+			}
+
+			if (route->source_pad >= sd->entity.num_pads)
+				return -EINVAL;
+
+			if (!(pads[route->source_pad].flags &
+			      MEDIA_PAD_FL_SOURCE))
+				return -EINVAL;
+		}
+
+		krouting.num_routes = routing->num_routes;
+		krouting.routes = routes;
+
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
+	}
+
 	default:
 		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
 	}
@@ -970,6 +1044,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 
 	mutex_destroy(&state->lock);
 
+	kvfree(state->routing.routes);
 	kvfree(state->pads);
 	kfree(state);
 }
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 7f9c83fac020..f318670a49c8 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -694,11 +694,25 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_krouting - subdev routing table
+ *
+ * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
+ *
+ * This structure contains the routing table for a subdev.
+ */
+struct v4l2_subdev_krouting {
+	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
+};
+
 /**
  * struct v4l2_subdev_state - Used for storing subdev state information.
  *
  * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
+ * @routing: routing table for the subdev
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
 struct v4l2_subdev_state {
 	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
+	struct v4l2_subdev_krouting routing;
 };
 
 /**
@@ -770,6 +785,9 @@ struct v4l2_subdev_state {
  *		     applied to the hardware. The operation shall fail if the
  *		     pad index it has been called on is not valid or in case of
  *		     unrecoverable failures.
+ *
+ * @set_routing: enable or disable data connection routes described in the
+ *		 subdevice routing table.
  */
 struct v4l2_subdev_pad_ops {
 	int (*init_cfg)(struct v4l2_subdev *sd,
@@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
 			       struct v4l2_mbus_config *config);
 	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
 			       struct v4l2_mbus_config *config);
+	int (*set_routing)(struct v4l2_subdev *sd,
+			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
+			   struct v4l2_subdev_krouting *route);
 };
 
 /**
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 658106f5b5dc..5ba409db47ff 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
 /* The v4l2 sub-device video device node is registered in read-only mode. */
 #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
 
+/*
+ * Is the route active? An active route will start when streaming is enabled
+ * on a video node.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
+
+/*
+ * Is the route immutable, i.e. can it be activated and inactivated?
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
+
+/*
+ * Is the route a source endpoint? A source endpoint route refers to a stream
+ * generated internally by the subdevice (usually a sensor), and thus there
+ * is no sink-side endpoint for the route. The sink_pad and sink_stream
+ * fields are unused.
+ * Set by the driver.
+ */
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
+
+/**
+ * struct v4l2_subdev_route - A route inside a subdev
+ *
+ * @sink_pad: the sink pad index
+ * @sink_stream: the sink stream identifier
+ * @source_pad: the source pad index
+ * @source_stream: the source stream identifier
+ * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_route {
+	__u32 sink_pad;
+	__u32 sink_stream;
+	__u32 source_pad;
+	__u32 source_stream;
+	__u32 flags;
+	__u32 reserved[5];
+};
+
+/**
+ * struct v4l2_subdev_routing - Subdev routing information
+ *
+ * @which: configuration type (from enum v4l2_subdev_format_whence)
+ * @routes: pointer to the routes array
+ * @num_routes: the total number of routes in the routes array
+ * @reserved: drivers and applications must zero this array
+ */
+struct v4l2_subdev_routing {
+	__u32 which;
+	__u64 routes;
+	__u32 num_routes;
+	__u32 reserved[5];
+};
+
 /* Backwards compatibility define --- to be removed */
 #define v4l2_subdev_edid v4l2_edid
 
@@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
 #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
 #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
 #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
+#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
+#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
 /* The following ioctls are identical to the ioctls in videodev2.h */
 #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
 #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
-- 
2.25.1

[PATCH v9 28/36] media: subdev: add v4l2_subdev_has_route()

[Tomi Valkeinen]

Add a v4l2_subdev_has_route() helper function which can be used for
media_entity_operations.has_route op.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++
 2 files changed, 47 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index b4e1f8772d96..14b8282fe45b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -998,6 +998,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1)
+{
+	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
+	struct v4l2_subdev_krouting *routing;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	state = v4l2_subdev_lock_active_state(sd);
+
+	routing = &state->routing;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
+		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
+			v4l2_subdev_unlock_state(state);
+			return true;
+		}
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	return false;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
+
 struct v4l2_subdev_state *
 __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 			  struct lock_class_key *lock_key)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index f318670a49c8..8b2cf3190276 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1226,6 +1226,22 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_link_validate(struct media_link *link);
 
+/**
+ * v4l2_subdev_has_route - MC has_route implementation for subdevs
+ *
+ * @entity: pointer to &struct media_entity
+ * @pad0: pad number for the first pad
+ * @pad1: pad number for the second pad
+ *
+ * This function looks at the routing in subdev's active state and returns if
+ * there is a route connecting pad0 and pad1.
+ *
+ * This function can be used as implementation for
+ * media_entity_operations.has_route.
+ */
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1);
+
 /**
  * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
  *
-- 
2.25.1

[PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

Add a helper function to set the subdev routing. The helper can be used
from subdev driver's set_routing op to store the routing table.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 +++++++++++++++
 2 files changed, 44 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 14b8282fe45b..af53f827ec27 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
 	mutex_unlock(&state->lock);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
+
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing)
+{
+	struct v4l2_subdev_krouting *dst = &state->routing;
+	const struct v4l2_subdev_krouting *src = routing;
+
+	lockdep_assert_held(&state->lock);
+
+	kvfree(dst->routes);
+	dst->routes = NULL;
+	dst->num_routes = 0;
+
+	if (src->num_routes > 0) {
+		dst->routes = kvmalloc_array(src->num_routes,
+					     sizeof(*src->routes), GFP_KERNEL);
+		if (!dst->routes)
+			return -ENOMEM;
+
+		memcpy(dst->routes, src->routes,
+		       src->num_routes * sizeof(*src->routes));
+		dst->num_routes = src->num_routes;
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 8b2cf3190276..b6c5cd00831e 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1437,4 +1437,20 @@ v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
 	return state;
 }
 
+/**
+ * v4l2_subdev_set_routing() - Set given routing to subdev state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ *
+ * This will release old routing table (if any) from the state, allocate
+ * enough space for the given routing, and copy the routing.
+ *
+ * This can be used from the subdev driver's set_routing op, after validating
+ * the routing.
+ */
+int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
+			    struct v4l2_subdev_state *state,
+			    struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

[PATCH v9 30/36] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

Add documentation related to multiplexed streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../driver-api/media/v4l2-subdev.rst          |   8 +
 .../userspace-api/media/v4l/dev-subdev.rst    | 164 ++++++++++++++++++
 2 files changed, 172 insertions(+)

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index c3fd57fff668..410e03a29f2a 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -546,6 +546,14 @@ subdev drivers. In these cases the called subdev ops must also handle the NULL
 case. This can be easily managed by the use of
 v4l2_subdev_validate_and_lock_state() helper.
 
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
+
 V4L2 sub-device functions and data structures
 ---------------------------------------------
 
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..ea3efa97bb08 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,167 @@ source pads.
     :maxdepth: 1
 
     subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, each
+link in the media grap and each pad in a subdevice passes through a single
+video stream. Thus each pad contains a format configuration for that single
+stream. In some cases a subdev can do stream processing and split a stream
+into two or compose two streams into one, but the inputs and outputs for the
+subdev are still a single stream per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
+bus carries multiple streams. Thus a camera could output two streams, a pixel
+stream and a metadata stream, and a bridge subdev could route the streams
+from multiple input pads into a single output pad.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, requires such a routing
+configuration where the link between those two types of drivers contain only
+a single stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink
+(e.g. a receiver in a SoC). Each media link carries all the streams from
+one end of the link to the other, whereas subdevices have routing tables
+which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a
+stream. In other words, configuration for a particular stream ID must exist
+on both sides of a media link, but another stream ID can be used for the same
+stream at the other side of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
+  source pad (pad 0), and outputs two streams, pixel data and metadata.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint), instead of just a pad.
+
+Presuming there are no format conversions in the pipeline, the userspace needs
+to configure all the route endpoints using four formats (two pixel formats
+and two metadata formats) with VIDIOC_SUBDEV_S_FMT.
-- 
2.25.1

[PATCH v9 31/36] media: subdev: add stream based configuration

[Tomi Valkeinen]

Add support to manage configurations (format, crop, compose) per stream,
instead of per pad. This is accomplished with data structures that hold
an array of all subdev's stream configurations.

The number of streams can vary at runtime based on routing. Every time
the routing is changed, the stream configurations need to be
re-initialized.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
 .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
 .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
 .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
 .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
 .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
 .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
 drivers/media/v4l2-core/v4l2-subdev.c         | 140 +++++++++++++++++-
 include/media/v4l2-subdev.h                   |  48 ++++++
 include/uapi/linux/v4l2-subdev.h              |  28 +++-
 10 files changed, 229 insertions(+), 22 deletions(-)

diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
index 3703943b412f..8def4c05d3da 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
@@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
       - Frame intervals to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
index c25a9896df0e..3ef361c0dca7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
@@ -97,7 +97,10 @@ information about try formats.
       - Frame sizes to be enumerated, from enum
 	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
index 417f1a19bcc4..248f6f9ee7c5 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
@@ -73,7 +73,10 @@ information about the try formats.
       - ``flags``
       - See :ref:`v4l2-subdev-mbus-code-flags`
     * - __u32
-      - ``reserved``\ [7]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [6]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
index bd15c0a5a66b..1d267f7e7991 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
@@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
       - ``rect``
       - Crop rectangle boundaries, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
index 7acdbb939d89..ed253a1e44b7 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
@@ -102,7 +102,10 @@ should be as close as possible to the original request.
       - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
 	details.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
index d7fe7543c506..842f962d2aea 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
@@ -90,7 +90,10 @@ the same sub-device is not defined.
       - ``interval``
       - Period, in seconds, between consecutive video frames.
     * - __u32
-      - ``reserved``\ [9]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [8]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
index f9172a42f036..6b629c19168c 100644
--- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
+++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
@@ -94,7 +94,10 @@ Selection targets and flags are documented in
       - ``r``
       - Selection rectangle, in pixels.
     * - __u32
-      - ``reserved``\ [8]
+      - ``stream``
+      - Stream identifier.
+    * - __u32
+      - ``reserved``\ [7]
       - Reserved for future extensions. Applications and drivers must set
 	the array to zero.
 
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index af53f827ec27..f63826755057 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -149,14 +149,45 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
 	return 0;
 }
 
-static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
+static int check_state_pads(struct v4l2_subdev *sd, u32 which,
+			    struct v4l2_subdev_state *state)
 {
+	if (sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)
+		return 0;
+
 	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
 		return -EINVAL;
 
 	return 0;
 }
 
+static int check_state_pad_stream(struct v4l2_subdev *sd,
+				  struct v4l2_subdev_state *state, u32 pad,
+				  u32 stream)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
+		return 0;
+
+	/*
+	 * We need to take the state lock to access the format, but as we then
+	 * have to unlock, nothing prevents someone changing the state before
+	 * this call thread enters the driver's op and the driver has the
+	 * change to lock the state.
+	 */
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, pad, stream);
+
+	v4l2_subdev_unlock_state(state);
+
+	if (!fmt)
+		return -EINVAL;
+
+	return 0;
+}
+
 static inline int check_format(struct v4l2_subdev *sd,
 			       struct v4l2_subdev_state *state,
 			       struct v4l2_subdev_format *format)
@@ -165,7 +196,8 @@ static inline int check_format(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
-	       check_state_pads(format->which, state);
+	       check_state_pads(sd, format->which, state) ? :
+	       check_state_pad_stream(sd, state, format->pad, format->stream);
 }
 
 static int call_get_fmt(struct v4l2_subdev *sd,
@@ -192,7 +224,8 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
-	       check_state_pads(code->which, state) ? :
+	       check_state_pads(sd, code->which, state) ? :
+	       check_state_pad_stream(sd, state, code->pad, code->stream) ? :
 	       sd->ops->pad->enum_mbus_code(sd, state, code);
 }
 
@@ -204,7 +237,8 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
-	       check_state_pads(fse->which, state) ? :
+	       check_state_pads(sd, fse->which, state) ? :
+	       check_state_pad_stream(sd, state, fse->pad, fse->stream) ? :
 	       sd->ops->pad->enum_frame_size(sd, state, fse);
 }
 
@@ -239,7 +273,8 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
-	       check_state_pads(fie->which, state) ? :
+	       check_state_pads(sd, fie->which, state) ? :
+	       check_state_pad_stream(sd, state, fie->pad, fie->stream) ? :
 	       sd->ops->pad->enum_frame_interval(sd, state, fie);
 }
 
@@ -251,7 +286,8 @@ static inline int check_selection(struct v4l2_subdev *sd,
 		return -EINVAL;
 
 	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
-	       check_state_pads(sel->which, state);
+	       check_state_pads(sd, sel->which, state) ? :
+	       check_state_pad_stream(sd, state, sel->pad, sel->stream);
 }
 
 static int call_get_selection(struct v4l2_subdev *sd,
@@ -867,6 +903,71 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
 	.poll = subdev_poll,
 };
 
+static int
+v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
+			 const struct v4l2_subdev_krouting *routing)
+{
+	u32 num_configs = 0;
+	unsigned int i;
+	u32 format_idx = 0;
+
+	kvfree(stream_configs->configs);
+	stream_configs->configs = NULL;
+	stream_configs->num_configs = 0;
+
+	/* Count number of formats needed */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		/*
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
+		 */
+		num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+	}
+
+	if (num_configs) {
+		stream_configs->configs =
+			kvcalloc(num_configs, sizeof(*stream_configs->configs),
+				 GFP_KERNEL);
+
+		if (!stream_configs->configs)
+			return -ENOMEM;
+
+		stream_configs->num_configs = num_configs;
+	}
+
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		u32 idx;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
+			idx = format_idx++;
+
+			stream_configs->configs[idx].pad = route->sink_pad;
+			stream_configs->configs[idx].stream = route->sink_stream;
+		}
+
+		idx = format_idx++;
+
+		stream_configs->configs[idx].pad = route->source_pad;
+		stream_configs->configs[idx].stream = route->source_stream;
+	}
+
+	return 0;
+}
+
 #ifdef CONFIG_MEDIA_CONTROLLER
 
 int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
@@ -1042,7 +1143,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
 
 	__mutex_init(&state->lock, lock_name, lock_key);
 
-	if (sd->entity.num_pads) {
+	/* Drivers that support streams do not need the legacy pad config */
+	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
 		state->pads = kvmalloc_array(sd->entity.num_pads,
 					     sizeof(*state->pads),
 					     GFP_KERNEL | __GFP_ZERO);
@@ -1076,6 +1178,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
 	mutex_destroy(&state->lock);
 
 	kvfree(state->routing.routes);
+	kvfree(state->stream_configs.configs);
 	kvfree(state->pads);
 	kfree(state);
 }
@@ -1176,6 +1279,27 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 		dst->num_routes = src->num_routes;
 	}
 
-	return 0;
+	return v4l2_init_stream_configs(&state->stream_configs, dst);
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
+
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+
+	lockdep_assert_held(&state->lock);
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i) {
+		if (stream_configs->configs[i].pad == pad &&
+		    stream_configs->configs[i].stream == stream)
+			return &stream_configs->configs[i].fmt;
+	}
+
+	return NULL;
+}
+EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index b6c5cd00831e..1dc824416c1b 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -694,6 +694,37 @@ struct v4l2_subdev_pad_config {
 	struct v4l2_rect try_compose;
 };
 
+/**
+ * struct v4l2_subdev_stream_config - Used for storing stream configuration.
+ *
+ * @pad: pad number
+ * @stream: stream number
+ * @fmt: &struct v4l2_mbus_framefmt
+ * @crop: &struct v4l2_rect to be used for crop
+ * @compose: &struct v4l2_rect to be used for compose
+ *
+ * This structure stores configuration for a stream.
+ */
+struct v4l2_subdev_stream_config {
+	u32 pad;
+	u32 stream;
+
+	struct v4l2_mbus_framefmt fmt;
+	struct v4l2_rect crop;
+	struct v4l2_rect compose;
+};
+
+/**
+ * struct v4l2_subdev_stream_configs - A collection of stream configs.
+ *
+ * @num_configs: number of entries in @config.
+ * @configs: an array of &struct v4l2_subdev_stream_configs.
+ */
+struct v4l2_subdev_stream_configs {
+	u32 num_configs;
+	struct v4l2_subdev_stream_config *configs;
+};
+
 /**
  * struct v4l2_subdev_krouting - subdev routing table
  *
@@ -713,6 +744,7 @@ struct v4l2_subdev_krouting {
  * @lock: mutex for the state
  * @pads: &struct v4l2_subdev_pad_config array
  * @routing: routing table for the subdev
+ * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
  *
  * This structure only needs to be passed to the pad op if the 'which' field
  * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
@@ -722,6 +754,7 @@ struct v4l2_subdev_state {
 	struct mutex lock;
 	struct v4l2_subdev_pad_config *pads;
 	struct v4l2_subdev_krouting routing;
+	struct v4l2_subdev_stream_configs stream_configs;
 };
 
 /**
@@ -1453,4 +1486,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_state_get_stream_format() - Get pointer to a stream format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
+ * stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
+			     u32 stream);
+
 #endif
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index 5ba409db47ff..f61ace2b5755 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @format: media bus format (format code and frame size)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_format {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_mbus_framefmt format;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -58,13 +60,15 @@ struct v4l2_subdev_format {
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @pad: pad number, as reported by the media API
  * @rect: pad crop rectangle boundaries
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_crop {
 	__u32 which;
 	__u32 pad;
 	struct v4l2_rect rect;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
@@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
  * @code: format code (MEDIA_BUS_FMT_ definitions)
  * @which: format type (from enum v4l2_subdev_format_whence)
  * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_mbus_code_enum {
@@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
 	__u32 code;
 	__u32 which;
 	__u32 flags;
-	__u32 reserved[7];
+	__u32 stream;
+	__u32 reserved[6];
 };
 
 /**
@@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
  * @min_height: minimum frame height, in pixels
  * @max_height: maximum frame height, in pixels
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_size_enum {
@@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
 	__u32 min_height;
 	__u32 max_height;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
  * struct v4l2_subdev_frame_interval - Pad-level frame rate
  * @pad: pad number, as reported by the media API
  * @interval: frame interval in seconds
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval {
 	__u32 pad;
 	struct v4l2_fract interval;
-	__u32 reserved[9];
+	__u32 stream;
+	__u32 reserved[8];
 };
 
 /**
@@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
  * @height: frame height in pixels
  * @interval: frame interval in seconds
  * @which: format type (from enum v4l2_subdev_format_whence)
+ * @stream: stream number, defined in subdev routing
  * @reserved: drivers and applications must zero this array
  */
 struct v4l2_subdev_frame_interval_enum {
@@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
 	__u32 height;
 	struct v4l2_fract interval;
 	__u32 which;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
@@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
  *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
  * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
  * @r: coordinates of the selection window
+ * @stream: stream number, defined in subdev routing
  * @reserved: for future use, set to zero for now
  *
  * Hardware may use multiple helper windows to process a video stream.
@@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
 	__u32 target;
 	__u32 flags;
 	struct v4l2_rect r;
-	__u32 reserved[8];
+	__u32 stream;
+	__u32 reserved[7];
 };
 
 /**
-- 
2.25.1

[PATCH v9 32/36] media: subdev: use streams in v4l2_subdev_link_validate()

[Tomi Valkeinen]

Update v4l2_subdev_link_validate() to use routing and streams for
validation.

Instead of just looking at the format on the pad on both ends of the
link, the routing tables are used to collect all the streams going from
the source to the sink over the link, and the streams' formats on both
ends of the link are verified.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 217 +++++++++++++++++++++++---
 1 file changed, 199 insertions(+), 18 deletions(-)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index f63826755057..37e2e1f907fc 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -16,6 +16,7 @@
 #include <linux/videodev2.h>
 #include <linux/export.h>
 #include <linux/version.h>
+#include <linux/sort.h>
 
 #include <media/v4l2-ctrls.h>
 #include <media/v4l2-device.h>
@@ -1050,6 +1051,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
 
 static int
 v4l2_subdev_link_validate_get_format(struct media_pad *pad,
+				     u32 stream,
 				     struct v4l2_subdev_format *fmt)
 {
 	if (is_media_entity_v4l2_subdev(pad->entity)) {
@@ -1061,6 +1063,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 
 		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
 		fmt->pad = pad->index;
+		fmt->stream = stream;
 		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
 	}
 
@@ -1071,31 +1074,209 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
 	return -EINVAL;
 }
 
-int v4l2_subdev_link_validate(struct media_link *link)
+static int cmp_u32(const void *a, const void *b)
 {
-	struct v4l2_subdev *sink;
-	struct v4l2_subdev_format sink_fmt, source_fmt;
-	int rval;
+	u32 a32 = *(u32 *)a;
+	u32 b32 = *(u32 *)b;
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->source, &source_fmt);
-	if (rval < 0)
-		return 0;
+	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
+}
+
+static int v4l2_link_validate_get_streams(struct media_link *link,
+					  bool is_source, u32 *out_num_streams,
+					  const u32 **out_streams,
+					  bool *allocated)
+{
+	static const u32 default_streams[] = { 0 };
+	struct v4l2_subdev_krouting *routing;
+	struct v4l2_subdev *subdev;
+	u32 num_streams;
+	u32 *streams;
+	unsigned int i;
+	struct v4l2_subdev_state *state;
+
+	if (is_source)
+		subdev = media_entity_to_v4l2_subdev(link->source->entity);
+	else
+		subdev = media_entity_to_v4l2_subdev(link->sink->entity);
 
-	rval = v4l2_subdev_link_validate_get_format(
-		link->sink, &sink_fmt);
-	if (rval < 0)
+	if (!(subdev->flags & V4L2_SUBDEV_FL_MULTIPLEXED)) {
+		*out_num_streams = 1;
+		*out_streams = default_streams;
+		*allocated = false;
 		return 0;
+	}
 
-	sink = media_entity_to_v4l2_subdev(link->sink->entity);
+	state = v4l2_subdev_lock_active_state(subdev);
 
-	rval = v4l2_subdev_call(sink, pad, link_validate, link,
-				&source_fmt, &sink_fmt);
-	if (rval != -ENOIOCTLCMD)
-		return rval;
+	routing = &state->routing;
+
+	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
+
+	if (!streams) {
+		v4l2_subdev_unlock_state(state);
+		return -ENOMEM;
+	}
+
+	num_streams = 0;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+		int j;
+		u32 route_pad;
+		u32 route_stream;
+		u32 link_pad;
+
+		if (is_source) {
+			route_pad = route->source_pad;
+			route_stream = route->source_stream;
+			link_pad = link->source->index;
+		} else {
+			route_pad = route->sink_pad;
+			route_stream = route->sink_stream;
+			link_pad = link->sink->index;
+		}
+
+		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
+			continue;
+
+		if (route_pad != link_pad)
+			continue;
+
+		/* look for duplicates... */
+		for (j = 0; j < num_streams; ++j) {
+			if (streams[j] == route_stream)
+				break;
+		}
+
+		/* ...and drop the stream if we already have it */
+		if (j != num_streams)
+			continue;
+
+		streams[num_streams++] = route_stream;
+	}
+
+	v4l2_subdev_unlock_state(state);
+
+	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
+
+	*out_num_streams = num_streams;
+	*out_streams = streams;
+	*allocated = true;
 
-	return v4l2_subdev_link_validate_default(
-		sink, link, &source_fmt, &sink_fmt);
+	return 0;
+}
+
+int v4l2_subdev_link_validate(struct media_link *link)
+{
+	struct v4l2_subdev *sink_subdev =
+		media_entity_to_v4l2_subdev(link->sink->entity);
+	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
+	u32 num_source_streams;
+	const u32 *source_streams;
+	bool source_allocated;
+	u32 num_sink_streams;
+	const u32 *sink_streams;
+	bool sink_allocated;
+	unsigned int sink_idx;
+	unsigned int source_idx;
+	int ret;
+
+	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
+		link->source->entity->name, link->source->index,
+		link->sink->entity->name, link->sink->index);
+
+	ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
+					     &source_streams,
+					     &source_allocated);
+	if (ret)
+		return ret;
+
+	ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
+					     &sink_streams, &sink_allocated);
+	if (ret)
+		goto free_source;
+
+	/* It is ok to have more source streams than sink streams */
+	if (num_source_streams < num_sink_streams) {
+		dev_err(dev,
+			"Not enough source streams: %d < %d\n",
+			num_source_streams, num_sink_streams);
+		ret = -EINVAL;
+		goto out;
+	}
+
+	/* Validate source and sink stream formats */
+
+	source_idx = 0;
+
+	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
+		struct v4l2_subdev_format sink_fmt, source_fmt;
+		u32 stream;
+
+		stream = sink_streams[sink_idx];
+
+		for (; source_idx < num_source_streams; ++source_idx) {
+			if (source_streams[source_idx] == stream)
+				break;
+		}
+
+		if (source_idx == num_source_streams) {
+			dev_err(dev, "No source stream for sink stream %u\n",
+				stream);
+			ret = -EINVAL;
+			goto out;
+		}
+
+		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
+			link->source->entity->name, link->source->index, stream,
+			link->sink->entity->name, link->sink->index, stream);
+
+		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
+							   &source_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->source->entity->name, link->source->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
+							   &sink_fmt);
+		if (ret < 0) {
+			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
+				link->sink->entity->name, link->sink->index,
+				stream);
+			ret = 0;
+			continue;
+		}
+
+		/* TODO: add stream number to link_validate() */
+		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
+				       &source_fmt, &sink_fmt);
+		if (!ret)
+			continue;
+
+		if (ret != -ENOIOCTLCMD)
+			goto out;
+
+		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
+							&source_fmt, &sink_fmt);
+
+		if (ret)
+			goto out;
+	}
+
+out:
+	if (sink_allocated)
+		kfree(sink_streams);
+
+free_source:
+	if (source_allocated)
+		kfree(source_streams);
+
+	return ret;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
 
-- 
2.25.1

[PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

Add two helper functions to make dealing with streams easier:

v4l2_state_find_opposite_end - given a routing table and a pad + stream,
return the pad + stream on the opposite side of the subdev.

v4l2_state_get_opposite_stream_format - return a pointer to the format
on the pad + stream on the opposite side from the given pad + stream.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
 2 files changed, 74 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 37e2e1f907fc..9eeadad997c8 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 	return NULL;
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
+
+int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
+				 u32 stream, u32 *other_pad, u32 *other_stream)
+{
+	unsigned int i;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		struct v4l2_subdev_route *route = &routing->routes[i];
+
+		if (route->source_pad == pad &&
+		    route->source_stream == stream) {
+			*other_pad = route->sink_pad;
+			*other_stream = route->sink_stream;
+			return 0;
+		}
+
+		if (route->sink_pad == pad && route->sink_stream == stream) {
+			*other_pad = route->source_pad;
+			*other_stream = route->source_stream;
+			return 0;
+		}
+	}
+
+	return -EINVAL;
+}
+EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
+
+struct v4l2_mbus_framefmt *
+v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
+				      u32 stream)
+{
+	u32 other_pad, other_stream;
+	int ret;
+
+	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
+					   &other_pad, &other_stream);
+	if (ret)
+		return NULL;
+
+	return v4l2_state_get_stream_format(state, other_pad, other_stream);
+}
+EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1dc824416c1b..4b9520410783 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1501,4 +1501,36 @@ struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream);
 
+/**
+ * v4l2_state_find_opposite_end() - Find the opposite stream
+ * @routing: routing used to find the opposite side
+ * @pad: pad id
+ * @stream: stream id
+ * @other_pad: pointer used to return the opposite pad
+ * @other_stream: pointer used to return the opposite stream
+ *
+ * This function uses the routing table to find the pad + stream which is
+ * opposite the given pad + stream.
+ *
+ * Returns 0 on success, or -EINVAL if no matching route is found.
+ */
+int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
+				 u32 stream, u32 *other_pad, u32 *other_stream);
+
+/**
+ * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
+ *					     format
+ * @state: subdevice state
+ * @pad: pad id
+ * @stream: stream id
+ *
+ * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
+ * that is opposite the given pad + stream in the subdev state.
+ *
+ * If the state does not contain the given pad + stream, NULL is returned.
+ */
+struct v4l2_mbus_framefmt *
+v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
+				      u32 stream);
+
 #endif
-- 
2.25.1

[PATCH v9 34/36] media: subdev: add v4l2_subdev_get_fmt() helper function

[Tomi Valkeinen]

Add v4l2_subdev_get_fmt() helper function which implements
v4l2_subdev_pad_ops.get_fmt using streams. Subdev drivers that do not
need to do anything special in their get_fmt op can use this helper
directly for v4l2_subdev_pad_ops.get_fmt.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 37 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 9eeadad997c8..d402ffeef560 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1526,3 +1526,24 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 	return v4l2_state_get_stream_format(state, other_pad, other_stream);
 }
 EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
+
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format)
+{
+	struct v4l2_mbus_framefmt *fmt;
+
+	v4l2_subdev_lock_state(state);
+
+	fmt = v4l2_state_get_stream_format(state, format->pad, format->stream);
+	if (!fmt) {
+		v4l2_subdev_unlock_state(state);
+		return -EINVAL;
+	}
+
+	format->format = *fmt;
+
+	v4l2_subdev_unlock_state(state);
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 4b9520410783..04fca595aa91 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1532,5 +1532,21 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
 struct v4l2_mbus_framefmt *
 v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 				      u32 stream);
+/**
+ * v4l2_subdev_get_fmt() - Fill format based on state
+ * @sd: subdevice
+ * @state: subdevice state
+ * @format: pointer to &struct v4l2_subdev_format
+ *
+ * Fill @format based on the pad and stream given in the @format struct.
+ *
+ * This function can be used by the subdev drivers to implement
+ * v4l2_subdev_pad_ops.get_fmt if the subdev driver does not need to do
+ * anything special in their get_fmt op.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
+			struct v4l2_subdev_format *format);
 
 #endif
-- 
2.25.1

[PATCH v9 35/36] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Tomi Valkeinen]

v4l2_subdev_set_routing_with_fmt() is the same as
v4l2_subdev_set_routing(), but additionally initializes all the streams
with the given format.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 16 ++++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index d402ffeef560..22a9bea0fa85 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1464,6 +1464,28 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt)
+{
+	struct v4l2_subdev_stream_configs *stream_configs;
+	unsigned int i;
+	int ret;
+
+	ret = v4l2_subdev_set_routing(sd, state, routing);
+	if (ret)
+		return ret;
+
+	stream_configs = &state->stream_configs;
+
+	for (i = 0; i < stream_configs->num_configs; ++i)
+		stream_configs->configs[i].fmt = *fmt;
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
+
 struct v4l2_mbus_framefmt *
 v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
 			     u32 stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 04fca595aa91..1a4df0aafe8a 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1486,6 +1486,22 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
 			    struct v4l2_subdev_krouting *routing);
 
+/**
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
+ * @sd: The subdevice
+ * @state: The subdevice state
+ * @routing: Routing that will be copied to subdev state
+ * @fmt: Format used to initialize all the streams
+ *
+ * This is the same as v4l2_subdev_set_routing, but additionally initializes
+ * all the streams using the given format.
+ */
+int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
+				     struct v4l2_subdev_state *state,
+				     struct v4l2_subdev_krouting *routing,
+				     const struct v4l2_mbus_framefmt *fmt);
+
 /**
  * v4l2_state_get_stream_format() - Get pointer to a stream format
  * @state: subdevice state
-- 
2.25.1

[PATCH v9 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Tomi Valkeinen]

Add a helper for verifying routing for the common case of
non-overlapping 1-to-1 streams.

Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
---
 drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
 include/media/v4l2-subdev.h           | 14 ++++++++++++++
 2 files changed, 38 insertions(+)

diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index 22a9bea0fa85..2a64ff003e4b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -1569,3 +1569,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 	return 0;
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
+
+int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
+{
+	unsigned int i, j;
+
+	for (i = 0; i < routing->num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routing->routes[i];
+
+		for (j = i + 1; j < routing->num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routing->routes[j];
+
+			if (route->sink_pad == r->sink_pad &&
+			    route->sink_stream == r->sink_stream)
+				return -EINVAL;
+
+			if (route->source_pad == r->source_pad &&
+			    route->source_stream == r->source_stream)
+				return -EINVAL;
+		}
+	}
+
+	return 0;
+}
+EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 1a4df0aafe8a..5e50f2ded653 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -1565,4 +1565,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
 int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 			struct v4l2_subdev_format *format);
 
+/**
+ * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
+ *				  1-to-1 streams
+ * @routing: routing to verify
+ *
+ * This verifies that the given routing contains only non-overlapping 1-to-1
+ * streams. In other words, no two streams have the same source or sink
+ * stream ID on a single pad. This is the most common case of routing
+ * supported by devices.
+ *
+ * Returns 0 on success, error value otherwise.
+ */
+int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
+
 #endif
-- 
2.25.1

Re: [PATCH v9 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

On 05/10/2021 11:57, Tomi Valkeinen wrote:
> Hi,
> 
> This is v9 of the multiplexed streams series. v8 can be found from:
> 
> https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/

Here is a diff between v8 and v9, which may help in reviews.

diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
index b28607c3fc79..410e03a29f2a 100644
--- a/Documentation/driver-api/media/v4l2-subdev.rst
+++ b/Documentation/driver-api/media/v4l2-subdev.rst
@@ -523,7 +523,7 @@ Centrally managed subdev active state
  
  Traditionally V4L2 subdev drivers maintained internal state for the active
  configuration for the subdev. This is often implemented as an array of struct
-v4l2_mbus_framefmt, on entry for each pad.
+v4l2_mbus_framefmt, one entry for each pad.
  
  In addition to the active configuration, each subdev filehandle has contained
  an array of struct v4l2_subdev_pad_config, managed by V4L2 core, which
@@ -531,17 +531,28 @@ contains the TRY configuration.
  
  To simplify the subdev drivers the V4L2 subdev API now optionally supports a
  centrally managed active configuration. A subdev driver must use
-v4l2_subdev_alloc_state() to initialize the active state between calls to
-media_entity_pads_init() and v4l2_*_register_subdev().
+v4l2_subdev_init_finalize() to initialize the active state between calls to
+media_entity_pads_init() and v4l2_*_register_subdev(), and must call
+v4l2_subdev_cleanup() to free the state.
  
-The active state must be locked before access, and can be done with e.g.
+The active state must be locked before access, and can be done with
  v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().
  
  The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
-Unfortunately all the subdev drivers do not comply with this yet, and may
-pass NULL for the ACTIVE case, so the called subdev ops must also handle the
-NULL case. This can be easily managed by the helpers
-v4l2_subdev_validate_state() or v4l2_subdev_validate_and_lock_state().
+Unfortunately all the subdev drivers do not comply with this yet, and may pass
+NULL for the ACTIVE case. This is only a problem for subdev drivers which use
+the cetrally managed active state and are used in media pipelines with older
+subdev drivers. In these cases the called subdev ops must also handle the NULL
+case. This can be easily managed by the use of
+v4l2_subdev_validate_and_lock_state() helper.
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+A subdevice driver can implement support for multiplexed streams by setting
+the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
+centrally managed subdev active state, routing and stream based
+configuration.
  
  V4L2 sub-device functions and data structures
  ---------------------------------------------
diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
index a67c2749089a..ea3efa97bb08 100644
--- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
+++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
@@ -503,3 +503,167 @@ source pads.
      :maxdepth: 1
  
      subdev-formats
+
+Streams, multiplexed media pads and internal routing
+----------------------------------------------------
+
+Commonly V4L2 subdevices support only separate video streams, that is, each
+link in the media grap and each pad in a subdevice passes through a single
+video stream. Thus each pad contains a format configuration for that single
+stream. In some cases a subdev can do stream processing and split a stream
+into two or compose two streams into one, but the inputs and outputs for the
+subdev are still a single stream per pad.
+
+Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
+bus carries multiple streams. Thus a camera could output two streams, a pixel
+stream and a metadata stream, and a bridge subdev could route the streams
+from multiple input pads into a single output pad.
+
+Subdevice drivers that support multiplexed streams are compatible with
+non-multiplexed subdev drivers, but, of course, requires such a routing
+configuration where the link between those two types of drivers contain only
+a single stream.
+
+Understanding streams
+^^^^^^^^^^^^^^^^^^^^^
+
+A stream is a stream of content (e.g. pixel data or metadata) flowing through
+the media pipeline from a source (e.g. a sensor) towards the final sink
+(e.g. a receiver in a SoC). Each media link carries all the streams from
+one end of the link to the other, whereas subdevices have routing tables
+which describe how the incoming streams from sink pads are routed to the
+source pads.
+
+A stream ID (often just "stream") is a media link-local identifier for a
+stream. In other words, configuration for a particular stream ID must exist
+on both sides of a media link, but another stream ID can be used for the same
+stream at the other side of the subdevice.
+
+A stream at a specific point in the media pipeline is identified with the
+subdev and a (pad, stream) pair. For subdevices that do not support
+multiplexed streams the 'stream' is always 0.
+
+Configuring streams
+^^^^^^^^^^^^^^^^^^^
+
+The configuration of the streams is done individually for each subdevice and
+the validity of the streams between subdevices is validated when the pipeline
+is started.
+
+There are three steps in configuring the streams:
+
+1) Set up links. Connect the pads between subdevices using the :ref:`Media
+Controller API <media_controller>`
+
+2) Routing. The routing table for the subdevice must be set with
+:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
+
+3) Configure streams. Each route endpoint must be configured
+with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
+
+Multiplexed streams setup example
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A simple example of a multiplexed stream setup might be as follows:
+
+- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
+  source pad (pad 0), and outputs two streams, pixel data and metadata.
+
+- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
+  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
+  streams.
+
+- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
+  connected to the bridge, and four source pads (pads 1-4), going to the DMA
+  engine. The receiver demultiplexes the incoming streams to the four source
+  pads.
+
+- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
+  single source pad in the receiver.
+
+The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
+exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
+modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
+
+To configure this pipeline, the userspace must take the following steps:
+
+1) Set up media links between entities: connect the sensors to the bridge,
+bridge to the receiver, and the receiver to the DMA engines. This step does
+not differ from normal non-multiplexed media controller setup.
+
+2) Configure routing.
+
+.. flat-table:: Sensor routing table (identical on both sensors)
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0 (unused)
+      - 0/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Pixel data stream. Source route, i.e. the sink fields are unused.
+    * - 0/0 (unused)
+      - 0/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
+      - Metadata stream. Source route, i.e. the sink fields are unused.
+
+.. flat-table:: Bridge routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/1
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 1/0
+      - 2/2
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 1/1
+      - 2/3
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+.. flat-table:: Receiver routing table
+    :header-rows:  1
+
+    * - Sink Pad/Stream
+      - Source Pad/Stream
+      - Routing Flags
+      - Comments
+    * - 0/0
+      - 1/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor A
+    * - 0/1
+      - 2/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor A
+    * - 0/2
+      - 3/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Pixel data stream from Sensor B
+    * - 0/3
+      - 4/0
+      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
+      - Metadata stream from Sensor B
+
+3) Configure streams
+
+After configuring the routing table, the next step is configuring the streams.
+This step is similar to configuring the pads in a non-multiplexed streams
+setup, with the difference that we need to configure each (pad, stream) pair
+(i.e. route endpoint), instead of just a pad.
+
+Presuming there are no format conversions in the pipeline, the userspace needs
+to configure all the route endpoints using four formats (two pixel formats
+and two metadata formats) with VIDIOC_SUBDEV_S_FMT.
diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
index 1de30d5b437f..e6bd94d63e4f 100644
--- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
+++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
@@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
  {
  	struct v4l2_subdev *sd = vin_to_source(vin);
  	struct v4l2_subdev_state *sd_state;
+	static struct lock_class_key key;
  	struct v4l2_subdev_format format = {
  		.which = which,
  		.pad = vin->parallel.source_pad,
@@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
  	u32 width, height;
  	int ret;
  
-	sd_state = v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE);
+	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
  	if (IS_ERR(sd_state))
  		return PTR_ERR(sd_state);
  
@@ -288,7 +289,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
  
  	rvin_format_align(vin, pix);
  done:
-	v4l2_free_subdev_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
  
  	return ret;
  }
diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
index 63ea5e472c33..e607c3ae2520 100644
--- a/drivers/media/platform/vsp1/vsp1_entity.c
+++ b/drivers/media/platform/vsp1/vsp1_entity.c
@@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
  		     const char *name, unsigned int num_pads,
  		     const struct v4l2_subdev_ops *ops, u32 function)
  {
+	static struct lock_class_key key;
  	struct v4l2_subdev *subdev;
  	unsigned int i;
  	int ret;
@@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
  	 * Allocate the pad configuration to store formats and selection
  	 * rectangles.
  	 */
-	entity->config = v4l2_alloc_subdev_state(&entity->subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
+	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
+						   "vsp1:config->lock", &key);
  	if (IS_ERR(entity->config)) {
  		media_entity_cleanup(&entity->subdev.entity);
  		return PTR_ERR(entity->config);
@@ -690,6 +692,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
  		entity->ops->destroy(entity);
  	if (entity->subdev.ctrl_handler)
  		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
-	v4l2_free_subdev_state(entity->config);
+	__v4l2_subdev_state_free(entity->config);
  	media_entity_cleanup(&entity->subdev.entity);
  }
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index e3a7793157db..2a64ff003e4b 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -27,11 +27,10 @@
  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
  {
-	static struct lock_class_key __key;
  	struct v4l2_subdev_state *state;
+	static struct lock_class_key key;
  
-	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_TRY,
-					  "v4l2_subdev_fh->state", &__key);
+	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
  	if (IS_ERR(state))
  		return PTR_ERR(state);
  
@@ -42,7 +41,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
  
  static void subdev_fh_free(struct v4l2_subdev_fh *fh)
  {
-	v4l2_free_subdev_state(fh->state);
+	__v4l2_subdev_state_free(fh->state);
  	fh->state = NULL;
  }
  
@@ -815,11 +814,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
  				return -EINVAL;
  		}
  
-		krouting.which = routing->which;
  		krouting.num_routes = routing->num_routes;
  		krouting.routes = routes;
  
-		return v4l2_subdev_call(sd, pad, set_routing, state, &krouting);
+		return v4l2_subdev_call(sd, pad, set_routing, state,
+					routing->which, &krouting);
  	}
  
  	default:
@@ -925,10 +924,11 @@ v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
  			continue;
  
  		/*
-		 * Each route needs a format on both ends of the route, except for
-		 * source streams which only need one format.
+		 * Each route needs a format on both ends of the route, except
+		 * for source streams which only need one format.
  		 */
-		num_configs += (route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
+		num_configs +=
+			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
  	}
  
  	if (num_configs) {
@@ -942,7 +942,10 @@ v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
  		stream_configs->num_configs = num_configs;
  	}
  
-	/* Fill in the 'pad' and stream' value for each item in the array from the routing table */
+	/*
+	 * Fill in the 'pad' and stream' value for each item in the array from
+	 * the routing table
+	 */
  	for (i = 0; i < routing->num_routes; ++i) {
  		struct v4l2_subdev_route *route = &routing->routes[i];
  		u32 idx;
@@ -1277,7 +1280,8 @@ int v4l2_subdev_link_validate(struct media_link *link)
  }
  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
  
-bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0, unsigned int pad1)
+bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
+			   unsigned int pad1)
  {
  	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
  	struct v4l2_subdev_krouting *routing;
@@ -1308,9 +1312,7 @@ bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0, unsig
  EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
  
  struct v4l2_subdev_state *
-__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
-			  enum v4l2_subdev_format_whence which,
-			  const char *lock_name,
+__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
  			  struct lock_class_key *lock_key)
  {
  	struct v4l2_subdev_state *state;
@@ -1322,8 +1324,6 @@ __v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
  
  	__mutex_init(&state->lock, lock_name, lock_key);
  
-	state->which = which;
-
  	/* Drivers that support streams do not need the legacy pad config */
  	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
  		state->pads = kvmalloc_array(sd->entity.num_pads,
@@ -1349,9 +1349,9 @@ __v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
  
  	return ERR_PTR(ret);
  }
-EXPORT_SYMBOL_GPL(__v4l2_alloc_subdev_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_alloc);
  
-void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
  {
  	if (!state)
  		return;
@@ -1363,7 +1363,7 @@ void v4l2_free_subdev_state(struct v4l2_subdev_state *state)
  	kvfree(state->pads);
  	kfree(state);
  }
-EXPORT_SYMBOL_GPL(v4l2_free_subdev_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_state_free);
  
  #endif /* CONFIG_MEDIA_CONTROLLER */
  
@@ -1394,12 +1394,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
  }
  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
  
-int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
-			      struct lock_class_key *key)
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key)
  {
  	struct v4l2_subdev_state *state;
  
-	state = __v4l2_alloc_subdev_state(sd, V4L2_SUBDEV_FORMAT_ACTIVE, name, key);
+	state = __v4l2_subdev_state_alloc(sd, name, key);
  	if (IS_ERR(state))
  		return PTR_ERR(state);
  
@@ -1407,14 +1407,14 @@ int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
  
  	return 0;
  }
-EXPORT_SYMBOL_GPL(__v4l2_subdev_alloc_state);
+EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
  
-void v4l2_subdev_free_state(struct v4l2_subdev *sd)
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
  {
-	v4l2_free_subdev_state(sd->state);
+	__v4l2_subdev_state_free(sd->state);
  	sd->state = NULL;
  }
-EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
+EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
  
  struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
  {
@@ -1449,18 +1449,15 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
  	dst->routes = NULL;
  	dst->num_routes = 0;
  
-	if (src->num_routes == 0) {
-		dst->which = src->which;
-	} else {
-		dst->routes = kvmalloc_array(src->num_routes, sizeof(*src->routes),
-					     GFP_KERNEL);
+	if (src->num_routes > 0) {
+		dst->routes = kvmalloc_array(src->num_routes,
+					     sizeof(*src->routes), GFP_KERNEL);
  		if (!dst->routes)
  			return -ENOMEM;
  
  		memcpy(dst->routes, src->routes,
  		       src->num_routes * sizeof(*src->routes));
  		dst->num_routes = src->num_routes;
-		dst->which = src->which;
  	}
  
  	return v4l2_init_stream_configs(&state->stream_configs, dst);
diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
index 691f5e04b0a1..3516735f0cad 100644
--- a/drivers/staging/media/tegra-video/vi.c
+++ b/drivers/staging/media/tegra-video/vi.c
@@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
  				      struct v4l2_pix_format *pix)
  {
  	const struct tegra_video_format *fmtinfo;
+	static struct lock_class_key key;
  	struct v4l2_subdev *subdev;
  	struct v4l2_subdev_format fmt;
  	struct v4l2_subdev_state *sd_state;
@@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
  	if (!subdev)
  		return -ENODEV;
  
-	sd_state = v4l2_alloc_subdev_state(subdev, V4L2_SUBDEV_FORMAT_ACTIVE);
+	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
+					     &key);
  	if (IS_ERR(sd_state))
  		return PTR_ERR(sd_state);
  	/*
@@ -558,7 +560,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
  	v4l2_fill_pix_format(pix, &fmt.format);
  	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
  
-	v4l2_free_subdev_state(sd_state);
+	__v4l2_subdev_state_free(sd_state);
  
  	return 0;
  }
diff --git a/include/media/media-entity.h b/include/media/media-entity.h
index cd1750e495df..f90cc6417d30 100644
--- a/include/media/media-entity.h
+++ b/include/media/media-entity.h
@@ -933,7 +933,8 @@ struct media_pad *__media_entity_next_routed_pad(struct media_pad *root,
  						 struct media_pad *iter);
  
  /**
- * media_entity_for_each_routed_pad - Iterate over entity pads connected by routes
+ * media_entity_for_each_routed_pad - Iterate over entity pads connected by
+ *				      routes
   *
   * @root: The root pad to which the iterated pads have a route
   * @iter: The iterator pad
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index 476d797ecc68..5e50f2ded653 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -728,24 +728,20 @@ struct v4l2_subdev_stream_configs {
  /**
   * struct v4l2_subdev_krouting - subdev routing table
   *
- * @which: format type (from enum v4l2_subdev_format_whence)
- * @routes: &struct v4l2_subdev_route
   * @num_routes: number of routes
+ * @routes: &struct v4l2_subdev_route
   *
- * This structure is used to translate arguments received from
- * VIDIOC_SUBDEV_G/S_ROUTING() ioctl to subdev device drivers operations.
+ * This structure contains the routing table for a subdev.
   */
  struct v4l2_subdev_krouting {
-	u32 which;
-	struct v4l2_subdev_route *routes;
  	unsigned int num_routes;
+	struct v4l2_subdev_route *routes;
  };
  
  /**
   * struct v4l2_subdev_state - Used for storing subdev state information.
   *
   * @lock: mutex for the state
- * @which: state type (from enum v4l2_subdev_format_whence)
   * @pads: &struct v4l2_subdev_pad_config array
   * @routing: routing table for the subdev
   * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
@@ -756,7 +752,6 @@ struct v4l2_subdev_krouting {
   */
  struct v4l2_subdev_state {
  	struct mutex lock;
-	u32 which;
  	struct v4l2_subdev_pad_config *pads;
  	struct v4l2_subdev_krouting routing;
  	struct v4l2_subdev_stream_configs stream_configs;
@@ -872,6 +867,7 @@ struct v4l2_subdev_pad_ops {
  			       struct v4l2_mbus_config *config);
  	int (*set_routing)(struct v4l2_subdev *sd,
  			   struct v4l2_subdev_state *state,
+			   enum v4l2_subdev_format_whence which,
  			   struct v4l2_subdev_krouting *route);
  };
  
@@ -950,7 +946,12 @@ struct v4l2_subdev_internal_ops {
  /*
   * Set this flag if this subdev supports multiplexed streams. This means
   * that the driver supports routing and handles the stream parameter in its
- * v4l2_subdev_pad_ops handlers.
+ * v4l2_subdev_pad_ops handlers. More specifically, this means:
+ *
+ * - Centrally managed active state is enabled
+ * - Legacy pad config is _not_ supported (state->pads)
+ * - Routing ioctls are available
+ * - Multiple streams per pad are supported
   */
  #define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
  
@@ -1008,7 +1009,7 @@ struct v4l2_subdev_platform_data {
   *		     device using v4l2_async_register_subdev_sensor().
   * @pdata: common part of subdevice platform data
   * @state: active state for the subdev (NULL for subdevs tracking the state
- * 	   internally)
+ *	   internally)
   *
   * Each instance of a subdev driver should create this struct, either
   * stand-alone or embedded in a larger struct.
@@ -1040,6 +1041,18 @@ struct v4l2_subdev {
  	struct v4l2_async_notifier *notifier;
  	struct v4l2_async_notifier *subdev_notifier;
  	struct v4l2_subdev_platform_data *pdata;
+
+	/*
+	 * The fields below are private, and should only be accessed via
+	 * appropriate functions.
+	 */
+
+	/*
+	 * TODO: state should most likely be changed from a pointer to an
+	 * embedded field. For the time being it's kept as a pointer to more
+	 * easily catch uses of state in the cases where the driver doesn't
+	 * support it.
+	 */
  	struct v4l2_subdev_state *state;
  };
  
@@ -1263,32 +1276,24 @@ bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
  			   unsigned int pad1);
  
  /**
- * v4l2_alloc_subdev_state - allocate v4l2_subdev_state
+ * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
   *
   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
- * @which: configuration type for the state (from enum v4l2_subdev_format_whence)
+ * @lock_name: name of the state lock
+ * @key: lock_class_key for the lock
   *
- * Must call v4l2_free_subdev_state() when state is no longer needed.
+ * Must call __v4l2_subdev_state_free() when state is no longer needed.
   */
-#define v4l2_alloc_subdev_state(sd, which)                                     \
-	({                                                                     \
-		static struct lock_class_key __key;                            \
-		const char *name = KBUILD_BASENAME                             \
-			":" __stringify(__LINE__) ":sd->state->lock";          \
-		__v4l2_alloc_subdev_state(sd, which, name, &__key);            \
-	})
-
-struct v4l2_subdev_state *
-__v4l2_alloc_subdev_state(struct v4l2_subdev *sd,
-			  enum v4l2_subdev_format_whence which,
-			  const char *lock_name, struct lock_class_key *key);
+struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
+						    const char *lock_name,
+						    struct lock_class_key *key);
  
  /**
- * v4l2_free_subdev_state - free a v4l2_subdev_state
+ * __v4l2_subdev_state_free - free a v4l2_subdev_state
   *
   * @state: v4l2_subdev_state to be freed.
   */
-void v4l2_free_subdev_state(struct v4l2_subdev_state *state);
+void __v4l2_subdev_state_free(struct v4l2_subdev_state *state);
  
  #endif /* CONFIG_MEDIA_CONTROLLER */
  
@@ -1358,33 +1363,36 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
  			      const struct v4l2_event *ev);
  
  /**
- * v4l2_subdev_alloc_state() - Allocate active subdev state for subdevice
- * @sd: The subdev for which the state is allocated
+ * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
+ * @sd: The subdev
   *
- * This will allocate a subdev state and store it to
- * &struct v4l2_subdev->state.
+ * This finalizes the initialization of the subdev, including allocation of
+ * the active state for the subdev.
   *
- * Must call v4l2_subdev_free_state() when the state is no longer needed.
+ * This must be called by the subdev drivers that use the centralized active
+ * state, after the subdev struct has been initialized and
+ * media_entity_pads_init() has been called.
+ *
+ * Must call v4l2_subdev_cleanup() when the subdev is being removed.
   */
-#define v4l2_subdev_alloc_state(sd)                                            \
+#define v4l2_subdev_init_finalize(sd)                                          \
  	({                                                                     \
  		static struct lock_class_key __key;                            \
  		const char *name = KBUILD_BASENAME                             \
-			":" __stringify(__LINE__) ":sd->state->lock";          \
-		__v4l2_subdev_alloc_state(sd, name, &__key);                   \
+			":" __stringify(__LINE__) ":subdev->state->lock";      \
+		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
  	})
  
-int __v4l2_subdev_alloc_state(struct v4l2_subdev *sd, const char *name,
-			      struct lock_class_key *key);
+int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
+				struct lock_class_key *key);
  
  /**
- * v4l2_subdev_free_state() - Free the active subdev state for subdevice
+ * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
   * @sd: The subdevice
   *
- * This will free the subdev's state and set
- * &struct v4l2_subdev->state to NULL.
+ * This will release the resources allocated in v4l2_subdev_init_finalize.
   */
-void v4l2_subdev_free_state(struct v4l2_subdev *sd);
+void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
  
  /**
   * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
@@ -1400,7 +1408,8 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
  }
  
  /**
- * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for subdevice
+ * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
+ *				     subdevice
   * @sd: The subdevice
   *
   * Return the locked active state for the subdevice, or NULL if the subdev
@@ -1429,34 +1438,26 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
  
  /**
- * v4l2_subdev_validate_state() - Gets the TRY or ACTIVE subdev state
+ * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev
+ *					   state
   * @sd: subdevice
   * @state: subdevice state as passed to the subdev op
   *
- * Subdev ops used to be sometimes called with NULL as the state for ACTIVE
- * case. Even if the v4l2 core now passes proper state for both TRY and
- * ACTIVE cases, a subdev driver may call an op in another subdev driver,
- * passing NULL.
+ * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
+ * NULL as the state parameter, as subdevs always used to have their active
+ * state stored privately.
   *
- * This function can be used as a helper to get the state also for the ACTIVE
- * case. The subdev driver that supports ACTIVE state can use this function
- * as the first thing in its ops, ensuring that the state variable contains
- * either the TRY or ACTIVE state.
- */
-static inline struct v4l2_subdev_state *
-v4l2_subdev_validate_state(struct v4l2_subdev *sd,
-			   struct v4l2_subdev_state *state)
-{
-	return state ? state : sd->state;
-}
-
-/**
- * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev state
- * @sd: subdevice
- * @state: subdevice state as passed to the subdev op
+ * However, newer state-aware subdev drivers, which store their active state in
+ * a common place, subdev->state, expect to always get a proper state as a
+ * parameter.
+ *
+ * These state-aware drivers can use v4l2_subdev_validate_and_lock_state()
+ * instead of v4l2_subdev_lock_state(). v4l2_subdev_validate_and_lock_state()
+ * solves the issue by using subdev->state in case the passed state is
+ * NULL.
   *
- * This is a helper function which does the same as v4l2_subdev_validate_state
- * () except that it also locks the state.
+ * This is a temporary helper function, and should be removed when we can ensure
+ * that all drivers pass proper state when calling other subdevs.
   */
  static inline struct v4l2_subdev_state *
  v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
@@ -1486,7 +1487,8 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
  			    struct v4l2_subdev_krouting *routing);
  
  /**
- * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev state
+ * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
+ *					state
   * @sd: The subdevice
   * @state: The subdevice state
   * @routing: Routing that will be copied to subdev state
@@ -1532,7 +1534,8 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
  				 u32 stream, u32 *other_pad, u32 *other_stream);
  
  /**
- * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream format
+ * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
+ *					     format
   * @state: subdevice state
   * @pad: pad id
   * @stream: stream id
@@ -1564,7 +1567,7 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
  
  /**
   * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
- * 				  1-to-1 streams
+ *				  1-to-1 streams
   * @routing: routing to verify
   *
   * This verifies that the given routing contains only non-overlapping 1-to-1
diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
index d00ef85db1c7..f61ace2b5755 100644
--- a/include/uapi/linux/v4l2-subdev.h
+++ b/include/uapi/linux/v4l2-subdev.h
@@ -206,13 +206,13 @@ struct v4l2_subdev_capability {
   * Is the route active? An active route will start when streaming is enabled
   * on a video node.
   */
-#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		BIT(0)
+#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
  
  /*
   * Is the route immutable, i.e. can it be activated and inactivated?
   * Set by the driver.
   */
-#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		BIT(1)
+#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
  
  /*
   * Is the route a source endpoint? A source endpoint route refers to a stream
@@ -221,7 +221,7 @@ struct v4l2_subdev_capability {
   * fields are unused.
   * Set by the driver.
   */
-#define V4L2_SUBDEV_ROUTE_FL_SOURCE		BIT(2)
+#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
  
  /**
   * struct v4l2_subdev_route - A route inside a subdev

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 05/10/2021 11:57, Tomi Valkeinen wrote:

> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..5ba409db47ff 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>   /* The v4l2 sub-device video device node is registered in read-only mode. */
>   #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>   
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)

These are still not correct. They should be unsigned. But I realized we 
have _BITUL() for this purpose, so I'll change the uapi headers to use 
that macro.

  Tomi

Re: [PATCH v9 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Oct 05, 2021 at 11:57:50AM +0300, Tomi Valkeinen wrote:
> Add a helper for verifying routing for the common case of
> non-overlapping 1-to-1 streams.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 14 ++++++++++++++
>  2 files changed, 38 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 22a9bea0fa85..2a64ff003e4b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1569,3 +1569,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
> +
> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
> +{
> +	unsigned int i, j;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		const struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		for (j = i + 1; j < routing->num_routes; ++j) {
> +			const struct v4l2_subdev_route *r = &routing->routes[j];
> +
> +			if (route->sink_pad == r->sink_pad &&
> +			    route->sink_stream == r->sink_stream)
> +				return -EINVAL;
> +
> +			if (route->source_pad == r->source_pad &&
> +			    route->source_stream == r->source_stream)
> +				return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1a4df0aafe8a..5e50f2ded653 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1565,4 +1565,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
>  int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  			struct v4l2_subdev_format *format);
>  
> +/**
> + * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
> + *				  1-to-1 streams

Let's reflect this in the function name then, we can call it
v4l2_routing_verify_1_to_1(). This will make it possible to add a
v4l2_subdev_routing_validate_1_to_n() later.

I'd rename verify to validate though, and add subdev to the function
name, calling it v4l2_subdev_routing_validate_1_to_1().

> + * @routing: routing to verify
> + *
> + * This verifies that the given routing contains only non-overlapping 1-to-1
> + * streams. In other words, no two streams have the same source or sink
> + * stream ID on a single pad. This is the most common case of routing
> + * supported by devices.
> + *
> + * Returns 0 on success, error value otherwise.
> + */
> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Oct 05, 2021 at 11:57:41AM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 ++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 178 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>  
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>  
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>  
>  	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>  
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 52309a299b03..b4e1f8772d96 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>  	}
>  
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -710,6 +716,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;

Do we really want to condition the routing API to multiplexed streams
support ? Internal subdev routing is useful for devices that don't
support multiplexed streams, and forcing a switch to the multiplexed
streams API for those concerns me. In particular, with the multiplexed
streams API, pad formats only exist once routes are created. This is a
major change compared to the existing API (and it led to the creation of
the V4L2_SUBDEV_ROUTE_FL_SOURCE flag, which I'm still not entirely sure
I like) and I don't have a good enough view of all its side effects yet.

This being said, if conditioning formats availability on pads to the
existence of routes causes issues for such devices, maybe they would be
better solved globally, as they could affect devices supporting
multiplexed streams too.

> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -970,6 +1044,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  
>  	mutex_destroy(&state->lock);
>  
> +	kvfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 7f9c83fac020..f318670a49c8 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,11 +694,25 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
>  struct v4l2_subdev_state {
>  	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>  
>  /**
> @@ -770,6 +785,9 @@ struct v4l2_subdev_state {
>   *		     applied to the hardware. The operation shall fail if the
>   *		     pad index it has been called on is not valid or in case of
>   *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
>  			       struct v4l2_mbus_config *config);
>  	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);
>  };
>  
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..5ba409db47ff 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device video device node is registered in read-only mode. */
>  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>  
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u64 routes;
> +	__u32 num_routes;
> +	__u32 reserved[5];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>  
> @@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Oct 05, 2021 at 11:57:43AM +0300, Tomi Valkeinen wrote:
> Add a helper function to set the subdev routing. The helper can be used
> from subdev driver's set_routing op to store the routing table.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 +++++++++++++++
>  2 files changed, 44 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 14b8282fe45b..af53f827ec27 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>  	mutex_unlock(&state->lock);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> +
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing)
> +{
> +	struct v4l2_subdev_krouting *dst = &state->routing;
> +	const struct v4l2_subdev_krouting *src = routing;
> +
> +	lockdep_assert_held(&state->lock);

Calling this function doesn't make much sense if the subdev doesn't have
the V4L2_SUBDEV_FL_MULTIPLEXED set, right ? If that's correct, should
this be documented below, and/or possibly checked here ?

	if (WARN_ON(!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)))
		return -E???;

> +
> +	kvfree(dst->routes);
> +	dst->routes = NULL;
> +	dst->num_routes = 0;
> +
> +	if (src->num_routes > 0) {
> +		dst->routes = kvmalloc_array(src->num_routes,
> +					     sizeof(*src->routes), GFP_KERNEL);

How many routes do we typically expect, is it worth a vmalloc ? If not,
kmemdup() could be a candidate (as I think by the time we get here,
num_routes should have been validated to not overflow, but please tell
me if I got this part wrong).

> +		if (!dst->routes)
> +			return -ENOMEM;
> +
> +		memcpy(dst->routes, src->routes,
> +		       src->num_routes * sizeof(*src->routes));
> +		dst->num_routes = src->num_routes;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 8b2cf3190276..b6c5cd00831e 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1437,4 +1437,20 @@ v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
>  	return state;
>  }
>  
> +/**
> + * v4l2_subdev_set_routing() - Set given routing to subdev state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + *
> + * This will release old routing table (if any) from the state, allocate
> + * enough space for the given routing, and copy the routing.
> + *
> + * This can be used from the subdev driver's set_routing op, after validating
> + * the routing.
> + */
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Laurent Pinchart]

Hi Tomi,

Thank you for the patch.

On Tue, Oct 05, 2021 at 11:57:47AM +0300, Tomi Valkeinen wrote:
> Add two helper functions to make dealing with streams easier:
> 
> v4l2_state_find_opposite_end - given a routing table and a pad + stream,
> return the pad + stream on the opposite side of the subdev.
> 
> v4l2_state_get_opposite_stream_format - return a pointer to the format
> on the pad + stream on the opposite side from the given pad + stream.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
>  2 files changed, 74 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 37e2e1f907fc..9eeadad997c8 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>  	return NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
> +
> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> +				 u32 stream, u32 *other_pad, u32 *other_stream)

This function should take a state pointer given its name. I would also
rename it to v4l2_subdev_state_find_opposite_end(). Same for
v4l2_state_get_opposite_stream_format() which should be
v4l2_subdev_state_get_opposite_stream_format().

> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (route->source_pad == pad &&
> +		    route->source_stream == stream) {
> +			*other_pad = route->sink_pad;
> +			*other_stream = route->sink_stream;

Can we support other_stream being NULL ? When the subdev implements the
routing API without multiplexed streams, the other_stream number will
always be 0 and it would be nice if the caller didn't have to declare a
placeholder variable.

There are less use cases for other_pad being NULL, but maybe we could
also allow that for consistency reasons ? Up to you.

> +			return 0;
> +		}
> +
> +		if (route->sink_pad == pad && route->sink_stream == stream) {
> +			*other_pad = route->source_pad;
> +			*other_stream = route->source_stream;
> +			return 0;
> +		}
> +	}
> +
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
> +
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> +				      u32 stream)
> +{
> +	u32 other_pad, other_stream;
> +	int ret;
> +
> +	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
> +					   &other_pad, &other_stream);
> +	if (ret)
> +		return NULL;
> +
> +	return v4l2_state_get_stream_format(state, other_pad, other_stream);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1dc824416c1b..4b9520410783 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1501,4 +1501,36 @@ struct v4l2_mbus_framefmt *
>  v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>  			     u32 stream);
>  
> +/**
> + * v4l2_state_find_opposite_end() - Find the opposite stream
> + * @routing: routing used to find the opposite side
> + * @pad: pad id
> + * @stream: stream id
> + * @other_pad: pointer used to return the opposite pad
> + * @other_stream: pointer used to return the opposite stream
> + *
> + * This function uses the routing table to find the pad + stream which is
> + * opposite the given pad + stream.
> + *
> + * Returns 0 on success, or -EINVAL if no matching route is found.
> + */
> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> +				 u32 stream, u32 *other_pad, u32 *other_stream);
> +
> +/**
> + * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
> + *					     format
> + * @state: subdevice state
> + * @pad: pad id
> + * @stream: stream id
> + *
> + * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
> + * that is opposite the given pad + stream in the subdev state.
> + *
> + * If the state does not contain the given pad + stream, NULL is returned.
> + */
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> +				      u32 stream);
> +
>  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Laurent Pinchart]

On Sun, Oct 10, 2021 at 05:15:15PM +0300, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Oct 05, 2021 at 11:57:47AM +0300, Tomi Valkeinen wrote:
> > Add two helper functions to make dealing with streams easier:
> > 
> > v4l2_state_find_opposite_end - given a routing table and a pad + stream,
> > return the pad + stream on the opposite side of the subdev.
> > 
> > v4l2_state_get_opposite_stream_format - return a pointer to the format
> > on the pad + stream on the opposite side from the given pad + stream.
> > 
> > Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> > ---
> >  drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
> >  include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
> >  2 files changed, 74 insertions(+)
> > 
> > diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> > index 37e2e1f907fc..9eeadad997c8 100644
> > --- a/drivers/media/v4l2-core/v4l2-subdev.c
> > +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> > @@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
> >  	return NULL;
> >  }
> >  EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
> > +
> > +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> > +				 u32 stream, u32 *other_pad, u32 *other_stream)
> 
> This function should take a state pointer given its name. I would also
> rename it to v4l2_subdev_state_find_opposite_end(). Same for
> v4l2_state_get_opposite_stream_format() which should be
> v4l2_subdev_state_get_opposite_stream_format().

Also, the state pointer passed to this function should be const.

> > +{
> > +	unsigned int i;
> > +
> > +	for (i = 0; i < routing->num_routes; ++i) {
> > +		struct v4l2_subdev_route *route = &routing->routes[i];
> > +
> > +		if (route->source_pad == pad &&
> > +		    route->source_stream == stream) {
> > +			*other_pad = route->sink_pad;
> > +			*other_stream = route->sink_stream;
> 
> Can we support other_stream being NULL ? When the subdev implements the
> routing API without multiplexed streams, the other_stream number will
> always be 0 and it would be nice if the caller didn't have to declare a
> placeholder variable.
> 
> There are less use cases for other_pad being NULL, but maybe we could
> also allow that for consistency reasons ? Up to you.
> 
> > +			return 0;
> > +		}
> > +
> > +		if (route->sink_pad == pad && route->sink_stream == stream) {
> > +			*other_pad = route->source_pad;
> > +			*other_stream = route->source_stream;
> > +			return 0;
> > +		}
> > +	}
> > +
> > +	return -EINVAL;
> > +}
> > +EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
> > +
> > +struct v4l2_mbus_framefmt *
> > +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> > +				      u32 stream)
> > +{
> > +	u32 other_pad, other_stream;
> > +	int ret;
> > +
> > +	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
> > +					   &other_pad, &other_stream);
> > +	if (ret)
> > +		return NULL;
> > +
> > +	return v4l2_state_get_stream_format(state, other_pad, other_stream);
> > +}
> > +EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
> > diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> > index 1dc824416c1b..4b9520410783 100644
> > --- a/include/media/v4l2-subdev.h
> > +++ b/include/media/v4l2-subdev.h
> > @@ -1501,4 +1501,36 @@ struct v4l2_mbus_framefmt *
> >  v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
> >  			     u32 stream);
> >  
> > +/**
> > + * v4l2_state_find_opposite_end() - Find the opposite stream
> > + * @routing: routing used to find the opposite side
> > + * @pad: pad id
> > + * @stream: stream id
> > + * @other_pad: pointer used to return the opposite pad
> > + * @other_stream: pointer used to return the opposite stream
> > + *
> > + * This function uses the routing table to find the pad + stream which is
> > + * opposite the given pad + stream.
> > + *
> > + * Returns 0 on success, or -EINVAL if no matching route is found.
> > + */
> > +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> > +				 u32 stream, u32 *other_pad, u32 *other_stream);
> > +
> > +/**
> > + * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
> > + *					     format
> > + * @state: subdevice state
> > + * @pad: pad id
> > + * @stream: stream id
> > + *
> > + * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
> > + * that is opposite the given pad + stream in the subdev state.
> > + *
> > + * If the state does not contain the given pad + stream, NULL is returned.
> > + */
> > +struct v4l2_mbus_framefmt *
> > +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> > +				      u32 stream);
> > +
> >  #endif

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Tomi Valkeinen]

On 10/10/2021 04:23, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Oct 05, 2021 at 11:57:50AM +0300, Tomi Valkeinen wrote:
>> Add a helper for verifying routing for the common case of
>> non-overlapping 1-to-1 streams.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 14 ++++++++++++++
>>   2 files changed, 38 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 22a9bea0fa85..2a64ff003e4b 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1569,3 +1569,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   	return 0;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
>> +
>> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
>> +{
>> +	unsigned int i, j;
>> +
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		const struct v4l2_subdev_route *route = &routing->routes[i];
>> +
>> +		for (j = i + 1; j < routing->num_routes; ++j) {
>> +			const struct v4l2_subdev_route *r = &routing->routes[j];
>> +
>> +			if (route->sink_pad == r->sink_pad &&
>> +			    route->sink_stream == r->sink_stream)
>> +				return -EINVAL;
>> +
>> +			if (route->source_pad == r->source_pad &&
>> +			    route->source_stream == r->source_stream)
>> +				return -EINVAL;
>> +		}
>> +	}
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index 1a4df0aafe8a..5e50f2ded653 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -1565,4 +1565,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
>>   int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>>   			struct v4l2_subdev_format *format);
>>   
>> +/**
>> + * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
>> + *				  1-to-1 streams
> 
> Let's reflect this in the function name then, we can call it
> v4l2_routing_verify_1_to_1(). This will make it possible to add a
> v4l2_subdev_routing_validate_1_to_n() later.

Sounds fine to me.

> I'd rename verify to validate though, and add subdev to the function
> name, calling it v4l2_subdev_routing_validate_1_to_1().

I don't mind changing the name, but what's the difference with verify 
and validate?

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

On 10/10/2021 10:36, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Oct 05, 2021 at 11:57:41AM +0300, Tomi Valkeinen wrote:
>> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>>
>> Add support for subdev internal routing. A route is defined as a single
>> stream from a sink pad to a source pad.
>>
>> The userspace can configure the routing via two new ioctls,
>> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
>> implement the functionality with v4l2_subdev_pad_ops.set_routing().
>>
>> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
>>
>> - Add sink and source streams for multiplexed links
>> - Copy the argument back in case of an error. This is needed to let the
>>    caller know the number of routes.
>>
>> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>>
>> - Expand and refine documentation.
>> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>>    version of the ioctl is not required.
>> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
>>
>> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>>
>> - Fix typecasing warnings
>> - Check sink & source pad types
>> - Add 'which' field
>> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
>> - Routing to subdev state
>> - Dropped get_routing subdev op
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>>   drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 22 ++++++++
>>   include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>>   4 files changed, 178 insertions(+), 1 deletion(-)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
>> index 05d5db3d85e5..8e9315ffcb99 100644
>> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
>> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
>> @@ -16,6 +16,7 @@
>>   #include <linux/kernel.h>
>>   #include <linux/version.h>
>>   
>> +#include <linux/v4l2-subdev.h>
>>   #include <linux/videodev2.h>
>>   
>>   #include <media/v4l2-common.h>
>> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>>   		ret = 1;
>>   		break;
>>   	}
>> +
>> +	case VIDIOC_SUBDEV_G_ROUTING:
>> +	case VIDIOC_SUBDEV_S_ROUTING: {
>> +		struct v4l2_subdev_routing *routing = parg;
>> +
>> +		if (routing->num_routes > 256)
>> +			return -EINVAL;
>> +
>> +		*user_ptr = u64_to_user_ptr(routing->routes);
>> +		*kernel_ptr = (void **)&routing->routes;
>> +		*array_size = sizeof(struct v4l2_subdev_route)
>> +			    * routing->num_routes;
>> +		ret = 1;
>> +		break;
>> +	}
>>   	}
>>   
>>   	return ret;
>> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>>   	/*
>>   	 * Some ioctls can return an error, but still have valid
>>   	 * results that must be returned.
>> +	 *
>> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
>> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
>> +	 * defined here as part of the 'v4l2_ioctls' array. As
>> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
>> +	 * in case of failure, but it is not defined here as part of the
>> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>>   	 */
>> -	if (err < 0 && !always_copy)
>> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>>   		goto out;
>>   
>>   out_array_args:
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 52309a299b03..b4e1f8772d96 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>>   		which = ((struct v4l2_subdev_selection *)arg)->which;
>>   		break;
>>   	}
>> +
>> +	case VIDIOC_SUBDEV_G_ROUTING:
>> +	case VIDIOC_SUBDEV_S_ROUTING: {
>> +		which = ((struct v4l2_subdev_routing *)arg)->which;
>> +		break;
>> +	}
>>   	}
>>   
>>   	return which == V4L2_SUBDEV_FORMAT_TRY ?
>> @@ -710,6 +716,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>>   	case VIDIOC_SUBDEV_QUERYSTD:
>>   		return v4l2_subdev_call(sd, video, querystd, arg);
>>   
>> +	case VIDIOC_SUBDEV_G_ROUTING: {
>> +		struct v4l2_subdev_routing *routing = arg;
>> +		struct v4l2_subdev_krouting *krouting;
>> +
>> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
>> +			return -ENOIOCTLCMD;
> 
> Do we really want to condition the routing API to multiplexed streams
> support ? Internal subdev routing is useful for devices that don't
> support multiplexed streams, and forcing a switch to the multiplexed
> streams API for those concerns me. In particular, with the multiplexed
> streams API, pad formats only exist once routes are created. This is a
> major change compared to the existing API (and it led to the creation of
> the V4L2_SUBDEV_ROUTE_FL_SOURCE flag, which I'm still not entirely sure
> I like) and I don't have a good enough view of all its side effects yet.
> 
> This being said, if conditioning formats availability on pads to the
> existence of routes causes issues for such devices, maybe they would be
> better solved globally, as they could affect devices supporting
> multiplexed streams too.

V4L2_SUBDEV_FL_MULTIPLEXED doesn't exactly mean "the subdev supports 
multiplexed streams". It means "the subdev supports central active 
state, routing and multiplexed streams".

I thought about keeping the meaning, but renaming the flag, but I have 
no good idea for the name.

I thought about splitting the flag into separate parts (those three 
above), but the features are kind of interleaved: 1) both routing and 
streams depend on the new state, 2) streams depends on routing, 3) 
routing depends on streams.

Those dependencies exist currently in the code. With some effort I 
believe some or all of the dependencies could be removed.

But I would have to ask, why? What's the use case and benefit?

"Multiplexed streams support in a subdev" doesn't mean it has to have 
multiple streams. In fact, I believe a subdev that supports just a 
single stream is easier to implement with the multiplexed streams API 
than with the older pad-based API.

If you add routing support to a subdev, you anyway have to rethink how 
it works and restructure the code, why not clean it up a bit more with 
the multiplexed streams API at the same time?

> with the multiplexed streams API, pad formats only exist once routes are created

This is true, but the driver can force the routes to always exist. It 
can do this in the case the device does not support multiple streams. 
The route can be disabled, which should allow setting the formats but 
still keeping the route inactive. And it should be fine to have e.g. a 
single sink pad and 2 source pads, with routes from both source pads to 
the same sink pad, as long as the other one is disabled. But I haven't 
tested this, there might be a bug there somewhere.

I'm not a fan of V4L2_SUBDEV_ROUTE_FL_SOURCE either, but it wasn't 
created for the problem you describe. V4L2_SUBDEV_ROUTE_FL_SOURCE means 
that a subdev has an output stream that isn't routed to an input stream. 
You're talking about a case where you want a (pad,stream) to be 
configurable even if it's not routed anywhere (yet).

I think conceptually the flag is fine (i.e. a subdev with an output 
stream that isn't routed to an input stream). The problem is more about 
the routing table, as that is the place to define "stream configuration 
points". This is something I raised some time back, thinking out loud if 
the routing table should be dropped, and instead (somehow) define these 
configuration points per pad. So a pad would have a userspace defined 
list of (input-stream, other-pad, other-stream) tuples, where "other" 
means the pad/stream on the other side of the subdev. That "other" pad 
should have a matching, but reverse, entry.

It would be a bit cumbersome, though, I think. The routing table is an 
easier way to manage this, with V4L2_SUBDEV_ROUTE_FL_SOURCE flag. But if 
needed, we could look further into separating the routing and the 
management of the "config points".

  Tomi

Re: [PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

On 10/10/2021 13:21, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Oct 05, 2021 at 11:57:43AM +0300, Tomi Valkeinen wrote:
>> Add a helper function to set the subdev routing. The helper can be used
>> from subdev driver's set_routing op to store the routing table.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 16 +++++++++++++++
>>   2 files changed, 44 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 14b8282fe45b..af53f827ec27 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>>   	mutex_unlock(&state->lock);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
>> +
>> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>> +			    struct v4l2_subdev_state *state,
>> +			    struct v4l2_subdev_krouting *routing)
>> +{
>> +	struct v4l2_subdev_krouting *dst = &state->routing;
>> +	const struct v4l2_subdev_krouting *src = routing;
>> +
>> +	lockdep_assert_held(&state->lock);
> 
> Calling this function doesn't make much sense if the subdev doesn't have
> the V4L2_SUBDEV_FL_MULTIPLEXED set, right ? If that's correct, should
> this be documented below, and/or possibly checked here ?
> 
> 	if (WARN_ON(!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)))
> 		return -E???;

That is true, but it's true for all the functions introduced in the 
series. Do we want to add the check everywhere? In many cases this 
should be quite obvious: if you haven't ported the subdev driver to the 
new API, you don't have routing table, and can't sensibly call this 
function. Or you don't have stream ID, so you can't sensibly call 
functions that take a stream number.

>> +
>> +	kvfree(dst->routes);
>> +	dst->routes = NULL;
>> +	dst->num_routes = 0;
>> +
>> +	if (src->num_routes > 0) {
>> +		dst->routes = kvmalloc_array(src->num_routes,
>> +					     sizeof(*src->routes), GFP_KERNEL);
> 
> How many routes do we typically expect, is it worth a vmalloc ? If not,

I have to say I have no clue if it's worth it. vmalloc is used for the 
pads, and was used in the original routing patches. I did wonder about 
this but just went with what's already there.

It's not easy to guess the expected number of routes. I'd say... 4 pads 
on each side and 8 streams per pad-pair sounds kind of a lot but still 
sounds feasible. That would be 32 routes. But that would be on the 
maximum side. On average, I guess we'll have... 1 or 2 routes? That's 
wha the sensors will have.

> kmemdup() could be a candidate (as I think by the time we get here,
> num_routes should have been validated to not overflow, but please tell
> me if I got this part wrong).

I'm not sure what you mean here. We are allocating (enough) new memory, 
it will never overflow.

>> +		if (!dst->routes)
>> +			return -ENOMEM;
>> +
>> +		memcpy(dst->routes, src->routes,
>> +		       src->num_routes * sizeof(*src->routes));
>> +		dst->num_routes = src->num_routes;
>> +	}
>> +
>> +	return 0;

  Tomi

Re: [PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Tomi Valkeinen]

On 10/10/2021 17:15, Laurent Pinchart wrote:
> Hi Tomi,
> 
> Thank you for the patch.
> 
> On Tue, Oct 05, 2021 at 11:57:47AM +0300, Tomi Valkeinen wrote:
>> Add two helper functions to make dealing with streams easier:
>>
>> v4l2_state_find_opposite_end - given a routing table and a pad + stream,
>> return the pad + stream on the opposite side of the subdev.
>>
>> v4l2_state_get_opposite_stream_format - return a pointer to the format
>> on the pad + stream on the opposite side from the given pad + stream.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
>>   include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
>>   2 files changed, 74 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index 37e2e1f907fc..9eeadad997c8 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>>   	return NULL;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
>> +
>> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
>> +				 u32 stream, u32 *other_pad, u32 *other_stream)
> 
> This function should take a state pointer given its name. I would also
> rename it to v4l2_subdev_state_find_opposite_end(). Same for
> v4l2_state_get_opposite_stream_format() which should be
> v4l2_subdev_state_get_opposite_stream_format().

It doesn't need the state. I think it's better to rename this to 
v4l2_routing_find_opposite_end(). Or 
v4l2_subdev_routing_find_opposite_end() if you want the "subdev" to be 
there.

>> +{
>> +	unsigned int i;
>> +
>> +	for (i = 0; i < routing->num_routes; ++i) {
>> +		struct v4l2_subdev_route *route = &routing->routes[i];
>> +
>> +		if (route->source_pad == pad &&
>> +		    route->source_stream == stream) {
>> +			*other_pad = route->sink_pad;
>> +			*other_stream = route->sink_stream;
> 
> Can we support other_stream being NULL ? When the subdev implements the
> routing API without multiplexed streams, the other_stream number will
> always be 0 and it would be nice if the caller didn't have to declare a
> placeholder variable.
> 
> There are less use cases for other_pad being NULL, but maybe we could
> also allow that for consistency reasons ? Up to you.

Sure. But are you sure the ignored stream is 0? Should the function 
verify that the stream is indeed 0 if other_stream is NULL?

For the pad, I don't know when it would be used, so it's perhaps better 
not to add that (as we can't have similar validation as for the stream).

  Tomi

Re: [PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Laurent Pinchart]

Hi Tomi,

On Mon, Oct 11, 2021 at 11:21:14AM +0300, Tomi Valkeinen wrote:
> On 10/10/2021 17:15, Laurent Pinchart wrote:
> > On Tue, Oct 05, 2021 at 11:57:47AM +0300, Tomi Valkeinen wrote:
> >> Add two helper functions to make dealing with streams easier:
> >>
> >> v4l2_state_find_opposite_end - given a routing table and a pad + stream,
> >> return the pad + stream on the opposite side of the subdev.
> >>
> >> v4l2_state_get_opposite_stream_format - return a pointer to the format
> >> on the pad + stream on the opposite side from the given pad + stream.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
> >>   include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
> >>   2 files changed, 74 insertions(+)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index 37e2e1f907fc..9eeadad997c8 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
> >>   	return NULL;
> >>   }
> >>   EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
> >> +
> >> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> >> +				 u32 stream, u32 *other_pad, u32 *other_stream)
> > 
> > This function should take a state pointer given its name. I would also
> > rename it to v4l2_subdev_state_find_opposite_end(). Same for
> > v4l2_state_get_opposite_stream_format() which should be
> > v4l2_subdev_state_get_opposite_stream_format().
> 
> It doesn't need the state. I think it's better to rename this to 
> v4l2_routing_find_opposite_end(). Or 
> v4l2_subdev_routing_find_opposite_end() if you want the "subdev" to be 
> there.

Is there any use case for calling this function on a
v4l2_subdev_krouting instance that isn't stored in a v4l2_subdev_state ?
If not, I'd prefer standardizing on operating on the state in all cases.

> >> +{
> >> +	unsigned int i;
> >> +
> >> +	for (i = 0; i < routing->num_routes; ++i) {
> >> +		struct v4l2_subdev_route *route = &routing->routes[i];
> >> +
> >> +		if (route->source_pad == pad &&
> >> +		    route->source_stream == stream) {
> >> +			*other_pad = route->sink_pad;
> >> +			*other_stream = route->sink_stream;
> > 
> > Can we support other_stream being NULL ? When the subdev implements the
> > routing API without multiplexed streams, the other_stream number will
> > always be 0 and it would be nice if the caller didn't have to declare a
> > placeholder variable.
> > 
> > There are less use cases for other_pad being NULL, but maybe we could
> > also allow that for consistency reasons ? Up to you.
> 
> Sure. But are you sure the ignored stream is 0? Should the function 
> verify that the stream is indeed 0 if other_stream is NULL?

I'd expect the routing table to have been validated at .set_routing()
time, with != 0 streams rejected at that point, so ignoring the stream
when calling v4l2_state_find_opposite_end() in drivers that don't
support multiplexed streams should be safe.

> For the pad, I don't know when it would be used, so it's perhaps better 
> not to add that (as we can't have similar validation as for the stream).

OK, we can only handle a NULL other_stream then.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 00/36] v4l: subdev internal routing and streams

[Hans Verkuil]

Hi Tomi,

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Hi,
> 
> This is v9 of the multiplexed streams series. v8 can be found from:
> 
> https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/
> 
> I have pushed my work branch to:
> 
> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v9
> 
> which contains the patches in this series, along with subdev drivers
> using multiplexed streams.
> 
> As can be guessed from the work branch, I have been testing this series
> with TI's FPDLink setup. I have also done a "backwards compatibility"
> test by dropping all multiplexed streams patches from the CAL driver
> (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
> single-stream configuration.

I hope to look at this series this week (fingers crossed), but I was asked to
give some input w.r.t. testing of this series:

I think before this can be merged we need:

1) libcamera tests. Since libcamera would probably be the most active user of this
API, and you have HW for it, it makes a lot of sense that there are decent tests
for the supported HW in libcamera. That takes care of the real-world tests.

2) obviously the existing utils in v4l-utils need to be adapted to understand any
new API elements.

3) compliance tests in v4l2-compliance for the new API. After I did a review of the
series we can see to what extent this is possible.

4) optionally (for now at least, I reserve the right to change my mind): it would
be very helpful if this can be added to vimc (or something similar), allowing for
testing the API without having real hardware, which is useful both for writing
the tests and for running regression tests regularly on a simple VM, without needing
special hardware.

Regards,

	Hans

> 
> The changes since v8:
> - Fixed the use of BIT() in uAPI header.
> - Documentation updates.
> - Added documentation about multiplexed streams to dev-subdev.rst (I had
>   this for v8, but missed it from the series...).
> - 'which' has been dropped from the subdev state. This affects the
>   subdev state allocation (no need to give the try/active parameter) and
>   set_routing() subdev op (need to pass 'which').
> - Dropped v4l2_subdev_validate_state(), as it's not needed.
> - Renamed v4l2_subdev_alloc_state() and v4l2_subdev_free_state() to
>   v4l2_subdev_init_finalize() and v4l2_subdev_cleanup(). I'm not super
>   happy with the names, but I think it's better to have general init &
>   cleanup functions, which can be used for other purposes too.
> - Renamed v4l2_alloc_subdev_state() and v4l2_free_subdev_state() to
>   __v4l2_subdev_state_alloc() and __v4l2_subdev_state_free() to give an
>   indication that they're not supposed to be used from the drivers.
> 
>  Tomi
> 
> Jacopo Mondi (2):
>   media: entity: Add iterator helper for entity pads
>   media: Documentation: Add GS_ROUTING documentation
> 
> Laurent Pinchart (4):
>   media: entity: Add has_route entity operation
>   media: entity: Add media_entity_has_route() function
>   media: entity: Use routing information during graph traversal
>   media: subdev: Add [GS]_ROUTING subdev ioctls and operations
> 
> Sakari Ailus (13):
>   media: entity: Use pad as a starting point for graph walk
>   media: entity: Use pads instead of entities in the media graph walk
>     stack
>   media: entity: Walk the graph based on pads
>   media: mc: Start walk from a specific pad in use count calculation
>   media: entity: Move the pipeline from entity to pads
>   media: entity: Use pad as the starting point for a pipeline
>   media: entity: Skip link validation for pads to which there is no
>     route
>   media: entity: Add an iterator helper for connected pads
>   media: entity: Add only connected pads to the pipeline
>   media: entity: Add debug information in graph walk route check
>   media: Add bus type to frame descriptors
>   media: Add CSI-2 bus configuration to frame descriptors
>   media: Add stream to frame descriptor
> 
> Tomi Valkeinen (17):
>   media: subdev: rename subdev-state alloc & free
>   media: subdev: add active state to struct v4l2_subdev
>   media: subdev: pass also the active state to subdevs from ioctls
>   media: subdev: add subdev state locking
>   media: subdev: Add v4l2_subdev_validate_and_lock_state()
>   media: Documentation: add documentation about subdev state
>   media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8
>   media: add V4L2_SUBDEV_FL_MULTIPLEXED
>   media: subdev: add v4l2_subdev_has_route()
>   media: subdev: add v4l2_subdev_set_routing helper()
>   media: Documentation: add multiplexed streams documentation
>   media: subdev: add stream based configuration
>   media: subdev: use streams in v4l2_subdev_link_validate()
>   media: subdev: add "opposite" stream helper funcs
>   media: subdev: add v4l2_subdev_get_fmt() helper function
>   media: subdev: add v4l2_subdev_set_routing_with_fmt() helper
>   media: subdev: add v4l2_routing_simple_verify() helper
> 
>  Documentation/driver-api/media/mc-core.rst    |  18 +-
>  .../driver-api/media/v4l2-subdev.rst          |  36 +
>  .../userspace-api/media/v4l/dev-subdev.rst    | 166 ++++
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
>  .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
>  .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
>  .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
>  .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++
>  .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
>  drivers/media/mc/mc-device.c                  |  13 +-
>  drivers/media/mc/mc-entity.c                  | 257 +++---
>  drivers/media/pci/intel/ipu3/ipu3-cio2-main.c |   6 +-
>  .../media/platform/exynos4-is/fimc-capture.c  |   8 +-
>  .../platform/exynos4-is/fimc-isp-video.c      |   8 +-
>  drivers/media/platform/exynos4-is/fimc-isp.c  |   2 +-
>  drivers/media/platform/exynos4-is/fimc-lite.c |  10 +-
>  drivers/media/platform/exynos4-is/media-dev.c |  20 +-
>  drivers/media/platform/omap3isp/isp.c         |   2 +-
>  drivers/media/platform/omap3isp/ispvideo.c    |  25 +-
>  drivers/media/platform/omap3isp/ispvideo.h    |   2 +-
>  .../media/platform/qcom/camss/camss-video.c   |   6 +-
>  drivers/media/platform/rcar-vin/rcar-core.c   |  16 +-
>  drivers/media/platform/rcar-vin/rcar-dma.c    |   8 +-
>  drivers/media/platform/rcar-vin/rcar-v4l2.c   |   5 +-
>  .../platform/rockchip/rkisp1/rkisp1-capture.c |   6 +-
>  .../media/platform/s3c-camif/camif-capture.c  |   6 +-
>  drivers/media/platform/stm32/stm32-dcmi.c     |   6 +-
>  .../platform/sunxi/sun4i-csi/sun4i_dma.c      |   6 +-
>  .../platform/sunxi/sun6i-csi/sun6i_video.c    |   6 +-
>  drivers/media/platform/ti-vpe/cal-video.c     |   6 +-
>  drivers/media/platform/vsp1/vsp1_entity.c     |   6 +-
>  drivers/media/platform/vsp1/vsp1_video.c      |  18 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  20 +-
>  drivers/media/platform/xilinx/xilinx-dma.h    |   2 +-
>  .../media/test-drivers/vimc/vimc-capture.c    |   6 +-
>  drivers/media/usb/au0828/au0828-core.c        |   8 +-
>  drivers/media/v4l2-core/v4l2-ioctl.c          |  25 +-
>  drivers/media/v4l2-core/v4l2-mc.c             |  43 +-
>  drivers/media/v4l2-core/v4l2-subdev.c         | 732 +++++++++++++++++-
>  drivers/staging/media/imx/imx-media-utils.c   |   8 +-
>  drivers/staging/media/ipu3/ipu3-v4l2.c        |   6 +-
>  drivers/staging/media/omap4iss/iss.c          |   2 +-
>  drivers/staging/media/omap4iss/iss_video.c    |  40 +-
>  drivers/staging/media/omap4iss/iss_video.h    |   2 +-
>  drivers/staging/media/tegra-video/tegra210.c  |   6 +-
>  drivers/staging/media/tegra-video/vi.c        |   6 +-
>  include/media/media-entity.h                  | 143 +++-
>  include/media/v4l2-subdev.h                   | 374 ++++++++-
>  include/uapi/linux/v4l2-subdev.h              |  85 +-
>  52 files changed, 1989 insertions(+), 369 deletions(-)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> 

Re: [PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Laurent Pinchart]

Hi Tomi,

On Mon, Oct 11, 2021 at 11:13:40AM +0300, Tomi Valkeinen wrote:
> On 10/10/2021 13:21, Laurent Pinchart wrote:
> > On Tue, Oct 05, 2021 at 11:57:43AM +0300, Tomi Valkeinen wrote:
> >> Add a helper function to set the subdev routing. The helper can be used
> >> from subdev driver's set_routing op to store the routing table.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
> >>   include/media/v4l2-subdev.h           | 16 +++++++++++++++
> >>   2 files changed, 44 insertions(+)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index 14b8282fe45b..af53f827ec27 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> >>   	mutex_unlock(&state->lock);
> >>   }
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> >> +
> >> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> >> +			    struct v4l2_subdev_state *state,
> >> +			    struct v4l2_subdev_krouting *routing)
> >> +{
> >> +	struct v4l2_subdev_krouting *dst = &state->routing;
> >> +	const struct v4l2_subdev_krouting *src = routing;
> >> +
> >> +	lockdep_assert_held(&state->lock);
> > 
> > Calling this function doesn't make much sense if the subdev doesn't have
> > the V4L2_SUBDEV_FL_MULTIPLEXED set, right ? If that's correct, should
> > this be documented below, and/or possibly checked here ?
> > 
> > 	if (WARN_ON(!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)))
> > 		return -E???;
> 
> That is true, but it's true for all the functions introduced in the 
> series. Do we want to add the check everywhere? In many cases this 
> should be quite obvious: if you haven't ported the subdev driver to the 
> new API, you don't have routing table, and can't sensibly call this 
> function. Or you don't have stream ID, so you can't sensibly call 
> functions that take a stream number.

That's right. I was mostly concern about driver conversion, it's likely
easy to forget to add V4L2_SUBDEV_FL_MULTIPLEXED and then spend time
debugging issues.

I'm fine either way for the WARN_ON, but would still like to capture
this in the documentation.

> >> +
> >> +	kvfree(dst->routes);
> >> +	dst->routes = NULL;
> >> +	dst->num_routes = 0;
> >> +
> >> +	if (src->num_routes > 0) {
> >> +		dst->routes = kvmalloc_array(src->num_routes,
> >> +					     sizeof(*src->routes), GFP_KERNEL);
> > 
> > How many routes do we typically expect, is it worth a vmalloc ? If not,
> 
> I have to say I have no clue if it's worth it. vmalloc is used for the 
> pads, and was used in the original routing patches. I did wonder about 
> this but just went with what's already there.
> 
> It's not easy to guess the expected number of routes. I'd say... 4 pads 
> on each side and 8 streams per pad-pair sounds kind of a lot but still 
> sounds feasible. That would be 32 routes. But that would be on the 
> maximum side. On average, I guess we'll have... 1 or 2 routes? That's 
> wha the sensors will have.
> 
> > kmemdup() could be a candidate (as I think by the time we get here,
> > num_routes should have been validated to not overflow, but please tell
> > me if I got this part wrong).
> 
> I'm not sure what you mean here. We are allocating (enough) new memory, 
> it will never overflow.

I mean an integer overflow in the size * nelems calculation. The array
allocation functions pay special attention to this.

> >> +		if (!dst->routes)
> >> +			return -ENOMEM;
> >> +
> >> +		memcpy(dst->routes, src->routes,
> >> +		       src->num_routes * sizeof(*src->routes));
> >> +		dst->num_routes = src->num_routes;
> >> +	}
> >> +
> >> +	return 0;

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Laurent Pinchart]

On Mon, Oct 11, 2021 at 10:25:49AM +0300, Tomi Valkeinen wrote:
> On 10/10/2021 04:23, Laurent Pinchart wrote:
> > Hi Tomi,
> > 
> > Thank you for the patch.
> > 
> > On Tue, Oct 05, 2021 at 11:57:50AM +0300, Tomi Valkeinen wrote:
> >> Add a helper for verifying routing for the common case of
> >> non-overlapping 1-to-1 streams.
> >>
> >> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> >> ---
> >>   drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
> >>   include/media/v4l2-subdev.h           | 14 ++++++++++++++
> >>   2 files changed, 38 insertions(+)
> >>
> >> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> >> index 22a9bea0fa85..2a64ff003e4b 100644
> >> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> >> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> >> @@ -1569,3 +1569,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> >>   	return 0;
> >>   }
> >>   EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
> >> +
> >> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
> >> +{
> >> +	unsigned int i, j;
> >> +
> >> +	for (i = 0; i < routing->num_routes; ++i) {
> >> +		const struct v4l2_subdev_route *route = &routing->routes[i];
> >> +
> >> +		for (j = i + 1; j < routing->num_routes; ++j) {
> >> +			const struct v4l2_subdev_route *r = &routing->routes[j];
> >> +
> >> +			if (route->sink_pad == r->sink_pad &&
> >> +			    route->sink_stream == r->sink_stream)
> >> +				return -EINVAL;
> >> +
> >> +			if (route->source_pad == r->source_pad &&
> >> +			    route->source_stream == r->source_stream)
> >> +				return -EINVAL;
> >> +		}
> >> +	}
> >> +
> >> +	return 0;
> >> +}
> >> +EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
> >> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> >> index 1a4df0aafe8a..5e50f2ded653 100644
> >> --- a/include/media/v4l2-subdev.h
> >> +++ b/include/media/v4l2-subdev.h
> >> @@ -1565,4 +1565,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> >>   int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> >>   			struct v4l2_subdev_format *format);
> >>   
> >> +/**
> >> + * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
> >> + *				  1-to-1 streams
> > 
> > Let's reflect this in the function name then, we can call it
> > v4l2_routing_verify_1_to_1(). This will make it possible to add a
> > v4l2_subdev_routing_validate_1_to_n() later.
> 
> Sounds fine to me.
> 
> > I'd rename verify to validate though, and add subdev to the function
> > name, calling it v4l2_subdev_routing_validate_1_to_1().
> 
> I don't mind changing the name, but what's the difference with verify 
> and validate?

verify = to check or prove the truth, validate = to check or prove the
validatity.

-- 
Regards,

Laurent Pinchart

Re: [PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Tomi Valkeinen]

On 12/10/2021 00:06, Laurent Pinchart wrote:
> Hi Tomi,
> 
> On Mon, Oct 11, 2021 at 11:13:40AM +0300, Tomi Valkeinen wrote:
>> On 10/10/2021 13:21, Laurent Pinchart wrote:
>>> On Tue, Oct 05, 2021 at 11:57:43AM +0300, Tomi Valkeinen wrote:
>>>> Add a helper function to set the subdev routing. The helper can be used
>>>> from subdev driver's set_routing op to store the routing table.
>>>>
>>>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>>>> ---
>>>>    drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>>>>    include/media/v4l2-subdev.h           | 16 +++++++++++++++
>>>>    2 files changed, 44 insertions(+)
>>>>
>>>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> index 14b8282fe45b..af53f827ec27 100644
>>>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>>>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>>>> @@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>>>>    	mutex_unlock(&state->lock);
>>>>    }
>>>>    EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
>>>> +
>>>> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>>>> +			    struct v4l2_subdev_state *state,
>>>> +			    struct v4l2_subdev_krouting *routing)
>>>> +{
>>>> +	struct v4l2_subdev_krouting *dst = &state->routing;
>>>> +	const struct v4l2_subdev_krouting *src = routing;
>>>> +
>>>> +	lockdep_assert_held(&state->lock);
>>>
>>> Calling this function doesn't make much sense if the subdev doesn't have
>>> the V4L2_SUBDEV_FL_MULTIPLEXED set, right ? If that's correct, should
>>> this be documented below, and/or possibly checked here ?
>>>
>>> 	if (WARN_ON(!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)))
>>> 		return -E???;
>>
>> That is true, but it's true for all the functions introduced in the
>> series. Do we want to add the check everywhere? In many cases this
>> should be quite obvious: if you haven't ported the subdev driver to the
>> new API, you don't have routing table, and can't sensibly call this
>> function. Or you don't have stream ID, so you can't sensibly call
>> functions that take a stream number.
> 
> That's right. I was mostly concern about driver conversion, it's likely
> easy to forget to add V4L2_SUBDEV_FL_MULTIPLEXED and then spend time
> debugging issues.
> 
> I'm fine either way for the WARN_ON, but would still like to capture
> this in the documentation.

There are perhaps two easy things to miss, adding 
V4L2_SUBDEV_FL_MULTIPLEXED and calling v4l2_subdev_init_finalize(). 
Maybe I can add some checks to catch these two.

>>>> +
>>>> +	kvfree(dst->routes);
>>>> +	dst->routes = NULL;
>>>> +	dst->num_routes = 0;
>>>> +
>>>> +	if (src->num_routes > 0) {
>>>> +		dst->routes = kvmalloc_array(src->num_routes,
>>>> +					     sizeof(*src->routes), GFP_KERNEL);
>>>
>>> How many routes do we typically expect, is it worth a vmalloc ? If not,
>>
>> I have to say I have no clue if it's worth it. vmalloc is used for the
>> pads, and was used in the original routing patches. I did wonder about
>> this but just went with what's already there.
>>
>> It's not easy to guess the expected number of routes. I'd say... 4 pads
>> on each side and 8 streams per pad-pair sounds kind of a lot but still
>> sounds feasible. That would be 32 routes. But that would be on the
>> maximum side. On average, I guess we'll have... 1 or 2 routes? That's
>> wha the sensors will have.
>>
>>> kmemdup() could be a candidate (as I think by the time we get here,
>>> num_routes should have been validated to not overflow, but please tell
>>> me if I got this part wrong).
>>
>> I'm not sure what you mean here. We are allocating (enough) new memory,
>> it will never overflow.
> 
> I mean an integer overflow in the size * nelems calculation. The array
> allocation functions pay special attention to this.

check_array_args() gives an error if there are more than 256 routes. It 
also does a copy of the userspace's routing table to a kvmalloc'd buffer 
(which is the source here), which, in case of overflow, would then 
already be broken. But the limit of 256 feels safe enough even if in the 
future the struct v4l2_subdev_route would grow a bit.

  Tomi

Re: [PATCH v9 01/36] media: subdev: rename subdev-state alloc & free

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> v4l2_subdev_alloc_state() and v4l2_subdev_free_state() are not supposed
> to be used by the drivers. However, we do have a few drivers that use
> those at the moment, so we need to expose these functions for the time
> being.
> 
> Prefix the functions with __ to mark the functions as internal.
> 
> At the same time, rename them to v4l2_subdev_state_alloc and
> v4l2_subdev_state_free to match the style used for other functions like
> video_device_alloc() and media_request_alloc().
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  4 ++--
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 ++--
>  drivers/media/v4l2-core/v4l2-subdev.c       | 12 ++++++------
>  drivers/staging/media/tegra-video/vi.c      |  4 ++--
>  include/media/v4l2-subdev.h                 | 10 +++++-----
>  5 files changed, 17 insertions(+), 17 deletions(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index 0d141155f0e3..ba1d16ab1651 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -252,7 +252,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>  
> -	sd_state = v4l2_subdev_alloc_state(sd);
> +	sd_state = __v4l2_subdev_state_alloc(sd);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  
> @@ -288,7 +288,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  
>  	rvin_format_align(vin, pix);
>  done:
> -	v4l2_subdev_free_state(sd_state);
> +	__v4l2_subdev_state_free(sd_state);
>  
>  	return ret;
>  }
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 823c15facd1b..869cadc1468d 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -675,7 +675,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = v4l2_subdev_alloc_state(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> @@ -690,6 +690,6 @@ void vsp1_entity_destroy(struct vsp1_entity *entity)
>  		entity->ops->destroy(entity);
>  	if (entity->subdev.ctrl_handler)
>  		v4l2_ctrl_handler_free(entity->subdev.ctrl_handler);
> -	v4l2_subdev_free_state(entity->config);
> +	__v4l2_subdev_state_free(entity->config);
>  	media_entity_cleanup(&entity->subdev.entity);
>  }
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 5d27a27cc2f2..fe49c86a9b02 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -28,7 +28,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = v4l2_subdev_alloc_state(sd);
> +	state = __v4l2_subdev_state_alloc(sd);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -39,7 +39,7 @@ static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  
>  static void subdev_fh_free(struct v4l2_subdev_fh *fh)
>  {
> -	v4l2_subdev_free_state(fh->state);
> +	__v4l2_subdev_state_free(fh->state);
>  	fh->state = NULL;
>  }
>  
> @@ -870,7 +870,7 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -903,9 +903,9 @@ struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd)
>  
>  	return ERR_PTR(ret);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_alloc_state);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_state_alloc);
>  
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
> +void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  {
>  	if (!state)
>  		return;
> @@ -913,7 +913,7 @@ void v4l2_subdev_free_state(struct v4l2_subdev_state *state)
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_free_state);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_state_free);
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>  
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index d321790b07d9..66b9ce160472 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -507,7 +507,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>  
> -	sd_state = v4l2_subdev_alloc_state(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> @@ -558,7 +558,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	v4l2_fill_pix_format(pix, &fmt.format);
>  	tegra_channel_fmt_align(chan, pix, fmtinfo->bpp);
>  
> -	v4l2_subdev_free_state(sd_state);
> +	__v4l2_subdev_state_free(sd_state);
>  
>  	return 0;
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 95ec18c2f49c..e52bf508c75b 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1135,20 +1135,20 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>  int v4l2_subdev_link_validate(struct media_link *link);
>  
>  /**
> - * v4l2_subdev_alloc_state - allocate v4l2_subdev_state
> + * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
>   *
> - * Must call v4l2_subdev_free_state() when state is no longer needed.
> + * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *v4l2_subdev_alloc_state(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
>  
>  /**
> - * v4l2_subdev_free_state - free a v4l2_subdev_state
> + * __v4l2_subdev_state_free - free a v4l2_subdev_state
>   *
>   * @state: v4l2_subdev_state to be freed.
>   */
> -void v4l2_subdev_free_state(struct v4l2_subdev_state *state);
> +void __v4l2_subdev_state_free(struct v4l2_subdev_state *state);
>  
>  #endif /* CONFIG_MEDIA_CONTROLLER */
>  
> 

Re: [PATCH v9 03/36] media: subdev: pass also the active state to subdevs from ioctls

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> At the moment when a subdev op is called, the TRY subdev state
> (subdev_fh->state) is passed as a parameter even for ACTIVE case, or

for -> for the

> alternatively a NULL can be passed for ACTIVE case. This used to make
> sense, as the ACTIVE state was handled internally by the subdev drivers.
> 
> We now have a state for ACTIVE case in a standard place, and can pass

for -> for the

> that alto to the drivers. This patch changes the subdev ioctls to either

alto -> also

> pass the TRY or ACTIVE state to the subdev.
> 
> Unfortunately many drivers call ops from other subdevs, and implicitly
> pass NULL as the state, so this is just a partial solution. A coccinelle
> spatch could perhaps be created which fixes the drivers' subdev calls.
> 
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
> 
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.
> 
> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_validate_and_lock_state() helper, introduced in a follow up
> patch.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

With those typos fixed:

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

	Hans

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 73 +++++++++++++++++++++++----
>  1 file changed, 63 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index bcaf66a1e3d9..aa757cc28879 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -353,6 +353,53 @@ const struct v4l2_subdev_ops v4l2_subdev_call_wrappers = {
>  EXPORT_SYMBOL(v4l2_subdev_call_wrappers);
>  
>  #if defined(CONFIG_VIDEO_V4L2_SUBDEV_API)
> +
> +static struct v4l2_subdev_state *
> +subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
> +		       unsigned int cmd, void *arg)
> +{
> +	u32 which;
> +
> +	switch (cmd) {
> +	default:
> +		return NULL;
> +
> +	case VIDIOC_SUBDEV_G_FMT:
> +	case VIDIOC_SUBDEV_S_FMT: {
> +		which = ((struct v4l2_subdev_format *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_G_CROP:
> +	case VIDIOC_SUBDEV_S_CROP: {
> +		which = ((struct v4l2_subdev_crop *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_MBUS_CODE: {
> +		which = ((struct v4l2_subdev_mbus_code_enum *)arg)->which;
> +		break;
> +	}
> +	case VIDIOC_SUBDEV_ENUM_FRAME_SIZE: {
> +		which = ((struct v4l2_subdev_frame_size_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_ENUM_FRAME_INTERVAL: {
> +		which = ((struct v4l2_subdev_frame_interval_enum *)arg)->which;
> +		break;
> +	}
> +
> +	case VIDIOC_SUBDEV_G_SELECTION:
> +	case VIDIOC_SUBDEV_S_SELECTION: {
> +		which = ((struct v4l2_subdev_selection *)arg)->which;
> +		break;
> +	}
> +	}
> +
> +	return which == V4L2_SUBDEV_FORMAT_TRY ?
> +			     subdev_fh->state :
> +			     v4l2_subdev_get_active_state(sd);
> +}
> +
>  static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  {
>  	struct video_device *vdev = video_devdata(file);
> @@ -360,8 +407,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	struct v4l2_fh *vfh = file->private_data;
>  	struct v4l2_subdev_fh *subdev_fh = to_v4l2_subdev_fh(vfh);
>  	bool ro_subdev = test_bit(V4L2_FL_SUBDEV_RO_DEVNODE, &vdev->flags);
> +	struct v4l2_subdev_state *state;
>  	int rval;
>  
> +	state = subdev_ioctl_get_state(sd, subdev_fh, cmd, arg);
> +
>  	switch (cmd) {
>  	case VIDIOC_SUBDEV_QUERYCAP: {
>  		struct v4l2_subdev_capability *cap = arg;
> @@ -484,7 +534,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, get_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, format);
>  	}
>  
>  	case VIDIOC_SUBDEV_S_FMT: {
> @@ -495,7 +545,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(format->reserved, 0, sizeof(format->reserved));
>  		memset(format->format.reserved, 0, sizeof(format->format.reserved));
> -		return v4l2_subdev_call(sd, pad, set_fmt, subdev_fh->state, format);
> +		return v4l2_subdev_call(sd, pad, set_fmt, state, format);
>  	}
>  
>  	case VIDIOC_SUBDEV_G_CROP: {
> @@ -509,7 +559,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.target = V4L2_SEL_TGT_CROP;
>  
>  		rval = v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, &sel);
> +			sd, pad, get_selection, state, &sel);
>  
>  		crop->rect = sel.r;
>  
> @@ -531,7 +581,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		sel.r = crop->rect;
>  
>  		rval = v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, &sel);
> +			sd, pad, set_selection, state, &sel);
>  
>  		crop->rect = sel.r;
>  
> @@ -542,7 +592,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_mbus_code_enum *code = arg;
>  
>  		memset(code->reserved, 0, sizeof(code->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_mbus_code, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_mbus_code, state,
>  					code);
>  	}
>  
> @@ -550,7 +600,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_size_enum *fse = arg;
>  
>  		memset(fse->reserved, 0, sizeof(fse->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_size, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_size, state,
>  					fse);
>  	}
>  
> @@ -575,7 +625,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  		struct v4l2_subdev_frame_interval_enum *fie = arg;
>  
>  		memset(fie->reserved, 0, sizeof(fie->reserved));
> -		return v4l2_subdev_call(sd, pad, enum_frame_interval, subdev_fh->state,
> +		return v4l2_subdev_call(sd, pad, enum_frame_interval, state,
>  					fie);
>  	}
>  
> @@ -584,7 +634,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, get_selection, subdev_fh->state, sel);
> +			sd, pad, get_selection, state, sel);
>  	}
>  
>  	case VIDIOC_SUBDEV_S_SELECTION: {
> @@ -595,7 +645,7 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  
>  		memset(sel->reserved, 0, sizeof(sel->reserved));
>  		return v4l2_subdev_call(
> -			sd, pad, set_selection, subdev_fh->state, sel);
> +			sd, pad, set_selection, state, sel);
>  	}
>  
>  	case VIDIOC_G_EDID: {
> @@ -829,10 +879,13 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
>  		struct v4l2_subdev *sd =
>  			media_entity_to_v4l2_subdev(pad->entity);
> +		struct v4l2_subdev_state *state;
> +
> +		state = v4l2_subdev_get_active_state(sd);
>  
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> -		return v4l2_subdev_call(sd, pad, get_fmt, NULL, fmt);
> +		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
>  	}
>  
>  	WARN(pad->entity->function != MEDIA_ENT_F_IO_V4L,
> 

Re: [PATCH v9 05/36] media: subdev: Add v4l2_subdev_validate_and_lock_state()

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> All suitable subdev ops are now passed either the TRY or the ACTIVE
> state by the v4l2 core. However, other subdev drivers can still call the
> ops passing NULL as the state, implying the active case.
> 
> For all current upstream drivers this doesn't matter, as they do not
> expect to get a valid state for ACTIVE case. But future drivers which
> support multiplexed streaming and routing will depend on getting a state
> for both active and try cases.
> 
> For new drivers we can mandate that the pipelines where the drivers are
> used need to pass the state properly, or preferably, not call such
> subdev ops at all.
> 
> However, if an existing subdev driver is changed to support multiplexed
> streams, the driver has to consider cases where its ops will be called
> with NULL state. The problem can easily be solved by using the
> v4l2_subdev_validate_and_lock_state() helper, introduced here.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 33 +++++++++++++++++++++++++++++++++
>  1 file changed, 33 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index a6e8f531a396..b75eeaee492b 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1313,4 +1313,37 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>   */
>  void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>  
> +/**
> + * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev

I think this name is confusing since it isn't validating anything.

I'd call it v4l2_subdev_lock_and_return_state().

Regards,

	Hans

> + *					   state
> + * @sd: subdevice
> + * @state: subdevice state as passed to the subdev op
> + *
> + * Due to legacy reasons, when subdev drivers call ops in other subdevs they use
> + * NULL as the state parameter, as subdevs always used to have their active
> + * state stored privately.
> + *
> + * However, newer state-aware subdev drivers, which store their active state in
> + * a common place, subdev->state, expect to always get a proper state as a
> + * parameter.
> + *
> + * These state-aware drivers can use v4l2_subdev_validate_and_lock_state()
> + * instead of v4l2_subdev_lock_state(). v4l2_subdev_validate_and_lock_state()
> + * solves the issue by using subdev->state in case the passed state is
> + * NULL.
> + *
> + * This is a temporary helper function, and should be removed when we can ensure
> + * that all drivers pass proper state when calling other subdevs.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
> +				    struct v4l2_subdev_state *state)
> +{
> +	state = state ? state : sd->state;
> +
> +	v4l2_subdev_lock_state(state);
> +
> +	return state;
> +}
> +
>  #endif
> 

Re: [PATCH v9 02/36] media: subdev: add active state to struct v4l2_subdev

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add a new 'state' field to struct v4l2_subdev to which we can store the
> active state of a subdev. This will place the subdev configuration into
> a known place, allowing us to use the state directly from the v4l2
> framework, thus simplifying the drivers.
> 
> Also add functions v4l2_subdev_init_finalize() and
> v4l2_subdev_cleanup(), which will allocate and free the active state.
> The functions are named in a generic way so that they can be also used
> for other subdev initialization work.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>  include/media/v4l2-subdev.h           | 51 +++++++++++++++++++++++++++
>  2 files changed, 72 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index fe49c86a9b02..bcaf66a1e3d9 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
> +
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +{
> +	struct v4l2_subdev_state *state;
> +
> +	state = __v4l2_subdev_state_alloc(sd);
> +	if (IS_ERR(state))
> +		return PTR_ERR(state);
> +
> +	sd->state = state;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
> +{
> +	__v4l2_subdev_state_free(sd->state);
> +	sd->state = NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index e52bf508c75b..3aaa7146e5ff 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
>   * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>   *		     device using v4l2_async_register_subdev_sensor().
>   * @pdata: common part of subdevice platform data
> + * @state: active state for the subdev (NULL for subdevs tracking the state
> + *	   internally)
>   *
>   * Each instance of a subdev driver should create this struct, either
>   * stand-alone or embedded in a larger struct.
> @@ -929,6 +931,19 @@ struct v4l2_subdev {
>  	struct v4l2_async_notifier *notifier;
>  	struct v4l2_async_notifier *subdev_notifier;
>  	struct v4l2_subdev_platform_data *pdata;
> +
> +	/*
> +	 * The fields below are private, and should only be accessed via
> +	 * appropriate functions.
> +	 */
> +
> +	/*
> +	 * TODO: state should most likely be changed from a pointer to an
> +	 * embedded field. For the time being it's kept as a pointer to more
> +	 * easily catch uses of state in the cases where the driver doesn't
> +	 * support it.
> +	 */
> +	struct v4l2_subdev_state *state;

Hmm:

struct v4l2_subdev_state {
        struct v4l2_subdev_pad_config *pads;
};

and:

struct v4l2_subdev_pad_config {
        struct v4l2_mbus_framefmt try_fmt;
        struct v4l2_rect try_crop;
        struct v4l2_rect try_compose;
};

So why is this the active state if struct v4l2_subdev_pad_config has try_* fields?

At the very least it should be mentioned somewhere in the code (and probably commit
log as well) that the try_ prefix is historical, or will be removed later, or something
like that.

Regards,

	Hans

>  };
>  
>  
> @@ -1217,4 +1232,40 @@ extern const struct v4l2_subdev_ops v4l2_subdev_call_wrappers;
>  void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  			      const struct v4l2_event *ev);
>  
> +/**
> + * v4l2_subdev_init_finalize() - Finalize the initialization of the subdevice
> + * @sd: The subdev
> + *
> + * This finalizes the initialization of the subdev, including allocation of
> + * the active state for the subdev.
> + *
> + * This must be called by the subdev drivers that use the centralized active
> + * state, after the subdev struct has been initialized and
> + * media_entity_pads_init() has been called.
> + *
> + * Must call v4l2_subdev_cleanup() when the subdev is being removed.
> + */
> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> + * @sd: The subdevice
> + *
> + * This will release the resources allocated in v4l2_subdev_init_finalize.
> + */
> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_get_active_state() - Return the active subdev state for subdevice
> + * @sd: The subdevice
> + *
> + * Return the active state for the subdevice, or NULL if the subdev does not
> + * support active state.
> + */
> +static inline struct v4l2_subdev_state *
> +v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
> +{
> +	return sd->state;
> +}
> +
>  #endif
> 

Re: [PATCH v9 04/36] media: subdev: add subdev state locking

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> The V4L2 subdevs have managed without centralized locking for the state
> (previously pad_config), as the TRY state is supposedly safe (although I
> believe two TRY ioctls for the same fd would race), and the ACTIVE
> state, and its locking, is managed by the drivers internally.
> 
> We now have ACTIVE state in a centralized position, and need locking.
> Strictly speaking the locking is only needed for new drivers that use
> the new state, as the current drivers continue behaving as they used to.
> 
> Add a mutex to the struct v4l2_subdev_state, along with a few helper
> functions for locking/unlocking.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>  drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>  drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>  drivers/staging/media/tegra-video/vi.c      |  4 +-
>  include/media/v4l2-subdev.h                 | 49 ++++++++++++++++++++-
>  5 files changed, 88 insertions(+), 10 deletions(-)
> 
> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> index ba1d16ab1651..e6bd94d63e4f 100644
> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  {
>  	struct v4l2_subdev *sd = vin_to_source(vin);
>  	struct v4l2_subdev_state *sd_state;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev_format format = {
>  		.which = which,
>  		.pad = vin->parallel.source_pad,
> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>  	u32 width, height;
>  	int ret;
>  
> -	sd_state = __v4l2_subdev_state_alloc(sd);
> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  
> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
> index 869cadc1468d..e607c3ae2520 100644
> --- a/drivers/media/platform/vsp1/vsp1_entity.c
> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
> @@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  		     const char *name, unsigned int num_pads,
>  		     const struct v4l2_subdev_ops *ops, u32 function)
>  {
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	unsigned int i;
>  	int ret;
> @@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>  	 * Allocate the pad configuration to store formats and selection
>  	 * rectangles.
>  	 */
> -	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
> +						   "vsp1:config->lock", &key);
>  	if (IS_ERR(entity->config)) {
>  		media_entity_cleanup(&entity->subdev.entity);
>  		return PTR_ERR(entity->config);
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index aa757cc28879..52309a299b03 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -27,8 +27,9 @@
>  static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>  {
>  	struct v4l2_subdev_state *state;
> +	static struct lock_class_key key;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
> +struct v4l2_subdev_state *
> +__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
> +			  struct lock_class_key *lock_key)
>  {
>  	struct v4l2_subdev_state *state;
>  	int ret;
> @@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>  	if (!state)
>  		return ERR_PTR(-ENOMEM);
>  
> +	__mutex_init(&state->lock, lock_name, lock_key);
> +
>  	if (sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
> @@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  	if (!state)
>  		return;
>  
> +	mutex_destroy(&state->lock);
> +
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>  
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key)
>  {
>  	struct v4l2_subdev_state *state;
>  
> -	state = __v4l2_subdev_state_alloc(sd);
> +	state = __v4l2_subdev_state_alloc(sd, name, key);
>  	if (IS_ERR(state))
>  		return PTR_ERR(state);
>  
> @@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>  
>  	return 0;
>  }
> -EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
> +EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
>  
>  void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  {
> @@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>  	sd->state = NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
> +
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
> +{
> +	mutex_lock(&sd->state->lock);
> +
> +	return sd->state;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);

Why would this function just lock the active state, but...

> +
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_lock(&state->lock);

...this function makes no mention of 'active'.

The naming is just weird.

Part of the problem is that sd->state doesn't mention in the field name
that this is the active state. I think the field name and the function
name should match. So it is either lock_state and state->lock, or it
is lock_active_state and act_state->lock.

Either works, but this odd mix of names is confusing.

Regards,

	Hans

> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_state);
> +
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
> +{
> +	mutex_unlock(&state->lock);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> diff --git a/drivers/staging/media/tegra-video/vi.c b/drivers/staging/media/tegra-video/vi.c
> index 66b9ce160472..3516735f0cad 100644
> --- a/drivers/staging/media/tegra-video/vi.c
> +++ b/drivers/staging/media/tegra-video/vi.c
> @@ -491,6 +491,7 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  				      struct v4l2_pix_format *pix)
>  {
>  	const struct tegra_video_format *fmtinfo;
> +	static struct lock_class_key key;
>  	struct v4l2_subdev *subdev;
>  	struct v4l2_subdev_format fmt;
>  	struct v4l2_subdev_state *sd_state;
> @@ -507,7 +508,8 @@ static int __tegra_channel_try_format(struct tegra_vi_channel *chan,
>  	if (!subdev)
>  		return -ENODEV;
>  
> -	sd_state = __v4l2_subdev_state_alloc(subdev);
> +	sd_state = __v4l2_subdev_state_alloc(subdev, "tegra:state->lock",
> +					     &key);
>  	if (IS_ERR(sd_state))
>  		return PTR_ERR(sd_state);
>  	/*
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 3aaa7146e5ff..a6e8f531a396 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -655,6 +655,7 @@ struct v4l2_subdev_pad_config {
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
> + * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
> @@ -662,6 +663,7 @@ struct v4l2_subdev_pad_config {
>   * %V4L2_SUBDEV_FORMAT_ACTIVE it is safe to pass %NULL.
>   */
>  struct v4l2_subdev_state {
> +	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
>  };
>  
> @@ -1153,10 +1155,14 @@ int v4l2_subdev_link_validate(struct media_link *link);
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
>   * @sd: pointer to &struct v4l2_subdev for which the state is being allocated.
> + * @lock_name: name of the state lock
> + * @key: lock_class_key for the lock
>   *
>   * Must call __v4l2_subdev_state_free() when state is no longer needed.
>   */
> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd);
> +struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd,
> +						    const char *lock_name,
> +						    struct lock_class_key *key);
>  
>  /**
>   * __v4l2_subdev_state_free - free a v4l2_subdev_state
> @@ -1245,7 +1251,16 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>   *
>   * Must call v4l2_subdev_cleanup() when the subdev is being removed.
>   */
> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd);
> +#define v4l2_subdev_init_finalize(sd)                                          \
> +	({                                                                     \
> +		static struct lock_class_key __key;                            \
> +		const char *name = KBUILD_BASENAME                             \
> +			":" __stringify(__LINE__) ":subdev->state->lock";      \
> +		__v4l2_subdev_init_finalize(sd, name, &__key);                 \
> +	})
> +
> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
> +				struct lock_class_key *key);
>  
>  /**
>   * v4l2_subdev_cleanup() - Release the resources needed by the subdevice
> @@ -1268,4 +1283,34 @@ v4l2_subdev_get_active_state(struct v4l2_subdev *sd)
>  	return sd->state;
>  }
>  
> +/**
> + * v4l2_subdev_lock_active_state() - Lock and return the active subdev state for
> + *				     subdevice
> + * @sd: The subdevice
> + *
> + * Return the locked active state for the subdevice, or NULL if the subdev
> + * does not support active state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd);
> +
> +/**
> + * v4l2_subdev_lock_state() - Lock the subdev state
> + * @state: The subdevice state
> + *
> + * Lock the given subdev state.
> + *
> + * Must be unlocked with v4l2_subdev_unlock_state() after use.
> + */
> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
> +
> +/**
> + * v4l2_subdev_unlock_state() - Unlock the subdev state
> + * @state: The subdevice state
> + *
> + * Unlock the given subdev state.
> + */
> +void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
> +
>  #endif
> 

Re: [PATCH v9 06/36] media: Documentation: add documentation about subdev state

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add documentation about centrally managed subdev state.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../driver-api/media/v4l2-subdev.rst          | 28 +++++++++++++++++++
>  1 file changed, 28 insertions(+)
> 
> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> index 7736da077fb8..c3fd57fff668 100644
> --- a/Documentation/driver-api/media/v4l2-subdev.rst
> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> @@ -518,6 +518,34 @@ The :c:func:`v4l2_i2c_new_subdev` function will call
>  :c:type:`i2c_board_info` structure using the ``client_type`` and the
>  ``addr`` to fill it.
>  
> +Centrally managed subdev active state
> +-------------------------------------
> +
> +Traditionally V4L2 subdev drivers maintained internal state for the active
> +configuration for the subdev. This is often implemented as an array of struct
> +v4l2_mbus_framefmt, one entry for each pad.
> +
> +In addition to the active configuration, each subdev filehandle has contained

s/contained//

> +an array of struct v4l2_subdev_pad_config, managed by V4L2 core, which
> +contains the TRY configuration.
> +
> +To simplify the subdev drivers the V4L2 subdev API now optionally supports a
> +centrally managed active configuration. A subdev driver must use
> +v4l2_subdev_init_finalize() to initialize the active state between calls to
> +media_entity_pads_init() and v4l2_*_register_subdev(), and must call
> +v4l2_subdev_cleanup() to free the state.
> +
> +The active state must be locked before access, and can be done with

and -> and that

> +v4l2_subdev_lock_state() or v4l2_subdev_lock_active_state().

Note my review of patch 4 regarding the naming of these functions.

> +
> +The V4L2 core will pass either the TRY or ACTIVE state to various subdev ops.
> +Unfortunately all the subdev drivers do not comply with this yet, and may pass

all the subdev drivers do not comply -> not all the subdev drivers comply

> +NULL for the ACTIVE case. This is only a problem for subdev drivers which use
> +the cetrally managed active state and are used in media pipelines with older

cetrally -> centrally

> +subdev drivers. In these cases the called subdev ops must also handle the NULL
> +case. This can be easily managed by the use of
> +v4l2_subdev_validate_and_lock_state() helper.
> +
>  V4L2 sub-device functions and data structures
>  ---------------------------------------------
>  
> 

Regards,

	Hans

Re: [PATCH v9 24/36] media: subdev: increase V4L2_FRAME_DESC_ENTRY_MAX to 8

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> V4L2_FRAME_DESC_ENTRY_MAX is currently set to 4. In theory it's possible
> to have an arbitrary amount of streams in a single pad, so preferably
> there should be no hardcoded maximum number.
> 
> However, I believe a reasonable max is 8, which would cover a CSI-2 pad
> with 4 streams of pixel data and 4 streams of metadata.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  include/media/v4l2-subdev.h | 6 +++++-
>  1 file changed, 5 insertions(+), 1 deletion(-)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index f8443475651d..b3a72c98a2d3 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -360,7 +360,11 @@ struct v4l2_mbus_frame_desc_entry {
>  	} bus;
>  };
>  
> -#define V4L2_FRAME_DESC_ENTRY_MAX	4
> + /*
> +  * FIXME: If this number is too small, it should be dropped altogether and the
> +  * API switched to a dynamic number of frame descriptor entries.
> +  */
> +#define V4L2_FRAME_DESC_ENTRY_MAX	8
>  
>  /**
>   * enum v4l2_mbus_frame_desc_type - media bus frame description type
> 

Re: [PATCH v9 07/36] media: entity: Use pad as a starting point for graph walk

[Hans Verkuil]

FYI: I'm not reviewing patches 7-23, lots of people looked at those already,
and I trust that code :-)

Regards,

	Hans

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> From: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> With the upcoming use of the recently added has_route() media entity op,
> all the pads in an entity will no longer be considered interconnected.
> This has an effect where the media graph is traversed: the starting pad
> does make a difference.
> 
> Prepare for this change by using pad instead of the entity as an argument
> for the graph walk operations. The actual graph traversal algorithm change
> is in further patches.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> Reviewed-by: Niklas Söderlund <niklas.soderlund+renesas@ragnatech.se>
> Reviewed-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Reviewed-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  Documentation/driver-api/media/mc-core.rst    |  2 +-
>  drivers/media/mc/mc-entity.c                  | 17 ++++++++---------
>  drivers/media/platform/exynos4-is/media-dev.c |  4 ++--
>  drivers/media/platform/omap3isp/ispvideo.c    |  2 +-
>  drivers/media/platform/vsp1/vsp1_video.c      |  2 +-
>  drivers/media/platform/xilinx/xilinx-dma.c    |  2 +-
>  drivers/media/v4l2-core/v4l2-mc.c             |  6 +++---
>  drivers/staging/media/omap4iss/iss_video.c    |  4 ++--
>  include/media/media-entity.h                  | 10 ++++------
>  9 files changed, 23 insertions(+), 26 deletions(-)
> 
> diff --git a/Documentation/driver-api/media/mc-core.rst b/Documentation/driver-api/media/mc-core.rst
> index 57b5bbba944e..ba0aee982124 100644
> --- a/Documentation/driver-api/media/mc-core.rst
> +++ b/Documentation/driver-api/media/mc-core.rst
> @@ -167,7 +167,7 @@ Drivers initiate a graph traversal by calling
>  :c:func:`media_graph_walk_start()`
>  
>  The graph structure, provided by the caller, is initialized to start graph
> -traversal at the given entity.
> +traversal at the given pad in an entity.
>  
>  Drivers can then retrieve the next entity by calling
>  :c:func:`media_graph_walk_next()`
> diff --git a/drivers/media/mc/mc-entity.c b/drivers/media/mc/mc-entity.c
> index f40f41977142..47cdaa301838 100644
> --- a/drivers/media/mc/mc-entity.c
> +++ b/drivers/media/mc/mc-entity.c
> @@ -291,17 +291,16 @@ void media_graph_walk_cleanup(struct media_graph *graph)
>  }
>  EXPORT_SYMBOL_GPL(media_graph_walk_cleanup);
>  
> -void media_graph_walk_start(struct media_graph *graph,
> -			    struct media_entity *entity)
> +void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad)
>  {
>  	media_entity_enum_zero(&graph->ent_enum);
> -	media_entity_enum_set(&graph->ent_enum, entity);
> +	media_entity_enum_set(&graph->ent_enum, pad->entity);
>  
>  	graph->top = 0;
>  	graph->stack[graph->top].entity = NULL;
> -	stack_push(graph, entity);
> -	dev_dbg(entity->graph_obj.mdev->dev,
> -		"begin graph walk at '%s'\n", entity->name);
> +	stack_push(graph, pad->entity);
> +	dev_dbg(pad->graph_obj.mdev->dev,
> +		"begin graph walk at '%s':%u\n", pad->entity->name, pad->index);
>  }
>  EXPORT_SYMBOL_GPL(media_graph_walk_start);
>  
> @@ -420,7 +419,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  			goto error_graph_walk_start;
>  	}
>  
> -	media_graph_walk_start(&pipe->graph, entity);
> +	media_graph_walk_start(&pipe->graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(graph))) {
>  		DECLARE_BITMAP(active, MEDIA_ENTITY_MAX_PADS);
> @@ -504,7 +503,7 @@ __must_check int __media_pipeline_start(struct media_entity *entity,
>  	 * Link validation on graph failed. We revert what we did and
>  	 * return the error.
>  	 */
> -	media_graph_walk_start(graph, entity_err);
> +	media_graph_walk_start(graph, entity_err->pads);
>  
>  	while ((entity_err = media_graph_walk_next(graph))) {
>  		/* Sanity check for negative stream_count */
> @@ -555,7 +554,7 @@ void __media_pipeline_stop(struct media_entity *entity)
>  	if (WARN_ON(!pipe))
>  		return;
>  
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(graph))) {
>  		/* Sanity check for negative stream_count */
> diff --git a/drivers/media/platform/exynos4-is/media-dev.c b/drivers/media/platform/exynos4-is/media-dev.c
> index fa648721eaab..4ec7f7a6b0f6 100644
> --- a/drivers/media/platform/exynos4-is/media-dev.c
> +++ b/drivers/media/platform/exynos4-is/media-dev.c
> @@ -1173,7 +1173,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	 * through active links. This is needed as we cannot power on/off the
>  	 * subdevs in random order.
>  	 */
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(graph))) {
>  		if (!is_media_entity_v4l2_video_device(entity))
> @@ -1188,7 +1188,7 @@ static int __fimc_md_modify_pipelines(struct media_entity *entity, bool enable,
>  	return 0;
>  
>  err:
> -	media_graph_walk_start(graph, entity_err);
> +	media_graph_walk_start(graph, entity_err->pads);
>  
>  	while ((entity_err = media_graph_walk_next(graph))) {
>  		if (!is_media_entity_v4l2_video_device(entity_err))
> diff --git a/drivers/media/platform/omap3isp/ispvideo.c b/drivers/media/platform/omap3isp/ispvideo.c
> index 8811d6dd4ee7..3c1485d59404 100644
> --- a/drivers/media/platform/omap3isp/ispvideo.c
> +++ b/drivers/media/platform/omap3isp/ispvideo.c
> @@ -234,7 +234,7 @@ static int isp_video_get_graph_data(struct isp_video *video,
>  		return ret;
>  	}
>  
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct isp_video *__video;
> diff --git a/drivers/media/platform/vsp1/vsp1_video.c b/drivers/media/platform/vsp1/vsp1_video.c
> index 044eb5778820..61e4fbaba7b7 100644
> --- a/drivers/media/platform/vsp1/vsp1_video.c
> +++ b/drivers/media/platform/vsp1/vsp1_video.c
> @@ -569,7 +569,7 @@ static int vsp1_video_pipeline_build(struct vsp1_pipeline *pipe,
>  	if (ret)
>  		return ret;
>  
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct v4l2_subdev *subdev;
> diff --git a/drivers/media/platform/xilinx/xilinx-dma.c b/drivers/media/platform/xilinx/xilinx-dma.c
> index 338c3661d809..af0d55ab6c15 100644
> --- a/drivers/media/platform/xilinx/xilinx-dma.c
> +++ b/drivers/media/platform/xilinx/xilinx-dma.c
> @@ -189,7 +189,7 @@ static int xvip_pipeline_validate(struct xvip_pipeline *pipe,
>  		return ret;
>  	}
>  
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		struct xvip_dma *dma;
> diff --git a/drivers/media/v4l2-core/v4l2-mc.c b/drivers/media/v4l2-core/v4l2-mc.c
> index b01474717dca..d215fe31b9a2 100644
> --- a/drivers/media/v4l2-core/v4l2-mc.c
> +++ b/drivers/media/v4l2-core/v4l2-mc.c
> @@ -436,7 +436,7 @@ static int pipeline_pm_use_count(struct media_entity *entity,
>  {
>  	int use = 0;
>  
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(graph))) {
>  		if (is_media_entity_v4l2_video_device(entity))
> @@ -499,7 +499,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
>  	if (!change)
>  		return 0;
>  
> -	media_graph_walk_start(graph, entity);
> +	media_graph_walk_start(graph, entity->pads);
>  
>  	while (!ret && (entity = media_graph_walk_next(graph)))
>  		if (is_media_entity_v4l2_subdev(entity))
> @@ -508,7 +508,7 @@ static int pipeline_pm_power(struct media_entity *entity, int change,
>  	if (!ret)
>  		return ret;
>  
> -	media_graph_walk_start(graph, first);
> +	media_graph_walk_start(graph, first->pads);
>  
>  	while ((first = media_graph_walk_next(graph))
>  	       && first != entity)
> diff --git a/drivers/staging/media/omap4iss/iss_video.c b/drivers/staging/media/omap4iss/iss_video.c
> index d0da083deed5..760cd0ab1feb 100644
> --- a/drivers/staging/media/omap4iss/iss_video.c
> +++ b/drivers/staging/media/omap4iss/iss_video.c
> @@ -217,7 +217,7 @@ iss_video_far_end(struct iss_video *video)
>  		return NULL;
>  	}
>  
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  
>  	while ((entity = media_graph_walk_next(&graph))) {
>  		if (entity == &video->video.entity)
> @@ -892,7 +892,7 @@ iss_video_streamon(struct file *file, void *fh, enum v4l2_buf_type type)
>  		goto err_media_pipeline_start;
>  
>  	mutex_lock(&mdev->graph_mutex);
> -	media_graph_walk_start(&graph, entity);
> +	media_graph_walk_start(&graph, entity->pads);
>  	while ((entity = media_graph_walk_next(&graph)))
>  		media_entity_enum_set(&pipe->ent_enum, entity);
>  	mutex_unlock(&mdev->graph_mutex);
> diff --git a/include/media/media-entity.h b/include/media/media-entity.h
> index 09737b47881f..b9bfcf34eb0a 100644
> --- a/include/media/media-entity.h
> +++ b/include/media/media-entity.h
> @@ -902,22 +902,20 @@ __must_check int media_graph_walk_init(
>  void media_graph_walk_cleanup(struct media_graph *graph);
>  
>  /**
> - * media_graph_walk_start - Start walking the media graph at a
> - *	given entity
> + * media_graph_walk_start - Start walking the media graph at a given pad
>   *
>   * @graph: Media graph structure that will be used to walk the graph
> - * @entity: Starting entity
> + * @pad: Starting pad
>   *
>   * Before using this function, media_graph_walk_init() must be
>   * used to allocate resources used for walking the graph. This
>   * function initializes the graph traversal structure to walk the
> - * entities graph starting at the given entity. The traversal
> + * entities graph starting at the given pad. The traversal
>   * structure must not be modified by the caller during graph
>   * traversal. After the graph walk, the resources must be released
>   * using media_graph_walk_cleanup().
>   */
> -void media_graph_walk_start(struct media_graph *graph,
> -			    struct media_entity *entity);
> +void media_graph_walk_start(struct media_graph *graph, struct media_pad *pad);
>  
>  /**
>   * media_graph_walk_next - Get the next entity in the graph
> 

Re: [PATCH v9 25/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
> the subdev supports the new API with multiplexed streams (routing,
> stream configs).
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index b3a72c98a2d3..7f9c83fac020 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -888,6 +888,17 @@ struct v4l2_subdev_internal_ops {
>   * should set this flag.
>   */
>  #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
> +/*
> + * Set this flag if this subdev supports multiplexed streams. This means
> + * that the driver supports routing and handles the stream parameter in its
> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
> + *
> + * - Centrally managed active state is enabled
> + * - Legacy pad config is _not_ supported (state->pads)

I'm not sure what is meant with this. Can you elaborate?

Regards,

	Hans

> + * - Routing ioctls are available
> + * - Multiple streams per pad are supported
> + */
> +#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
>  
>  struct regulator_bulk_data;
>  
> 

Re: [PATCH v9 25/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
> the subdev supports the new API with multiplexed streams (routing,
> stream configs).
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  include/media/v4l2-subdev.h | 11 +++++++++++
>  1 file changed, 11 insertions(+)
> 
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index b3a72c98a2d3..7f9c83fac020 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -888,6 +888,17 @@ struct v4l2_subdev_internal_ops {
>   * should set this flag.
>   */
>  #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
> +/*
> + * Set this flag if this subdev supports multiplexed streams. This means
> + * that the driver supports routing and handles the stream parameter in its
> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
> + *
> + * - Centrally managed active state is enabled
> + * - Legacy pad config is _not_ supported (state->pads)
> + * - Routing ioctls are available
> + * - Multiple streams per pad are supported
> + */
> +#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)

This is an internal kAPI flag. But wouldn't it be useful to have this as
an entity flag instead? E.g. MEDIA_ENT_FL_MULTIPLEXED

Or perhaps as a Sub-Device Capabilities Flag V4L2_SUBDEV_CAP_MULTIPLEXED.
Or a combination. It is useful if you can detect this from userspace without
having to call G_ROUTING to see if that exists.

You can ignore this if something like that is added in later patches.

Regards,

	Hans

>  
>  struct regulator_bulk_data;
>  
> 

Re: [PATCH v9 26/36] media: Documentation: Add GS_ROUTING documentation

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> From: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> Add documentation for VIDIOC_SUBDEV_G/S_ROUTING ioctl and add
> description of multiplexed media pads and internal routing to the
> V4L2-subdev documentation section.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  .../userspace-api/media/v4l/dev-subdev.rst    |   2 +
>  .../userspace-api/media/v4l/user-func.rst     |   1 +
>  .../media/v4l/vidioc-subdev-g-routing.rst     | 146 ++++++++++++++++++
>  3 files changed, 149 insertions(+)
>  create mode 100644 Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> 
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index fd1de0a73a9f..a67c2749089a 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -29,6 +29,8 @@ will feature a character device node on which ioctls can be called to
>  
>  -  negotiate image formats on individual pads
>  
> +-  inspect and modify internal data routing between pads of the same entity
> +
>  Sub-device character device nodes, conventionally named
>  ``/dev/v4l-subdev*``, use major number 81.
>  
> diff --git a/Documentation/userspace-api/media/v4l/user-func.rst b/Documentation/userspace-api/media/v4l/user-func.rst
> index 53e604bd7d60..228c1521f190 100644
> --- a/Documentation/userspace-api/media/v4l/user-func.rst
> +++ b/Documentation/userspace-api/media/v4l/user-func.rst
> @@ -70,6 +70,7 @@ Function Reference
>      vidioc-subdev-g-crop
>      vidioc-subdev-g-fmt
>      vidioc-subdev-g-frame-interval
> +    vidioc-subdev-g-routing
>      vidioc-subdev-g-selection
>      vidioc-subdev-querycap
>      vidioc-subscribe-event
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> new file mode 100644
> index 000000000000..41f4873c49f7
> --- /dev/null
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-routing.rst
> @@ -0,0 +1,146 @@
> +.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later
> +.. c:namespace:: V4L
> +
> +.. _VIDIOC_SUBDEV_G_ROUTING:
> +
> +******************************************************
> +ioctl VIDIOC_SUBDEV_G_ROUTING, VIDIOC_SUBDEV_S_ROUTING
> +******************************************************
> +
> +Name
> +====
> +
> +VIDIOC_SUBDEV_G_ROUTING - VIDIOC_SUBDEV_S_ROUTING - Get or set routing between streams of media pads in a media entity.
> +
> +
> +Synopsis
> +========
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_G_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_G_ROUTING
> +
> +.. c:function:: int ioctl( int fd, VIDIOC_SUBDEV_S_ROUTING, struct v4l2_subdev_routing *argp )
> +    :name: VIDIOC_SUBDEV_S_ROUTING
> +
> +
> +Arguments
> +=========
> +
> +``fd``
> +    File descriptor returned by :ref:`open() <func-open>`.
> +
> +``argp``
> +    Pointer to struct :c:type:`v4l2_subdev_routing`.
> +
> +
> +Description
> +===========
> +
> +These ioctls are used to get and set the routing in a media entity.
> +The routing configuration determines the flows of data inside an entity.
> +
> +Drivers report their current routing tables using the
> +``VIDIOC_SUBDEV_G_ROUTING`` ioctl and application may enable or disable routes
> +with the VIDIOC_SUBDEV_S_ROUTING ioctl, by adding or removing routes and setting
> +or clearing the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag of the  ``flags`` field of
> +a struct :c:type:`v4l2_subdev_route`.
> +
> +A special case for routing are routes marked with
> +``V4L2_SUBDEV_ROUTE_FL_SOURCE`` flag. These routes are used to describe
> +source endpoints on sensors and the sink fields are unused.
> +
> +When inspecting routes through VIDIOC_SUBDEV_G_ROUTING and the application
> +provided ``num_routes`` is not big enough to contain all the available routes
> +the subdevice exposes, drivers return the ENOSPC error code and adjust the
> +value of the ``num_routes`` field. Application should then reserve enough memory
> +for all the route entries and call VIDIOC_SUBDEV_G_ROUTING again.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_routing
> +
> +.. flat-table:: struct v4l2_subdev_routing
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``which``
> +      - Format to modified, from enum
> +        :ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
> +    * - struct :c:type:`v4l2_subdev_route`
> +      - ``routes[]``
> +      - Array of struct :c:type:`v4l2_subdev_route` entries
> +    * - __u32
> +      - ``num_routes``
> +      - Number of entries of the routes array
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{4.4cm}|p{4.4cm}|p{8.7cm}|
> +
> +.. c:type:: v4l2_subdev_route
> +
> +.. flat-table:: struct v4l2_subdev_route
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       1 1 2
> +
> +    * - __u32
> +      - ``sink_pad``
> +      - Sink pad number.
> +    * - __u32
> +      - ``sink_stream``
> +      - Sink pad stream number.
> +    * - __u32
> +      - ``source_pad``
> +      - Source pad number.
> +    * - __u32
> +      - ``source_stream``
> +      - Source pad stream number.
> +    * - __u32
> +      - ``flags``
> +      - Route enable/disable flags
> +	:ref:`v4l2_subdev_routing_flags <v4l2-subdev-routing-flags>`.
> +    * - __u32
> +      - ``reserved``\ [5]
> +      - Reserved for future extensions. Applications and drivers must set
> +	the array to zero.
> +
> +.. tabularcolumns:: |p{6.6cm}|p{2.2cm}|p{8.7cm}|
> +
> +.. _v4l2-subdev-routing-flags:
> +
> +.. flat-table:: enum v4l2_subdev_routing_flags
> +    :header-rows:  0
> +    :stub-columns: 0
> +    :widths:       3 1 4
> +
> +    * - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - 0
> +      - The route is enabled. Set by applications.
> +    * - V4L2_SUBDEV_ROUTE_FL_IMMUTABLE
> +      - 1
> +      - The route is immutable. Set by the driver.
> +    * - V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - 2
> +      - The route is a source route, and the ``sink_pad`` and ``sink_stream``
> +        fields are unused. Set by the driver.
> +
> +Return Value
> +============
> +
> +On success 0 is returned, on error -1 and the ``errno`` variable is set
> +appropriately. The generic error codes are described at the
> +:ref:`Generic Error Codes <gen-errors>` chapter.
> +
> +ENOSPC
> +   The number of provided route entries is less than the available ones.
> +
> +EINVAL
> +   The sink or source pad identifiers reference a non-existing pad, or reference
> +   pads of different types (ie. the sink_pad identifiers refers to a source pad)
> +   or the sink or source stream identifiers reference a non-existing stream on
> +   the sink or source pad.
> 

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
> 
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
> 
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
> 
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
> 
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
> 
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 ++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 178 insertions(+), 1 deletion(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>  
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>  
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>  
>  	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>  
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 52309a299b03..b4e1f8772d96 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>  	}
>  
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -710,6 +716,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>  
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -970,6 +1044,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  
>  	mutex_destroy(&state->lock);
>  
> +	kvfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 7f9c83fac020..f318670a49c8 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,11 +694,25 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};
> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
>  struct v4l2_subdev_state {
>  	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>  
>  /**
> @@ -770,6 +785,9 @@ struct v4l2_subdev_state {
>   *		     applied to the hardware. The operation shall fail if the
>   *		     pad index it has been called on is not valid or in case of
>   *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
>  			       struct v4l2_mbus_config *config);
>  	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);
>  };
>  
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..5ba409db47ff 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device video device node is registered in read-only mode. */
>  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>  
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u64 routes;

Move routes to after num_routes, otherwise you will get padding after 'which'
on some architectures, which would require messy compat32 conversion.

> +	__u32 num_routes;
> +	__u32 reserved[5];

And make this reserved[6] to again ensure 64 bit alignment of the whole struct.

> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>  
> @@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
> 

Regards,

	Hans

Re: [PATCH v9 28/36] media: subdev: add v4l2_subdev_has_route()

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add a v4l2_subdev_has_route() helper function which can be used for
> media_entity_operations.has_route op.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 31 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 ++++++++++++++
>  2 files changed, 47 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index b4e1f8772d96..14b8282fe45b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -998,6 +998,37 @@ int v4l2_subdev_link_validate(struct media_link *link)
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> +bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
> +			   unsigned int pad1)
> +{
> +	struct v4l2_subdev *sd = media_entity_to_v4l2_subdev(entity);
> +	struct v4l2_subdev_krouting *routing;
> +	unsigned int i;
> +	struct v4l2_subdev_state *state;
> +
> +	state = v4l2_subdev_lock_active_state(sd);
> +
> +	routing = &state->routing;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		if ((route->sink_pad == pad0 && route->source_pad == pad1) ||
> +		    (route->source_pad == pad0 && route->sink_pad == pad1)) {
> +			v4l2_subdev_unlock_state(state);
> +			return true;
> +		}
> +	}
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	return false;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_has_route);
> +
>  struct v4l2_subdev_state *
>  __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
>  			  struct lock_class_key *lock_key)
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index f318670a49c8..8b2cf3190276 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1226,6 +1226,22 @@ int v4l2_subdev_link_validate_default(struct v4l2_subdev *sd,
>   */
>  int v4l2_subdev_link_validate(struct media_link *link);
>  
> +/**
> + * v4l2_subdev_has_route - MC has_route implementation for subdevs
> + *
> + * @entity: pointer to &struct media_entity
> + * @pad0: pad number for the first pad
> + * @pad1: pad number for the second pad
> + *
> + * This function looks at the routing in subdev's active state and returns if
> + * there is a route connecting pad0 and pad1.
> + *
> + * This function can be used as implementation for
> + * media_entity_operations.has_route.
> + */
> +bool v4l2_subdev_has_route(struct media_entity *entity, unsigned int pad0,
> +			   unsigned int pad1);
> +
>  /**
>   * __v4l2_subdev_state_alloc - allocate v4l2_subdev_state
>   *
> 

Re: [PATCH v9 29/36] media: subdev: add v4l2_subdev_set_routing helper()

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add a helper function to set the subdev routing. The helper can be used
> from subdev driver's set_routing op to store the routing table.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 28 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 +++++++++++++++
>  2 files changed, 44 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 14b8282fe45b..af53f827ec27 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1151,3 +1151,31 @@ void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state)
>  	mutex_unlock(&state->lock);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
> +
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing)
> +{
> +	struct v4l2_subdev_krouting *dst = &state->routing;
> +	const struct v4l2_subdev_krouting *src = routing;
> +
> +	lockdep_assert_held(&state->lock);
> +
> +	kvfree(dst->routes);
> +	dst->routes = NULL;
> +	dst->num_routes = 0;
> +
> +	if (src->num_routes > 0) {
> +		dst->routes = kvmalloc_array(src->num_routes,
> +					     sizeof(*src->routes), GFP_KERNEL);
> +		if (!dst->routes)
> +			return -ENOMEM;
> +
> +		memcpy(dst->routes, src->routes,
> +		       src->num_routes * sizeof(*src->routes));
> +		dst->num_routes = src->num_routes;
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 8b2cf3190276..b6c5cd00831e 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1437,4 +1437,20 @@ v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
>  	return state;
>  }
>  
> +/**
> + * v4l2_subdev_set_routing() - Set given routing to subdev state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + *
> + * This will release old routing table (if any) from the state, allocate
> + * enough space for the given routing, and copy the routing.
> + *
> + * This can be used from the subdev driver's set_routing op, after validating
> + * the routing.
> + */
> +int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
> +			    struct v4l2_subdev_state *state,
> +			    struct v4l2_subdev_krouting *routing);
> +
>  #endif
> 

Re: [PATCH v9 30/36] media: Documentation: add multiplexed streams documentation

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add documentation related to multiplexed streams.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../driver-api/media/v4l2-subdev.rst          |   8 +
>  .../userspace-api/media/v4l/dev-subdev.rst    | 164 ++++++++++++++++++
>  2 files changed, 172 insertions(+)
> 
> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
> index c3fd57fff668..410e03a29f2a 100644
> --- a/Documentation/driver-api/media/v4l2-subdev.rst
> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
> @@ -546,6 +546,14 @@ subdev drivers. In these cases the called subdev ops must also handle the NULL
>  case. This can be easily managed by the use of
>  v4l2_subdev_validate_and_lock_state() helper.
>  
> +Streams, multiplexed media pads and internal routing
> +----------------------------------------------------
> +
> +A subdevice driver can implement support for multiplexed streams by setting
> +the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
> +centrally managed subdev active state, routing and stream based
> +configuration.
> +
>  V4L2 sub-device functions and data structures
>  ---------------------------------------------
>  
> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> index a67c2749089a..ea3efa97bb08 100644
> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
> @@ -503,3 +503,167 @@ source pads.
>      :maxdepth: 1
>  
>      subdev-formats
> +
> +Streams, multiplexed media pads and internal routing
> +----------------------------------------------------
> +
> +Commonly V4L2 subdevices support only separate video streams, that is, each
> +link in the media grap and each pad in a subdevice passes through a single

passes -> pass
grap -> graph

> +video stream. Thus each pad contains a format configuration for that single
> +stream. In some cases a subdev can do stream processing and split a stream
> +into two or compose two streams into one, but the inputs and outputs for the
> +subdev are still a single stream per pad.
> +
> +Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
> +bus carries multiple streams. Thus a camera could output two streams, a pixel
> +stream and a metadata stream, and a bridge subdev could route the streams
> +from multiple input pads into a single output pad.
> +
> +Subdevice drivers that support multiplexed streams are compatible with
> +non-multiplexed subdev drivers, but, of course, requires such a routing

requires such -> require

> +configuration where the link between those two types of drivers contain only

contain -> contains

> +a single stream.
> +
> +Understanding streams
> +^^^^^^^^^^^^^^^^^^^^^
> +
> +A stream is a stream of content (e.g. pixel data or metadata) flowing through
> +the media pipeline from a source (e.g. a sensor) towards the final sink
> +(e.g. a receiver in a SoC). Each media link carries all the streams from

'receiver' is a bit vague. What sort of receiver are we talking about here?
What about: 'a receiver and demultiplexer in a SoC'. This implies that it is
the block that can receive a multiplexed stream and demux it. The demuxing
part is the critical feature of this block IMHO.

> +one end of the link to the other, whereas subdevices have routing tables

'whereas' is an odd word. Would simply 'and' not be a better word to use here?

> +which describe how the incoming streams from sink pads are routed to the
> +source pads.
> +
> +A stream ID (often just "stream") is a media link-local identifier for a
> +stream. In other words, configuration for a particular stream ID must exist

configuration -> a configuration

> +on both sides of a media link, but another stream ID can be used for the same
> +stream at the other side of the subdevice.
> +
> +A stream at a specific point in the media pipeline is identified with the
> +subdev and a (pad, stream) pair. For subdevices that do not support
> +multiplexed streams the 'stream' is always 0.

I find this section confusing. I think there are two issues here: what exactly is
meant with 'media link': a link between two pads from different entities?

And I also think that 'subdev' is sometimes used where 'media entity' is actually
meant. E.g. "at the other side of the subdevice" makes no sense, but "between two
other entities" makes more sense.

> +
> +Configuring streams
> +^^^^^^^^^^^^^^^^^^^
> +
> +The configuration of the streams is done individually for each subdevice and
> +the validity of the streams between subdevices is validated when the pipeline
> +is started.

Here too I think you really mean 'entities'. A subdev device node controls an
entity. It might be me, but when I think of a media stream I see it as a data
stream running through entities, and subdevs are just the interfaces through
which the entities are controlled.

> +
> +There are three steps in configuring the streams:
> +
> +1) Set up links. Connect the pads between subdevices using the :ref:`Media
> +Controller API <media_controller>`
> +
> +2) Routing. The routing table for the subdevice must be set with
> +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
> +
> +3) Configure streams. Each route endpoint must be configured
> +with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
> +
> +Multiplexed streams setup example
> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
> +
> +A simple example of a multiplexed stream setup might be as follows:
> +
> +- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
> +  source pad (pad 0), and outputs two streams, pixel data and metadata.
> +
> +- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
> +  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
> +  streams.
> +
> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
> +  connected to the bridge, and four source pads (pads 1-4), going to the DMA
> +  engine. The receiver demultiplexes the incoming streams to the four source
> +  pads.
> +
> +- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
> +  single source pad in the receiver.
> +
> +The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
> +
> +To configure this pipeline, the userspace must take the following steps:
> +
> +1) Set up media links between entities: connect the sensors to the bridge,
> +bridge to the receiver, and the receiver to the DMA engines. This step does
> +not differ from normal non-multiplexed media controller setup.
> +
> +2) Configure routing.
> +
> +.. flat-table:: Sensor routing table (identical on both sensors)
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0 (unused)
> +      - 0/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - Pixel data stream. Source route, i.e. the sink fields are unused.
> +    * - 0/0 (unused)
> +      - 0/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
> +      - Metadata stream. Source route, i.e. the sink fields are unused.
> +
> +.. flat-table:: Bridge routing table
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from Sensor A
> +    * - 0/1
> +      - 2/1
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from Sensor A
> +    * - 1/0
> +      - 2/2
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from Sensor B
> +    * - 1/1
> +      - 2/3
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from Sensor B
> +
> +.. flat-table:: Receiver routing table
> +    :header-rows:  1
> +
> +    * - Sink Pad/Stream
> +      - Source Pad/Stream
> +      - Routing Flags
> +      - Comments
> +    * - 0/0
> +      - 1/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from Sensor A
> +    * - 0/1
> +      - 2/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from Sensor A
> +    * - 0/2
> +      - 3/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Pixel data stream from Sensor B
> +    * - 0/3
> +      - 4/0
> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
> +      - Metadata stream from Sensor B
> +
> +3) Configure streams
> +
> +After configuring the routing table, the next step is configuring the streams.
> +This step is similar to configuring the pads in a non-multiplexed streams
> +setup, with the difference that we need to configure each (pad, stream) pair
> +(i.e. route endpoint), instead of just a pad.
> +
> +Presuming there are no format conversions in the pipeline, the userspace needs

Presuming -> Assuming

> +to configure all the route endpoints using four formats (two pixel formats
> +and two metadata formats) with VIDIOC_SUBDEV_S_FMT.
> 

Actually, if there are format conversions, wouldn't you still do the same thing?

Is that 'Presuming...pipeline,' needed at all?

Regards,

	Hans

Re: [PATCH v9 31/36] media: subdev: add stream based configuration

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add support to manage configurations (format, crop, compose) per stream,

Isn't that 'pad+stream'? This sentence suggests that 'stream' replaces 'pad',
but it is 'in addition to', right?

> instead of per pad. This is accomplished with data structures that hold
> an array of all subdev's stream configurations.
> 
> The number of streams can vary at runtime based on routing. Every time
> the routing is changed, the stream configurations need to be
> re-initialized.

Is this documented? I can't remember reading this, but I may have missed this.
And what if the new routing leaves one stream unchanged, and changes a second,
does that still mean that both stream configurations need to be re-initialized,
or just that of the changed stream?

If changing the routing, even if the route of an existing stream stays unchanged,
will require this re-initialization, then that has to be documented clearly, including
in the affected ioctls.

Regards,

	Hans

> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  .../v4l/vidioc-subdev-enum-frame-interval.rst |   5 +-
>  .../v4l/vidioc-subdev-enum-frame-size.rst     |   5 +-
>  .../v4l/vidioc-subdev-enum-mbus-code.rst      |   5 +-
>  .../media/v4l/vidioc-subdev-g-crop.rst        |   5 +-
>  .../media/v4l/vidioc-subdev-g-fmt.rst         |   5 +-
>  .../v4l/vidioc-subdev-g-frame-interval.rst    |   5 +-
>  .../media/v4l/vidioc-subdev-g-selection.rst   |   5 +-
>  drivers/media/v4l2-core/v4l2-subdev.c         | 140 +++++++++++++++++-
>  include/media/v4l2-subdev.h                   |  48 ++++++
>  include/uapi/linux/v4l2-subdev.h              |  28 +++-
>  10 files changed, 229 insertions(+), 22 deletions(-)
> 
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> index 3703943b412f..8def4c05d3da 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-interval.rst
> @@ -92,7 +92,10 @@ multiple pads of the same sub-device is not defined.
>        - Frame intervals to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> index c25a9896df0e..3ef361c0dca7 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-frame-size.rst
> @@ -97,7 +97,10 @@ information about try formats.
>        - Frame sizes to be enumerated, from enum
>  	:ref:`v4l2_subdev_format_whence <v4l2-subdev-format-whence>`.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> index 417f1a19bcc4..248f6f9ee7c5 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-enum-mbus-code.rst
> @@ -73,7 +73,10 @@ information about the try formats.
>        - ``flags``
>        - See :ref:`v4l2-subdev-mbus-code-flags`
>      * - __u32
> -      - ``reserved``\ [7]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [6]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> index bd15c0a5a66b..1d267f7e7991 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-crop.rst
> @@ -96,7 +96,10 @@ modified format should be as close as possible to the original request.
>        - ``rect``
>        - Crop rectangle boundaries, in pixels.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> index 7acdbb939d89..ed253a1e44b7 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-fmt.rst
> @@ -102,7 +102,10 @@ should be as close as possible to the original request.
>        - Definition of an image format, see :c:type:`v4l2_mbus_framefmt` for
>  	details.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> index d7fe7543c506..842f962d2aea 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-frame-interval.rst
> @@ -90,7 +90,10 @@ the same sub-device is not defined.
>        - ``interval``
>        - Period, in seconds, between consecutive video frames.
>      * - __u32
> -      - ``reserved``\ [9]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [8]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> index f9172a42f036..6b629c19168c 100644
> --- a/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> +++ b/Documentation/userspace-api/media/v4l/vidioc-subdev-g-selection.rst
> @@ -94,7 +94,10 @@ Selection targets and flags are documented in
>        - ``r``
>        - Selection rectangle, in pixels.
>      * - __u32
> -      - ``reserved``\ [8]
> +      - ``stream``
> +      - Stream identifier.
> +    * - __u32
> +      - ``reserved``\ [7]
>        - Reserved for future extensions. Applications and drivers must set
>  	the array to zero.
>  
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index af53f827ec27..f63826755057 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -149,14 +149,45 @@ static inline int check_pad(struct v4l2_subdev *sd, u32 pad)
>  	return 0;
>  }
>  
> -static int check_state_pads(u32 which, struct v4l2_subdev_state *state)
> +static int check_state_pads(struct v4l2_subdev *sd, u32 which,
> +			    struct v4l2_subdev_state *state)
>  {
> +	if (sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED)
> +		return 0;
> +
>  	if (which == V4L2_SUBDEV_FORMAT_TRY && (!state || !state->pads))
>  		return -EINVAL;
>  
>  	return 0;
>  }
>  
> +static int check_state_pad_stream(struct v4l2_subdev *sd,
> +				  struct v4l2_subdev_state *state, u32 pad,
> +				  u32 stream)
> +{
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +		return 0;
> +
> +	/*
> +	 * We need to take the state lock to access the format, but as we then
> +	 * have to unlock, nothing prevents someone changing the state before
> +	 * this call thread enters the driver's op and the driver has the
> +	 * change to lock the state.
> +	 */
> +	v4l2_subdev_lock_state(state);
> +
> +	fmt = v4l2_state_get_stream_format(state, pad, stream);
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	if (!fmt)
> +		return -EINVAL;
> +
> +	return 0;
> +}
> +
>  static inline int check_format(struct v4l2_subdev *sd,
>  			       struct v4l2_subdev_state *state,
>  			       struct v4l2_subdev_format *format)
> @@ -165,7 +196,8 @@ static inline int check_format(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	return check_which(format->which) ? : check_pad(sd, format->pad) ? :
> -	       check_state_pads(format->which, state);
> +	       check_state_pads(sd, format->which, state) ? :
> +	       check_state_pad_stream(sd, state, format->pad, format->stream);
>  }
>  
>  static int call_get_fmt(struct v4l2_subdev *sd,
> @@ -192,7 +224,8 @@ static int call_enum_mbus_code(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	return check_which(code->which) ? : check_pad(sd, code->pad) ? :
> -	       check_state_pads(code->which, state) ? :
> +	       check_state_pads(sd, code->which, state) ? :
> +	       check_state_pad_stream(sd, state, code->pad, code->stream) ? :
>  	       sd->ops->pad->enum_mbus_code(sd, state, code);
>  }
>  
> @@ -204,7 +237,8 @@ static int call_enum_frame_size(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	return check_which(fse->which) ? : check_pad(sd, fse->pad) ? :
> -	       check_state_pads(fse->which, state) ? :
> +	       check_state_pads(sd, fse->which, state) ? :
> +	       check_state_pad_stream(sd, state, fse->pad, fse->stream) ? :
>  	       sd->ops->pad->enum_frame_size(sd, state, fse);
>  }
>  
> @@ -239,7 +273,8 @@ static int call_enum_frame_interval(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	return check_which(fie->which) ? : check_pad(sd, fie->pad) ? :
> -	       check_state_pads(fie->which, state) ? :
> +	       check_state_pads(sd, fie->which, state) ? :
> +	       check_state_pad_stream(sd, state, fie->pad, fie->stream) ? :
>  	       sd->ops->pad->enum_frame_interval(sd, state, fie);
>  }
>  
> @@ -251,7 +286,8 @@ static inline int check_selection(struct v4l2_subdev *sd,
>  		return -EINVAL;
>  
>  	return check_which(sel->which) ? : check_pad(sd, sel->pad) ? :
> -	       check_state_pads(sel->which, state);
> +	       check_state_pads(sd, sel->which, state) ? :
> +	       check_state_pad_stream(sd, state, sel->pad, sel->stream);
>  }
>  
>  static int call_get_selection(struct v4l2_subdev *sd,
> @@ -867,6 +903,71 @@ const struct v4l2_file_operations v4l2_subdev_fops = {
>  	.poll = subdev_poll,
>  };
>  
> +static int
> +v4l2_init_stream_configs(struct v4l2_subdev_stream_configs *stream_configs,
> +			 const struct v4l2_subdev_krouting *routing)
> +{
> +	u32 num_configs = 0;
> +	unsigned int i;
> +	u32 format_idx = 0;
> +
> +	kvfree(stream_configs->configs);
> +	stream_configs->configs = NULL;
> +	stream_configs->num_configs = 0;
> +
> +	/* Count number of formats needed */
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		/*
> +		 * Each route needs a format on both ends of the route, except
> +		 * for source streams which only need one format.
> +		 */
> +		num_configs +=
> +			(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE) ? 1 : 2;
> +	}
> +
> +	if (num_configs) {
> +		stream_configs->configs =
> +			kvcalloc(num_configs, sizeof(*stream_configs->configs),
> +				 GFP_KERNEL);
> +
> +		if (!stream_configs->configs)
> +			return -ENOMEM;
> +
> +		stream_configs->num_configs = num_configs;
> +	}
> +
> +	/*
> +	 * Fill in the 'pad' and stream' value for each item in the array from
> +	 * the routing table
> +	 */
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +		u32 idx;
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +			idx = format_idx++;
> +
> +			stream_configs->configs[idx].pad = route->sink_pad;
> +			stream_configs->configs[idx].stream = route->sink_stream;
> +		}
> +
> +		idx = format_idx++;
> +
> +		stream_configs->configs[idx].pad = route->source_pad;
> +		stream_configs->configs[idx].stream = route->source_stream;
> +	}
> +
> +	return 0;
> +}
> +
>  #ifdef CONFIG_MEDIA_CONTROLLER
>  
>  int v4l2_subdev_get_fwnode_pad_1_to_1(struct media_entity *entity,
> @@ -1042,7 +1143,8 @@ __v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
>  
>  	__mutex_init(&state->lock, lock_name, lock_key);
>  
> -	if (sd->entity.num_pads) {
> +	/* Drivers that support streams do not need the legacy pad config */
> +	if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED) && sd->entity.num_pads) {
>  		state->pads = kvmalloc_array(sd->entity.num_pads,
>  					     sizeof(*state->pads),
>  					     GFP_KERNEL | __GFP_ZERO);
> @@ -1076,6 +1178,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>  	mutex_destroy(&state->lock);
>  
>  	kvfree(state->routing.routes);
> +	kvfree(state->stream_configs.configs);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> @@ -1176,6 +1279,27 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>  		dst->num_routes = src->num_routes;
>  	}
>  
> -	return 0;
> +	return v4l2_init_stream_configs(&state->stream_configs, dst);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
> +
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
> +			     u32 stream)
> +{
> +	struct v4l2_subdev_stream_configs *stream_configs;
> +	unsigned int i;
> +
> +	lockdep_assert_held(&state->lock);
> +
> +	stream_configs = &state->stream_configs;
> +
> +	for (i = 0; i < stream_configs->num_configs; ++i) {
> +		if (stream_configs->configs[i].pad == pad &&
> +		    stream_configs->configs[i].stream == stream)
> +			return &stream_configs->configs[i].fmt;
> +	}
> +
> +	return NULL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index b6c5cd00831e..1dc824416c1b 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,6 +694,37 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>  
> +/**
> + * struct v4l2_subdev_stream_config - Used for storing stream configuration.
> + *
> + * @pad: pad number
> + * @stream: stream number
> + * @fmt: &struct v4l2_mbus_framefmt
> + * @crop: &struct v4l2_rect to be used for crop
> + * @compose: &struct v4l2_rect to be used for compose
> + *
> + * This structure stores configuration for a stream.
> + */
> +struct v4l2_subdev_stream_config {
> +	u32 pad;
> +	u32 stream;
> +
> +	struct v4l2_mbus_framefmt fmt;
> +	struct v4l2_rect crop;
> +	struct v4l2_rect compose;
> +};
> +
> +/**
> + * struct v4l2_subdev_stream_configs - A collection of stream configs.
> + *
> + * @num_configs: number of entries in @config.
> + * @configs: an array of &struct v4l2_subdev_stream_configs.
> + */
> +struct v4l2_subdev_stream_configs {
> +	u32 num_configs;
> +	struct v4l2_subdev_stream_config *configs;
> +};
> +
>  /**
>   * struct v4l2_subdev_krouting - subdev routing table
>   *
> @@ -713,6 +744,7 @@ struct v4l2_subdev_krouting {
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
>   * @routing: routing table for the subdev
> + * @stream_configs: stream configurations (only for V4L2_SUBDEV_FL_MULTIPLEXED)
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -722,6 +754,7 @@ struct v4l2_subdev_state {
>  	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
>  	struct v4l2_subdev_krouting routing;
> +	struct v4l2_subdev_stream_configs stream_configs;
>  };
>  
>  /**
> @@ -1453,4 +1486,19 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>  			    struct v4l2_subdev_state *state,
>  			    struct v4l2_subdev_krouting *routing);
>  
> +/**
> + * v4l2_state_get_stream_format() - Get pointer to a stream format
> + * @state: subdevice state
> + * @pad: pad id
> + * @stream: stream id
> + *
> + * This returns a pointer to &struct v4l2_mbus_framefmt for the given pad +
> + * stream in the subdev state.
> + *
> + * If the state does not contain the given pad + stream, NULL is returned.
> + */
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
> +			     u32 stream);
> +
>  #endif
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 5ba409db47ff..f61ace2b5755 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -44,13 +44,15 @@ enum v4l2_subdev_format_whence {
>   * @which: format type (from enum v4l2_subdev_format_whence)
>   * @pad: pad number, as reported by the media API
>   * @format: media bus format (format code and frame size)
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_format {
>  	__u32 which;
>  	__u32 pad;
>  	struct v4l2_mbus_framefmt format;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> @@ -58,13 +60,15 @@ struct v4l2_subdev_format {
>   * @which: format type (from enum v4l2_subdev_format_whence)
>   * @pad: pad number, as reported by the media API
>   * @rect: pad crop rectangle boundaries
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_crop {
>  	__u32 which;
>  	__u32 pad;
>  	struct v4l2_rect rect;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  #define V4L2_SUBDEV_MBUS_CODE_CSC_COLORSPACE	0x00000001
> @@ -80,6 +84,7 @@ struct v4l2_subdev_crop {
>   * @code: format code (MEDIA_BUS_FMT_ definitions)
>   * @which: format type (from enum v4l2_subdev_format_whence)
>   * @flags: flags set by the driver, (V4L2_SUBDEV_MBUS_CODE_*)
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_mbus_code_enum {
> @@ -88,7 +93,8 @@ struct v4l2_subdev_mbus_code_enum {
>  	__u32 code;
>  	__u32 which;
>  	__u32 flags;
> -	__u32 reserved[7];
> +	__u32 stream;
> +	__u32 reserved[6];
>  };
>  
>  /**
> @@ -101,6 +107,7 @@ struct v4l2_subdev_mbus_code_enum {
>   * @min_height: minimum frame height, in pixels
>   * @max_height: maximum frame height, in pixels
>   * @which: format type (from enum v4l2_subdev_format_whence)
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_frame_size_enum {
> @@ -112,19 +119,22 @@ struct v4l2_subdev_frame_size_enum {
>  	__u32 min_height;
>  	__u32 max_height;
>  	__u32 which;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
>   * struct v4l2_subdev_frame_interval - Pad-level frame rate
>   * @pad: pad number, as reported by the media API
>   * @interval: frame interval in seconds
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_frame_interval {
>  	__u32 pad;
>  	struct v4l2_fract interval;
> -	__u32 reserved[9];
> +	__u32 stream;
> +	__u32 reserved[8];
>  };
>  
>  /**
> @@ -136,6 +146,7 @@ struct v4l2_subdev_frame_interval {
>   * @height: frame height in pixels
>   * @interval: frame interval in seconds
>   * @which: format type (from enum v4l2_subdev_format_whence)
> + * @stream: stream number, defined in subdev routing
>   * @reserved: drivers and applications must zero this array
>   */
>  struct v4l2_subdev_frame_interval_enum {
> @@ -146,7 +157,8 @@ struct v4l2_subdev_frame_interval_enum {
>  	__u32 height;
>  	struct v4l2_fract interval;
>  	__u32 which;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> @@ -158,6 +170,7 @@ struct v4l2_subdev_frame_interval_enum {
>   *	    defined in v4l2-common.h; V4L2_SEL_TGT_* .
>   * @flags: constraint flags, defined in v4l2-common.h; V4L2_SEL_FLAG_*.
>   * @r: coordinates of the selection window
> + * @stream: stream number, defined in subdev routing
>   * @reserved: for future use, set to zero for now
>   *
>   * Hardware may use multiple helper windows to process a video stream.
> @@ -170,7 +183,8 @@ struct v4l2_subdev_selection {
>  	__u32 target;
>  	__u32 flags;
>  	struct v4l2_rect r;
> -	__u32 reserved[8];
> +	__u32 stream;
> +	__u32 reserved[7];
>  };
>  
>  /**
> 

Re: [PATCH v9 32/36] media: subdev: use streams in v4l2_subdev_link_validate()

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Update v4l2_subdev_link_validate() to use routing and streams for
> validation.
> 
> Instead of just looking at the format on the pad on both ends of the
> link, the routing tables are used to collect all the streams going from
> the source to the sink over the link, and the streams' formats on both
> ends of the link are verified.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 217 +++++++++++++++++++++++---
>  1 file changed, 199 insertions(+), 18 deletions(-)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index f63826755057..37e2e1f907fc 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -16,6 +16,7 @@
>  #include <linux/videodev2.h>
>  #include <linux/export.h>
>  #include <linux/version.h>
> +#include <linux/sort.h>
>  
>  #include <media/v4l2-ctrls.h>
>  #include <media/v4l2-device.h>
> @@ -1050,6 +1051,7 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate_default);
>  
>  static int
>  v4l2_subdev_link_validate_get_format(struct media_pad *pad,
> +				     u32 stream,
>  				     struct v4l2_subdev_format *fmt)
>  {
>  	if (is_media_entity_v4l2_subdev(pad->entity)) {
> @@ -1061,6 +1063,7 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  
>  		fmt->which = V4L2_SUBDEV_FORMAT_ACTIVE;
>  		fmt->pad = pad->index;
> +		fmt->stream = stream;
>  		return v4l2_subdev_call(sd, pad, get_fmt, state, fmt);
>  	}
>  
> @@ -1071,31 +1074,209 @@ v4l2_subdev_link_validate_get_format(struct media_pad *pad,
>  	return -EINVAL;
>  }
>  
> -int v4l2_subdev_link_validate(struct media_link *link)
> +static int cmp_u32(const void *a, const void *b)
>  {
> -	struct v4l2_subdev *sink;
> -	struct v4l2_subdev_format sink_fmt, source_fmt;
> -	int rval;
> +	u32 a32 = *(u32 *)a;
> +	u32 b32 = *(u32 *)b;
>  
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->source, &source_fmt);
> -	if (rval < 0)
> -		return 0;
> +	return a32 > b32 ? 1 : (a32 < b32 ? -1 : 0);
> +}
> +
> +static int v4l2_link_validate_get_streams(struct media_link *link,
> +					  bool is_source, u32 *out_num_streams,
> +					  const u32 **out_streams,
> +					  bool *allocated)
> +{
> +	static const u32 default_streams[] = { 0 };

Hmm, I'd just write 'static const u32 default_stream = 0;'

> +	struct v4l2_subdev_krouting *routing;
> +	struct v4l2_subdev *subdev;
> +	u32 num_streams;
> +	u32 *streams;
> +	unsigned int i;
> +	struct v4l2_subdev_state *state;
> +
> +	if (is_source)
> +		subdev = media_entity_to_v4l2_subdev(link->source->entity);
> +	else
> +		subdev = media_entity_to_v4l2_subdev(link->sink->entity);
>  
> -	rval = v4l2_subdev_link_validate_get_format(
> -		link->sink, &sink_fmt);
> -	if (rval < 0)
> +	if (!(subdev->flags & V4L2_SUBDEV_FL_MULTIPLEXED)) {
> +		*out_num_streams = 1;
> +		*out_streams = default_streams;

and here write = &default_stream;

That feels more natural.

> +		*allocated = false;
>  		return 0;
> +	}
>  
> -	sink = media_entity_to_v4l2_subdev(link->sink->entity);
> +	state = v4l2_subdev_lock_active_state(subdev);
>  
> -	rval = v4l2_subdev_call(sink, pad, link_validate, link,
> -				&source_fmt, &sink_fmt);
> -	if (rval != -ENOIOCTLCMD)
> -		return rval;
> +	routing = &state->routing;
> +
> +	streams = kmalloc_array(routing->num_routes, sizeof(u32), GFP_KERNEL);
> +
> +	if (!streams) {
> +		v4l2_subdev_unlock_state(state);
> +		return -ENOMEM;
> +	}
> +
> +	num_streams = 0;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +		int j;
> +		u32 route_pad;
> +		u32 route_stream;
> +		u32 link_pad;
> +
> +		if (is_source) {
> +			route_pad = route->source_pad;
> +			route_stream = route->source_stream;
> +			link_pad = link->source->index;
> +		} else {
> +			route_pad = route->sink_pad;
> +			route_stream = route->sink_stream;
> +			link_pad = link->sink->index;
> +		}
> +
> +		if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_ACTIVE))
> +			continue;
> +
> +		if (route_pad != link_pad)
> +			continue;
> +
> +		/* look for duplicates... */
> +		for (j = 0; j < num_streams; ++j) {
> +			if (streams[j] == route_stream)
> +				break;
> +		}
> +
> +		/* ...and drop the stream if we already have it */
> +		if (j != num_streams)
> +			continue;
> +
> +		streams[num_streams++] = route_stream;
> +	}
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	sort(streams, num_streams, sizeof(u32), &cmp_u32, NULL);
> +
> +	*out_num_streams = num_streams;
> +	*out_streams = streams;
> +	*allocated = true;
>  
> -	return v4l2_subdev_link_validate_default(
> -		sink, link, &source_fmt, &sink_fmt);
> +	return 0;
> +}
> +
> +int v4l2_subdev_link_validate(struct media_link *link)
> +{
> +	struct v4l2_subdev *sink_subdev =
> +		media_entity_to_v4l2_subdev(link->sink->entity);
> +	struct device *dev = sink_subdev->entity.graph_obj.mdev->dev;
> +	u32 num_source_streams;
> +	const u32 *source_streams;
> +	bool source_allocated;
> +	u32 num_sink_streams;
> +	const u32 *sink_streams;
> +	bool sink_allocated;
> +	unsigned int sink_idx;
> +	unsigned int source_idx;
> +	int ret;
> +
> +	dev_dbg(dev, "validating link \"%s\":%u -> \"%s\":%u\n",
> +		link->source->entity->name, link->source->index,
> +		link->sink->entity->name, link->sink->index);
> +
> +	ret = v4l2_link_validate_get_streams(link, true, &num_source_streams,
> +					     &source_streams,
> +					     &source_allocated);
> +	if (ret)
> +		return ret;
> +
> +	ret = v4l2_link_validate_get_streams(link, false, &num_sink_streams,
> +					     &sink_streams, &sink_allocated);
> +	if (ret)
> +		goto free_source;
> +
> +	/* It is ok to have more source streams than sink streams */

Why? Because that means that those extra streams are just ignored on the sink side
of the link?

In any case, state the reason why it is ok.

> +	if (num_source_streams < num_sink_streams) {
> +		dev_err(dev,
> +			"Not enough source streams: %d < %d\n",
> +			num_source_streams, num_sink_streams);
> +		ret = -EINVAL;
> +		goto out;
> +	}
> +
> +	/* Validate source and sink stream formats */
> +
> +	source_idx = 0;
> +
> +	for (sink_idx = 0; sink_idx < num_sink_streams; ++sink_idx) {
> +		struct v4l2_subdev_format sink_fmt, source_fmt;
> +		u32 stream;
> +
> +		stream = sink_streams[sink_idx];
> +
> +		for (; source_idx < num_source_streams; ++source_idx) {
> +			if (source_streams[source_idx] == stream)
> +				break;
> +		}
> +
> +		if (source_idx == num_source_streams) {
> +			dev_err(dev, "No source stream for sink stream %u\n",
> +				stream);
> +			ret = -EINVAL;
> +			goto out;
> +		}
> +
> +		dev_dbg(dev, "validating stream \"%s\":%u:%u -> \"%s\":%u:%u\n",
> +			link->source->entity->name, link->source->index, stream,
> +			link->sink->entity->name, link->sink->index, stream);
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->source, stream,
> +							   &source_fmt);
> +		if (ret < 0) {
> +			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +				link->source->entity->name, link->source->index,
> +				stream);
> +			ret = 0;
> +			continue;
> +		}
> +
> +		ret = v4l2_subdev_link_validate_get_format(link->sink, stream,
> +							   &sink_fmt);
> +		if (ret < 0) {
> +			dev_dbg(dev, "Failed to get format for \"%s\":%u:%u (but that's ok)\n",
> +				link->sink->entity->name, link->sink->index,
> +				stream);
> +			ret = 0;
> +			continue;
> +		}
> +
> +		/* TODO: add stream number to link_validate() */
> +		ret = v4l2_subdev_call(sink_subdev, pad, link_validate, link,
> +				       &source_fmt, &sink_fmt);
> +		if (!ret)
> +			continue;
> +
> +		if (ret != -ENOIOCTLCMD)
> +			goto out;
> +
> +		ret = v4l2_subdev_link_validate_default(sink_subdev, link,
> +							&source_fmt, &sink_fmt);
> +
> +		if (ret)
> +			goto out;
> +	}
> +
> +out:
> +	if (sink_allocated)
> +		kfree(sink_streams);
> +
> +free_source:
> +	if (source_allocated)
> +		kfree(source_streams);
> +
> +	return ret;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>  
> 

Regards,

	Hans

Re: [PATCH v9 33/36] media: subdev: add "opposite" stream helper funcs

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add two helper functions to make dealing with streams easier:
> 
> v4l2_state_find_opposite_end - given a routing table and a pad + stream,
> return the pad + stream on the opposite side of the subdev.
> 
> v4l2_state_get_opposite_stream_format - return a pointer to the format
> on the pad + stream on the opposite side from the given pad + stream.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 42 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 32 ++++++++++++++++++++
>  2 files changed, 74 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 37e2e1f907fc..9eeadad997c8 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1484,3 +1484,45 @@ v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>  	return NULL;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_state_get_stream_format);
> +
> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> +				 u32 stream, u32 *other_pad, u32 *other_stream)
> +{
> +	unsigned int i;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		if (route->source_pad == pad &&
> +		    route->source_stream == stream) {
> +			*other_pad = route->sink_pad;
> +			*other_stream = route->sink_stream;
> +			return 0;
> +		}
> +
> +		if (route->sink_pad == pad && route->sink_stream == stream) {
> +			*other_pad = route->source_pad;
> +			*other_stream = route->source_stream;
> +			return 0;
> +		}
> +	}
> +
> +	return -EINVAL;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_state_find_opposite_end);
> +
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> +				      u32 stream)
> +{
> +	u32 other_pad, other_stream;
> +	int ret;
> +
> +	ret = v4l2_state_find_opposite_end(&state->routing, pad, stream,
> +					   &other_pad, &other_stream);
> +	if (ret)
> +		return NULL;
> +
> +	return v4l2_state_get_stream_format(state, other_pad, other_stream);
> +}
> +EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1dc824416c1b..4b9520410783 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1501,4 +1501,36 @@ struct v4l2_mbus_framefmt *
>  v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>  			     u32 stream);
>  
> +/**
> + * v4l2_state_find_opposite_end() - Find the opposite stream
> + * @routing: routing used to find the opposite side
> + * @pad: pad id
> + * @stream: stream id
> + * @other_pad: pointer used to return the opposite pad
> + * @other_stream: pointer used to return the opposite stream
> + *
> + * This function uses the routing table to find the pad + stream which is
> + * opposite the given pad + stream.
> + *
> + * Returns 0 on success, or -EINVAL if no matching route is found.
> + */
> +int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
> +				 u32 stream, u32 *other_pad, u32 *other_stream);
> +
> +/**
> + * v4l2_state_get_opposite_stream_format() - Get pointer to opposite stream
> + *					     format
> + * @state: subdevice state
> + * @pad: pad id
> + * @stream: stream id
> + *
> + * This returns a pointer to &struct v4l2_mbus_framefmt for the pad + stream
> + * that is opposite the given pad + stream in the subdev state.
> + *
> + * If the state does not contain the given pad + stream, NULL is returned.
> + */
> +struct v4l2_mbus_framefmt *
> +v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
> +				      u32 stream);
> +
>  #endif
> 

Re: [PATCH v9 34/36] media: subdev: add v4l2_subdev_get_fmt() helper function

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add v4l2_subdev_get_fmt() helper function which implements
> v4l2_subdev_pad_ops.get_fmt using streams. Subdev drivers that do not
> need to do anything special in their get_fmt op can use this helper
> directly for v4l2_subdev_pad_ops.get_fmt.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 ++++++++++++++++
>  2 files changed, 37 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 9eeadad997c8..d402ffeef560 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1526,3 +1526,24 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
>  	return v4l2_state_get_stream_format(state, other_pad, other_stream);
>  }
>  EXPORT_SYMBOL_GPL(v4l2_state_get_opposite_stream_format);
> +
> +int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> +			struct v4l2_subdev_format *format)
> +{
> +	struct v4l2_mbus_framefmt *fmt;
> +
> +	v4l2_subdev_lock_state(state);
> +
> +	fmt = v4l2_state_get_stream_format(state, format->pad, format->stream);
> +	if (!fmt) {
> +		v4l2_subdev_unlock_state(state);
> +		return -EINVAL;
> +	}
> +
> +	format->format = *fmt;
> +
> +	v4l2_subdev_unlock_state(state);
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 4b9520410783..04fca595aa91 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1532,5 +1532,21 @@ int v4l2_state_find_opposite_end(struct v4l2_subdev_krouting *routing, u32 pad,
>  struct v4l2_mbus_framefmt *
>  v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
>  				      u32 stream);
> +/**
> + * v4l2_subdev_get_fmt() - Fill format based on state
> + * @sd: subdevice
> + * @state: subdevice state
> + * @format: pointer to &struct v4l2_subdev_format
> + *
> + * Fill @format based on the pad and stream given in the @format struct.
> + *
> + * This function can be used by the subdev drivers to implement
> + * v4l2_subdev_pad_ops.get_fmt if the subdev driver does not need to do
> + * anything special in their get_fmt op.
> + *
> + * Returns 0 on success, error value otherwise.
> + */
> +int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
> +			struct v4l2_subdev_format *format);
>  
>  #endif
> 

Re: [PATCH v9 35/36] media: subdev: add v4l2_subdev_set_routing_with_fmt() helper

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> v4l2_subdev_set_routing_with_fmt() is the same as
> v4l2_subdev_set_routing(), but additionally initializes all the streams
> with the given format.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 22 ++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 16 ++++++++++++++++
>  2 files changed, 38 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index d402ffeef560..22a9bea0fa85 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1464,6 +1464,28 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
>  
> +int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_state *state,
> +				     struct v4l2_subdev_krouting *routing,
> +				     const struct v4l2_mbus_framefmt *fmt)
> +{
> +	struct v4l2_subdev_stream_configs *stream_configs;
> +	unsigned int i;
> +	int ret;
> +
> +	ret = v4l2_subdev_set_routing(sd, state, routing);
> +	if (ret)
> +		return ret;
> +
> +	stream_configs = &state->stream_configs;
> +
> +	for (i = 0; i < stream_configs->num_configs; ++i)
> +		stream_configs->configs[i].fmt = *fmt;
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing_with_fmt);
> +
>  struct v4l2_mbus_framefmt *
>  v4l2_state_get_stream_format(struct v4l2_subdev_state *state, unsigned int pad,
>  			     u32 stream)
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 04fca595aa91..1a4df0aafe8a 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1486,6 +1486,22 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
>  			    struct v4l2_subdev_state *state,
>  			    struct v4l2_subdev_krouting *routing);
>  
> +/**
> + * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
> + *					state
> + * @sd: The subdevice
> + * @state: The subdevice state
> + * @routing: Routing that will be copied to subdev state
> + * @fmt: Format used to initialize all the streams
> + *
> + * This is the same as v4l2_subdev_set_routing, but additionally initializes
> + * all the streams using the given format.
> + */
> +int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
> +				     struct v4l2_subdev_state *state,
> +				     struct v4l2_subdev_krouting *routing,
> +				     const struct v4l2_mbus_framefmt *fmt);
> +
>  /**
>   * v4l2_state_get_stream_format() - Get pointer to a stream format
>   * @state: subdevice state
> 

Re: [PATCH v9 36/36] media: subdev: add v4l2_routing_simple_verify() helper

[Hans Verkuil]

On 05/10/2021 10:57, Tomi Valkeinen wrote:
> Add a helper for verifying routing for the common case of
> non-overlapping 1-to-1 streams.
> 
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>

Reviewed-by: Hans Verkuil <hverkuil-cisco@xs4all.nl>

> ---
>  drivers/media/v4l2-core/v4l2-subdev.c | 24 ++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 14 ++++++++++++++
>  2 files changed, 38 insertions(+)
> 
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 22a9bea0fa85..2a64ff003e4b 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -1569,3 +1569,27 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  	return 0;
>  }
>  EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
> +
> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
> +{
> +	unsigned int i, j;
> +
> +	for (i = 0; i < routing->num_routes; ++i) {
> +		const struct v4l2_subdev_route *route = &routing->routes[i];
> +
> +		for (j = i + 1; j < routing->num_routes; ++j) {
> +			const struct v4l2_subdev_route *r = &routing->routes[j];
> +
> +			if (route->sink_pad == r->sink_pad &&
> +			    route->sink_stream == r->sink_stream)
> +				return -EINVAL;
> +
> +			if (route->source_pad == r->source_pad &&
> +			    route->source_stream == r->source_stream)
> +				return -EINVAL;
> +		}
> +	}
> +
> +	return 0;
> +}
> +EXPORT_SYMBOL_GPL(v4l2_routing_simple_verify);
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 1a4df0aafe8a..5e50f2ded653 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -1565,4 +1565,18 @@ v4l2_state_get_opposite_stream_format(struct v4l2_subdev_state *state, u32 pad,
>  int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
>  			struct v4l2_subdev_format *format);
>  
> +/**
> + * v4l2_routing_simple_verify() - Verify that all streams are non-overlapping
> + *				  1-to-1 streams
> + * @routing: routing to verify
> + *
> + * This verifies that the given routing contains only non-overlapping 1-to-1
> + * streams. In other words, no two streams have the same source or sink
> + * stream ID on a single pad. This is the most common case of routing
> + * supported by devices.
> + *
> + * Returns 0 on success, error value otherwise.
> + */
> +int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
> +
>  #endif
> 

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

Hi Tomi,

On Tue, Oct 05, 2021 at 11:57:41AM +0300, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
>
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
>
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
>
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>
>
> - Add sink and source streams for multiplexed links
> - Copy the argument back in case of an error. This is needed to let the
>   caller know the number of routes.
>
> Signed-off-by: Sakari Ailus <sakari.ailus@linux.intel.com>
>
> - Expand and refine documentation.
> - Make the 'routes' pointer a __u64 __user pointer so that a compat32
>   version of the ioctl is not required.
> - Add struct v4l2_subdev_krouting to be used for subdevice operations.
>
> Signed-off-by: Jacopo Mondi <jacopo+renesas@jmondi.org>
>
> - Fix typecasing warnings
> - Check sink & source pad types
> - Add 'which' field
> - Add V4L2_SUBDEV_ROUTE_FL_SOURCE
> - Routing to subdev state
> - Dropped get_routing subdev op
>
> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
> ---
>  drivers/media/v4l2-core/v4l2-ioctl.c  | 25 ++++++++-
>  drivers/media/v4l2-core/v4l2-subdev.c | 75 +++++++++++++++++++++++++++
>  include/media/v4l2-subdev.h           | 22 ++++++++
>  include/uapi/linux/v4l2-subdev.h      | 57 ++++++++++++++++++++
>  4 files changed, 178 insertions(+), 1 deletion(-)
>
> diff --git a/drivers/media/v4l2-core/v4l2-ioctl.c b/drivers/media/v4l2-core/v4l2-ioctl.c
> index 05d5db3d85e5..8e9315ffcb99 100644
> --- a/drivers/media/v4l2-core/v4l2-ioctl.c
> +++ b/drivers/media/v4l2-core/v4l2-ioctl.c
> @@ -16,6 +16,7 @@
>  #include <linux/kernel.h>
>  #include <linux/version.h>
>
> +#include <linux/v4l2-subdev.h>
>  #include <linux/videodev2.h>
>
>  #include <media/v4l2-common.h>
> @@ -3065,6 +3066,21 @@ static int check_array_args(unsigned int cmd, void *parg, size_t *array_size,
>  		ret = 1;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = parg;
> +
> +		if (routing->num_routes > 256)
> +			return -EINVAL;
> +
> +		*user_ptr = u64_to_user_ptr(routing->routes);
> +		*kernel_ptr = (void **)&routing->routes;
> +		*array_size = sizeof(struct v4l2_subdev_route)
> +			    * routing->num_routes;
> +		ret = 1;
> +		break;
> +	}
>  	}
>
>  	return ret;
> @@ -3328,8 +3344,15 @@ video_usercopy(struct file *file, unsigned int orig_cmd, unsigned long arg,
>  	/*
>  	 * Some ioctls can return an error, but still have valid
>  	 * results that must be returned.
> +	 *
> +	 * FIXME: subdev IOCTLS are partially handled here and partially in
> +	 * v4l2-subdev.c and the 'always_copy' flag can only be set for IOCTLS
> +	 * defined here as part of the 'v4l2_ioctls' array. As
> +	 * VIDIOC_SUBDEV_G_ROUTING needs to return results to applications even
> +	 * in case of failure, but it is not defined here as part of the
> +	 * 'v4l2_ioctls' array, insert an ad-hoc check to address that.
>  	 */
> -	if (err < 0 && !always_copy)
> +	if (err < 0 && !always_copy && cmd != VIDIOC_SUBDEV_G_ROUTING)
>  		goto out;
>
>  out_array_args:
> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
> index 52309a299b03..b4e1f8772d96 100644
> --- a/drivers/media/v4l2-core/v4l2-subdev.c
> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
> @@ -394,6 +394,12 @@ subdev_ioctl_get_state(struct v4l2_subdev *sd, struct v4l2_subdev_fh *subdev_fh,
>  		which = ((struct v4l2_subdev_selection *)arg)->which;
>  		break;
>  	}
> +
> +	case VIDIOC_SUBDEV_G_ROUTING:
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		which = ((struct v4l2_subdev_routing *)arg)->which;
> +		break;
> +	}
>  	}
>
>  	return which == V4L2_SUBDEV_FORMAT_TRY ?
> @@ -710,6 +716,74 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
>  	case VIDIOC_SUBDEV_QUERYSTD:
>  		return v4l2_subdev_call(sd, video, querystd, arg);
>
> +	case VIDIOC_SUBDEV_G_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_krouting *krouting;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		krouting = &state->routing;
> +
> +		if (routing->num_routes < krouting->num_routes) {
> +			routing->num_routes = krouting->num_routes;
> +			return -ENOSPC;
> +		}
> +
> +		memcpy((struct v4l2_subdev_route *)(uintptr_t)routing->routes,
> +		       krouting->routes,
> +		       krouting->num_routes * sizeof(*krouting->routes));
> +		routing->num_routes = krouting->num_routes;
> +
> +		return 0;
> +	}
> +
> +	case VIDIOC_SUBDEV_S_ROUTING: {
> +		struct v4l2_subdev_routing *routing = arg;
> +		struct v4l2_subdev_route *routes =
> +			(struct v4l2_subdev_route *)(uintptr_t)routing->routes;
> +		struct v4l2_subdev_krouting krouting = {};
> +		unsigned int i;
> +
> +		if (!(sd->flags & V4L2_SUBDEV_FL_MULTIPLEXED))
> +			return -ENOIOCTLCMD;
> +
> +		if (routing->which != V4L2_SUBDEV_FORMAT_TRY && ro_subdev)
> +			return -EPERM;
> +
> +		memset(routing->reserved, 0, sizeof(routing->reserved));
> +
> +		for (i = 0; i < routing->num_routes; ++i) {
> +			const struct v4l2_subdev_route *route = &routes[i];
> +			const struct media_pad *pads = sd->entity.pads;
> +
> +			/* Do not check sink pad for source routes */
> +			if (!(route->flags & V4L2_SUBDEV_ROUTE_FL_SOURCE)) {
> +				if (route->sink_pad >= sd->entity.num_pads)
> +					return -EINVAL;
> +
> +				if (!(pads[route->sink_pad].flags &
> +				      MEDIA_PAD_FL_SINK))
> +					return -EINVAL;
> +			}
> +
> +			if (route->source_pad >= sd->entity.num_pads)
> +				return -EINVAL;
> +
> +			if (!(pads[route->source_pad].flags &
> +			      MEDIA_PAD_FL_SOURCE))
> +				return -EINVAL;
> +		}
> +
> +		krouting.num_routes = routing->num_routes;
> +		krouting.routes = routes;
> +
> +		return v4l2_subdev_call(sd, pad, set_routing, state,
> +					routing->which, &krouting);
> +	}
> +
>  	default:
>  		return v4l2_subdev_call(sd, core, ioctl, cmd, arg);
>  	}
> @@ -970,6 +1044,7 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>
>  	mutex_destroy(&state->lock);
>
> +	kvfree(state->routing.routes);
>  	kvfree(state->pads);
>  	kfree(state);
>  }
> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
> index 7f9c83fac020..f318670a49c8 100644
> --- a/include/media/v4l2-subdev.h
> +++ b/include/media/v4l2-subdev.h
> @@ -694,11 +694,25 @@ struct v4l2_subdev_pad_config {
>  	struct v4l2_rect try_compose;
>  };
>
> +/**
> + * struct v4l2_subdev_krouting - subdev routing table
> + *
> + * @num_routes: number of routes
> + * @routes: &struct v4l2_subdev_route
> + *
> + * This structure contains the routing table for a subdev.
> + */
> +struct v4l2_subdev_krouting {
> +	unsigned int num_routes;
> +	struct v4l2_subdev_route *routes;
> +};

Re-reading v8
https://patchwork.linuxtv.org/project/linux-media/patch/20210830110116.488338-3-tomi.valkeinen@ideasonboard.com/#131630

my understanding was that your intention was to pass to the
set_routing() op the num_routes, *routes and which and drop
v4l2_subdev_krouting

> +
>  /**
>   * struct v4l2_subdev_state - Used for storing subdev state information.
>   *
>   * @lock: mutex for the state
>   * @pads: &struct v4l2_subdev_pad_config array
> + * @routing: routing table for the subdev
>   *
>   * This structure only needs to be passed to the pad op if the 'which' field
>   * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> @@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
>  struct v4l2_subdev_state {
>  	struct mutex lock;
>  	struct v4l2_subdev_pad_config *pads;
> +	struct v4l2_subdev_krouting routing;
>  };
>
>  /**
> @@ -770,6 +785,9 @@ struct v4l2_subdev_state {
>   *		     applied to the hardware. The operation shall fail if the
>   *		     pad index it has been called on is not valid or in case of
>   *		     unrecoverable failures.
> + *
> + * @set_routing: enable or disable data connection routes described in the
> + *		 subdevice routing table.
>   */
>  struct v4l2_subdev_pad_ops {
>  	int (*init_cfg)(struct v4l2_subdev *sd,
> @@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
>  			       struct v4l2_mbus_config *config);
>  	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>  			       struct v4l2_mbus_config *config);
> +	int (*set_routing)(struct v4l2_subdev *sd,
> +			   struct v4l2_subdev_state *state,
> +			   enum v4l2_subdev_format_whence which,
> +			   struct v4l2_subdev_krouting *route);

But it seems you've instead coalesced v4l2_subdev_routing_table into
v4l2_subdev_krouting (which I like!)

But why keep 'which' as a standalone parameter instead of adding it to
v4l2_subdev_krouting ? This operation signature is a bit of an
outliner as other ops have 'which' embedded in op-specific
structures.

Thanks
  j

>  };
>
>  /**
> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..5ba409db47ff 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>  /* The v4l2 sub-device video device node is registered in read-only mode. */
>  #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)
> +
> +/**
> + * struct v4l2_subdev_route - A route inside a subdev
> + *
> + * @sink_pad: the sink pad index
> + * @sink_stream: the sink stream identifier
> + * @source_pad: the source pad index
> + * @source_stream: the source stream identifier
> + * @flags: route flags V4L2_SUBDEV_ROUTE_FL_*
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_route {
> +	__u32 sink_pad;
> +	__u32 sink_stream;
> +	__u32 source_pad;
> +	__u32 source_stream;
> +	__u32 flags;
> +	__u32 reserved[5];
> +};
> +
> +/**
> + * struct v4l2_subdev_routing - Subdev routing information
> + *
> + * @which: configuration type (from enum v4l2_subdev_format_whence)
> + * @routes: pointer to the routes array
> + * @num_routes: the total number of routes in the routes array
> + * @reserved: drivers and applications must zero this array
> + */
> +struct v4l2_subdev_routing {
> +	__u32 which;
> +	__u64 routes;
> +	__u32 num_routes;
> +	__u32 reserved[5];
> +};
> +
>  /* Backwards compatibility define --- to be removed */
>  #define v4l2_subdev_edid v4l2_edid
>
> @@ -203,6 +258,8 @@ struct v4l2_subdev_capability {
>  #define VIDIOC_SUBDEV_S_CROP			_IOWR('V', 60, struct v4l2_subdev_crop)
>  #define VIDIOC_SUBDEV_G_SELECTION		_IOWR('V', 61, struct v4l2_subdev_selection)
>  #define VIDIOC_SUBDEV_S_SELECTION		_IOWR('V', 62, struct v4l2_subdev_selection)
> +#define VIDIOC_SUBDEV_G_ROUTING			_IOWR('V', 38, struct v4l2_subdev_routing)
> +#define VIDIOC_SUBDEV_S_ROUTING			_IOWR('V', 39, struct v4l2_subdev_routing)
>  /* The following ioctls are identical to the ioctls in videodev2.h */
>  #define VIDIOC_SUBDEV_G_STD			_IOR('V', 23, v4l2_std_id)
>  #define VIDIOC_SUBDEV_S_STD			_IOW('V', 24, v4l2_std_id)
> --
> 2.25.1
>

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 15/10/2021 15:13, Jacopo Mondi wrote:

>> +/**
>> + * struct v4l2_subdev_krouting - subdev routing table
>> + *
>> + * @num_routes: number of routes
>> + * @routes: &struct v4l2_subdev_route
>> + *
>> + * This structure contains the routing table for a subdev.
>> + */
>> +struct v4l2_subdev_krouting {
>> +	unsigned int num_routes;
>> +	struct v4l2_subdev_route *routes;
>> +};
> 
> Re-reading v8
> https://patchwork.linuxtv.org/project/linux-media/patch/20210830110116.488338-3-tomi.valkeinen@ideasonboard.com/#131630
> 
> my understanding was that your intention was to pass to the
> set_routing() op the num_routes, *routes and which and drop
> v4l2_subdev_krouting

No, that was not what I meant. But I did write it a bit unclearly, I 
admit...

I think I meant "drop v4l2_subdev_krouting, and pass 
v4l2_subdev_routing_table to the ops".

>> +
>>   /**
>>    * struct v4l2_subdev_state - Used for storing subdev state information.
>>    *
>>    * @lock: mutex for the state
>>    * @pads: &struct v4l2_subdev_pad_config array
>> + * @routing: routing table for the subdev
>>    *
>>    * This structure only needs to be passed to the pad op if the 'which' field
>>    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
>> @@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
>>   struct v4l2_subdev_state {
>>   	struct mutex lock;
>>   	struct v4l2_subdev_pad_config *pads;
>> +	struct v4l2_subdev_krouting routing;
>>   };
>>
>>   /**
>> @@ -770,6 +785,9 @@ struct v4l2_subdev_state {
>>    *		     applied to the hardware. The operation shall fail if the
>>    *		     pad index it has been called on is not valid or in case of
>>    *		     unrecoverable failures.
>> + *
>> + * @set_routing: enable or disable data connection routes described in the
>> + *		 subdevice routing table.
>>    */
>>   struct v4l2_subdev_pad_ops {
>>   	int (*init_cfg)(struct v4l2_subdev *sd,
>> @@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
>>   			       struct v4l2_mbus_config *config);
>>   	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
>>   			       struct v4l2_mbus_config *config);
>> +	int (*set_routing)(struct v4l2_subdev *sd,
>> +			   struct v4l2_subdev_state *state,
>> +			   enum v4l2_subdev_format_whence which,
>> +			   struct v4l2_subdev_krouting *route);
> 
> But it seems you've instead coalesced v4l2_subdev_routing_table into
> v4l2_subdev_krouting (which I like!)
> 
> But why keep 'which' as a standalone parameter instead of adding it to
> v4l2_subdev_krouting ? This operation signature is a bit of an
> outliner as other ops have 'which' embedded in op-specific
> structures.

The whole point here was to remove 'which'. Otherwise the routing table 
contains 'which', and the drivers somehow need to come up with a correct 
'which' value, which was causing trouble.

E.g. in init_cfg, you want to set up routing. But init_cfg is supposed 
to be 'which-less', the init should be the same for both active and try 
states.

The only use of 'which' in the routing table is when calling 
set_routing(). It's never needed after that. So it's pointless to store 
the 'which' value in the state.

The other ops have a struct as a parameter that comes directly from the 
userspace, and thus contains the 'which'. But any data they store to the 
state does not contain 'which'.

So, we could do the same, have the structs as they are in the mail you 
linked. But then we have the struct v4l2_subdev_krouting, which is only 
used when calling set_routing, and the only purpose of that struct would 
be to avoid passing 'which' as a parameter to set_routing...

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

[-- Attachment #1: Type: text/plain, Size: 6859 bytes --]

Hi Tomi,

On Fri, Oct 15, 2021 at 03:36:58PM +0300, Tomi Valkeinen wrote:
> Hi,
>
> On 15/10/2021 15:13, Jacopo Mondi wrote:
>
> > > +/**
> > > + * struct v4l2_subdev_krouting - subdev routing table
> > > + *
> > > + * @num_routes: number of routes
> > > + * @routes: &struct v4l2_subdev_route
> > > + *
> > > + * This structure contains the routing table for a subdev.
> > > + */
> > > +struct v4l2_subdev_krouting {
> > > +	unsigned int num_routes;
> > > +	struct v4l2_subdev_route *routes;
> > > +};
> >
> > Re-reading v8
> > https://patchwork.linuxtv.org/project/linux-media/patch/20210830110116.488338-3-tomi.valkeinen@ideasonboard.com/#131630
> >
> > my understanding was that your intention was to pass to the
> > set_routing() op the num_routes, *routes and which and drop
> > v4l2_subdev_krouting
>
> No, that was not what I meant. But I did write it a bit unclearly, I
> admit...
>
> I think I meant "drop v4l2_subdev_krouting, and pass
> v4l2_subdev_routing_table to the ops".
>
> > > +
> > >   /**
> > >    * struct v4l2_subdev_state - Used for storing subdev state information.
> > >    *
> > >    * @lock: mutex for the state
> > >    * @pads: &struct v4l2_subdev_pad_config array
> > > + * @routing: routing table for the subdev
> > >    *
> > >    * This structure only needs to be passed to the pad op if the 'which' field
> > >    * of the main argument is set to %V4L2_SUBDEV_FORMAT_TRY. For
> > > @@ -707,6 +721,7 @@ struct v4l2_subdev_pad_config {
> > >   struct v4l2_subdev_state {
> > >   	struct mutex lock;
> > >   	struct v4l2_subdev_pad_config *pads;
> > > +	struct v4l2_subdev_krouting routing;
> > >   };
> > >
> > >   /**
> > > @@ -770,6 +785,9 @@ struct v4l2_subdev_state {
> > >    *		     applied to the hardware. The operation shall fail if the
> > >    *		     pad index it has been called on is not valid or in case of
> > >    *		     unrecoverable failures.
> > > + *
> > > + * @set_routing: enable or disable data connection routes described in the
> > > + *		 subdevice routing table.
> > >    */
> > >   struct v4l2_subdev_pad_ops {
> > >   	int (*init_cfg)(struct v4l2_subdev *sd,
> > > @@ -814,6 +832,10 @@ struct v4l2_subdev_pad_ops {
> > >   			       struct v4l2_mbus_config *config);
> > >   	int (*set_mbus_config)(struct v4l2_subdev *sd, unsigned int pad,
> > >   			       struct v4l2_mbus_config *config);
> > > +	int (*set_routing)(struct v4l2_subdev *sd,
> > > +			   struct v4l2_subdev_state *state,
> > > +			   enum v4l2_subdev_format_whence which,
> > > +			   struct v4l2_subdev_krouting *route);
> >
> > But it seems you've instead coalesced v4l2_subdev_routing_table into
> > v4l2_subdev_krouting (which I like!)
> >
> > But why keep 'which' as a standalone parameter instead of adding it to
> > v4l2_subdev_krouting ? This operation signature is a bit of an
> > outliner as other ops have 'which' embedded in op-specific
> > structures.
>
> The whole point here was to remove 'which'. Otherwise the routing table
> contains 'which', and the drivers somehow need to come up with a correct
> 'which' value, which was causing trouble.
>
> E.g. in init_cfg, you want to set up routing. But init_cfg is supposed to be
> 'which-less', the init should be the same for both active and try states.
>
> The only use of 'which' in the routing table is when calling set_routing().
> It's never needed after that. So it's pointless to store the 'which' value
> in the state.
>
> The other ops have a struct as a parameter that comes directly from the
> userspace, and thus contains the 'which'. But any data they store to the
> state does not contain 'which'.
>
> So, we could do the same, have the structs as they are in the mail you
> linked. But then we have the struct v4l2_subdev_krouting, which is only used
> when calling set_routing, and the only purpose of that struct would be to
> avoid passing 'which' as a parameter to set_routing...

Ok, I understand this allow you to

        int _set_routing(sd, state, krouting)
        {
                v4l2_subdev_set_routing(krouting);

        }

        int set_routing(sd, state, which, krouting)
        {
                _set_routing(sd, state, krouting);
                if (which == ACTIVE)
                        apply_to_hw();
        }

        pad_ops ops = {
                .set_routing = set_routing,
        };

        int init_cfg(sd, state)
        {
                routes = {
                        ...
                };

                krouting routing {
                        .routes = routes,
                        .num_routes = 1,
                };

                _set_routing(sd, state, &routing);
        }

We're moving the issue with init_cfg() not being aware of the
active/try to the set_routing() operation signature (and in cascade to
all operation that operate on krouting from where 'which' had been
removed to comply with init_cfg).

I was actually advocating instead for adding back which to
krouting and have all the other v4l2_subdev functions that operates on
routing and do not care about 'which' to use the raw routes and their
number


        --- v4l2-subdev.h/c ---

        struct krouting {
                which;
                num_routes;
                routes;
        };

        int v4l2_subdev_set_routing(sd, state, routes, num_routes)
        {

                /* Copy routes into state */

        }

        ---- subdevice driver ----

        int _set_routing(sd, state, routes, num_routes)
        {
                return v4l2_subdev_set_routing(sd, state, routes,
                                               num_routes);

        }

        int set_routing(sd, state, krouting)
        {
                _set_routing(sd, state, krouting->routes, krouting->num_routes)

                if (krouting->which == ACTIVE)
                        apply_to_hw();
        }

        pad_ops ops = {
                .set_routing = set_routing,
        };

        int init_cfg(sd, state)
        {
                routes routing[] = {
                        ...
                };

                _set_routing(sd, state, routing, ARRAY_SIZE(routing));
        }

So that
- krouting contains 'which' but is only used in the set_routing
  operation
- init_cfg doesn't have to initialize a krouting just for the purpose
  of pleasing the v4l2_subdev function signatures

It requires to remove krouting from all the v4l2_subdev_*route*()
functions in v4l2_subdev.c and have them work on the raw routes
('routes' and 'num_routes' could as well be grouped in one structure
like you had in v8).

As you asked for opinions and I failed to provide one in v8 it
wouldn't be fair to ask you to backtrack just to see how it looks
like, so here you have a patch to be applied on top of your branch for
you to take into consideration (test on my gmsl work on top of your
v9).

Thanks
   j


>
>  Tomi

[-- Attachment #2: tomi_v9_krouting-which.patch --]
[-- Type: text/plain, Size: 12713 bytes --]

diff --git a/drivers/media/i2c/ds90ub913.c b/drivers/media/i2c/ds90ub913.c
index d675faa2c571..e6c7ca7941d5 100644
--- a/drivers/media/i2c/ds90ub913.c
+++ b/drivers/media/i2c/ds90ub913.c
@@ -267,7 +267,8 @@ static const struct v4l2_subdev_video_ops ub913_video_ops = {
 
 static int _ub913_set_routing(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *state,
-			     struct v4l2_subdev_krouting *routing)
+			     struct v4l2_subdev_route *routes,
+			     unsigned int num_routes)
 {
 	const struct v4l2_mbus_framefmt format = {
 		.width = 640,
@@ -286,16 +287,17 @@ static int _ub913_set_routing(struct v4l2_subdev *sd,
 	 * frame desc is made dynamically allocated.
 	 */
 
-	if (routing->num_routes > V4L2_FRAME_DESC_ENTRY_MAX)
+	if (num_routes > V4L2_FRAME_DESC_ENTRY_MAX)
 		return -EINVAL;
 
-	ret = v4l2_routing_simple_verify(routing);
+	ret = v4l2_routing_simple_verify(routes, num_routes);
 	if (ret)
 		return ret;
 
 	v4l2_subdev_lock_state(state);
 
-	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routing, &format);
+	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routes, num_routes,
+					       &format);
 
 	v4l2_subdev_unlock_state(state);
 
@@ -307,15 +309,14 @@ static int _ub913_set_routing(struct v4l2_subdev *sd,
 
 static int ub913_set_routing(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *state,
-			     enum v4l2_subdev_format_whence which,
 			     struct v4l2_subdev_krouting *routing)
 {
 	struct ub913_data *priv = sd_to_ub913(sd);
 
-	if (which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->streaming)
+	if (routing->which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->streaming)
 		return -EBUSY;
 
-	return _ub913_set_routing(sd, state, routing);
+	return _ub913_set_routing(sd, state, routing->routes, routing->num_routes);
 }
 
 static int ub913_get_source_frame_desc(struct ub913_data *priv,
@@ -454,12 +455,7 @@ static int ub913_init_cfg(struct v4l2_subdev *sd,
 		},
 	};
 
-	struct v4l2_subdev_krouting routing = {
-		.num_routes = ARRAY_SIZE(routes),
-		.routes = routes,
-	};
-
-	return _ub913_set_routing(sd, state, &routing);
+	return _ub913_set_routing(sd, state, routes, 1);
 }
 
 static const struct v4l2_subdev_pad_ops ub913_pad_ops = {
diff --git a/drivers/media/i2c/ds90ub960.c b/drivers/media/i2c/ds90ub960.c
index c655cb3e1ad6..1a3020e6c171 100644
--- a/drivers/media/i2c/ds90ub960.c
+++ b/drivers/media/i2c/ds90ub960.c
@@ -1443,7 +1443,8 @@ static const struct v4l2_subdev_video_ops ub960_video_ops = {
 
 static int _ub960_set_routing(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *state,
-			     struct v4l2_subdev_krouting *routing)
+			     struct v4l2_subdev_route *routes,
+			     unsigned int num_routes)
 {
 	const struct v4l2_mbus_framefmt format = {
 		.width = 640,
@@ -1462,16 +1463,17 @@ static int _ub960_set_routing(struct v4l2_subdev *sd,
 	 * frame desc is made dynamically allocated.
 	 */
 
-	if (routing->num_routes > V4L2_FRAME_DESC_ENTRY_MAX)
+	if (num_routes > V4L2_FRAME_DESC_ENTRY_MAX)
 		return -EINVAL;
 
-	ret = v4l2_routing_simple_verify(routing);
+	ret = v4l2_routing_simple_verify(routes, num_routes);
 	if (ret)
 		return ret;
 
 	v4l2_subdev_lock_state(state);
 
-	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routing, &format);
+	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routes, num_routes,
+					       &format);
 
 	v4l2_subdev_unlock_state(state);
 
@@ -1483,15 +1485,15 @@ static int _ub960_set_routing(struct v4l2_subdev *sd,
 
 static int ub960_set_routing(struct v4l2_subdev *sd,
 			     struct v4l2_subdev_state *state,
-			     enum v4l2_subdev_format_whence which,
 			     struct v4l2_subdev_krouting *routing)
 {
 	struct ub960_data *priv = sd_to_ub960(sd);
 
-	if (which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->streaming)
+	if (routing->which == V4L2_SUBDEV_FORMAT_ACTIVE && priv->streaming)
 		return -EBUSY;
 
-	return _ub960_set_routing(sd, state, routing);
+	return _ub960_set_routing(sd, state, routing->routes,
+				  routing->num_routes);
 }
 
 static int ub960_get_source_frame_desc(struct ub960_data *priv,
@@ -1676,12 +1678,7 @@ static int ub960_init_cfg(struct v4l2_subdev *sd,
 		},
 	};
 
-	struct v4l2_subdev_krouting routing = {
-		.num_routes = ARRAY_SIZE(routes),
-		.routes = routes,
-	};
-
-	return _ub960_set_routing(sd, state, &routing);
+	return _ub960_set_routing(sd, state, routes, 1);
 }
 
 static const struct v4l2_subdev_pad_ops ub960_pad_ops = {
diff --git a/drivers/media/i2c/ov1063x.c b/drivers/media/i2c/ov1063x.c
index 2f1d0c2c7cd5..fb76a67f9f55 100644
--- a/drivers/media/i2c/ov1063x.c
+++ b/drivers/media/i2c/ov1063x.c
@@ -1205,18 +1205,12 @@ static int _ov10635_set_routing(struct v4l2_subdev *sd,
 				 V4L2_SUBDEV_ROUTE_FL_SOURCE,
 		}
 	};
-
-	struct v4l2_subdev_krouting routing = {
-		.num_routes = ARRAY_SIZE(routes),
-		.routes = routes,
-	};
-
 	int ret;
 
 	if (enable_embedded_data)
 		routes[1].flags |= V4L2_SUBDEV_ROUTE_FL_ACTIVE;
 
-	ret = v4l2_subdev_set_routing(sd, state, &routing);
+	ret = v4l2_subdev_set_routing(sd, state, routes, ARRAY_SIZE(routes));
 	if (ret)
 		return ret;
 
@@ -1467,7 +1461,6 @@ static int ov1063x_get_frame_desc(struct v4l2_subdev *sd, unsigned int pad,
 
 static int ov1063x_set_routing(struct v4l2_subdev *sd,
 			       struct v4l2_subdev_state *state,
-			       enum v4l2_subdev_format_whence which,
 			       struct v4l2_subdev_krouting *routing)
 {
 	bool enable_embedded;
diff --git a/drivers/media/platform/ti-vpe/cal-camerarx.c b/drivers/media/platform/ti-vpe/cal-camerarx.c
index 3ccd06d37924..682fea8110da 100644
--- a/drivers/media/platform/ti-vpe/cal-camerarx.c
+++ b/drivers/media/platform/ti-vpe/cal-camerarx.c
@@ -836,8 +836,8 @@ static int cal_camerarx_sd_set_fmt(struct v4l2_subdev *sd,
 }
 
 static int _cal_camerarx_sd_set_routing(struct v4l2_subdev *sd,
-				       struct v4l2_subdev_state *state,
-				       struct v4l2_subdev_krouting *routing)
+					struct v4l2_subdev_route *routes,
+					unsigned int num_routes)
 {
 	static const struct v4l2_mbus_framefmt format = {
 		.width = 640,
@@ -849,9 +849,10 @@ static int _cal_camerarx_sd_set_routing(struct v4l2_subdev *sd,
 		.quantization = V4L2_QUANTIZATION_LIM_RANGE,
 		.xfer_func = V4L2_XFER_FUNC_SRGB,
 	};
+	struct v4l2_subdev_state *state;
 	int ret;
 
-	ret = v4l2_routing_simple_verify(routing);
+	ret = v4l2_routing_simple_verify(routes, num_routes);
 	if (ret)
 		return ret;
 
@@ -859,7 +860,7 @@ static int _cal_camerarx_sd_set_routing(struct v4l2_subdev *sd,
 
 	v4l2_subdev_lock_state(state);
 
-	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routing, &format);
+	ret = v4l2_subdev_set_routing_with_fmt(sd, state, routes, num_routes, &format);
 
 	v4l2_subdev_unlock_state(state);
 
@@ -871,10 +872,10 @@ static int _cal_camerarx_sd_set_routing(struct v4l2_subdev *sd,
 
 static int cal_camerarx_sd_set_routing(struct v4l2_subdev *sd,
 				       struct v4l2_subdev_state *state,
-				       enum v4l2_subdev_format_whence which,
 				       struct v4l2_subdev_krouting *routing)
 {
-	return _cal_camerarx_sd_set_routing(sd, state, routing);
+	return _cal_camerarx_sd_set_routing(sd, state, routing->routes,
+					    routing->num_routes);
 }
 
 static int cal_camerarx_sd_init_cfg(struct v4l2_subdev *sd,
@@ -888,13 +889,8 @@ static int cal_camerarx_sd_init_cfg(struct v4l2_subdev *sd,
 		.flags = V4L2_SUBDEV_ROUTE_FL_ACTIVE,
 	} };
 
-	struct v4l2_subdev_krouting routing = {
-		.num_routes = 1,
-		.routes = routes,
-	};
-
 	/* Initialize routing to single route to the fist source pad */
-	return _cal_camerarx_sd_set_routing(sd, state, &routing);
+	return _cal_camerarx_sd_set_routing(sd, state, routes, 1);
 }
 
 static const struct v4l2_subdev_video_ops cal_camerarx_video_ops = {
diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
index a325257590a1..4be91cbd72d5 100644
--- a/drivers/media/v4l2-core/v4l2-subdev.c
+++ b/drivers/media/v4l2-core/v4l2-subdev.c
@@ -817,11 +817,11 @@ static long subdev_do_ioctl(struct file *file, unsigned int cmd, void *arg)
 				return -EINVAL;
 		}
 
+		krouting.which = routing->which;
 		krouting.num_routes = routing->num_routes;
 		krouting.routes = routes;
 
-		return v4l2_subdev_call(sd, pad, set_routing, state,
-					routing->which, &krouting);
+		return v4l2_subdev_call(sd, pad, set_routing, state, &krouting);
 	}
 
 	default:
@@ -1441,10 +1441,10 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_unlock_state);
 
 int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
-			    struct v4l2_subdev_krouting *routing)
+			    struct v4l2_subdev_route *routes,
+			    unsigned int num_routes)
 {
 	struct v4l2_subdev_krouting *dst = &state->routing;
-	const struct v4l2_subdev_krouting *src = routing;
 
 	lockdep_assert_held(&state->lock);
 
@@ -1452,15 +1452,14 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 	dst->routes = NULL;
 	dst->num_routes = 0;
 
-	if (src->num_routes > 0) {
-		dst->routes = kvmalloc_array(src->num_routes,
-					     sizeof(*src->routes), GFP_KERNEL);
+	if (num_routes > 0) {
+		dst->routes = kvmalloc_array(num_routes,
+					     sizeof(routes[0]), GFP_KERNEL);
 		if (!dst->routes)
 			return -ENOMEM;
 
-		memcpy(dst->routes, src->routes,
-		       src->num_routes * sizeof(*src->routes));
-		dst->num_routes = src->num_routes;
+		memcpy(dst->routes, routes, num_routes * sizeof(routes[0]));
+		dst->num_routes = num_routes;
 	}
 
 	return v4l2_init_stream_configs(&state->stream_configs, dst);
@@ -1469,14 +1468,15 @@ EXPORT_SYMBOL_GPL(v4l2_subdev_set_routing);
 
 int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
 				     struct v4l2_subdev_state *state,
-				     struct v4l2_subdev_krouting *routing,
+				     struct v4l2_subdev_route *routes,
+				     unsigned int num_routes,
 				     const struct v4l2_mbus_framefmt *fmt)
 {
 	struct v4l2_subdev_stream_configs *stream_configs;
 	unsigned int i;
 	int ret;
 
-	ret = v4l2_subdev_set_routing(sd, state, routing);
+	ret = v4l2_subdev_set_routing(sd, state, routes, num_routes);
 	if (ret)
 		return ret;
 
@@ -1573,15 +1573,16 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
 }
 EXPORT_SYMBOL_GPL(v4l2_subdev_get_fmt);
 
-int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing)
+int v4l2_routing_simple_verify(const struct v4l2_subdev_route *routes,
+			       unsigned int num_routes)
 {
 	unsigned int i, j;
 
-	for (i = 0; i < routing->num_routes; ++i) {
-		const struct v4l2_subdev_route *route = &routing->routes[i];
+	for (i = 0; i < num_routes; ++i) {
+		const struct v4l2_subdev_route *route = &routes[i];
 
-		for (j = i + 1; j < routing->num_routes; ++j) {
-			const struct v4l2_subdev_route *r = &routing->routes[j];
+		for (j = i + 1; j < num_routes; ++j) {
+			const struct v4l2_subdev_route *r = &routes[j];
 
 			if (route->sink_pad == r->sink_pad &&
 			    route->sink_stream == r->sink_stream)
diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
index d0354e507970..9f1ad7eeb37a 100644
--- a/include/media/v4l2-subdev.h
+++ b/include/media/v4l2-subdev.h
@@ -737,6 +737,7 @@ struct v4l2_subdev_stream_configs {
  * This structure contains the routing table for a subdev.
  */
 struct v4l2_subdev_krouting {
+	int which;
 	unsigned int num_routes;
 	struct v4l2_subdev_route *routes;
 };
@@ -870,7 +871,6 @@ struct v4l2_subdev_pad_ops {
 			       struct v4l2_mbus_config *config);
 	int (*set_routing)(struct v4l2_subdev *sd,
 			   struct v4l2_subdev_state *state,
-			   enum v4l2_subdev_format_whence which,
 			   struct v4l2_subdev_krouting *route);
 };
 
@@ -1487,7 +1487,8 @@ v4l2_subdev_validate_and_lock_state(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
 			    struct v4l2_subdev_state *state,
-			    struct v4l2_subdev_krouting *routing);
+			    struct v4l2_subdev_route *routes,
+			    unsigned int num_routes);
 
 /**
  * v4l2_subdev_set_routing_with_fmt() - Set given routing and format to subdev
@@ -1502,7 +1503,8 @@ int v4l2_subdev_set_routing(struct v4l2_subdev *sd,
  */
 int v4l2_subdev_set_routing_with_fmt(struct v4l2_subdev *sd,
 				     struct v4l2_subdev_state *state,
-				     struct v4l2_subdev_krouting *routing,
+				     struct v4l2_subdev_route *routes,
+				     unsigned int num_routes,
 				     const struct v4l2_mbus_framefmt *fmt);
 
 /**
@@ -1580,6 +1582,7 @@ int v4l2_subdev_get_fmt(struct v4l2_subdev *sd, struct v4l2_subdev_state *state,
  *
  * Returns 0 on success, error value otherwise.
  */
-int v4l2_routing_simple_verify(const struct v4l2_subdev_krouting *routing);
+int v4l2_routing_simple_verify(const struct v4l2_subdev_route *routes,
+			       unsigned int num_routes);
 
 #endif

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 15/10/2021 17:05, Jacopo Mondi wrote:

> Ok, I understand this allow you to
> 
>          int _set_routing(sd, state, krouting)
>          {
>                  v4l2_subdev_set_routing(krouting);
> 
>          }
> 
>          int set_routing(sd, state, which, krouting)
>          {
>                  _set_routing(sd, state, krouting);
>                  if (which == ACTIVE)
>                          apply_to_hw();
>          }
> 
>          pad_ops ops = {
>                  .set_routing = set_routing,
>          };
> 
>          int init_cfg(sd, state)
>          {
>                  routes = {
>                          ...
>                  };
> 
>                  krouting routing {
>                          .routes = routes,
>                          .num_routes = 1,
>                  };
> 
>                  _set_routing(sd, state, &routing);
>          }

Yes, although I would guess that the likely use of which in set_routing is

if (which == ACTIVE && priv->streaming)
	return -EBUSY;

> We're moving the issue with init_cfg() not being aware of the
> active/try to the set_routing() operation signature (and in cascade to
> all operation that operate on krouting from where 'which' had been
> removed to comply with init_cfg).

Hmm, what operations? set_routing() subdev op is the only place that 
needs the which, afaics. The issue with the previous series was that we 
needed 'which' in places where we shouldn't. In the v9, that issue is gone.

Or do you see issues with your series caused by this change?

> I was actually advocating instead for adding back which to
> krouting and have all the other v4l2_subdev functions that operates on
> routing and do not care about 'which' to use the raw routes and their
> number
> 
> 
>          --- v4l2-subdev.h/c ---
> 
>          struct krouting {
>                  which;
>                  num_routes;
>                  routes;
>          };
> 
>          int v4l2_subdev_set_routing(sd, state, routes, num_routes)
>          {
> 
>                  /* Copy routes into state */
> 
>          }
> 
>          ---- subdevice driver ----
> 
>          int _set_routing(sd, state, routes, num_routes)
>          {
>                  return v4l2_subdev_set_routing(sd, state, routes,
>                                                 num_routes);
> 
>          }
> 
>          int set_routing(sd, state, krouting)
>          {
>                  _set_routing(sd, state, krouting->routes, krouting->num_routes)
> 
>                  if (krouting->which == ACTIVE)
>                          apply_to_hw();
>          }
> 
>          pad_ops ops = {
>                  .set_routing = set_routing,
>          };
> 
>          int init_cfg(sd, state)
>          {
>                  routes routing[] = {
>                          ...
>                  };
> 
>                  _set_routing(sd, state, routing, ARRAY_SIZE(routing));
>          }
> 
> So that
> - krouting contains 'which' but is only used in the set_routing
>    operation
> - init_cfg doesn't have to initialize a krouting just for the purpose
>    of pleasing the v4l2_subdev function signatures

Why do you want the routes and num_routes handled separately? Isn't it 
nicer if they're grouped together in a struct?

> It requires to remove krouting from all the v4l2_subdev_*route*()
> functions in v4l2_subdev.c and have them work on the raw routes
> ('routes' and 'num_routes' could as well be grouped in one structure
> like you had in v8).

If they're grouped, then... Wouldn't we just be back in the case I 
described in the mail you linked, having a struct for the sole purpose 
of passing parameters to set_routing?

> As you asked for opinions and I failed to provide one in v8 it
> wouldn't be fair to ask you to backtrack just to see how it looks
> like, so here you have a patch to be applied on top of your branch for
> you to take into consideration (test on my gmsl work on top of your
> v9).

So it's not quite clear to me what's the issue you see here.

Is it just that you don't like passing 'which' separately to 
set_routing(), but other ops get it in a struct? If that's a problem, we 
can create the extra struct used solely for calling set_routing.

Or do you think it's better to have routes and num_routes used 
separately, as you do in your patch? If so, why, what's the benefit?

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

Hi Tomi,

On Fri, Oct 15, 2021 at 06:17:45PM +0300, Tomi Valkeinen wrote:
> Hi,
>
> On 15/10/2021 17:05, Jacopo Mondi wrote:
>
> > Ok, I understand this allow you to
> >
> >          int _set_routing(sd, state, krouting)
> >          {
> >                  v4l2_subdev_set_routing(krouting);
> >
> >          }
> >
> >          int set_routing(sd, state, which, krouting)
> >          {
> >                  _set_routing(sd, state, krouting);
> >                  if (which == ACTIVE)
> >                          apply_to_hw();
> >          }
> >
> >          pad_ops ops = {
> >                  .set_routing = set_routing,
> >          };
> >
> >          int init_cfg(sd, state)
> >          {
> >                  routes = {
> >                          ...
> >                  };
> >
> >                  krouting routing {
> >                          .routes = routes,
> >                          .num_routes = 1,
> >                  };
> >
> >                  _set_routing(sd, state, &routing);
> >          }
>
> Yes, although I would guess that the likely use of which in set_routing is
>
> if (which == ACTIVE && priv->streaming)
> 	return -EBUSY;

Not really. In my use case in example, I need to know if a route is
active as according to the routing table I have to update the value of
a control (the total pixel rate of the demultiplexer) and I need to
update the mask of active (routed) GMSL link upon which several other
operations and HW configurations to be applied at s_stream() time
depend on.

>
> > We're moving the issue with init_cfg() not being aware of the
> > active/try to the set_routing() operation signature (and in cascade to
> > all operation that operate on krouting from where 'which' had been
> > removed to comply with init_cfg).
>
> Hmm, what operations? set_routing() subdev op is the only place that needs
> the which, afaics. The issue with the previous series was that we needed
> 'which' in places where we shouldn't. In the v9, that issue is gone.
>
> Or do you see issues with your series caused by this change?
>
> > I was actually advocating instead for adding back which to
> > krouting and have all the other v4l2_subdev functions that operates on
> > routing and do not care about 'which' to use the raw routes and their
> > number
> >
> >
> >          --- v4l2-subdev.h/c ---
> >
> >          struct krouting {
> >                  which;
> >                  num_routes;
> >                  routes;
> >          };
> >
> >          int v4l2_subdev_set_routing(sd, state, routes, num_routes)
> >          {
> >
> >                  /* Copy routes into state */
> >
> >          }
> >
> >          ---- subdevice driver ----
> >
> >          int _set_routing(sd, state, routes, num_routes)
> >          {
> >                  return v4l2_subdev_set_routing(sd, state, routes,
> >                                                 num_routes);
> >
> >          }
> >
> >          int set_routing(sd, state, krouting)
> >          {
> >                  _set_routing(sd, state, krouting->routes, krouting->num_routes)
> >
> >                  if (krouting->which == ACTIVE)
> >                          apply_to_hw();
> >          }
> >
> >          pad_ops ops = {
> >                  .set_routing = set_routing,
> >          };
> >
> >          int init_cfg(sd, state)
> >          {
> >                  routes routing[] = {
> >                          ...
> >                  };
> >
> >                  _set_routing(sd, state, routing, ARRAY_SIZE(routing));
> >          }
> >
> > So that
> > - krouting contains 'which' but is only used in the set_routing
> >    operation
> > - init_cfg doesn't have to initialize a krouting just for the purpose
> >    of pleasing the v4l2_subdev function signatures
>
> Why do you want the routes and num_routes handled separately? Isn't it nicer
> if they're grouped together in a struct?
>
> > It requires to remove krouting from all the v4l2_subdev_*route*()
> > functions in v4l2_subdev.c and have them work on the raw routes
> > ('routes' and 'num_routes' could as well be grouped in one structure
> > like you had in v8).
>
> If they're grouped, then... Wouldn't we just be back in the case I described
> in the mail you linked, having a struct for the sole purpose of passing
> parameters to set_routing?
>
> > As you asked for opinions and I failed to provide one in v8 it
> > wouldn't be fair to ask you to backtrack just to see how it looks
> > like, so here you have a patch to be applied on top of your branch for
> > you to take into consideration (test on my gmsl work on top of your
> > v9).
>
> So it's not quite clear to me what's the issue you see here.
>
> Is it just that you don't like passing 'which' separately to set_routing(),
> but other ops get it in a struct? If that's a problem, we can create the
> extra struct used solely for calling set_routing.
>
> Or do you think it's better to have routes and num_routes used separately,
> as you do in your patch? If so, why, what's the benefit?
>

I started from the set_routing() signature being an out-liner and I
wanted to see what did it take to change it on top of v9.

I concur grouping routes and num_routes like you did in v8 is better
than having them separate, but my purpose was removing the
'which' parameter from set_routing() and that's why the quickly
sketched patch I sent: to have a comparison of how it looks like
(similar to your v8, I know but I failed to express myself at that
time) and get opinions.

Do you still think it's better to have all v4l2_subdev*route*() functions
to operate on krouting and have 'which' as an additional paramter ?

>  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

Hi,

On 15/10/2021 18:28, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Fri, Oct 15, 2021 at 06:17:45PM +0300, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 15/10/2021 17:05, Jacopo Mondi wrote:
>>
>>> Ok, I understand this allow you to
>>>
>>>           int _set_routing(sd, state, krouting)
>>>           {
>>>                   v4l2_subdev_set_routing(krouting);
>>>
>>>           }
>>>
>>>           int set_routing(sd, state, which, krouting)
>>>           {
>>>                   _set_routing(sd, state, krouting);
>>>                   if (which == ACTIVE)
>>>                           apply_to_hw();
>>>           }
>>>
>>>           pad_ops ops = {
>>>                   .set_routing = set_routing,
>>>           };
>>>
>>>           int init_cfg(sd, state)
>>>           {
>>>                   routes = {
>>>                           ...
>>>                   };
>>>
>>>                   krouting routing {
>>>                           .routes = routes,
>>>                           .num_routes = 1,
>>>                   };
>>>
>>>                   _set_routing(sd, state, &routing);
>>>           }
>>
>> Yes, although I would guess that the likely use of which in set_routing is
>>
>> if (which == ACTIVE && priv->streaming)
>> 	return -EBUSY;
> 
> Not really. In my use case in example, I need to know if a route is
> active as according to the routing table I have to update the value of
> a control (the total pixel rate of the demultiplexer) and I need to
> update the mask of active (routed) GMSL link upon which several other
> operations and HW configurations to be applied at s_stream() time
> depend on.

Are you allowing changing routing while streaming is enabled? You have 
to be super careful with that... I haven't tested it at all, but I guess 
if no active routes nor pads with active routes are changed, it should 
be ok.

>>> We're moving the issue with init_cfg() not being aware of the
>>> active/try to the set_routing() operation signature (and in cascade to
>>> all operation that operate on krouting from where 'which' had been
>>> removed to comply with init_cfg).
>>
>> Hmm, what operations? set_routing() subdev op is the only place that needs
>> the which, afaics. The issue with the previous series was that we needed
>> 'which' in places where we shouldn't. In the v9, that issue is gone.
>>
>> Or do you see issues with your series caused by this change?
>>
>>> I was actually advocating instead for adding back which to
>>> krouting and have all the other v4l2_subdev functions that operates on
>>> routing and do not care about 'which' to use the raw routes and their
>>> number
>>>
>>>
>>>           --- v4l2-subdev.h/c ---
>>>
>>>           struct krouting {
>>>                   which;
>>>                   num_routes;
>>>                   routes;
>>>           };
>>>
>>>           int v4l2_subdev_set_routing(sd, state, routes, num_routes)
>>>           {
>>>
>>>                   /* Copy routes into state */
>>>
>>>           }
>>>
>>>           ---- subdevice driver ----
>>>
>>>           int _set_routing(sd, state, routes, num_routes)
>>>           {
>>>                   return v4l2_subdev_set_routing(sd, state, routes,
>>>                                                  num_routes);
>>>
>>>           }
>>>
>>>           int set_routing(sd, state, krouting)
>>>           {
>>>                   _set_routing(sd, state, krouting->routes, krouting->num_routes)
>>>
>>>                   if (krouting->which == ACTIVE)
>>>                           apply_to_hw();
>>>           }
>>>
>>>           pad_ops ops = {
>>>                   .set_routing = set_routing,
>>>           };
>>>
>>>           int init_cfg(sd, state)
>>>           {
>>>                   routes routing[] = {
>>>                           ...
>>>                   };
>>>
>>>                   _set_routing(sd, state, routing, ARRAY_SIZE(routing));
>>>           }
>>>
>>> So that
>>> - krouting contains 'which' but is only used in the set_routing
>>>     operation
>>> - init_cfg doesn't have to initialize a krouting just for the purpose
>>>     of pleasing the v4l2_subdev function signatures
>>
>> Why do you want the routes and num_routes handled separately? Isn't it nicer
>> if they're grouped together in a struct?
>>
>>> It requires to remove krouting from all the v4l2_subdev_*route*()
>>> functions in v4l2_subdev.c and have them work on the raw routes
>>> ('routes' and 'num_routes' could as well be grouped in one structure
>>> like you had in v8).
>>
>> If they're grouped, then... Wouldn't we just be back in the case I described
>> in the mail you linked, having a struct for the sole purpose of passing
>> parameters to set_routing?
>>
>>> As you asked for opinions and I failed to provide one in v8 it
>>> wouldn't be fair to ask you to backtrack just to see how it looks
>>> like, so here you have a patch to be applied on top of your branch for
>>> you to take into consideration (test on my gmsl work on top of your
>>> v9).
>>
>> So it's not quite clear to me what's the issue you see here.
>>
>> Is it just that you don't like passing 'which' separately to set_routing(),
>> but other ops get it in a struct? If that's a problem, we can create the
>> extra struct used solely for calling set_routing.
>>
>> Or do you think it's better to have routes and num_routes used separately,
>> as you do in your patch? If so, why, what's the benefit?
>>
> 
> I started from the set_routing() signature being an out-liner and I
> wanted to see what did it take to change it on top of v9.

But all that needs is a new struct, which has 'which' and the krouting 
as members, no?

> I concur grouping routes and num_routes like you did in v8 is better
> than having them separate, but my purpose was removing the
> 'which' parameter from set_routing() and that's why the quickly
> sketched patch I sent: to have a comparison of how it looks like
> (similar to your v8, I know but I failed to express myself at that
> time) and get opinions.
> 
> Do you still think it's better to have all v4l2_subdev*route*() functions
> to operate on krouting and have 'which' as an additional paramter ?

I don't have a strong opinion on the 'which' parameter to set_routing. I 
agree it looks odd compared to all the other ops, but then, I don't know 
if that's enough of a reason to create a new struct. Having another 
struct for routing might cause more confusion than what it helps.

As for changing all the functions which currently get 
v4l2_subdev_krouting as a parameter to get routes pointer and 
num_routes... I do much prefer using v4l2_subdev_krouting. Routes and 
num_routes always go together, so why not group them into a struct?

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

Hi Tomi,

On Fri, Oct 15, 2021 at 07:10:02PM +0300, Tomi Valkeinen wrote:
> Hi,
>
> On 15/10/2021 18:28, Jacopo Mondi wrote:
> > Hi Tomi,
> >
> > On Fri, Oct 15, 2021 at 06:17:45PM +0300, Tomi Valkeinen wrote:
> > > Hi,
> > >
> > > On 15/10/2021 17:05, Jacopo Mondi wrote:
> > >
> > > > Ok, I understand this allow you to
> > > >
> > > >           int _set_routing(sd, state, krouting)
> > > >           {
> > > >                   v4l2_subdev_set_routing(krouting);
> > > >
> > > >           }
> > > >
> > > >           int set_routing(sd, state, which, krouting)
> > > >           {
> > > >                   _set_routing(sd, state, krouting);
> > > >                   if (which == ACTIVE)
> > > >                           apply_to_hw();
> > > >           }
> > > >
> > > >           pad_ops ops = {
> > > >                   .set_routing = set_routing,
> > > >           };
> > > >
> > > >           int init_cfg(sd, state)
> > > >           {
> > > >                   routes = {
> > > >                           ...
> > > >                   };
> > > >
> > > >                   krouting routing {
> > > >                           .routes = routes,
> > > >                           .num_routes = 1,
> > > >                   };
> > > >
> > > >                   _set_routing(sd, state, &routing);
> > > >           }
> > >
> > > Yes, although I would guess that the likely use of which in set_routing is
> > >
> > > if (which == ACTIVE && priv->streaming)
> > > 	return -EBUSY;
> >
> > Not really. In my use case in example, I need to know if a route is
> > active as according to the routing table I have to update the value of
> > a control (the total pixel rate of the demultiplexer) and I need to
> > update the mask of active (routed) GMSL link upon which several other
> > operations and HW configurations to be applied at s_stream() time
> > depend on.
>
> Are you allowing changing routing while streaming is enabled? You have to be

No :) What I meant is that I need to know if a route is active or not
for other reasons, not just for disallow route changes while
streaming.

> super careful with that... I haven't tested it at all, but I guess if no
> active routes nor pads with active routes are changed, it should be ok.
>
> > > > We're moving the issue with init_cfg() not being aware of the
> > > > active/try to the set_routing() operation signature (and in cascade to
> > > > all operation that operate on krouting from where 'which' had been
> > > > removed to comply with init_cfg).
> > >
> > > Hmm, what operations? set_routing() subdev op is the only place that needs
> > > the which, afaics. The issue with the previous series was that we needed
> > > 'which' in places where we shouldn't. In the v9, that issue is gone.
> > >
> > > Or do you see issues with your series caused by this change?
> > >
> > > > I was actually advocating instead for adding back which to
> > > > krouting and have all the other v4l2_subdev functions that operates on
> > > > routing and do not care about 'which' to use the raw routes and their
> > > > number
> > > >
> > > >
> > > >           --- v4l2-subdev.h/c ---
> > > >
> > > >           struct krouting {
> > > >                   which;
> > > >                   num_routes;
> > > >                   routes;
> > > >           };
> > > >
> > > >           int v4l2_subdev_set_routing(sd, state, routes, num_routes)
> > > >           {
> > > >
> > > >                   /* Copy routes into state */
> > > >
> > > >           }
> > > >
> > > >           ---- subdevice driver ----
> > > >
> > > >           int _set_routing(sd, state, routes, num_routes)
> > > >           {
> > > >                   return v4l2_subdev_set_routing(sd, state, routes,
> > > >                                                  num_routes);
> > > >
> > > >           }
> > > >
> > > >           int set_routing(sd, state, krouting)
> > > >           {
> > > >                   _set_routing(sd, state, krouting->routes, krouting->num_routes)
> > > >
> > > >                   if (krouting->which == ACTIVE)
> > > >                           apply_to_hw();
> > > >           }
> > > >
> > > >           pad_ops ops = {
> > > >                   .set_routing = set_routing,
> > > >           };
> > > >
> > > >           int init_cfg(sd, state)
> > > >           {
> > > >                   routes routing[] = {
> > > >                           ...
> > > >                   };
> > > >
> > > >                   _set_routing(sd, state, routing, ARRAY_SIZE(routing));
> > > >           }
> > > >
> > > > So that
> > > > - krouting contains 'which' but is only used in the set_routing
> > > >     operation
> > > > - init_cfg doesn't have to initialize a krouting just for the purpose
> > > >     of pleasing the v4l2_subdev function signatures
> > >
> > > Why do you want the routes and num_routes handled separately? Isn't it nicer
> > > if they're grouped together in a struct?
> > >
> > > > It requires to remove krouting from all the v4l2_subdev_*route*()
> > > > functions in v4l2_subdev.c and have them work on the raw routes
> > > > ('routes' and 'num_routes' could as well be grouped in one structure
> > > > like you had in v8).
> > >
> > > If they're grouped, then... Wouldn't we just be back in the case I described
> > > in the mail you linked, having a struct for the sole purpose of passing
> > > parameters to set_routing?
> > >
> > > > As you asked for opinions and I failed to provide one in v8 it
> > > > wouldn't be fair to ask you to backtrack just to see how it looks
> > > > like, so here you have a patch to be applied on top of your branch for
> > > > you to take into consideration (test on my gmsl work on top of your
> > > > v9).
> > >
> > > So it's not quite clear to me what's the issue you see here.
> > >
> > > Is it just that you don't like passing 'which' separately to set_routing(),
> > > but other ops get it in a struct? If that's a problem, we can create the
> > > extra struct used solely for calling set_routing.
> > >
> > > Or do you think it's better to have routes and num_routes used separately,
> > > as you do in your patch? If so, why, what's the benefit?
> > >
> >
> > I started from the set_routing() signature being an out-liner and I
> > wanted to see what did it take to change it on top of v9.
>
> But all that needs is a new struct, which has 'which' and the krouting as
> members, no?

>
> > I concur grouping routes and num_routes like you did in v8 is better
> > than having them separate, but my purpose was removing the
> > 'which' parameter from set_routing() and that's why the quickly
> > sketched patch I sent: to have a comparison of how it looks like
> > (similar to your v8, I know but I failed to express myself at that
> > time) and get opinions.
> >
> > Do you still think it's better to have all v4l2_subdev*route*() functions
> > to operate on krouting and have 'which' as an additional paramter ?
>
> I don't have a strong opinion on the 'which' parameter to set_routing. I
> agree it looks odd compared to all the other ops, but then, I don't know if
> that's enough of a reason to create a new struct. Having another struct for
> routing might cause more confusion than what it helps.
>

I see.

> As for changing all the functions which currently get v4l2_subdev_krouting
> as a parameter to get routes pointer and num_routes... I do much prefer
> using v4l2_subdev_krouting. Routes and num_routes always go together, so why
> not group them into a struct?

I'm not against a new structure to group routes and num_routes,
krouting can hold that struct and which like it was done in v8.

Minor anyway, just wanted to provide you the patch for a quick
comparison.

Thanks
   j

>
>  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

On 15/10/2021 19:47, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Fri, Oct 15, 2021 at 07:10:02PM +0300, Tomi Valkeinen wrote:
>> Hi,
>>
>> On 15/10/2021 18:28, Jacopo Mondi wrote:
>>> Hi Tomi,
>>>
>>> On Fri, Oct 15, 2021 at 06:17:45PM +0300, Tomi Valkeinen wrote:
>>>> Hi,
>>>>
>>>> On 15/10/2021 17:05, Jacopo Mondi wrote:
>>>>
>>>>> Ok, I understand this allow you to
>>>>>
>>>>>            int _set_routing(sd, state, krouting)
>>>>>            {
>>>>>                    v4l2_subdev_set_routing(krouting);
>>>>>
>>>>>            }
>>>>>
>>>>>            int set_routing(sd, state, which, krouting)
>>>>>            {
>>>>>                    _set_routing(sd, state, krouting);
>>>>>                    if (which == ACTIVE)
>>>>>                            apply_to_hw();
>>>>>            }
>>>>>
>>>>>            pad_ops ops = {
>>>>>                    .set_routing = set_routing,
>>>>>            };
>>>>>
>>>>>            int init_cfg(sd, state)
>>>>>            {
>>>>>                    routes = {
>>>>>                            ...
>>>>>                    };
>>>>>
>>>>>                    krouting routing {
>>>>>                            .routes = routes,
>>>>>                            .num_routes = 1,
>>>>>                    };
>>>>>
>>>>>                    _set_routing(sd, state, &routing);
>>>>>            }
>>>>
>>>> Yes, although I would guess that the likely use of which in set_routing is
>>>>
>>>> if (which == ACTIVE && priv->streaming)
>>>> 	return -EBUSY;
>>>
>>> Not really. In my use case in example, I need to know if a route is
>>> active as according to the routing table I have to update the value of
>>> a control (the total pixel rate of the demultiplexer) and I need to
>>> update the mask of active (routed) GMSL link upon which several other
>>> operations and HW configurations to be applied at s_stream() time
>>> depend on.
>>
>> Are you allowing changing routing while streaming is enabled? You have to be
> 
> No :) What I meant is that I need to know if a route is active or not
> for other reasons, not just for disallow route changes while
> streaming.

I'm interested to hear why =). I think set_routing should mostly just 
check if the routing is valid, and that check is the same for active and 
try cases.

In your example above you call apply_to_hw(). I think that should not be 
done. The commit to HW should be done in s_stream().

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Jacopo Mondi]

Hi Tomi,

On Fri, Oct 15, 2021 at 07:55:39PM +0300, Tomi Valkeinen wrote:
> On 15/10/2021 19:47, Jacopo Mondi wrote:
> > Hi Tomi,
> >
> > On Fri, Oct 15, 2021 at 07:10:02PM +0300, Tomi Valkeinen wrote:
> > > Hi,
> > >
> > > On 15/10/2021 18:28, Jacopo Mondi wrote:
> > > > Hi Tomi,
> > > >
> > > > On Fri, Oct 15, 2021 at 06:17:45PM +0300, Tomi Valkeinen wrote:
> > > > > Hi,
> > > > >
> > > > > On 15/10/2021 17:05, Jacopo Mondi wrote:
> > > > >
> > > > > > Ok, I understand this allow you to
> > > > > >
> > > > > >            int _set_routing(sd, state, krouting)
> > > > > >            {
> > > > > >                    v4l2_subdev_set_routing(krouting);
> > > > > >
> > > > > >            }
> > > > > >
> > > > > >            int set_routing(sd, state, which, krouting)
> > > > > >            {
> > > > > >                    _set_routing(sd, state, krouting);
> > > > > >                    if (which == ACTIVE)
> > > > > >                            apply_to_hw();
> > > > > >            }
> > > > > >
> > > > > >            pad_ops ops = {
> > > > > >                    .set_routing = set_routing,
> > > > > >            };
> > > > > >
> > > > > >            int init_cfg(sd, state)
> > > > > >            {
> > > > > >                    routes = {
> > > > > >                            ...
> > > > > >                    };
> > > > > >
> > > > > >                    krouting routing {
> > > > > >                            .routes = routes,
> > > > > >                            .num_routes = 1,
> > > > > >                    };
> > > > > >
> > > > > >                    _set_routing(sd, state, &routing);
> > > > > >            }
> > > > >
> > > > > Yes, although I would guess that the likely use of which in set_routing is
> > > > >
> > > > > if (which == ACTIVE && priv->streaming)
> > > > > 	return -EBUSY;
> > > >
> > > > Not really. In my use case in example, I need to know if a route is
> > > > active as according to the routing table I have to update the value of
> > > > a control (the total pixel rate of the demultiplexer) and I need to
> > > > update the mask of active (routed) GMSL link upon which several other
> > > > operations and HW configurations to be applied at s_stream() time
> > > > depend on.
> > >
> > > Are you allowing changing routing while streaming is enabled? You have to be
> >
> > No :) What I meant is that I need to know if a route is active or not
> > for other reasons, not just for disallow route changes while
> > streaming.
>
> I'm interested to hear why =). I think set_routing should mostly just check
> if the routing is valid, and that check is the same for active and try
> cases.
>
> In your example above you call apply_to_hw(). I think that should not be
> done. The commit to HW should be done in s_stream().

I agree, and I have moved most register writes at that time.

However I need to update the mask of active sources in order to be
able to compute the the pixel rate control as the total pixel rate is the
sum of the active routes.


>
>  Tomi

Re: [PATCH v9 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

Hi Hans,

On 11/10/2021 18:29, Hans Verkuil wrote:
> Hi Tomi,
> 
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Hi,
>>
>> This is v9 of the multiplexed streams series. v8 can be found from:
>>
>> https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/
>>
>> I have pushed my work branch to:
>>
>> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v9
>>
>> which contains the patches in this series, along with subdev drivers
>> using multiplexed streams.
>>
>> As can be guessed from the work branch, I have been testing this series
>> with TI's FPDLink setup. I have also done a "backwards compatibility"
>> test by dropping all multiplexed streams patches from the CAL driver
>> (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
>> single-stream configuration.
> 
> I hope to look at this series this week (fingers crossed), but I was asked to
> give some input w.r.t. testing of this series:

Thanks for the reviews! I'll start updating the series accordingly.

> I think before this can be merged we need:
> 
> 1) libcamera tests. Since libcamera would probably be the most active user of this
> API, and you have HW for it, it makes a lot of sense that there are decent tests
> for the supported HW in libcamera. That takes care of the real-world tests.

I agree, libcamera would be a good userspace test. Laurent has been 
working on that.

> 2) obviously the existing utils in v4l-utils need to be adapted to understand any
> new API elements.

Yes. I think it's "just" two things that are needed: ability to set a 
routing table (that might be quite messy via the cmdline for larger 
routing tables) and ability to set format and other parameters with a 
(pad,stream) tuple, instead of just pad.

> 3) compliance tests in v4l2-compliance for the new API. After I did a review of the
> series we can see to what extent this is possible.

One thing we have to fix are the problems caused by adding the 'stream' 
field to many structs, but I think fixing that is trivial.

Actually testing routing and streams is a bit more difficult.

> 4) optionally (for now at least, I reserve the right to change my mind): it would
> be very helpful if this can be added to vimc (or something similar), allowing for
> testing the API without having real hardware, which is useful both for writing
> the tests and for running regression tests regularly on a simple VM, without needing
> special hardware.

I haven't studied the vimc code, but maybe a metadata stream would be an 
easy addition.

  Tomi

Re: [PATCH v9 00/36] v4l: subdev internal routing and streams

[Jacopo Mondi]

Hi Tomi,

On Tue, Oct 26, 2021 at 02:54:45PM +0300, Tomi Valkeinen wrote:
> Hi Hans,
>
> On 11/10/2021 18:29, Hans Verkuil wrote:
> > Hi Tomi,
> >
> > On 05/10/2021 10:57, Tomi Valkeinen wrote:
> > > Hi,
> > >
> > > This is v9 of the multiplexed streams series. v8 can be found from:
> > >
> > > https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/
> > >
> > > I have pushed my work branch to:
> > >
> > > git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v9
> > >
> > > which contains the patches in this series, along with subdev drivers
> > > using multiplexed streams.
> > >
> > > As can be guessed from the work branch, I have been testing this series
> > > with TI's FPDLink setup. I have also done a "backwards compatibility"
> > > test by dropping all multiplexed streams patches from the CAL driver
> > > (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
> > > single-stream configuration.
> >
> > I hope to look at this series this week (fingers crossed), but I was asked to
> > give some input w.r.t. testing of this series:
>
> Thanks for the reviews! I'll start updating the series accordingly.
>
> > I think before this can be merged we need:
> >
> > 1) libcamera tests. Since libcamera would probably be the most active user of this
> > API, and you have HW for it, it makes a lot of sense that there are decent tests
> > for the supported HW in libcamera. That takes care of the real-world tests.
>
> I agree, libcamera would be a good userspace test. Laurent has been working
> on that.
>
> > 2) obviously the existing utils in v4l-utils need to be adapted to understand any
> > new API elements.
>
> Yes. I think it's "just" two things that are needed: ability to set a
> routing table (that might be quite messy via the cmdline for larger routing
> tables) and ability to set format and other parameters with a (pad,stream)
> tuple, instead of just pad.
>

For testing my series I have sketeched out
https://git.sr.ht/~jmondi_/v4l2-utils

I can work to make the patches there upstreamable

Thanks
   j


> > 3) compliance tests in v4l2-compliance for the new API. After I did a review of the
> > series we can see to what extent this is possible.
>
> One thing we have to fix are the problems caused by adding the 'stream'
> field to many structs, but I think fixing that is trivial.
>
> Actually testing routing and streams is a bit more difficult.
>
> > 4) optionally (for now at least, I reserve the right to change my mind): it would
> > be very helpful if this can be added to vimc (or something similar), allowing for
> > testing the API without having real hardware, which is useful both for writing
> > the tests and for running regression tests regularly on a simple VM, without needing
> > special hardware.
>
> I haven't studied the vimc code, but maybe a metadata stream would be an
> easy addition.
>
>  Tomi
>

Re: [PATCH v9 00/36] v4l: subdev internal routing and streams

[Tomi Valkeinen]

On 26/10/2021 15:32, Jacopo Mondi wrote:
> Hi Tomi,
> 
> On Tue, Oct 26, 2021 at 02:54:45PM +0300, Tomi Valkeinen wrote:
>> Hi Hans,
>>
>> On 11/10/2021 18:29, Hans Verkuil wrote:
>>> Hi Tomi,
>>>
>>> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>>>> Hi,
>>>>
>>>> This is v9 of the multiplexed streams series. v8 can be found from:
>>>>
>>>> https://lore.kernel.org/all/20210830110116.488338-1-tomi.valkeinen@ideasonboard.com/
>>>>
>>>> I have pushed my work branch to:
>>>>
>>>> git://git.kernel.org/pub/scm/linux/kernel/git/tomba/linux.git multistream/work-v9
>>>>
>>>> which contains the patches in this series, along with subdev drivers
>>>> using multiplexed streams.
>>>>
>>>> As can be guessed from the work branch, I have been testing this series
>>>> with TI's FPDLink setup. I have also done a "backwards compatibility"
>>>> test by dropping all multiplexed streams patches from the CAL driver
>>>> (the CSI-2 RX on the TI SoC), and using the FPDLink drivers with
>>>> single-stream configuration.
>>>
>>> I hope to look at this series this week (fingers crossed), but I was asked to
>>> give some input w.r.t. testing of this series:
>>
>> Thanks for the reviews! I'll start updating the series accordingly.
>>
>>> I think before this can be merged we need:
>>>
>>> 1) libcamera tests. Since libcamera would probably be the most active user of this
>>> API, and you have HW for it, it makes a lot of sense that there are decent tests
>>> for the supported HW in libcamera. That takes care of the real-world tests.
>>
>> I agree, libcamera would be a good userspace test. Laurent has been working
>> on that.
>>
>>> 2) obviously the existing utils in v4l-utils need to be adapted to understand any
>>> new API elements.
>>
>> Yes. I think it's "just" two things that are needed: ability to set a
>> routing table (that might be quite messy via the cmdline for larger routing
>> tables) and ability to set format and other parameters with a (pad,stream)
>> tuple, instead of just pad.
>>
> 
> For testing my series I have sketeched out
> https://git.sr.ht/~jmondi_/v4l2-utils
> 
> I can work to make the patches there upstreamable

Thanks, that would be great! I haven't tested your patches yet, but with 
a quick glance, looks good. The other options also need the stream 
parameter.

There's also media-ctl. It's not quite clear to me what we should have 
in media-ctl and what in v4l2-ctl.

  Tomi

Re: [PATCH v9 02/36] media: subdev: add active state to struct v4l2_subdev

[Tomi Valkeinen]

On 12/10/2021 17:32, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Add a new 'state' field to struct v4l2_subdev to which we can store the
>> active state of a subdev. This will place the subdev configuration into
>> a known place, allowing us to use the state directly from the v4l2
>> framework, thus simplifying the drivers.
>>
>> Also add functions v4l2_subdev_init_finalize() and
>> v4l2_subdev_cleanup(), which will allocate and free the active state.
>> The functions are named in a generic way so that they can be also used
>> for other subdev initialization work.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/v4l2-core/v4l2-subdev.c | 21 +++++++++++
>>   include/media/v4l2-subdev.h           | 51 +++++++++++++++++++++++++++
>>   2 files changed, 72 insertions(+)
>>
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index fe49c86a9b02..bcaf66a1e3d9 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -943,3 +943,24 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   	v4l2_subdev_notify(sd, V4L2_DEVICE_NOTIFY_EVENT, (void *)ev);
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>> +
>> +int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>> +{
>> +	struct v4l2_subdev_state *state;
>> +
>> +	state = __v4l2_subdev_state_alloc(sd);
>> +	if (IS_ERR(state))
>> +		return PTR_ERR(state);
>> +
>> +	sd->state = state;
>> +
>> +	return 0;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
>> +
>> +void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>> +{
>> +	__v4l2_subdev_state_free(sd->state);
>> +	sd->state = NULL;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index e52bf508c75b..3aaa7146e5ff 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -898,6 +898,8 @@ struct v4l2_subdev_platform_data {
>>    * @subdev_notifier: A sub-device notifier implicitly registered for the sub-
>>    *		     device using v4l2_async_register_subdev_sensor().
>>    * @pdata: common part of subdevice platform data
>> + * @state: active state for the subdev (NULL for subdevs tracking the state
>> + *	   internally)
>>    *
>>    * Each instance of a subdev driver should create this struct, either
>>    * stand-alone or embedded in a larger struct.
>> @@ -929,6 +931,19 @@ struct v4l2_subdev {
>>   	struct v4l2_async_notifier *notifier;
>>   	struct v4l2_async_notifier *subdev_notifier;
>>   	struct v4l2_subdev_platform_data *pdata;
>> +
>> +	/*
>> +	 * The fields below are private, and should only be accessed via
>> +	 * appropriate functions.
>> +	 */
>> +
>> +	/*
>> +	 * TODO: state should most likely be changed from a pointer to an
>> +	 * embedded field. For the time being it's kept as a pointer to more
>> +	 * easily catch uses of state in the cases where the driver doesn't
>> +	 * support it.
>> +	 */
>> +	struct v4l2_subdev_state *state;
> 
> Hmm:
> 
> struct v4l2_subdev_state {
>          struct v4l2_subdev_pad_config *pads;
> };
> 
> and:
> 
> struct v4l2_subdev_pad_config {
>          struct v4l2_mbus_framefmt try_fmt;
>          struct v4l2_rect try_crop;
>          struct v4l2_rect try_compose;
> };
> 
> So why is this the active state if struct v4l2_subdev_pad_config has try_* fields?
> 
> At the very least it should be mentioned somewhere in the code (and probably commit
> log as well) that the try_ prefix is historical, or will be removed later, or something
> like that.

I'll add a note about it. I did not want to do renames like this yet, as 
they'll affect multiple drivers, creating a conflict hell when rebasing 
this series.

Also note that at the moment those try_* fields are only used in 
try-context. Upstream drivers do not support active state, so they'll 
only get the TRY version of v4l2_subdev_state. And the new drivers that 
support active state do not use the 'pads' field nor v4l2_subdev_pad_config.

  Tomi

Re: [PATCH v9 04/36] media: subdev: add subdev state locking

[Tomi Valkeinen]

On 12/10/2021 17:36, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> The V4L2 subdevs have managed without centralized locking for the state
>> (previously pad_config), as the TRY state is supposedly safe (although I
>> believe two TRY ioctls for the same fd would race), and the ACTIVE
>> state, and its locking, is managed by the drivers internally.
>>
>> We now have ACTIVE state in a centralized position, and need locking.
>> Strictly speaking the locking is only needed for new drivers that use
>> the new state, as the current drivers continue behaving as they used to.
>>
>> Add a mutex to the struct v4l2_subdev_state, along with a few helper
>> functions for locking/unlocking.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   drivers/media/platform/rcar-vin/rcar-v4l2.c |  3 +-
>>   drivers/media/platform/vsp1/vsp1_entity.c   |  4 +-
>>   drivers/media/v4l2-core/v4l2-subdev.c       | 38 +++++++++++++---
>>   drivers/staging/media/tegra-video/vi.c      |  4 +-
>>   include/media/v4l2-subdev.h                 | 49 ++++++++++++++++++++-
>>   5 files changed, 88 insertions(+), 10 deletions(-)
>>
>> diff --git a/drivers/media/platform/rcar-vin/rcar-v4l2.c b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> index ba1d16ab1651..e6bd94d63e4f 100644
>> --- a/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> +++ b/drivers/media/platform/rcar-vin/rcar-v4l2.c
>> @@ -244,6 +244,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>   {
>>   	struct v4l2_subdev *sd = vin_to_source(vin);
>>   	struct v4l2_subdev_state *sd_state;
>> +	static struct lock_class_key key;
>>   	struct v4l2_subdev_format format = {
>>   		.which = which,
>>   		.pad = vin->parallel.source_pad,
>> @@ -252,7 +253,7 @@ static int rvin_try_format(struct rvin_dev *vin, u32 which,
>>   	u32 width, height;
>>   	int ret;
>>   
>> -	sd_state = __v4l2_subdev_state_alloc(sd);
>> +	sd_state = __v4l2_subdev_state_alloc(sd, "rvin:state->lock", &key);
>>   	if (IS_ERR(sd_state))
>>   		return PTR_ERR(sd_state);
>>   
>> diff --git a/drivers/media/platform/vsp1/vsp1_entity.c b/drivers/media/platform/vsp1/vsp1_entity.c
>> index 869cadc1468d..e607c3ae2520 100644
>> --- a/drivers/media/platform/vsp1/vsp1_entity.c
>> +++ b/drivers/media/platform/vsp1/vsp1_entity.c
>> @@ -613,6 +613,7 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>>   		     const char *name, unsigned int num_pads,
>>   		     const struct v4l2_subdev_ops *ops, u32 function)
>>   {
>> +	static struct lock_class_key key;
>>   	struct v4l2_subdev *subdev;
>>   	unsigned int i;
>>   	int ret;
>> @@ -675,7 +676,8 @@ int vsp1_entity_init(struct vsp1_device *vsp1, struct vsp1_entity *entity,
>>   	 * Allocate the pad configuration to store formats and selection
>>   	 * rectangles.
>>   	 */
>> -	entity->config = __v4l2_subdev_state_alloc(&entity->subdev);
>> +	entity->config = __v4l2_subdev_state_alloc(&entity->subdev,
>> +						   "vsp1:config->lock", &key);
>>   	if (IS_ERR(entity->config)) {
>>   		media_entity_cleanup(&entity->subdev.entity);
>>   		return PTR_ERR(entity->config);
>> diff --git a/drivers/media/v4l2-core/v4l2-subdev.c b/drivers/media/v4l2-core/v4l2-subdev.c
>> index aa757cc28879..52309a299b03 100644
>> --- a/drivers/media/v4l2-core/v4l2-subdev.c
>> +++ b/drivers/media/v4l2-core/v4l2-subdev.c
>> @@ -27,8 +27,9 @@
>>   static int subdev_fh_init(struct v4l2_subdev_fh *fh, struct v4l2_subdev *sd)
>>   {
>>   	struct v4l2_subdev_state *state;
>> +	static struct lock_class_key key;
>>   
>> -	state = __v4l2_subdev_state_alloc(sd);
>> +	state = __v4l2_subdev_state_alloc(sd, "fh->state->lock", &key);
>>   	if (IS_ERR(state))
>>   		return PTR_ERR(state);
>>   
>> @@ -923,7 +924,9 @@ int v4l2_subdev_link_validate(struct media_link *link)
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_link_validate);
>>   
>> -struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>> +struct v4l2_subdev_state *
>> +__v4l2_subdev_state_alloc(struct v4l2_subdev *sd, const char *lock_name,
>> +			  struct lock_class_key *lock_key)
>>   {
>>   	struct v4l2_subdev_state *state;
>>   	int ret;
>> @@ -932,6 +935,8 @@ struct v4l2_subdev_state *__v4l2_subdev_state_alloc(struct v4l2_subdev *sd)
>>   	if (!state)
>>   		return ERR_PTR(-ENOMEM);
>>   
>> +	__mutex_init(&state->lock, lock_name, lock_key);
>> +
>>   	if (sd->entity.num_pads) {
>>   		state->pads = kvmalloc_array(sd->entity.num_pads,
>>   					     sizeof(*state->pads),
>> @@ -963,6 +968,8 @@ void __v4l2_subdev_state_free(struct v4l2_subdev_state *state)
>>   	if (!state)
>>   		return;
>>   
>> +	mutex_destroy(&state->lock);
>> +
>>   	kvfree(state->pads);
>>   	kfree(state);
>>   }
>> @@ -997,11 +1004,12 @@ void v4l2_subdev_notify_event(struct v4l2_subdev *sd,
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_notify_event);
>>   
>> -int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>> +int __v4l2_subdev_init_finalize(struct v4l2_subdev *sd, const char *name,
>> +				struct lock_class_key *key)
>>   {
>>   	struct v4l2_subdev_state *state;
>>   
>> -	state = __v4l2_subdev_state_alloc(sd);
>> +	state = __v4l2_subdev_state_alloc(sd, name, key);
>>   	if (IS_ERR(state))
>>   		return PTR_ERR(state);
>>   
>> @@ -1009,7 +1017,7 @@ int v4l2_subdev_init_finalize(struct v4l2_subdev *sd)
>>   
>>   	return 0;
>>   }
>> -EXPORT_SYMBOL_GPL(v4l2_subdev_init_finalize);
>> +EXPORT_SYMBOL_GPL(__v4l2_subdev_init_finalize);
>>   
>>   void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>>   {
>> @@ -1017,3 +1025,23 @@ void v4l2_subdev_cleanup(struct v4l2_subdev *sd)
>>   	sd->state = NULL;
>>   }
>>   EXPORT_SYMBOL_GPL(v4l2_subdev_cleanup);
>> +
>> +struct v4l2_subdev_state *v4l2_subdev_lock_active_state(struct v4l2_subdev *sd)
>> +{
>> +	mutex_lock(&sd->state->lock);
>> +
>> +	return sd->state;
>> +}
>> +EXPORT_SYMBOL_GPL(v4l2_subdev_lock_active_state);
> 
> Why would this function just lock the active state, but...
> 
>> +
>> +void v4l2_subdev_lock_state(struct v4l2_subdev_state *state)
>> +{
>> +	mutex_lock(&state->lock);
> 
> ...this function makes no mention of 'active'.
> 
> The naming is just weird.

The first function locks the active state of the subdev, the second is 
used to lock a state, whether it's active or try.

> Part of the problem is that sd->state doesn't mention in the field name
> that this is the active state. I think the field name and the function
> name should match. So it is either lock_state and state->lock, or it
> is lock_active_state and act_state->lock.

I can rename the field to active_state.

  Tomi

Re: [PATCH v9 05/36] media: subdev: Add v4l2_subdev_validate_and_lock_state()

[Tomi Valkeinen]

On 12/10/2021 17:16, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> All suitable subdev ops are now passed either the TRY or the ACTIVE
>> state by the v4l2 core. However, other subdev drivers can still call the
>> ops passing NULL as the state, implying the active case.
>>
>> For all current upstream drivers this doesn't matter, as they do not
>> expect to get a valid state for ACTIVE case. But future drivers which
>> support multiplexed streaming and routing will depend on getting a state
>> for both active and try cases.
>>
>> For new drivers we can mandate that the pipelines where the drivers are
>> used need to pass the state properly, or preferably, not call such
>> subdev ops at all.
>>
>> However, if an existing subdev driver is changed to support multiplexed
>> streams, the driver has to consider cases where its ops will be called
>> with NULL state. The problem can easily be solved by using the
>> v4l2_subdev_validate_and_lock_state() helper, introduced here.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   include/media/v4l2-subdev.h | 33 +++++++++++++++++++++++++++++++++
>>   1 file changed, 33 insertions(+)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index a6e8f531a396..b75eeaee492b 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -1313,4 +1313,37 @@ void v4l2_subdev_lock_state(struct v4l2_subdev_state *state);
>>    */
>>   void v4l2_subdev_unlock_state(struct v4l2_subdev_state *state);
>>   
>> +/**
>> + * v4l2_subdev_validate_and_lock_state() - Gets locked TRY or ACTIVE subdev
> 
> I think this name is confusing since it isn't validating anything.
> 
> I'd call it v4l2_subdev_lock_and_return_state().

Yes, I really didn't like the name either.

v4l2_subdev_lock_and_return_state() isn't quite perfect either, as it 
sounds like it just locks and returns the given state. But I think it's 
better than the current name, so I'll change.

  Tomi

Re: [PATCH v9 25/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

On 12/10/2021 17:48, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
>> the subdev supports the new API with multiplexed streams (routing,
>> stream configs).
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   include/media/v4l2-subdev.h | 11 +++++++++++
>>   1 file changed, 11 insertions(+)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index b3a72c98a2d3..7f9c83fac020 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -888,6 +888,17 @@ struct v4l2_subdev_internal_ops {
>>    * should set this flag.
>>    */
>>   #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
>> +/*
>> + * Set this flag if this subdev supports multiplexed streams. This means
>> + * that the driver supports routing and handles the stream parameter in its
>> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
>> + *
>> + * - Centrally managed active state is enabled
>> + * - Legacy pad config is _not_ supported (state->pads)
> 
> I'm not sure what is meant with this. Can you elaborate?

It means that a subdev with V4L2_SUBDEV_FL_MULTIPLEXED cannot use the 
legacy pads array in the state (state->pads is NULL), as multiplexed 
streams requires streams support. For streams you need the routing & 
stream configs, added in the following patches.

  Tomi

> Regards,
> 
> 	Hans
> 
>> + * - Routing ioctls are available
>> + * - Multiple streams per pad are supported
>> + */
>> +#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
>>   
>>   struct regulator_bulk_data;
>>   
>>
> 

Re: [PATCH v9 25/36] media: add V4L2_SUBDEV_FL_MULTIPLEXED

[Tomi Valkeinen]

On 12/10/2021 17:55, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Add subdev flag V4L2_SUBDEV_FL_MULTIPLEXED. It is used to indicate that
>> the subdev supports the new API with multiplexed streams (routing,
>> stream configs).
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   include/media/v4l2-subdev.h | 11 +++++++++++
>>   1 file changed, 11 insertions(+)
>>
>> diff --git a/include/media/v4l2-subdev.h b/include/media/v4l2-subdev.h
>> index b3a72c98a2d3..7f9c83fac020 100644
>> --- a/include/media/v4l2-subdev.h
>> +++ b/include/media/v4l2-subdev.h
>> @@ -888,6 +888,17 @@ struct v4l2_subdev_internal_ops {
>>    * should set this flag.
>>    */
>>   #define V4L2_SUBDEV_FL_HAS_EVENTS		(1U << 3)
>> +/*
>> + * Set this flag if this subdev supports multiplexed streams. This means
>> + * that the driver supports routing and handles the stream parameter in its
>> + * v4l2_subdev_pad_ops handlers. More specifically, this means:
>> + *
>> + * - Centrally managed active state is enabled
>> + * - Legacy pad config is _not_ supported (state->pads)
>> + * - Routing ioctls are available
>> + * - Multiple streams per pad are supported
>> + */
>> +#define V4L2_SUBDEV_FL_MULTIPLEXED		(1U << 4)
> 
> This is an internal kAPI flag. But wouldn't it be useful to have this as
> an entity flag instead? E.g. MEDIA_ENT_FL_MULTIPLEXED
> 
> Or perhaps as a Sub-Device Capabilities Flag V4L2_SUBDEV_CAP_MULTIPLEXED.
> Or a combination. It is useful if you can detect this from userspace without
> having to call G_ROUTING to see if that exists.

This is probably true, but I haven't had a need for it yet so I didn't 
add a flag. Another reason is that I'm not quite happy with the 
V4L2_SUBDEV_FL_MULTIPLEXED. We might want to split it to smaller bits, 
perhaps separating routing and multiplexed streams.

So I'd prefer to wait until we actually have a need to detect this from 
the userspace, and hopefully we then know better how these features 
behave and what we want to expose.

This need might come when adding support to v4l2 tools and libcamera.

  Tomi

Re: [PATCH v9 30/36] media: Documentation: add multiplexed streams documentation

[Tomi Valkeinen]

On 12/10/2021 18:34, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Add documentation related to multiplexed streams.
>>
>> Signed-off-by: Tomi Valkeinen <tomi.valkeinen@ideasonboard.com>
>> ---
>>   .../driver-api/media/v4l2-subdev.rst          |   8 +
>>   .../userspace-api/media/v4l/dev-subdev.rst    | 164 ++++++++++++++++++
>>   2 files changed, 172 insertions(+)
>>
>> diff --git a/Documentation/driver-api/media/v4l2-subdev.rst b/Documentation/driver-api/media/v4l2-subdev.rst
>> index c3fd57fff668..410e03a29f2a 100644
>> --- a/Documentation/driver-api/media/v4l2-subdev.rst
>> +++ b/Documentation/driver-api/media/v4l2-subdev.rst
>> @@ -546,6 +546,14 @@ subdev drivers. In these cases the called subdev ops must also handle the NULL
>>   case. This can be easily managed by the use of
>>   v4l2_subdev_validate_and_lock_state() helper.
>>   
>> +Streams, multiplexed media pads and internal routing
>> +----------------------------------------------------
>> +
>> +A subdevice driver can implement support for multiplexed streams by setting
>> +the V4L2_SUBDEV_FL_MULTIPLEXED subdev flag and implementing support for
>> +centrally managed subdev active state, routing and stream based
>> +configuration.
>> +
>>   V4L2 sub-device functions and data structures
>>   ---------------------------------------------
>>   
>> diff --git a/Documentation/userspace-api/media/v4l/dev-subdev.rst b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> index a67c2749089a..ea3efa97bb08 100644
>> --- a/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> +++ b/Documentation/userspace-api/media/v4l/dev-subdev.rst
>> @@ -503,3 +503,167 @@ source pads.
>>       :maxdepth: 1
>>   
>>       subdev-formats
>> +
>> +Streams, multiplexed media pads and internal routing
>> +----------------------------------------------------
>> +
>> +Commonly V4L2 subdevices support only separate video streams, that is, each
>> +link in the media grap and each pad in a subdevice passes through a single
> 
> passes -> pass
> grap -> graph
> 
>> +video stream. Thus each pad contains a format configuration for that single
>> +stream. In some cases a subdev can do stream processing and split a stream
>> +into two or compose two streams into one, but the inputs and outputs for the
>> +subdev are still a single stream per pad.
>> +
>> +Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, a single
>> +bus carries multiple streams. Thus a camera could output two streams, a pixel
>> +stream and a metadata stream, and a bridge subdev could route the streams
>> +from multiple input pads into a single output pad.
>> +
>> +Subdevice drivers that support multiplexed streams are compatible with
>> +non-multiplexed subdev drivers, but, of course, requires such a routing
> 
> requires such -> require
> 
>> +configuration where the link between those two types of drivers contain only
> 
> contain -> contains
> 
>> +a single stream.
>> +
>> +Understanding streams
>> +^^^^^^^^^^^^^^^^^^^^^
>> +
>> +A stream is a stream of content (e.g. pixel data or metadata) flowing through
>> +the media pipeline from a source (e.g. a sensor) towards the final sink
>> +(e.g. a receiver in a SoC). Each media link carries all the streams from
> 
> 'receiver' is a bit vague. What sort of receiver are we talking about here?
> What about: 'a receiver and demultiplexer in a SoC'. This implies that it is
> the block that can receive a multiplexed stream and demux it. The demuxing
> part is the critical feature of this block IMHO.
> 
>> +one end of the link to the other, whereas subdevices have routing tables
> 
> 'whereas' is an odd word. Would simply 'and' not be a better word to use here?
> 
>> +which describe how the incoming streams from sink pads are routed to the
>> +source pads.
>> +
>> +A stream ID (often just "stream") is a media link-local identifier for a
>> +stream. In other words, configuration for a particular stream ID must exist
> 
> configuration -> a configuration
> 
>> +on both sides of a media link, but another stream ID can be used for the same
>> +stream at the other side of the subdevice.
>> +
>> +A stream at a specific point in the media pipeline is identified with the
>> +subdev and a (pad, stream) pair. For subdevices that do not support
>> +multiplexed streams the 'stream' is always 0.
> 
> I find this section confusing. I think there are two issues here: what exactly is
> meant with 'media link': a link between two pads from different entities?

Yes. Isn't "media link" the common word for it?

> And I also think that 'subdev' is sometimes used where 'media entity' is actually
> meant. E.g. "at the other side of the subdevice" makes no sense, but "between two
> other entities" makes more sense.

That's true. Here we talk specifically about media entities which are 
v4l2 subdevs. Should I say "v4l2 subdev media entities"?

>> +
>> +Configuring streams
>> +^^^^^^^^^^^^^^^^^^^
>> +
>> +The configuration of the streams is done individually for each subdevice and
>> +the validity of the streams between subdevices is validated when the pipeline
>> +is started.
> 
> Here too I think you really mean 'entities'. A subdev device node controls an
> entity. It might be me, but when I think of a media stream I see it as a data
> stream running through entities, and subdevs are just the interfaces through
> which the entities are controlled.

I just group the media entity and the subdev together =). I understand 
they're separate things, or at least were designed so originally.

But what belongs to the media entity and what belongs to the v4l2 
subdev? If streams belong to the media side, should the routing be moved 
to media entities (and the ioctl too)?

Above says "The configuration of the streams is done individually for 
each subdevice". We call an ioctl in the subdev, and the subdev driver 
stores the stream configuration in the subdev's active state. Isn't that 
just "subdev" stuff then, media entity is not involved?

>> +
>> +There are three steps in configuring the streams:
>> +
>> +1) Set up links. Connect the pads between subdevices using the :ref:`Media
>> +Controller API <media_controller>`
>> +
>> +2) Routing. The routing table for the subdevice must be set with
>> +:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl.
>> +
>> +3) Configure streams. Each route endpoint must be configured
>> +with :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>`.
>> +
>> +Multiplexed streams setup example
>> +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>> +
>> +A simple example of a multiplexed stream setup might be as follows:
>> +
>> +- Two identical sensors (Sensor A and Sensor B). Each sensor has a single
>> +  source pad (pad 0), and outputs two streams, pixel data and metadata.
>> +
>> +- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the
>> +  sensors (pads 0, 1), and one source pad (pad 2), which outputs all 4
>> +  streams.
>> +
>> +- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0),
>> +  connected to the bridge, and four source pads (pads 1-4), going to the DMA
>> +  engine. The receiver demultiplexes the incoming streams to the four source
>> +  pads.
>> +
>> +- Four DMA Engines in the SoC (DMA Engine). Each DMA engine is connected to a
>> +  single source pad in the receiver.
>> +
>> +The sensors, the bridge and the receiver are modeled as V4L2 subdevices,
>> +exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are
>> +modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes.
>> +
>> +To configure this pipeline, the userspace must take the following steps:
>> +
>> +1) Set up media links between entities: connect the sensors to the bridge,
>> +bridge to the receiver, and the receiver to the DMA engines. This step does
>> +not differ from normal non-multiplexed media controller setup.
>> +
>> +2) Configure routing.
>> +
>> +.. flat-table:: Sensor routing table (identical on both sensors)
>> +    :header-rows:  1
>> +
>> +    * - Sink Pad/Stream
>> +      - Source Pad/Stream
>> +      - Routing Flags
>> +      - Comments
>> +    * - 0/0 (unused)
>> +      - 0/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
>> +      - Pixel data stream. Source route, i.e. the sink fields are unused.
>> +    * - 0/0 (unused)
>> +      - 0/1
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE | V4L2_SUBDEV_ROUTE_FL_SOURCE
>> +      - Metadata stream. Source route, i.e. the sink fields are unused.
>> +
>> +.. flat-table:: Bridge routing table
>> +    :header-rows:  1
>> +
>> +    * - Sink Pad/Stream
>> +      - Source Pad/Stream
>> +      - Routing Flags
>> +      - Comments
>> +    * - 0/0
>> +      - 2/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Pixel data stream from Sensor A
>> +    * - 0/1
>> +      - 2/1
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Metadata stream from Sensor A
>> +    * - 1/0
>> +      - 2/2
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Pixel data stream from Sensor B
>> +    * - 1/1
>> +      - 2/3
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Metadata stream from Sensor B
>> +
>> +.. flat-table:: Receiver routing table
>> +    :header-rows:  1
>> +
>> +    * - Sink Pad/Stream
>> +      - Source Pad/Stream
>> +      - Routing Flags
>> +      - Comments
>> +    * - 0/0
>> +      - 1/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Pixel data stream from Sensor A
>> +    * - 0/1
>> +      - 2/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Metadata stream from Sensor A
>> +    * - 0/2
>> +      - 3/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Pixel data stream from Sensor B
>> +    * - 0/3
>> +      - 4/0
>> +      - V4L2_SUBDEV_ROUTE_FL_ACTIVE
>> +      - Metadata stream from Sensor B
>> +
>> +3) Configure streams
>> +
>> +After configuring the routing table, the next step is configuring the streams.
>> +This step is similar to configuring the pads in a non-multiplexed streams
>> +setup, with the difference that we need to configure each (pad, stream) pair
>> +(i.e. route endpoint), instead of just a pad.
>> +
>> +Presuming there are no format conversions in the pipeline, the userspace needs
> 
> Presuming -> Assuming
> 
>> +to configure all the route endpoints using four formats (two pixel formats
>> +and two metadata formats) with VIDIOC_SUBDEV_S_FMT.
>>
> 
> Actually, if there are format conversions, wouldn't you still do the same thing?
> 
> Is that 'Presuming...pipeline,' needed at all?

I think what I wrote is confusing. I was just highlighting that when you 
have a pipeline with no conversions, you need to use exactly four 
formats (v4l2_mbus_framefmt) on all the stream endpoints across all the 
subdevs. If your pipeline does conversions, you need to use more formats 
to reflect the conversion.

I'll rewrite it.

  Tomi

Re: [PATCH v9 27/36] media: subdev: Add [GS]_ROUTING subdev ioctls and operations

[Tomi Valkeinen]

On 05/10/2021 11:57, Tomi Valkeinen wrote:
> From: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> 
> Add support for subdev internal routing. A route is defined as a single
> stream from a sink pad to a source pad.
> 
> The userspace can configure the routing via two new ioctls,
> VIDIOC_SUBDEV_G_ROUTING and VIDIOC_SUBDEV_S_ROUTING, and subdevs can
> implement the functionality with v4l2_subdev_pad_ops.set_routing().
> 
> Signed-off-by: Laurent Pinchart <laurent.pinchart@ideasonboard.com>
> Signed-off-by: Michal Simek <michal.simek@xilinx.com>

<snip>

> diff --git a/include/uapi/linux/v4l2-subdev.h b/include/uapi/linux/v4l2-subdev.h
> index 658106f5b5dc..5ba409db47ff 100644
> --- a/include/uapi/linux/v4l2-subdev.h
> +++ b/include/uapi/linux/v4l2-subdev.h
> @@ -188,6 +188,61 @@ struct v4l2_subdev_capability {
>   /* The v4l2 sub-device video device node is registered in read-only mode. */
>   #define V4L2_SUBDEV_CAP_RO_SUBDEV		0x00000001
>   
> +/*
> + * Is the route active? An active route will start when streaming is enabled
> + * on a video node.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_ACTIVE		(1 << 0)
> +
> +/*
> + * Is the route immutable, i.e. can it be activated and inactivated?
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_IMMUTABLE		(1 << 1)
> +
> +/*
> + * Is the route a source endpoint? A source endpoint route refers to a stream
> + * generated internally by the subdevice (usually a sensor), and thus there
> + * is no sink-side endpoint for the route. The sink_pad and sink_stream
> + * fields are unused.
> + * Set by the driver.
> + */
> +#define V4L2_SUBDEV_ROUTE_FL_SOURCE		(1 << 2)

I think we should have one more flag.

IMMUTABLE means that the route cannot be inactivated (or activated, but 
a route that's always inactive doesn't make sense), and even if it's not 
mentioned above, I think IMMUTABLE also means the route cannot be 
removed from the routing table, or changed in any way.

We should have a new flag that indicates that the route must exist, but 
it can be activated/inactivated. V4L2_SUBDEV_ROUTE_FL_STATIC? 
V4L2_SUBDEV_ROUTE_FL_IMMUTABLE would be a "superset" of 
V4L2_SUBDEV_ROUTE_FL_STATIC.

Or maybe we should have neither... Both are a bit challenging, as the 
routing system depends on user defined stream identifiers, which in 
practice means that only sensors might have IMMUTABLE or STATIC source 
routes. For every other subdev in the pipeline the userspace must be 
able to change the stream identifiers.

Perhaps just V4L2_SUBDEV_ROUTE_FL_SOURCE would be enough, which would 
also imply STATIC.

  Tomi

Re: [PATCH v9 31/36] media: subdev: add stream based configuration

[Tomi Valkeinen]

On 13/10/2021 10:17, Hans Verkuil wrote:
> On 05/10/2021 10:57, Tomi Valkeinen wrote:
>> Add support to manage configurations (format, crop, compose) per stream,
> 
> Isn't that 'pad+stream'? This sentence suggests that 'stream' replaces 'pad',
> but it is 'in addition to', right?

To configure a stream, or rather a specific "endpoint" along the stream, 
you need the subdev, pad id and stream id. But I don't think we need to 
mention subdev+pad+stream every time, as a stream is unique and it 
should usually be obvious what is meant.

Then again, "a stream" might mean the whole stream, starting from the 
source subdev and going to the final sink subdev. So perhaps we're 
missing some vocabulary here.

>> instead of per pad. This is accomplished with data structures that hold
>> an array of all subdev's stream configurations.
>>
>> The number of streams can vary at runtime based on routing. Every time
>> the routing is changed, the stream configurations need to be
>> re-initialized.
> 
> Is this documented? I can't remember reading this, but I may have missed this.

No, it isn't. I've added some docs now.

> And what if the new routing leaves one stream unchanged, and changes a second,
> does that still mean that both stream configurations need to be re-initialized,
> or just that of the changed stream?

It is the subdev driver that handles this, but I think (and what my 
drivers do) all streams for that subdev need to be re-configured.

Preserving (some) configs when setting new routing is possible, but I 
think it can be difficult to define exactly how set-route should behave 
and we should define the behavior very clearly so that the userspace can 
depend on it. Also, this needs to be implemented separately for each 
subdev driver, and it can be very easy to make mistakes with it in the 
driver code.

Consider e.g. changing the sink side of a route, but keeping source side 
unchanged. It would make sense to keep the source side config intact. 
But that config may no longer be valid as the sink side changes.

Or maybe the first route is unchanged, but changing the second route 
(perhaps the second route now starts or ends at the same pad used in the 
first route) might cause the first route to not be valid anymore.

So, all in all, I think it's much easier for both the kernel to reset 
the configs and for the userspace re-configure everything. But I'm open 
to other opinions =).

> If changing the routing, even if the route of an existing stream stays unchanged,
> will require this re-initialization, then that has to be documented clearly, including
> in the affected ioctls.

Yes, I'll add this to docs.

  Tomi